Apparatus and method for generating training data

Abstract

An apparatus and method for generating training data, the apparatus for generating training data comprising: a communication device that receives a daytime road image; a memory that stores a training data generation model for generating a night composite image from the daytime road image; and a processor that generates a night composite image by reducing the intensity of the daytime road image based on the daytime road image being input as an input value to the training data generation model and reflecting the night characteristics.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2023-0100618 filed on August 1, 2023, which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] The present disclosure relates to an apparatus and method for generating training data, and more particularly to a method for generating a data set necessary for training a deep learning model. Background Art

[0004] Recently, with the development of computer vision technology based on deep neural networks in autonomous driving technology, various artificial intelligence models such as object detection, semantic segmentation, depth estimation, and lane detection have been studied. Such artificial intelligence models can perform supervised learning on a large dataset constructed by obtaining road images and performing labeling for matching the obtained images with the target of each model, thereby achieving successful performance improvement.

[0005] In this regard, through the AI-based autonomous driving technology currently being studied, a data set for training AI models is constructed for autonomous driving in daytime environments. However, because autonomous driving technology needs to be applied to night environments as well as daytime environments, it is necessary to construct a data set that includes night road images.

[0006] In the current processing, a lot of cost and time are consumed in the process of acquiring road images, such as building daytime datasets, generating labels suitable for several artificial intelligence models, and building high-quality training datasets.

[0007] The information included in the Background of the Disclosure is only for enhancement of understanding of the general background of the disclosure and should not be taken as an acknowledgment or any form of suggestion that this information forms the prior art already known to a person skilled in the art. Summary of the invention

[0008] Various aspects of the present disclosure are directed to providing an apparatus and method for generating training data reflecting features of night road images using training of a generative adversary network (GAN), wherein the generative adversary network uses histogram vectors in a daytime road dataset including labels.

[0009] Another aspect of the present disclosure provides an apparatus and method for generating training data to obtain road images in a daytime environment including various road traffic conditions and construct a night training data set using labels generated in the road images in the daytime road images without changes.

[0010] The technical problems to be solved by the present disclosure are not limited to the above-mentioned technical problems, and any other technical problems not mentioned herein will be clearly understood by those skilled in the art to which the present disclosure belongs from the following description.

[0011] According to one aspect of the present disclosure, an apparatus for generating training data may include: a communication device that receives a daytime road image; a memory that stores a training data generation model for generating a night composite image from the daytime road image; and a processor that is input into the training data generation model based on the daytime road image as an input value to reduce the intensity of the daytime road image and reflect night features to generate a night composite image.

[0012] The training data generation model may include: a generator that reduces the intensity of a daytime road image and reflects night features to generate a night synthetic image; and a discriminator that learns the difference between the night synthetic image and a night reference image to distinguish the night synthetic image.

[0013] The generator may include a first generator that reduces the intensity of the daytime road image and a second generator that reflects nighttime features in the daytime road image.

[0014] The first generator may convert the daytime road image into a grayscale image and extract a histogram vector from the grayscale image, may extract an intensity reduction parameter from the histogram vector, and may reflect the intensity reduction parameter in the daytime road image to reduce the intensity of the daytime road image.

[0015] The processor may input the daytime road image to the first generator to reduce the intensity, and may input the intensity-reduced daytime road image to the second generator to generate a nighttime composite image.

[0016] The processor may input the daytime road image to the second generator to reflect nighttime characteristics, and may input the daytime road image reflecting nighttime characteristics to the first generator to generate a nighttime composite image.

[0017] The processor may input the daytime road image to the first generator and the second generator simultaneously to generate a nighttime composite image.

[0018] When it is found that the error is greater than a predetermined reference value, the processor may transform the daytime road image so that the generator reduces the error between the night synthetic image and the night reference image to generate the night synthetic image.

[0019] The processor is configured to determine the night synthetic image as training data when it is inferred that an error between the night synthetic image and the night reference image is less than or equal to a predetermined reference value.

[0020] The processor may reflect label information included in the daytime road image in the nighttime synthetic image to generate training data.

[0021] According to another aspect of the present disclosure, a method for generating training data may include: receiving a daytime road image; reducing the intensity of the daytime road image based on the daytime road image being input as an input value into a training data generation model, the training data generation model being used to generate the daytime road image as a night composite image; and reflecting night features to generate a night composite image.

[0022] The training data generation model may include: a generator that reduces the intensity of a daytime road image and reflects night features to generate a night synthetic image; and a discriminator that learns the difference between the night synthetic image and a night reference image to distinguish the night synthetic image.

[0023] The generator may include a first generator that reduces the intensity of the daytime road image and a second generator that reflects nighttime features in the daytime road image.

[0024] The step of reducing the intensity of the daytime road image may include: converting the daytime road image into a grayscale image; extracting a histogram vector from the grayscale image; extracting an intensity reduction parameter from the histogram vector; and reflecting the intensity reduction parameter in the daytime road image to reduce the intensity of the daytime road image.

[0025] The method may further include inputting the daytime road image to the first generator to reduce intensity, and inputting the daytime road image with reduced intensity to the second generator to generate the night composite image.

[0026] The method may further include inputting the daytime road image into the second generator to reflect nighttime characteristics, and inputting the daytime road image reflecting nighttime characteristics into the first generator to generate a nighttime composite image.

[0027] The method may further include simultaneously inputting the daytime road image to the first generator and the second generator to generate a nighttime composite image.

[0028] The method may further include transforming the daytime road image so that the generator reduces an error between the night synthetic image and the night reference image based on the error being greater than a predetermined reference value to generate the night synthetic image.

[0029] The method may further include determining the night synthetic image as training data based on an error between the night synthetic image and the night reference image being less than or equal to a predetermined reference value.

[0030] Determining the training data may include reflecting label information included in the daytime road image in the night synthetic image to generate the training data.

[0031] The methods and apparatus of the present disclosure have other features and advantages that will be apparent from or set forth in more detail in the accompanying drawings and the following detailed description incorporated herein, which together serve to explain certain principles of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a diagram showing a control block diagram of an apparatus for generating training data according to an exemplary embodiment of the present disclosure;

[0033] Figure 2 is a diagram showing a schematic block diagram of a device for generating training data according to an exemplary embodiment of the present disclosure generating and using training data;

[0034] Figure 3 is a diagram showing a detailed block diagram of a training data generation model according to an exemplary embodiment of the present disclosure;

[0035] Figure 4 is a view showing a detailed structure of a generator and a discriminator in a training data generation model according to an exemplary embodiment of the present invention;

[0036] Figure 5 is a diagram showing generation of a night road image from a day road image by generating a model through training data according to an exemplary embodiment of the present disclosure;

[0037] Figure 6 is a diagram showing that a generator of an apparatus for generating training data according to an exemplary embodiment of the present disclosure reduces the intensity of a daytime road image;

[0038] Figure 7 A diagram showing an example of arranging the order of generators and discriminators in a training data generation model according to an exemplary embodiment of the present invention;

[0039] Figure 8 is a diagram illustrating another example of an order of setting a generator and a discriminator in a training data generation model according to an exemplary embodiment of the present disclosure;

[0040] Fig. 9 is a diagram illustrating another example of an order of setting a generator and a discriminator in a training data generation model according to an exemplary embodiment of the present disclosure;

[0041] Fig.10 is a diagram illustrating a control flow chart of a method for generating training data according to an exemplary embodiment of the present disclosure; and

[0042] Fig.11 is a diagram showing an exemplary embodiment of the present disclosure. Fig.10A diagram of a control flow graph of the following method for generating training data.

[0043] It will be appreciated that the accompanying drawings are not necessarily drawn to scale, and that they present somewhat simplified representations of various features illustrating the basic principles of the present disclosure. The specific design features of the present disclosure as included herein (including, for example, specific dimensions, orientations, locations, and shapes) will be determined in part by the specific intended application and use environment.

[0044] In the drawings, reference numbers refer to the same or equivalent parts of the present disclosure throughout the several figures of the drawing. DETAILED DESCRIPTION

[0045] Reference will now be made in detail to various embodiments of the present disclosure, examples of which are shown in the accompanying drawings and described below. Although the present disclosure will be described in conjunction with exemplary embodiments of the present disclosure, it should be understood that this specification is not intended to limit the present disclosure to those exemplary embodiments of the present disclosure. On the other hand, the present disclosure is intended to cover not only the exemplary embodiments of the present disclosure, but also various replacements, modifications, equivalents and other embodiments that may be included in the spirit and scope of the present disclosure as defined by the appended claims.

[0046] It should be understood that the various embodiments of the present invention and the terms used therein are not intended to limit the technical features set forth herein to the various exemplary embodiments, and include various changes, equivalents, or replacements for the corresponding embodiments.

[0047] Regarding the description of the drawings, like reference numerals may be used to refer to like or related elements.

[0048] The singular form of a noun corresponding to an item may include the item or a plurality of items unless the relevant context clearly indicates otherwise.

[0049] As used herein, each of the expressions "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include any and all combinations of one or more of the items listed with the corresponding expression in the expression.

[0050] Terms such as “first” and “second” or “first” and “second” may be used to simply distinguish a corresponding component from another component, and do not limit the components in other aspects (for example, importance or order).

[0051] It should be understood that if any (e.g., first) component is referred to as being "coupled to", "coupled to", "connected to" or "connected to" another (e.g., second) component, with or without the term "operably" or "communicatively", this means that the element can be coupled to another element directly (e.g., wired), wirelessly or via a third component.

[0052] The terms “comprises,” “includes,” etc. specify the existence of the stated features, quantities, steps, operations, components, parts, or combinations thereof, but do not exclude the existence or addition of one or more other features, quantities, steps, operations, components, parts, or combinations thereof.

[0053] When one component is “connected,” “coupled,” “supported” or “contacting” another component, this includes the case where the components are directly connected, coupled, supported or touching each other and the case where the components are indirectly connected, coupled, supported or touching each other through a third component.

[0054] When a component is located “on” another component, this includes the case where the other component exists between the two components as well as the case where the one component is adjacent to the other component.

[0055] The term "and / or" includes a plurality of related described components or any combination of the plurality of related described components.

[0056] Hereinafter, a description of operating principles and exemplary embodiments of the present disclosure will be provided with reference to the accompanying drawings.

[0057] Figure 1 is a diagram illustrating an apparatus for generating training data according to an exemplary embodiment of the present disclosure.

[0058] Reference Figure 1 According to various exemplary embodiments of the present disclosure, the apparatus 1 for generating training data may include at least one processor 100, a memory 110, and a communication device 120, and may communicate with an external device 2 through the communication device 120 to generate training data 400.

[0059] The external device 2 in the exemplary embodiment of the present disclosure may refer to a device configured to obtain a daytime road image 200, and may include, for example, a camera or a server device provided in a vehicle. When the external device 2 is a camera provided in a vehicle, it may capture a road image during the day to obtain a daytime road image 200, and transmit the daytime road image 200 to the device 1 for generating training data through the communication device 120. When the external device 2 is a server device, it may transmit the captured daytime road image 200 to the device 1 for generating training data through the communication device 120. Therefore, when the external device 2 is a device configured to transmit the daytime road image 200 to the device 1 for generating training data, there is no limitation on its configuration.

[0060] The apparatus 1 for generating training data according to various exemplary embodiments of the present disclosure may refer to all electronic devices, each of which includes a processor 100 and a memory 110, and may include, for example, a server device, a personal computer, a terminal, a portable phone, a smart phone, a handheld device, and a wearable device. The above electronic devices can be loaded into a vehicle for operation. Hereinafter, a detailed description of each component of the apparatus 1 for generating training data will be provided.

[0061] The memory 110 may store various information required for driving the apparatus 1 for generating training data. In detail, the memory 110 may store an operating system and a program required for driving the apparatus 1 for generating training data, or may store data required for driving the apparatus 1 for generating training data.

[0062] For example, the memory 110 may store a Generative Adversarial Network (GAN) algorithm including a generator 310 and a generator 320 for generating training data 400 and a discriminator 330. The memory 110 may store parameters for the GAN algorithm and a loss function.

[0063] The memory 110 may include a volatile memory 110 for temporarily storing data, such as a state random access memory (SRAM) or a dynamic random access memory (DRAM). In addition, the memory 110 may include a non-volatile memory 110 for storing data for a long time, such as a read-only memory (ROM), an erasable programmable ROM (EPROM), and an electrically erasable programmable ROM (EEPROM).

[0064] The communication device 120 may include a wired communication device 122 and a wireless communication device 121 to communicate with the external device 2 .

[0065] The wired communication device 122 can access a wired communication network and can communicate with the external device 2 through the wired communication network. For example, the wired communication device 122 can access the wired communication network through Ethernet (IEEE 802.3 technical standard), or can access the wired communication network through controller area network (CAN) communication, and can send and receive data with the external device 2 through the wired communication network.

[0066] The wireless communication device 121 may include at least one of a short-range communication module and a long-range communication module.

[0067] The short-range communication module may communicate with an external device 2 adjacent to the apparatus 1 for generating training data using a short-range communication method. Here, the short-range communication module may use one of Bluetooth, Bluetooth Low Energy, Infrared Data Association (IrDA), ZigBee, Wireless Fidelity (Wi-Fi), Wi-Fi Direct, Ultra Wideband (UWB), or Near Field Communication (NFC).

[0068] The remote communication module may include a communication module for performing various types of remote communications, and may include a mobile communication device 120. The mobile communication device 120 may send and receive wireless signals with at least one of a base station, an external terminal, or an external device 2 on a mobile communication network. In addition, the remote communication module may communicate with an external device 2 or an external device 2 such as another electronic device through a surrounding access point (AP). The AP may connect a local area network (LAN) connected to the device 1 for generating training data and a wide area network (WAN) connected to a communication server. Therefore, the device 1 for generating training data may be connected to the communication server through the WAN to communicate with the communication server.

[0069] The processor 100 may output a control signal to generally control the apparatus 1 for generating training data. The processor 100 may include one central processing unit (CPU) or multiple CPUs and a graphics processing unit (GPU). At this time, the processor 100 may be implemented as an array of multiple logic gates, and may be implemented as a combination of a general-purpose microprocessor 100 and a memory 110 storing a program executable by the microprocessor 100.

[0070] The processor 100 may transform the received daytime road image 200 for generating the training data 400 for machine learning into the nighttime synthetic image 325 to generate the training data 400. In detail, the processor 100 may reduce the intensity of the daytime road image 200 based on the daytime road image 200 as an input value input to the training data generation model 300, and may reflect nighttime features to generate the nighttime synthetic image 325.

[0071] In addition, the processor 100 can convert the daytime road image 200 into a grayscale image and extract a histogram vector from the grayscale image. The processor 100 can extract an intensity reduction parameter from the histogram vector and reflect the intensity reduction parameter in the daytime road image 200 to reduce the intensity of the daytime road image 200.

[0072] In addition, according to an exemplary embodiment of the present disclosure, the processor 100 may input the daytime road image 200 to the first generator 310 to reduce the intensity, and may input the daytime road image 200 with reduced intensity to the second generator 320 to generate the night synthetic image 325. According to another exemplary embodiment of the present disclosure, the processor 100 may input the daytime road image 200 to the second generator 320 to reflect night characteristics, and may input the daytime road image 200 with night characteristics reflected to the first generator 310 to generate the night synthetic image 325. According to another exemplary embodiment of the present disclosure, the processor 100 may input the daytime road image 200 to the first generator 310 and the second generator 320 at the same time to generate the night synthetic image 325.

[0073] Furthermore, based on the error being greater than a predetermined reference value, the processor 100 may transform the daytime road image 200 to generate the night composite image 325 , so that the generators 310 and 320 reduce the error between the night composite image 325 and the night reference image 210 .

[0074] Furthermore, the processor 100 may be configured to determine the night synthetic image 325 as the training data 400 based on the error between the night synthetic image 325 and the night reference image 210 being less than or equal to a predetermined reference value.

[0075] Therefore, according to various exemplary embodiments of the present disclosure, the device 1 for generating training data can receive a daytime road image 200 to generate a virtual night road image, and the virtual night road image can be used as training data 400 for an artificial intelligence model for autonomous driving, thereby reducing the time and effort required to collect night road images independently of the daytime road images 200.

[0076] Hereinafter, a detailed process of generating the training data 400 by the training data generation model 300 included in the apparatus 1 for generating training data will be described.

[0077] Figure 2 2 is a diagram showing a schematic block diagram of a device for generating training data according to an exemplary embodiment of the present disclosure generating and using training data.

[0078] refer to Figure 2, the apparatus 1 for generating training data may obtain a daytime road image 200 from the outside, and may input the daytime road image 200 into the training data generation model 300. As described above, the training data generation model 300 may include a GAN model including a plurality of generators 310 and 320. The daytime road image 200 is input as an input value to the training data generation model 300, and a night synthetic image 325 may be output as an output value. At this time, the training data generation model 300 may include a method for reducing the intensity of the daytime road image 200 as an input value and reflecting night features to examine the relevance of the night synthetic image 325 as an output value.

[0079] Therefore, the night synthetic image 325 generated by the training data generation model 300 may be used as the training data 400 for training the autonomous driving deep learning model 500 .

[0080] Figure 2 The autonomous driving deep learning model 500 shown in FIG. 5 may refer to an artificial neural network that is loaded into a vehicle or stored in an external server to recognize the surrounding environment of the vehicle and generate vehicle control data 600 for autonomous driving.

[0081] The autonomous driving deep learning model 500 can train an artificial neural network by using the training data 400 generated from the apparatus 1 for generating training data as training data of the artificial neural network.

[0082] Examples of learning algorithms may be, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0083] The artificial neural network may include multiple neural network layers. Each of the neural network layers may include multiple weight values ​​and may perform neural network operations through the results of operations of previous layers and operations in multiple weights. Multiple weight values ​​of multiple neural network layers may be optimized through the results of training artificial intelligence models. For example, multiple weight values ​​may be updated so that the loss value or cost value obtained by the artificial intelligence model during the training process is reduced or minimized.

[0084] The artificial neural network may include a deep neural network (DNN). For example, the artificial neural network may include, but is not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RMB), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q network, etc.

[0085] The autonomous driving deep learning model 500 may receive daytime road images 200 and nighttime road images from an external device 2 such as a camera, and may analyze the daytime road images 200 and nighttime road images through an artificial neural network to generate vehicle data for autonomous driving.

[0086] The apparatus 1 for generating training data according to various exemplary embodiments of the present disclosure may be intended to generate night road images based on a GAN model, rather than collecting night road images for use as training data for the autonomous driving deep learning model 500 to train the autonomous driving deep learning model 500 to generate vehicle control data 600. At this time, the apparatus 1 for generating training data may use the labels of the daytime road images 200 without change to reduce the time and cost for collecting night road images.

[0087] Figure 3 is a diagram illustrating a detailed block diagram of a training data generation model according to an exemplary embodiment of the present disclosure.

[0088] refer to Figure 3 , the training data generation model 300 may include a first generator 310 for reducing the intensity of the daytime road image 200 and a second generator 320 for reflecting night features in the daytime road image 200. In addition, the training data generation model 300 may include a discriminator 330 for distinguishing the truth of the night synthetic image 325 generated from the first generator 310 and the second generator 320.

[0089] In other words, the apparatus 1 for generating training data may include a training data generation model 300. The training data generation model 300 may include a plurality of generators 310 and 320. The first generator 310 and the second generator 320 (ie, the two generators 310 and 320) are Figure 3 However, when the generators include structures configured to share the generation capabilities of a single generator, there is no limit on the number of generators.

[0090] The apparatus 1 for generating training data according to various exemplary embodiments of the present disclosure may include two generators 310 and 320 to share the work of a single generator between the two generators 310 and 320. Therefore, the work capacity of a single generator may be shared between the two generators 310 and 320.

[0091] In other words, because the work of generating the training data 400, which should be performed by a single generator in the prior art, is divided and performed by two generators 310 and 320 having different roles, the work processing capability can be improved.

[0092] In detail, one generator reduces the strength of the input value and reflects the night characteristics in the prior art. However, in an exemplary embodiment of the present disclosure, when the first generator 310 reduces the strength of the input value and the second generator 320 reflects the night characteristics of the input value, work efficiency can be increased.

[0093] As described above, the training data generation model 300 included in the apparatus 1 for generating training data may include a Generative Adversary Network (GAN) algorithm.

[0094] A GAN algorithm may refer to a generative model using an artificial neural network, and may refer to a model for generating fake data having characteristics similar to real data using such a generative model.

[0095] In addition, the GAN algorithm may allow the generators 310 and 320 and the discriminator 330 to compete with each other in an adversarial manner to develop the generators 310 and 320 and the discriminator 330. In an exemplary embodiment of the present disclosure, the generators 310 and 320 for generating the night synthetic image 325 and the discriminator 330 for distinguishing whether the night synthetic image 325 is forged may implement an adversarial competitive relationship. Here, the generators 310 and 320 may be configured to generate false data to generate false data similar to real data and deceive the discriminator 330, and the discriminator 330 may distinguish between real data and false data. When the generators 310 and 320 and the discriminator 330 are trained, the generators 310 and 320 for generating false data that cannot be distinguished from real data may be obtained.

[0096] In other words, the apparatus 1 for generating training data according to various exemplary embodiments of the present disclosure may be configured to generate the night synthetic image 325 through the two generators 310 and 320 , and may discriminate whether the night synthetic image 325 is real data or fake data through the discriminator 330 .

[0097] At this time, when the night synthetic image 325 is discriminated as false data, the generators 310 and 320 may correct parameters so that the night synthetic image 325 is discriminated as real data by the discriminator 330 to generate the night synthetic image 325 .

[0098] The night synthetic image 325 generated by the apparatus 1 for generating training data according to various exemplary embodiments of the present disclosure may include an image of the exterior of the vehicle captured in a time zone when the daytime of a certain time zone has passed, and an image of the exterior of the vehicle captured after sunset. The night synthetic image 325 may include an image obtained by randomly generating a night road image captured in a low-illuminance night environment in contrast to the daytime road image 200.

[0099] The discriminator 330 may compare the night synthetic image 325 generated by the two generators 310 and 320 with the night reference image 210, which is a real night road image, to determine an error between the night synthetic image 325 and the night reference image 210, and may discriminate the night synthetic image 325 as false based on the determined error being greater than a predetermined reference value. On the other hand, the discriminator 330 may be configured to determine the night synthetic image 325 as the training data 400 based on the determined error being less than or equal to the predetermined reference value, and may use the training data 400 as training data for the autonomous driving deep learning model 500.

[0100] The Pix2Pix algorithm may be used as the GAN algorithm used by the apparatus 1 for generating training data according to an exemplary embodiment of the present disclosure. However, the GAN algorithm may be configured to process and learn images like Cycle-GAN, which may be included in the training data generation model 300 without limitation.

[0101] Because the daytime road image 200 and the label data 220 of the artificial intelligence model (or object recognition, segmentation, depth recognition, etc.) for the daytime road image 200 are included in the training data 400 in the daytime environment (which is previously obtained by the device 1 for generating training data according to an exemplary embodiment of the present disclosure), the label data 220 for the daytime road image 200 can be applied unchanged to the night synthetic image 325 generated by the training data generation model 300.

[0102] This is because the original information of the daytime road image 200 is maintained unchanged in the process of generating the night synthetic image 325, and only the environment of the image is converted from the daytime environment to the nighttime environment. Therefore, the apparatus 1 for generating training data according to various exemplary embodiments of the present disclosure can construct the training data 400 including the night road image and the label data 220 of the night road image, and can use the training data 400 to train the network, thereby shortening the process of constructing the night road image 400.

[0103] Figure 4 2 is a view showing a detailed structure of a generator and a discriminator in a training data generation model according to an exemplary embodiment of the present invention.

[0104] Reference Figure 4 , the two generators 310 and 320 may include a U-NET model as an image segmentation algorithm.

[0105] The U-NET model may be an encoder-decoder based model. Generally, an encoder-decoder model may increase the number of channels and may reduce the dimension of capturing features of an input image in an encoding step, and may reduce the number of channels in a decoding step using only information encoded into a low dimension, and may increase the dimension to restore a high-dimensional image.

[0106] Here, the U-NET model can extract the features of the image using high-dimensional information as well as low-dimensional information, and can simultaneously identify the exact location of the image object. Figure 4 As shown, the encoder of the U-NET model can pass the input image through a convolution block a including convolution and ReLU activation functions, and can pass the input image through residual blocks b and c to extract deep features. In addition, the decoder of the U-NET model can reflect the features of the encoder in the output features of the encoder using a skip connection d. Thereafter, the output value of the U-Net model can be upsampled to output an image with the same resolution as the input image.

[0107] Next, Figure 4 The discriminator 330 may pass the input image through a leaky convolution block e including convolution and a LeakyReLU activation function to extract features, and may finally pass the input image through a fully connected layer f to output the true probability of the input image. At this time, the true value probability may be output as a value between "0" and "1", which may mean that when it is close to "0", it is identified as a false image and when it is close to "1", it is identified as a real image.

[0108] The training data generation model 300 according to various exemplary embodiments of the present disclosure may send the training data 400 generated by the two generators 310 and 320 and the discriminator 330 described above to the autonomous driving deep learning model 500 to use the training data 400 as training data. At this time, the training data generation model 300 may simplify the labeling process by using the label data 220 of the daytime road image 200 without change.

[0109] Figure 5 2 is a diagram illustrating generating a night road image from a day road image by generating a model through training data according to an exemplary embodiment of the present disclosure.

[0110] refer to Figure 5 , the apparatus 1 for generating training data may receive a daytime road image 200 from an external device 2. Label data 220 may be included in the received daytime road image 200. The label data 220 may refer to data divided into separate objects, such as vehicles, lines, buildings, and pedestrians, where different labels are assigned to separate objects.

[0111] The memory 110 of the apparatus 1 for generating training data may store an actually captured night road image. The training data generation model 300 may use the real night road image stored in the memory 110 as the night reference image 210 for the discriminator 330 to determine whether the image is true. At this time, the night reference image 210 may include night features. The night features may include vehicle images, line images, building images, and pedestrian images in a low-light environment.

[0112] Thus, the two generators 310 and 320 of the training data generation model 300 may reduce the intensity of the daytime road image 200 and may reflect the above-mentioned night characteristics to generate the night synthetic image 325. The discriminator 330 may compare the night synthetic image 325 with the night reference image 210, and may be configured to determine the night synthetic image 325 as the training data 400 when the error between the night synthetic image 325 and the night reference image 210 is less than or equal to a predetermined reference value. At this time, there may be various modifications in the order in which the input values ​​are input to the two generators 310 and 320. Figure 7 Describe this.

[0113] A process of generating the night synthetic image 325 in the apparatus 1 for generating training data according to an exemplary embodiment of the present disclosure is expressed as Equations 1 to 4 below.

[0114] [Equation 1]

[0115] (I D ×α)+F N =I N

[0116] [Equation 2]

[0117] α=G α (I D )

[0118] [Equation 3]

[0119] F N =I N -(I D ×α)

[0120] [Equation 4]

[0121] F N =G N (I D ×α)

[0122] In detail, I in the above equation 1 D Can represent daytime images, I N may represent a night image, and α may represent an intensity reduction parameter as determined in Equation 2 above. G in Equation 2 aboveα may be a generator, which may refer to the first generator 310. F in the above equation 3 N It can refer to night characteristics. G in equation 4 above N It may be a night feature generator, which may refer to the second generator 320. In other words, the residual of the image reflecting the night feature and the intensity-reduced image may be learned based on the above equation.

[0123] Figure 6 is a diagram illustrating that a generator of an apparatus for generating training data according to an exemplary embodiment of the present disclosure reduces the intensity of a daytime road image.

[0124] refer to Figure 6 , the first generator 310 of the apparatus 1 for generating training data according to various exemplary embodiments of the present disclosure may change the daytime road image 200 into a grayscale image 201 to reduce the intensity of the daytime road image 200. Grayscale may refer to the grayscale of a single channel, which includes different colors and different contrasts.

[0125] Thereafter, the first generator 310 may extract a histogram vector 202 from the grayscale image 201, and the extracted histogram vector 202 may be input to a multi-layer perceptron (MLP) structure 203 to extract a 1×1 size intensity reduction parameter. At this time, the last layer of the MLP structure 203 may be extracted as a sigmoid function to include values ​​between “0” and “1”.

[0126] Thereafter, the processor 100 may multiply the daytime road image 200 by the intensity reduction parameter to reduce the intensity of the entire image, thereby generating a reduced intensity image 204 .

[0127] Figure 7 1 is a diagram illustrating an example of arranging an order of generators and discriminators in a training data generation model according to an exemplary embodiment of the present invention.

[0128] refer to Figure 7 As described above, the training data generation model 300 may include a first generator 310 for reducing the intensity of the daytime road image 200 and a second generator 320 for reflecting night features in the daytime road image 200. In addition, the training data generation model 300 may include a discriminator 330 for discriminating the authenticity of the night synthetic image 325 generated from the first generator 310 and the second generator 320. Figure 7 An exemplary embodiment in which the output result of the first generator 310 is input as an input value of the second generator 320 is shown.

[0129] In detail, the apparatus 1 for generating training data may receive a daytime road image 200 from the outside. The daytime road image 200 may be input as an input value of the first generator 310. Thereafter, the first generator 310 may reduce the intensity of the received daytime road image 200 to generate an intensity-reduced image 315, and may input the intensity-reduced image 315 as an input value to the second generator 320.

[0130] Thereafter, the second generator 320 may reflect the night feature in the input intensity reduced image 315 to generate a night synthetic image 325. The processor 100 may input the generated night synthetic image 325 as an input value to the discriminator 330. The discriminator 330 may be configured to determine an error between the input night synthetic image 325 and the night reference image 210, and may be configured to determine the night synthetic image 325 as the training data 400 when the determined error is less than or equal to a predetermined value.

[0131] In addition, when the determined error is greater than a predetermined value, the first generator 310 and the second generator 320 may adjust the variables to reduce the error, thereby generating a night synthetic image 325. Therefore, the apparatus 1 for generating training data according to various exemplary embodiments of the present disclosure may be configured to generate a night synthetic image 325 based on a daytime road image 200 and may use the night synthetic image 325 as the training data 400 of the autonomous driving deep learning model 500 when the night synthetic image 325 is greater than a predetermined standard.

[0132] The experimental result values ​​of the hyperparameters and loss function of the model 300 generated based on the training data can be as follows Figure 7 , Figure 8 and Fig. 9 As shown, the arrangement of the first generator 310 and the second generator 320 is changed.

[0133] The loss function in the exemplary embodiment of the present disclosure is defined as the following Equation 5 and Equation 6.

[0134] [Equation 5]

[0135] L G =(E X [(D(G N (G α (X)))-1) 2 ])

[0136] [Equation 6]

[0137] L D =E Y [(D(Y)-1) 2 ]+E X [(D(G N (Gα (X))) 2 ]

[0138] Here, X can represent simulated data, Y can represent real-world data, and G a The first generator 310, G N may represent the second generator 320, D may represent the discriminator 330, and E may represent the predicted value. In other words, the training data generation model 300 can learn to reduce Figure 5 and Figure 6 Parameters in the direction of the loss function.

[0139] Figure 8 is a diagram illustrating another example of an order of setting a generator and a discriminator in a training data generation model according to an exemplary embodiment of the present disclosure.

[0140] Figure 8 An exemplary embodiment in which the output result of the second generator 320 is input as an input value of the first generator 310 is shown.

[0141] In detail, the apparatus 1 for generating training data may receive a daytime road image 200 from the outside. The daytime road image 200 may be input as an input value of the second generator 320. Thereafter, the second generator 320 may reflect night features in the received daytime road image 200 to generate a night feature reflected image 316, and may input the night feature reflected image 316 as an input value to the first generator 310.

[0142] Thereafter, the first generator 310 may reduce the intensity of the input night feature reflection image 316 to generate a night synthetic image 325. The processor 100 may input the generated night synthetic image 325 as an input value to the discriminator 330. The discriminator 330 may be configured to determine an error between the input night synthetic image 325 and the night reference image 210, and may be configured to determine the night synthetic image 325 as the training data 400 when the determined error is less than or equal to a predetermined value.

[0143] In addition, when the determined error is greater than a predetermined value, the first generator 310 and the second generator 320 may adjust the variables to reduce the error, thereby generating the night synthetic image 325 .

[0144] Therefore, according to various exemplary embodiments of the present disclosure, the device 1 for generating training data can be configured to generate a night synthetic image 325 based on the daytime road image 200 and the night synthetic image 325 can be used as training data 400 of the autonomous driving deep learning model 500 when the night synthetic image 325 is greater than a predetermined standard.

[0145] Fig. 9is a diagram illustrating another example of an order of setting a generator and a discriminator in a training data generation model according to an exemplary embodiment of the present disclosure.

[0146] Fig. 9 An exemplary embodiment is shown in which the processor 100 simultaneously inputs the daytime road image 200 to the first generator 310 and the second generator 320 .

[0147] In detail, the apparatus 1 for generating training data may receive a daytime road image 200 from the outside. The daytime road image 200 may be input as an input value of the first generator 310 and the second generator 320. Thereafter, the first generator 310 may reduce the intensity of the received daytime road image 200 to generate an intensity-reduced image, and the second generator 320 may reflect night features in the received daytime road image 200 to generate a night feature-reflected image.

[0148] Thereafter, the processor 100 may combine the intensity-reduced image generated by the first generator 310 and the night feature-reflected image generated by the second generator 320 to generate a night synthetic image 325. The processor 100 may input the generated night synthetic image 325 as an input value to the discriminator 330. The discriminator 330 may be configured to determine an error between the input night synthetic image 325 and the night reference image 210, and may be configured to determine the night synthetic image 325 as the training data 400 when the determined error is less than or equal to a predetermined value.

[0149] In addition, when the determined error is greater than a predetermined value, the first generator 310 and the second generator 320 may adjust the variables to reduce the error, thereby generating a night synthetic image 325. Therefore, the apparatus 1 for generating training data according to various exemplary embodiments of the present disclosure may be configured to generate a night synthetic image 325 based on a daytime road image 200 and may use the night synthetic image 325 as the training data 400 of the autonomous driving deep learning model 500 when the night synthetic image 325 is greater than a predetermined standard.

[0150] Furthermore, additional generators may be provided in addition to the first generator 310 and the second generator 320. A portion of the work performed by the first generator 310 and the second generator 320 may be shared between the additional generators, thus optimizing the efficiency of the training data generation model 300.

[0151] Fig.10 is a diagram illustrating a control flow chart of a method for generating training data according to an exemplary embodiment of the present disclosure. Fig.11 is a diagram showing an exemplary embodiment of the present disclosure. Fig.10 Schematic diagram of a control flow diagram of a method for generating training data following .

[0152] refer to Fig.10 In operation 1000, the apparatus 1 for generating training data according to various exemplary embodiments of the present disclosure may receive a daytime road image 200 from the outside. Thereafter, in operation 1010, the processor 100 may input the daytime road image 200 to the first generator 310. In operation 1020, the first generator 310 may convert the daytime road image 200 into a grayscale image, which is a single-channel grayscale image including different colors and different contrasts.

[0153] In operation 1030, the first generator 310 may extract a histogram vector indicating a brightness distribution of the grayscale image from the converted grayscale image. In operation 1040, the first generator 310 may extract an intensity reduction parameter from the histogram vector.

[0154] Thereafter, in operation 1050, the first generator 310 may reflect the intensity reduction parameter in the received daytime road image 200. In detail, the first generator 310 may multiply the daytime road image 200 by the intensity reduction parameter to reduce the intensity of the entire image.

[0155] Thereafter, in operation 1060, the processor 100 may input the intensity-reduced image in which the intensity-reduction parameter is reflected to the second generator 320. In operation 1070, the processor 100 may be configured to generate a night synthetic image 325 by reflecting night features including vehicle images, line images, building images, and pedestrian images in a low-illuminance environment in the intensity-reduced image.

[0156] In operation 1080, the processor 100 may input the generated night synthetic image 325 to the discriminator 330. In operation 1090, the processor 100 may learn the difference between the night synthetic image 325 and the night reference image 210, and may be configured to determine whether the night road image is true.

[0157] Thereafter, when the error between the night synthetic image 325 and the night reference image 210 is less than or equal to the reference value, the discriminator 330 may be configured to determine the night synthetic image 325 as the training data 400 in operation 1100. When the error is greater than the reference value, the discriminator 330 may correct the parameters in a direction of reducing the error to generate the night synthetic image 325.

[0158] The processor 100 may be configured to generate the training data 400 about the night road image based on the above series of processes. As described above, the order in which the first generator 310 and the second generator 320 are arranged may be changed based on the result value of the hyperparameter and the loss function.

[0159] As a result, the apparatus 1 for generating training data according to various exemplary embodiments of the present disclosure may be configured to generate a night synthetic image 325 based on GAN using a histogram vector, and may convert a daytime road image 200 including label data 220 into a night synthetic image 325 reflecting the characteristics of a night road environment. Therefore, the apparatus 1 for generating training data according to various exemplary embodiments of the present disclosure may obtain training data 400 without collecting and labeling a real night road image, thereby reducing the time and effort for constructing the training data 400 of the night road image.

[0160] Meanwhile, the exemplary embodiments of the disclosure of the present invention may be implemented in the form of a storage medium storing instructions executable by a computer. When executed by the processor 100, the instructions may be stored in the form of program code and may be configured to generate a program module and perform the operations of the exemplary embodiments of the disclosure of the present invention. The storage medium may be implemented as a computer-readable storage medium.

[0161] Computer-readable storage media may include all types of storage media that store instructions decoded by a computer. For example, a computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic tape, a disk, a flash memory 110, an optical data storage device, etc.

[0162] In addition, the computer-readable storage medium may be provided in the form of a non-transitory storage medium. In this document, the term "non-transitory storage medium" simply means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between the time when data is semi-permanently stored in the storage medium and the time when data is temporarily stored in the storage medium. For example, the "non-transitory storage medium" may include a buffer that temporarily stores data.

[0163] According to an exemplary embodiment of the present disclosure, the method according to various embodiments included in the present invention may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or via an application store (e.g., PlayStore). TM ) online distribution (e.g., download or upload), or directly distributed between two user devices (e.g., smart phones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium, such as the memory 110 of a manufacturer's server, a server of an application store, or a relay server.

[0164] According to an exemplary embodiment of the present disclosure, a device for generating training data may be configured to generate composite data by training a generated adversary network (GAN) using histogram vectors in a daytime road dataset including labels to reflect features of night road images, thereby reducing the time and cost required to construct a training dataset in a night environment.

[0165] According to an exemplary embodiment of the present disclosure, an apparatus for generating training data can use labels of daytime road images without changing them, omitting the work of generating labels for each artificial intelligence model, which takes a lot of time and money, thereby improving efficiency.

[0166] According to an exemplary embodiment of the present disclosure, the device for generating training data can improve the ability to generate images in a GAN algorithm using a 2-stage based generator.

[0167] For ease of explanation and accurate definition of the appended claims, the terms "upper", "lower", "inner", "outer", "up", "lower", "upward", "downward", "front", "rear", "back", "inner", "outer", "inward", "outward", "inside", "outer", "inner", "external", "forward", and "rearward" are used to describe features of the exemplary embodiments with reference to the locations of such features shown in the drawings. It should be further understood that the term "connected" or its derivatives refer to both direct and indirect connections.

[0168] In this specification, unless otherwise stated, a singular expression includes a plural expression unless the context clearly indicates otherwise.

[0169] In the exemplary embodiments of the present disclosure, it should be understood that terms such as “including” or “having” are intended to specify the presence of features, quantities, steps, operations, elements, parts, or a combination thereof described in the specification, and do not exclude the possibility of adding or existing one or more other features, quantities, steps, operations, elements, parts, or a combination thereof.

[0170] For the purpose of illustration and description, the foregoing descriptions of specific exemplary embodiments of the present disclosure have been presented. They are not intended to be exhaustive or to limit the present disclosure to the precise form disclosed, and it is apparent that many modifications and variations are possible in light of the above teachings. In order to illustrate certain principles of the present invention and their practical applications, exemplary embodiments have been selected and described to enable other persons skilled in the art to carry out and utilize various exemplary embodiments of the present disclosure and various substitutions and modifications thereof. The scope of the present disclosure is intended to be limited by the appended claims and their equivalents.

Claims

1. A device for generating training data, the device comprising: a communication device configured to receive a daytime road image; A memory storing a training data generation model for generating a nighttime synthetic image based on the daytime road image; as well as A processor is operably connected to the communication device and the memory, and is configured to reduce the intensity of the daytime road image and reflect night features based on the daytime road image as an input value input into the training data generation model to generate the night composite image.

2. The device according to claim 1, wherein: The training data generation model includes: a generator configured to reduce the intensity of the daytime road image and reflect the nighttime features to generate the nighttime composite image; and A discriminator is configured to learn the difference between the night synthetic image and a night reference image to discriminate the night synthetic image.

3. The device according to claim 2, wherein: The generator comprises: A first generator configured to reduce the intensity of the daytime road image; and The second generator is configured to reflect the night features in the daytime road image.

4. The device according to claim 3, wherein: The first generator is further configured to convert the daytime road image into a grayscale image and extract a histogram vector from the grayscale image, extract an intensity reduction parameter from the histogram vector, and reflect the intensity reduction parameter in the daytime road image to reduce the intensity of the daytime road image.

5. The device according to claim 3, wherein: The processor is further configured to input the daytime road image to the first generator to reduce intensity, and input the daytime road image with reduced intensity to the second generator to generate the night composite image.

6. The device according to claim 3, wherein: The processor is further configured to input the daytime road image to the second generator to reflect the night characteristics, and input the daytime road image reflecting the night characteristics to the first generator to generate the night composite image.

7. The device according to claim 3, wherein: The processor is further configured to simultaneously input the daytime road image to the first generator and the second generator to generate the nighttime composite image.

8. The device according to claim 2, wherein: The processor is further configured to transform the daytime road image when it is found that the error between the night composite image and the night reference image is greater than a predetermined reference value, so that the generator reduces the error to generate the night composite image.

9. The device according to claim 2, wherein: The processor is further configured to determine the night synthetic image as the training data when it is found that the error between the night synthetic image and the night reference image is less than or equal to a predetermined reference value.

10. The device according to claim 9, wherein: The processor is further configured to reflect label information included in the daytime road image in the nighttime synthetic image to generate the training data.

11. A method for generating training data, the method comprising: receiving daytime road images; The processor reduces the intensity of the daytime road image based on the daytime road image being input as an input value to a training data generation model, wherein the training data generation model is used to generate a nighttime synthetic image; and The processor reflects night characteristics to generate the night composite image.

12. The method according to claim 11, wherein: The training data generation model includes: a generator configured to reduce the intensity of the daytime road image and reflect the nighttime features to generate the nighttime composite image; and A discriminator is configured to learn the difference between the night synthetic image and a night reference image to discriminate the night synthetic image.

13. The method according to claim 12, wherein: The generator comprises: A first generator configured to reduce the intensity of the daytime road image; and The second generator is configured to reflect the night features in the daytime road image.

14. The method according to claim 11, wherein: Reducing the intensity of the daytime road image includes: Converting the daytime road image into a grayscale image; Extracting a histogram vector from the grayscale image; extracting an intensity reduction parameter from the histogram vector; and The intensity reduction parameter is reflected in the daytime road image to reduce the intensity of the daytime road image.

15. The method according to claim 13, further comprising: The daytime road image is input to the first generator by the processor to reduce the intensity, and the daytime road image with reduced intensity is input to the second generator by the processor to generate the night composite image.

16. The method according to claim 13, further comprising: The processor inputs the daytime road image to the second generator to reflect the night characteristics, and the processor inputs the daytime road image reflecting the night characteristics to the first generator to generate the night composite image.

17. The method according to claim 13, further comprising: The processor simultaneously inputs the daytime road image to the first generator and the second generator to generate the nighttime composite image.

18. The method according to claim 12, further comprising: When it is found that the error between the night composite image and the night reference image is greater than a predetermined reference value, the processor transforms the daytime road image so that the generator reduces the error to generate the night composite image.

19. The method according to claim 12, further comprising: When it is found that the error between the night composite image and the night reference image is less than or equal to a predetermined reference value, the processor determines the night composite image as the training data.

20. The method according to claim 19, wherein: Determining the training data includes: The label information included in the daytime road image is reflected in the nighttime synthetic image to generate the training data.

Images