Training data generation method and device for lane classification deep learning model

Abstract

The present invention relates to a method and device for generating training data for a lane classification deep learning model. The method, in one embodiment of the present invention, is performed by an electronic device and utilizes images of a white lane and other color lanes to generate composite images of the other color lanes to generate training data for the lane classification deep learning model. The method comprises: calculating a ratio of the three color channels of red (R), green (G), and blue (B) in the other color lanes in the image of the other color lanes, using one channel (a reference color channel) as a reference; and applying the calculated ratio to the other two channels relative to the reference color channel of the white lane in the image of the white lane to generate the composite image of the other color lane.

Description

Technical Field

[0001] The present invention relates to a method and apparatus for generating training data for a lane classification deep learning model, and more particularly to a method and apparatus for increasing training data for lanes with an unbalanced number. Background Art

[0002] Smart cars are vehicles equipped with automated driving systems. These vehicles are not directly controlled by a driver, but rather are controlled automatically by computers, such as computers, that automatically manage driving operations such as driving, stopping, turning, accelerating, and decelerating. Key challenges for these smart cars include maintaining lane control, maintaining a safe distance from neighboring vehicles, detecting approaching obstacles and avoiding collisions, and controlling vehicle speed based on traffic conditions and road conditions.

[0003] Recent advances in information and communication technology have led to the rapid development and practical application of safety driving assistance systems such as lane departure warning systems (LDWS) and lane keeping systems, as well as automated vehicle control systems. Lane recognition, in particular, is a core technology for smart cars and is attracting significant international attention, leading to active research.

[0004] Lane recognition significantly impacts safe driving, and therefore various sensors are used to estimate and determine lane positions. Specifically, multiple sensors, such as image sensors, radar, and lidar sensors, are used individually or in fusion to implement intelligent vehicle control systems for lane recognition and object detection in front of the vehicle. Among these, lane recognition using image sensors is widely used because it can recognize a wide range of information at a low cost and can be applied to a variety of existing visual processing algorithms.

[0005] On the other hand, lanes on roads are marked with various colors, and their colors and types have various traffic sign functions. White lanes, yellow lanes, blue lanes, etc., have various traffic sign functions such as one-way lanes, two-way lanes, and special lanes, depending on their colors.

[0006] Therefore, smart cars need to accurately classify lane colors and types. This color-based lane classification can be achieved using balanced training data for each lane color and the latest deep learning models.

[0007] However, on real roads, white lanes are common, while yellow and blue lanes are very rare, creating an imbalance in their number. This means that for lane classification training, it's impossible to collect the same amount of training data for each lane color using only real-world road images. Deep learning models trained on this imbalanced training data tend to favor frequently occurring scenarios, resulting in reduced classification accuracy. Summary of the Invention

[0008] (1) Technical issues to be resolved

[0009] In order to solve the problems of the prior art as described above, an object of the present invention is to provide a training data generation technology for a lane classification deep learning model that can increase training data for an unbalanced number of lanes.

[0010] However, the problems to be solved by the present invention are not limited to the problems mentioned above, and ordinary technicians in the technical field to which the present invention belongs can clearly understand other problems not mentioned from the following description.

[0011] (2) Technical solution

[0012] A method according to an embodiment of the present invention for solving the above-mentioned problem is performed by an electronic device or an in-vehicle system, and utilizes images of a white lane and lanes of other colors to generate composite images of lanes of other colors to generate training data for a lane classification deep learning model. The method is characterized in that it includes: a step of calculating the ratio of the three color channels of red (R), green (G), and blue (B) possessed by the other color lanes in the image of the other color lanes with one channel (a reference color channel) as a reference to the other two channels; and a step of scaling the other two channels relative to the reference color channel of the white lane in the image of the white lane by applying the above-calculated ratio to generate a composite image of the other color lanes.

[0013] Among the three color channels, the color value of the reference color channel may be the largest.

[0014] The above-mentioned other color lanes may be yellow lanes, and the above-mentioned reference color channel may be an R channel.

[0015] The above-mentioned other color lanes may be blue lanes, and the above-mentioned reference color channel may be a B channel.

[0016] The calculating step may include calculating the ratio for each row of a lane of another color having fading, and the generating step may include applying the calculated ratio for each row of the white lane.

[0017] The calculating step may include calculating the ratio for a plurality of lanes of other colors to calculate an average value thereof, and the generating step may include applying the ratio of the average value to the white lane.

[0018] The above-mentioned calculation step may include the step of calculating the above-mentioned ratio for different rows of other color lanes with fading, calculating the ratio for different rows of multiple other color lanes to calculate the average value thereof, and the above-mentioned generation step may include the step of applying the calculated average value of the above-mentioned ratios for different rows to each row of the white lane.

[0019] The generating step may include a step of labeling the generated composite images of lanes of other colors relative to lanes of corresponding colors.

[0020] An apparatus according to one embodiment of the present invention generates training data for a lane classification deep learning model, and is characterized in that the apparatus includes: a memory for storing images of white lanes and lanes of other colors; and a control unit for generating composite images of lanes of other colors as the training data using the stored images of white lanes and lanes of other colors.

[0021] Another embodiment of the present invention generates training data for a lane classification deep learning model, comprising: a communication unit configured to receive images of a white lane and other color lanes; and a control unit configured to generate a composite image of the other color lanes as the training data using the received images of the white lane and other color lanes.

[0022] The above-mentioned control unit can calculate the ratio of the other two channels of red (R), green (G) and blue (B) possessed by the other color lanes in the image of the other color lanes based on one channel (reference color channel) as the reference, and can apply the above-mentioned calculated ratio to the other two channels relative to the reference color channel of the white lane in the image of the white lane to scale them to generate a composite image of the other color lanes.

[0023] Among the three color channels, the color value of the reference color channel may be the largest.

[0024] The control unit may calculate the ratio for each line of the lanes of other colors that have faded colors, and may apply the calculated ratio for each line to each line of the white lane.

[0025] The control unit may calculate the ratio for a plurality of other color lanes to calculate an average value thereof, and may apply the ratio of the average value to the white lane.

[0026] The control unit may calculate the ratio for each row of a lane of another color that has faded color, may calculate the ratio for each row of a plurality of lanes of another color to calculate an average value thereof, and may apply the average value of the calculated ratio for each row of the white lane.

[0027] The control unit may label the generated composite images of lanes of other colors relative to the corresponding color lanes.

[0028] The device of one embodiment of the present invention may further include a camera for capturing images of the vehicle's surroundings. The images of the vehicle's surroundings captured by the camera may be used as a system within the vehicle.

[0029] (3) Beneficial effects

[0030] The present invention configured as described above has the advantage of being able to increase training data for lanes of other colors in order to adjust the imbalance in training data between the white lane and lanes of other colors for the deep learning model for color-based lane classification.

[0031] That is, the present invention can increase the training data of the corresponding lanes by generating synthetic images of the blue and yellow lanes with low frequency.

[0032] Furthermore, the present invention has the advantage of converting the image of the white lane in the RGB channel of the ratio reference lane based on the ratio of other color channels relative to the reference color channel, thereby more easily generating synthetic images of other color lanes that are close to reality.

[0033] Furthermore, the present invention has the advantage of being able to generate a faded composite image reflecting an image appearing on a white lane.

[0034] Furthermore, the present invention has the advantage of contributing to the generation of a deep learning model for lane classification with significantly improved classification performance, such as classification accuracy, by providing a relatively simple training data expansion technology.

[0035] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and those skilled in the art can clearly understand other effects not mentioned from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 FIG. 1 is a block diagram showing the structure of a training data generating apparatus 100 according to an embodiment of the present invention.

[0037] Figure 2 A block diagram showing the structure of the control unit 150 in the generating device 100 according to an embodiment of the present invention.

[0038] Figure 3 A flowchart showing a generation method according to an embodiment of the present invention.

[0039] Figure 4 A more detailed flowchart showing the process of generating a synthetic image of a blue lane.

[0040] Figure 5 A more detailed flowchart showing the process for generating a synthetic image of a yellow lane.

[0041] Figures 6a to 6c They respectively show an image of a white lane, a synthetic image of a blue lane generated using the generation method of the present invention, and a synthetic image of a yellow lane.

[0042] Figure 7Indicates the original lane and the lane generated using the generation method of the present invention.

[0043] Figure 8 A chart showing the performance of a deep learning model trained using the original training data.

[0044] Figure 9 A graph showing the performance of a deep learning model trained using training data augmented by the generation method according to the present invention.

[0045] Figure 10 A block diagram showing the structure of an in-vehicle camera system 200 according to an embodiment of the present invention.

[0046] Description of Reference Numerals

[0047] 100: Generating device 110: Input unit

[0048] 120: Communication unit 130: Display

[0049] 140: Memory 150: Control unit

[0050] 151: Ratio calculation unit 152: Image generation unit DETAILED DESCRIPTION

[0051] The following detailed description in conjunction with the accompanying drawings further clarifies the above-mentioned objects and means of the present invention, as well as the effects based thereon, so that those skilled in the art can easily implement the technical concepts of the present invention. Furthermore, in the course of describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, such detailed description will be omitted.

[0052] The terms used in this specification are intended to illustrate the embodiments and do not limit the present invention. In this specification, unless otherwise specified, the singular includes the plural, as appropriate. In this specification, terms such as "including," "having," "having," or "having" do not preclude the presence or addition of one or more structural elements other than the ones mentioned.

[0053] In this specification, terms such as "or" and "at least one" may refer to one of the listed terms or a combination of two or more. For example, "A or B" or "at least one of A and B" may include only A or B or both A and B.

[0054] In this specification, descriptions based on "for example" and the like, as with reference to characteristics, variables or values, may not be exactly consistent with the information presented, and the effects of deformation including allowable errors, measurement errors, limits of measurement accuracy and other factors that are generally known should not limit the implementation of the invention of the various embodiments of the invention.

[0055] In this specification, when it is stated that a structural element is "connected" or "connected" to another structural element, it may be "directly connected" or "connected" to the other structural element, but it should be understood that other structural elements may exist in between. Conversely, when it is stated that a structural element is "directly connected" or "directly connected" to another structural element, it should be understood that no other structural elements exist in between.

[0056] In this specification, when a structural element is described as being "on" or "in contact with" another structural element, it may be directly attached to or connected to the other structural element, but it should be understood that other structural elements may be present in between. Conversely, when a structural element is described as being "directly above" or "directly in contact with" another structural element, it should be understood that no other structural elements are present in between. Other expressions describing the relationship between structural elements, such as "between" and "directly between," are equally construed.

[0057] In this specification, terms such as "first" and "second" may be used to describe various structural elements, but these structural elements should not be limited to these terms. Furthermore, these terms should not be interpreted as limiting the order of the structural elements; they can be used to distinguish one structural element from another. For example, the "first structural element" can be named "the second structural element," and similarly, the "second structural element" can be named "the first structural element."

[0058] Unless otherwise defined, all terms used in this specification are to be used with the meanings commonly understood by those skilled in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted in an ideal or excessive manner unless specifically defined otherwise.

[0059] Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

[0060] Figure 1 FIG. 1 is a block diagram showing the structure of a training data generating apparatus 100 according to an embodiment of the present invention.

[0061] The training data generating apparatus 100 according to one embodiment of the present invention is an apparatus for generating training data for training a deep learning model for lane classification. In this case, the deep learning model is a model trained using various deep learning techniques.

[0062] In this case, deep learning models represent the relationship between input and output data as multiple layers (i.e., layers). This representation is called an "artificial neural network." Each layer within a neural network is composed of at least one filter, and each filter has a matrix of weights. In other words, each element (pixel) in the corresponding filter matrix corresponds to a weight value.

[0063] For example, deep learning technology may include deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), restricted Boltzmann machine (RBM), deep belief network (DBN), deep Q network (Deep Q-Networks), auto-encoder (Auto-encoder), variational auto-encoder (Variational Auto-encoder), etc., but is not limited to these.

[0064] The lane classification deep learning model is a model trained using deep learning techniques to classify the color of lanes in an image containing lanes. Of course, the lane classification deep learning model can also classify lane types (dashed, solid, etc.) in addition to lane color.

[0065] On the other hand, training a high-performance deep learning model with high accuracy requires a large amount of diverse training data. However, it is difficult to collect diverse and dense data that can represent all scenarios.

[0066] In particular, deep learning models for lane classification should be trained with more, equal, or similar images than for white lanes and lanes of other colors. In this case, white lanes can represent one-way traffic, yellow lanes can represent two-way traffic, and blue lanes can represent special traffic, but are not limited to these.

[0067] However, on real roads, white lanes are prevalent, while lanes of other colors (such as yellow and blue) are relatively rare. Consequently, the training data used to train deep learning models for lane classification is imbalanced. Specifically, images of white lanes are uneven, making them easily available. However, images of yellow and blue lanes are limited in availability, so fewer images can be collected than for white lanes. Consequently, deep learning models trained using this training data inevitably favor frequently occurring scenarios. Furthermore, when deep learning models are trained using this imbalanced training data, prediction inequality occurs. Specifically, the prediction performance (accuracy, etc.) for lanes of other colors is significantly lower than that for white lanes. This imbalance is particularly prevalent in the color-based lane classification problem using deep learning models using semantic segmentation.

[0068] The generation device 100 can provide a solution to the problem of imbalanced training data in color-based lane classification by providing a new training data augmentation technique. In this case, the training data is increased to training data that can be used by the operation, thereby increasing the imbalanced training data to adjust the balanced strategy.

[0069] The generating device 100 is a computing electronic device that generates composite images of other color lanes using images of corresponding lanes for unbalanced training data (such as images of white lanes and other color lanes, with more white lane images), thereby enhancing the training data. Figure 1 As shown, it may include an input unit 110 , a communication unit 120 , a display 130 , a memory 140 and a control unit 150 .

[0070] Of course, the generation device 100 may be a device for training a deep learning model for lane classification. In this case, the generation device 100 may train the deep learning model for lane classification using not only lane images but also synthetic lane images generated by the generation device 100 itself.

[0071] Alternatively, the generating device 100 may be installed within a vehicle or separately from the vehicle. In this case, the vehicle may be a smart car that includes an autonomous driving system. Specifically, a smart car is one in which driving operations such as driving, stopping, turning, accelerating, and decelerating are automatically performed by a computer, rather than by a driver.

[0072] For example, the generating device 100 separately provided with the vehicle may be a desktop personal computer, a laptop personal computer, a tablet personal computer, a netbook computer, a workstation, a personal digital assistant (PDA), a smartphone, a smart pad, or a mobile phone, etc., but is not limited thereto.

[0073] The input unit 110 generates input data corresponding to various user inputs and may include various input units. For example, the input unit 110 may include a keyboard, a keypad, a dome switch, a touch panel, a touch key, a touch pad, a mouse, a menu button, etc., but is not limited thereto.

[0074] Communication unit 120 is a structure for communicating with other devices. For example, communication unit 120 can perform wireless communication such as fifth-generation communication (5G), long-term evolution-advanced (LTE-A), long-term evolution (LTE), Bluetooth, Bluetooth low energy (BLE), near-field communication (NFC), and wireless fidelity (WiFi), or wired communication such as cable communication, but is not limited to these. Communication unit 120 can receive lane images and the like from another device and transmit generated lane composite images and the like to another device.

[0075] Display 130 displays various image data on a screen and can be composed of a non-luminescent panel or a luminescent panel. For example, display 130 may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. Furthermore, display 130 can be implemented as a touch screen, etc., in conjunction with input unit 110.

[0076] The memory 140 stores various information required for the operation of the generation device 100. This stored information may include, but is not limited to, lane images, synthesized lane images, and program information related to the generation method described below. For example, the memory 140 may include, but is not limited to, a hard disk type, a magnetic media type, a compact disc read-only memory (CD-ROM), an optical recording media type, a magneto-optical media type, a multimedia card micro type, a flash memory type, a read-only memory type, or a random access memory type. Furthermore, depending on its purpose and location, the memory 140 may be a cache, a buffer, a main memory, an auxiliary memory, or a separate storage system, but is not limited to these.

[0077] The control unit 150 can perform various control operations for the generation device 100. Specifically, the control unit 150 can control the execution of the generation method described below, and can control the operation of the remaining components of the generation device 100, namely, the input unit 110, the communication unit 120, the display 130, the memory 140, etc. For example, the control unit 150 may include a processor as hardware or a software process executed in the processor, but is not limited thereto.

[0078] Figure 2 A block diagram showing the structure of the control unit 150 in the generating device 100 according to an embodiment of the present invention.

[0079] The control unit 150 controls the generation method according to an embodiment of the present invention, such as Figure 2 As shown, the ratio calculation unit 151 and the image generation unit 152 may be included. For example, the ratio calculation unit 151 and the image generation unit 152 may be a hardware structure of the control unit 150 or a software process executed in the control unit 150, but the present invention is not limited thereto.

[0080] The generation method of the present invention is described in more detail below. However, for ease of explanation, the other color lanes are described as blue lanes or yellow lanes, but the present invention is not limited thereto and can be other color lanes.

[0081] Figure 3 A flow chart showing a generation method according to an embodiment of the present invention is shown. Figure 4 represents a more detailed flowchart for generating a synthetic image of a blue lane, Figure 5 A more detailed flowchart showing the process for generating a synthetic image of a yellow lane.

[0082] In the generation method of one embodiment of the present invention, images of white lanes and lanes of other colors are used to generate composite images of lanes of other colors to generate training data for a lane classification deep learning model, such as Figure 3 As shown, the method may include steps S210 and S220 (steps S210' and S220'). That is, the method for generating a synthetic image of a blue lane may include steps S210 and S220, and the method for generating a synthetic image of a yellow lane may include steps S210' and S220'.

[0083] First, in step S210 (step S210′), the ratio calculation unit 151 calculates the ratio of the three color channels (red (R), green (G), and blue (B)) of the other color lane (hereinafter referred to as the "ratio reference lane") in an image of the other color lane (hereinafter referred to as the "ratio reference image"), using one channel (the reference color channel) as a reference to the other two channels. In this case, the reference color channel is the channel that contributes most to the image of the corresponding color lane (i.e., the ratio reference lane), and its color value (RGB value) may be the largest among the three channels.

[0084] The ratio-reference image includes a blue ratio-reference lane, with the B channel being the reference color channel. This is because the B channel has the largest color value and contributes most to the blue ratio-reference lane in the image. Specifically, within the blue lane image, the ratios of the three color channels corresponding to the blue lane portion are calculated, specifically the ratios of the R and G channel color values ​​relative to the color value of the B channel, which serves as the reference color channel. Since the blue lane portion includes multiple pixels, each pixel having color values ​​for the three color channels (i.e., RGB values), the average of the color values ​​for each color channel in these pixels is calculated. Next, for these three average values, the ratios of the average values ​​of the other two channels (i.e., the R and G channels) relative to the B channel average are calculated. For example, if the color averages of the R, G, and B channels relative to the pixels in the blue lane portion are 10, 9, and 200, respectively, the R channel ratio, with the B channel as the reference, is 10 / 200, and the G channel ratio is 9 / 200 (hereinafter referred to as the "first example").

[0085] Furthermore, when the ratio-reference image includes a yellow ratio-reference lane, the reference color channel is the R channel. This is because the R channel has the largest color value and contributes most to the yellow ratio-reference lane in the image. Specifically, in the yellow lane image, the ratios of the three color channels corresponding to the yellow lane portion are calculated, specifically the ratios of the color values ​​of the G and B channels relative to the color value of the R channel, which serves as the reference color channel. Since the yellow lane portion includes multiple pixels, each pixel has color values ​​for the three color channels (i.e., RGB values), the average of the color values ​​of each color channel in these pixels is calculated. Next, for each of these three average values, the ratios of the average values ​​of the other two channels (i.e., the G and B channels) relative to the R channel average are calculated. For example, if the color averages of the R, G, and B channels relative to the pixels in the yellow lane portion are 150, 13, and 15, respectively, the ratio of the G channel relative to the R channel is 13 / 150, and the ratio of the B channel is 15 / 150 (hereinafter referred to as the "second example").

[0086] Then, in step S220 (step S220'), the image generating unit 152 applies the ratio calculated in step S210 (step S210') to the other two channels of the reference color channel of the white lane in the image of the white lane for scaling to generate a composite image of the other color lanes.

[0087] In the first example, the base color channel is the B channel. Therefore, in the image of the white lane, the R channel ratio (10 / 200) and the G channel ratio (9 / 200) based on the B channel are applied to the white lane portion. That is, in the image of the white lane, the white lane portion includes multiple pixels, each of which has color values ​​(i.e., RGB values) for the three color channels. In this case, the B channel remains unchanged for each pixel, while the color values ​​of the remaining R and G channels are converted. This conversion is performed by scaling according to the derived R channel ratio (10 / 200) and G channel ratio (9 / 200), thereby generating a composite image of the blue lane. For example, in the image of the white lane, if the RGB values ​​of a pixel in the white lane portion are (200, 199, 198), the B channel color value of 198 remains unchanged. The R channel is scaled to convert the color value to 200×(10 / 200), and the G channel is scaled to convert the color value to 199×(9 / 200), thereby converting the corresponding pixel into a pixel in the blue lane. Of course, performing the same transformation on the remaining pixels of the white lane produces a composite image of the blue lane.

[0088] Furthermore, in the second example, the reference color channel is the R channel. Therefore, in the image of the white lane, the G channel ratio (13 / 150) and the B channel ratio (15 / 150), based on the R channel, are applied to the white lane portion. That is, in the image of the white lane, the white lane portion includes multiple pixels, each of which has color values ​​(i.e., RGB values) for the three color channels. In this case, the B channel remains unchanged for each pixel, while the color values ​​of the remaining R and G channels are converted. This conversion is performed by scaling according to the derived G channel ratio (13 / 150) and B channel ratio (15 / 150), thereby generating a composite image of the yellow lane. For example, in the image of the white lane, if the RGB values ​​of a pixel in the white lane portion are (240, 241, 242), the R channel color value of 240 remains unchanged. The G channel is scaled to convert it to a color value of 241×(13 / 150), and the B channel is scaled to convert it to a color value of 242×(15 / 150), thereby converting the corresponding pixel into a pixel in the yellow lane. Of course, performing the same transformation on the remaining pixels of the white lane produces a composite image of the yellow lane.

[0089] Of course, multiple ratio reference images may be used in step S210 (step S210'). In this case, the aforementioned ratios are calculated for each of the ratio reference images in step S210 (step S210') to calculate the average of their ratios. Subsequently, in step S220 (step S220'), the aforementioned conversion is performed relative to the image of the white lane using the average of the calculated ratios to generate a composite image.

[0090] When two ratio reference images are used, in step S210 (step S210'), after calculating the ratio (first ratio) relative to the first ratio reference image and the ratio (second ratio) relative to the second ratio reference image, the average of the first and second ratios is calculated. Then, in step S220 (step S220'), the above-described conversion is performed on the image of the white lane using the average of the calculated first and second ratios to generate a composite image.

[0091] On the other hand, in the ratio reference image, the ratio reference lane portion does not have predetermined RGB values ​​and may have fading, such as gradation. In this case, fading means that the color of the ratio reference lane formed by multiple pixel rows gradually becomes lighter as the pixel rows move toward one side (or as the pixel rows move outward) in the corresponding ratio reference image.

[0092] To generate a composite image that more clearly reflects this fading, in step S210 (step S210'), the aforementioned ratios may be calculated for each pixel row in the ratio reference lane within the ratio reference image. Subsequently, in step S220 (step S220'), the aforementioned conversion is performed using the calculated ratios for each pixel row to generate a composite image.

[0093] When the faded ratio reference image includes three pixel rows, in step S210 (step S210'), the average ratio of the pixels in the first pixel row (first row ratio), the average ratio of the pixels in the second pixel row (second row ratio), and the average ratio of the pixels in the third pixel row (third row ratio) are calculated. Then, in step S220 (step S220'), the calculated first to third row ratios are used to perform the aforementioned conversion based on the different calculated row ratios on each row in the image of the white lane to generate a composite image. Specifically, the conversion is performed using the first row ratio on the first pixel row of the white lane, the second row ratio on the second pixel row of the white lane, and the third row ratio on the third pixel row of the white lane.

[0094] In particular, fading can be reflected simultaneously using multiple ratio reference images. In this case, in step S210 (step S210'), the ratio is calculated for each pixel row of the ratio reference lane with fading, and the ratio is calculated for each pixel row of the multiple ratio reference lanes to calculate the average value thereof. Thereafter, in step S220 (step S220'), the above-mentioned conversion can be performed on the average value of the calculated ratios of the different pixel rows for each pixel row of the white lane to generate a composite image. Figure 4 and Figure 5 Explain it in more detail.

[0095] That is, refer to Figure 4 and Figure 5 , step S210 (step S210') may include steps S211 to S213 (steps S211' to S213'), and step S220 (step S220') may include steps S221 to S223 (steps S221' to S223'), which represent the case of reflecting fading while using multiple ratio reference images. However, Figure 5 Except for the situation Figure 4 Except that "blue" is replaced by "yellow", the contents are the same, so the following only describes Figure 4 .

[0096] First, in step S211, it is determined whether any of the multiple blue lane images (i.e., ratio reference images) contain any blue lane images for which the ratio is not reflected. Specifically, in step S210, multiple faded blue lane images are sequentially selected, and the ratios of the selected blue lane images are calculated. In step S211, it is determined whether any blue lane images for which such calculation has not been performed still exist. If such blue lane images exist, the process proceeds to step S212; otherwise, the process proceeds to step S213.

[0097] In step S212, for the image of a blue lane for which the ratio has not been calculated, the ratio is calculated for each pixel row of the corresponding blue lane. When the calculation is completed, the process returns to step S211. In this way, the ratio can be calculated for each pixel row of all blue lane images.

[0098] Next, in step S221, it is determined whether any of the multiple white lane images have not yet undergone the conversion based on the ratio. Specifically, in step S220, the multiple white lane images are sequentially selected, and the blue lane conversion is performed on the selected white lane images using the ratio calculated in step S213. In step S221, it is determined whether any white lane images have not yet undergone this conversion. If such white lane images exist, the process proceeds to step S222; otherwise, the process ends.

[0099] Then, in step S222 , the average value of the different pixel row ratios calculated in step S213 is applied to different pixel rows of the corresponding white lane in an image of a white lane that has not been converted according to the ratio application, so as to be converted into a blue lane.

[0100] Then, in step S223, a composite image of the blue lane is generated based on the conversion in step S222. When the composite image generation is completed, the process returns to step S221. In this way, composite images of the blue lane can be generated according to the ratio applied to all white lane images.

[0101] Meanwhile, in step S220 (step S220'), the image generation unit 152 may label the generated composite images of other lane colors relative to the corresponding lane colors. When generating a composite image of the blue lane according to the first example, the corresponding composite image may be labeled as the blue lane. Furthermore, when generating a composite image of the yellow lane according to the second example, the corresponding composite image may be labeled as the yellow lane. These labels may be used as the resulting values ​​(labels) of the training data in subsequent deep learning training using the corresponding composite images.

[0102] Figure 6a An image representing a white lane, Figure 6b Express Figure 6a The synthetic image of the blue lane generated by the generation method of the present invention represents the Figure 6a The synthetic image of the yellow lane generated using the generation method of the present invention (right side). Figure 7 Indicates the original lane (short dashed line frame) and the lane generated using the generation method of the present invention (long dashed line frame). Figure 8 A performance chart showing the deep learning model trained using the original training data. Figure 9 A graph showing the performance of a deep learning model trained using training data augmented by the generation method according to the present invention.

[0103] Reference Figures 6a to 9 , it can be seen that the generation method of the present invention can generate a synthetic image with characteristics very similar to the ratio reference lane (blue lane or yellow lane) for the image of the white lane. Therefore, the present invention can adjust the balance of the unbalanced training data of white, yellow and blue lanes with a uniform distribution. In particular, referring to Figure 8 Even with the latest deep learning algorithm, when trained with the original unbalanced training data, there is a problem that the performance such as the classification accuracy relative to the blue and yellow lanes is greatly reduced. On the contrary, when trained with the training data generated by the enhancement technology of the present invention while using the same deep learning algorithm, such as Figure 9 As shown, the classification accuracy (performance) relative to the blue and yellow lanes can be greatly improved.

[0104] The above-mentioned generating method can be loaded into the memory 140 and executed under the control of the control unit 150. Such a program can be stored in the memory 340 in various types of non-transitory computer readable media. Non-transitory computer readable media include various types of tangible storage media.

[0105] For example, non-transitory computer-readable media include magnetic recording media (e.g., floppy disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), compact disk read-only memory (CD-ROM), compact disk recordable (CD-R), compact disk erasable / rewritable (CD-R / W), semiconductor memory (e.g., mask read-only memory (mask ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), flash ROM, random access memory (RAM)), but are not limited to these.

[0106] Furthermore, the program can be supplied via various types of transitory computer-readable media. For example, transitory computer-readable media may include, but are not limited to, electrical signals, optical signals, or electromagnetic waves. Specifically, the program can be supplied to the control unit 350 via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0107] Figure 10 A block diagram showing the structure of an in-vehicle camera system 200 according to an embodiment of the present invention.

[0108] Meanwhile, camera system 200 is a system installed within a vehicle and performs various controls on camera 210 and on the images of the vehicle's surroundings captured by camera 210. Camera system 200 includes: camera 210, which is optical, etc.; memory 220, which stores various information such as the images captured by camera 210; and control unit 230, which uses the information in memory 220 for control. Specifically, camera system 200 can be the aforementioned generation device 100 applied as a system within a vehicle.

[0109] In this case, camera 210 may be an optical camera, etc., capable of capturing various images of the vehicle's surroundings, including lanes. Furthermore, memory 220 and control unit 230 can perform the aforementioned functions and generation methods, corresponding to memory 140 and control unit 150 of generation device 100. These have already been described and will not be further described. Furthermore, in addition to camera 210, memory 220, and control unit 230, camera system 200 may also include input unit 110, communication unit 120, display 130, etc., of generation device 100. These have already been described and will not be further described.

[0110] The present invention, constructed as described above, has the advantage of increasing training data for lanes of other colors to correct the imbalance in training data between white lanes and lanes of other colors for a deep learning model for color-based lane classification. Specifically, by generating synthetic images of the less frequent blue and yellow lanes, the present invention increases training data for these lanes. Furthermore, by converting the image of the white lane in the RGB channels of the ratio reference lane based on the ratio of the other color channels relative to the reference color channel, the present invention can more easily generate synthetic images of lanes of other colors that are closer to reality. Furthermore, the present invention can generate synthetic images that reflect a faded image of the image appearing in the white lane. Furthermore, by providing a relatively simple training data expansion technique, the present invention can contribute to the generation of a deep learning model for lane classification with significantly improved classification performance, such as classification accuracy.

[0111] In the detailed description of the present invention, specific embodiments are described, but various modifications may be made without departing from the scope of the present invention. Therefore, the scope of the present invention is not limited to the described embodiments, but should be determined according to the scope of the appended invention claims and their equivalents.

Claims

1. A method, performed by an electronic device or in-vehicle system, utilizing images of a white lane and other colored lanes to generate composite images of other colored lanes to generate training data for a lane classification deep learning model, characterized in that: include: In the image of the other color lane, for the three color channels of red (R), green (G) and blue (B) of the other color lane, the ratio of the other two channels is calculated based on one channel, namely the reference color channel; as well as The step of applying the calculated ratio to the other two channels relative to the base color channel of the white lane in the image of the white lane for scaling to generate a composite image of the other color lanes.

2. The method according to claim 1, characterized in that Among the three color channels, the reference color channel has the largest color value.

3. The method according to claim 1, characterized in that The other color lane is a yellow lane, and the reference color channel is an R channel.

4. The method according to claim 1, wherein The other color lane is a blue lane, and the reference color channel is a B channel.

5. The method according to claim 1, wherein The step of calculating includes the step of calculating the ratio for lanes of other colors with fading in different rows, The step of generating comprises the step of applying the calculated different row ratios to the respective rows of the white lane.

6. The method according to claim 1, characterized in that The step of calculating includes calculating the ratio for a plurality of other color lanes to calculate an average value thereof, The step of generating includes the step of applying a ratio of the average value to the white lane.

7. The method according to claim 1, characterized in that The calculating step includes calculating the ratio for lanes of other colors with fading according to different rows, and calculating the ratio for a plurality of lanes of other colors according to the different rows to calculate the average value thereof. The step of generating comprises the step of applying the calculated average value of the different row ratios to the respective rows of the white lane.

8. The method according to claim 1, characterized in that The generating step includes the step of labeling the generated composite images of other color lanes relative to the corresponding color lanes.

9. A device for generating training data for a lane classification deep learning model, characterized in that: include: a memory for storing images of white lanes and other color lanes; as well as The control unit generates a composite image of the other color lanes as the training data using the stored images of the white lane and the other color lanes, The control unit calculates the ratio of the other two channels of red (R), green (G) and blue (B) in the image of the other color lane based on one channel, that is, the reference color channel. The control unit applies the calculated ratio to the other two channels of the image of the white lane relative to the reference color channel of the white lane for scaling to generate a composite image of the other color lanes.

10. The device according to claim 9, characterized in that Among the three color channels, the reference color channel has the largest color value.

11. The device according to claim 9, characterized in that The control unit calculates the ratio for each line of the other color lane having faded color, and applies the calculated ratio for each line to each line of the white lane.

12. The device according to claim 9, characterized in that The control unit calculates the ratio for a plurality of lanes of other colors to calculate an average value thereof, and applies the ratio of the average value to the white lane.

13. The device according to claim 9, characterized in that The control unit calculates the ratio for each line of the other color lane with fading, calculates the ratio for each line of a plurality of other color lanes to calculate an average value, and applies the calculated average value of the ratio for each line to each line of the white lane.

14. The device according to claim 9, characterized in that The control unit labels the generated composite images of lanes of other colors relative to the corresponding color lanes.

15. The device according to claim 9, characterized in that The device also includes a camera for capturing images of the surroundings of the vehicle. The device utilizes the images of the surroundings of the vehicle captured by the camera and is suitable for use as a system within the vehicle.

16. A device for generating training data for a lane classification deep learning model, characterized in that include: The communication department receives images of the white lane and other color lanes; as well as The control unit generates a composite image of the other color lanes as the training data using the received images of the white lane and the other color lanes, The control unit calculates the ratio of the other two channels of red (R), green (G) and blue (B) in the image of the other color lane based on one channel, that is, the reference color channel. The control unit applies the calculated ratio to the other two channels of the image of the white lane relative to the reference color channel of the white lane for scaling to generate a composite image of the other color lanes.

Images