On-board active learning method and device for training perception networks of autonomous vehicles

Abstract

Provided is an on-board active learning method for training a perception network of an autonomous vehicle. The method comprises: (a) acquiring driving video and sensory information from the autonomous vehicle's camera and sensors, inputting frames in the driving video and the sensory information into a scene code generation module to generate a scene code, wherein the scene code includes information about the scene in the frame and about driving events; and (b) utilizing the scene code and object detection information of each frame, selecting frames whose object detection information meets a preset condition as specific frames, or utilizing the scene code and object detection information to select frames that match a learning strategy as specific frames, and storing the specific frames and specific scene codes in a frame storage unit.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. patent application No. 63 / 014,877, filed on April 24, 2020, and U.S. patent application No. 17 / 204,287, filed on March 17, 2021, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present invention relates to an on-vehicle active learning method and device for training a perception network of an autonomous driving vehicle. More specifically, the present invention relates to an on-vehicle active learning method and device for selecting learning data for training the perception network from real-time data of the autonomous driving vehicle and using the selected learning data to train the perception network. Background Art

[0004] In recent years, research has been ongoing into methods for performing object recognition and the like using machine learning.

[0005] Deep learning is a type of machine learning that uses a neural network with multiple hidden layers between the input layer and the output layer, which has high recognition performance.

[0006] In addition, neural networks using deep learning typically learn by using backpropagation of losses.

[0007] To train such deep learning networks, existing techniques involve collecting raw data according to a data collection strategy. Human annotators then annotate the collected raw data to generate new training data. The deep learning network is then trained using both the new and original training data. Human engineers then use performance analysis results to modify and improve the learning algorithm used to train the deep learning network. Furthermore, the analysis results are used to adjust the data collection strategy and recheck annotations for errors, making appropriate corrections.

[0008] However, in such existing methods, as the performance of deep learning networks improves, difficult samples useful for training become scarce, so the effectiveness of using new training data to improve the performance of deep learning networks decreases, and the return on investment of data annotation by human annotators decreases.

[0009] On the other hand, autonomous vehicles drive according to the vehicle's operating information and driving environment without any action from the driver. They use a deep learning-based perception network to detect the driving environment, such as objects around the vehicle, lane lines, traffic signals, etc.

[0010] Such autonomous vehicles require online learning, i.e., learning while the perception network is installed, to update the perception network. However, due to the limited storage capacity of the vehicle's embedded system, data sampling is required from a database (e.g., the cloud) storing learning data to obtain the learning data required to train the perception network.

[0011] For data sampling, existing technologies use random sampling methods, metadata sampling methods, and manual curation sampling methods, but such existing sampling methods are not suitable for in-vehicle learning.

[0012] Therefore, the existing technology has the disadvantage of performing active learning after storing all data offline in a database (eg, cloud). Summary of the Invention

[0013] Technical issues

[0014] The present invention aims to solve all the above problems.

[0015] Another object of the present invention is to provide a method capable of performing active learning online.

[0016] Another object of the present invention is to improve the learning effect of perceptual networks using new training data.

[0017] Another object of the present invention is to provide a method for performing on-board learning for a perception network of an autonomous vehicle.

[0018] Technical Solution

[0019] In order to achieve the above-mentioned object of the present invention and realize the characteristic effects of the present invention described below, the characteristic structure of the present invention is as follows.

[0020] According to one aspect of the present invention, there is provided an on-vehicle active learning method for training a perception network of an autonomous vehicle, comprising: (a) during driving of the autonomous vehicle, after obtaining driving video from a camera and sensing information from a vehicle sensor, the on-vehicle active learning device inputs consecutive frames in the driving video and the sensing information into a scene code generation module, so that the scene code generation module generates a scene code with reference to the frames and the sensing information, wherein the scene code includes information about the scene of each frame and information about a driving event; and (b) the on-vehicle active learning device performs at least one of the following processes: (i) utilizing The present invention relates to a process of: (i) using the scene code of each frame and the object detection information in each frame detected by the object detector to select the frame whose object detection information in the driving event meets the preset conditions as the specific frame for training the perception network of the autonomous driving vehicle, and matching the specific frame and the corresponding specific scene code and storing them in the frame storage unit; and (ii) using the scene code and the object detection information to select the frame that matches the learning strategy of the perception network of the autonomous driving vehicle as the specific frame, and matching the specific frame and the corresponding specific scene code and storing them in the frame storage unit.

[0021] The on-board active learning method may further include (c) the on-board active learning device (c1) sampling the specific frame stored in the frame storage unit through the specific scene code to generate learning data, and (c2) using the learning data to train the perception network of the autonomous driving vehicle on-board.

[0022] In (c), the on-board active learning device performs at least one of the following processes: (i) in step (c1), when sampling the specific frame, only a part of the majority category is selected with reference to the scene code and the minority category is undersampled using as much data as possible, or oversampling is performed to generate the same number of copies of the minority category as the majority category to generate the learning data and train the perception network, (ii) in step (c2), when training the perception network using the learning data, (ii-1) a weighted balanced loss is generated with respect to the learning data corresponding to the scene code through weighted balanced, and (ii-2) the perception network is trained by back propagation using the weighted balanced loss.

[0023] In (a), the on-board active learning device enables the scene code generation module to (i) perform a learning operation on each of the frames through a deep learning-based scene classifier to classify the scene of each of the frames into the categories of driving environment and driving road, thereby generating a scene category code for each of the frames, (ii) enables the driving event detection module to detect driving events occurring during the driving of the autonomous driving vehicle from the frames and the sensing information of each of the frames to generate a driving event code, and (iii) uses the scene category code and the driving event code of each of the frames to generate a scene code corresponding to each of the frames.

[0024] The on-board active learning device enables the scene code generation module to (i) generate a frame-based event code by detecting scene changes in the frame through the driving event detection module, and (ii) generate a vehicle-based event code by detecting the operating state of the autonomous driving vehicle in the sensing information, thereby generating the driving event code.

[0025] In (b), the on-board active learning device selects a frame in which no object is detected from a collision area in an accident event as the specific frame with reference to the scene code, wherein the collision area is an area in the frame where the predicted object is located when the autonomous driving vehicle collides with the object or predicts that a collision will occur.

[0026] In (b), the on-board active learning device selects a frame in which an object is detected from a collision area in a normal event as the specific frame with reference to the scene code, wherein the collision area is an area in the frame where the predicted object is located when the autonomous driving vehicle collides with the object or predicts that a collision will occur.

[0027] In (b), the on-board active learning device selects a frame containing an object whose confidence score about the object in the object detection information is lower than or equal to a preset value as the specific frame.

[0028] In (b), the on-board active learning device refers to the scene code and selects a frame in which a pedestrian is detected in a rare driving environment as the specific frame.

[0029] According to another aspect of the present invention, there is provided an on-vehicle active learning device for training a perception network of an autonomous vehicle, comprising: a memory storing instructions for performing on-vehicle active learning on consecutive frames in a video acquired from the autonomous vehicle; and a processor performing on-vehicle active learning operations according to the instructions stored in the memory, the processor performing at least one of the following processes: (I) during driving of the autonomous vehicle, after acquiring driving video from a lens and sensing information from a vehicle sensor, inputting consecutive frames in the driving video and the sensing information into a scene code generation module, so that the scene code generation module generates scene codes with reference to the frames and the sensing information, wherein the scene code comprises including information about the scene of each of the frames and information about the driving event; and (II)(i) using the scene code of each of the frames and the object detection information in each of the frames detected by the object detector, the frames in which the object detection information of the driving event meets the preset conditions are selected as specific frames for training the perception network of the autonomous driving vehicle, and the specific frames and the specific scene codes corresponding thereto are matched and stored in the frame storage unit; and (ii) using the scene code and the object detection information, the frames that match the learning strategy of the perception network of the autonomous driving vehicle are selected as specific frames, and the specific frames and the specific scene codes corresponding thereto are matched and stored in the frame storage unit.

[0030] The processor may further execute (III) the processor further executes the following process, (III-1) sampling the specific frame stored in the frame storage unit through the specific scene code to generate learning data, and (III-2) using the learning data to train the perception network of the autonomous driving vehicle on-board.

[0031] In the process (III), the processor performs at least one of the following processes: (i) in the process (III-1), when sampling the specific frame, only a part of the majority category is selected with reference to the scene code and under-sampling is performed using as much data as possible in the minority category, or oversampling is performed by generating data of the minority category copies to the same number as the majority category, thereby generating the learning data and training the perception network; (ii) and in the process (III-1), when training the perception network using the learning data, (ii-1) the weighted balanced loss of the learning data corresponding to the scene code is generated through weighted balanced, and (ii-2) the perception network is trained by back propagation using the weighted balanced loss.

[0032] In the process (I), the processor enables the scene code generation module to (i) perform a learning operation on each of the frames through a deep learning-based scene classifier to classify the scene of each of the frames into the categories of driving environment and driving road, thereby generating a scene category code for each of the frames; (ii) enables the scene code generation module to detect driving events occurring during the driving process of the autonomous driving vehicle from the frames and the sensing information of each of the frames through the driving event detection module, thereby generating a driving event code; and (iii) enables the scene code generation module to use the scene category code and the driving event code of each of the frames to generate a scene code corresponding to each of the frames.

[0033] The processor (i) causes the scene code generation module to detect scene changes in the frame through the driving event detection module to generate a frame-based event code, and (ii) causes the scene code generation module to detect the operating state of the autonomous driving vehicle in the sensing information through the driving event to generate a vehicle-based event code, thereby generating the driving event code.

[0034] In the process (II), the processor selects a frame in which no object is detected from a collision area in an accident event as the specific frame with reference to the scene code, wherein the collision area is an area in the frame where the object is predicted to be located when the autonomous driving vehicle collides with the object or predicts that a collision will occur.

[0035] In the process (II), the processor selects a frame in which an object is detected from a collision area in a normal event as the specific frame with reference to the scene code, wherein the collision area is an area in the frame where the object is predicted to be located when the autonomous driving vehicle collides with the object or predicts that a collision will occur.

[0036] In the process (II), the processor selects a frame containing an object whose confidence score about the object in the object detection information is lower than or equal to a preset value as the specific frame.

[0037] In the process (II), the processor selects a frame in which a pedestrian is detected in a rare driving environment as the specific frame with reference to the scene code.

[0038] In addition, the present invention also provides a computer-readable recording medium for recording a computer program for executing the method of the present invention.

[0039] Beneficial effects

[0040] The present invention assigns scene codes based on video images input from the vehicle and determines whether they are helpful for learning, and then stores them in the vehicle's storage unit, thereby improving the learning effect of the perception network through training data.

[0041] In addition, the present invention performs sampling and balancing of the training data used for learning according to the scene code, thereby enabling active learning to be performed online (ie, in the vehicle itself).

[0042] In addition, the present invention samples and balances the training data used for learning according to the scene code, thereby enabling on-board training of the perception network of the autonomous driving vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The following drawings used to illustrate the embodiments of the present invention are only part of the embodiments of the present invention. Technical personnel in the field to which the present invention belongs (hereinafter referred to as "technicians") can obtain other drawings based on the following drawings without making any creative work.

[0044] Figure 1 2 is a diagram schematically illustrating an on-board active learning device for on-board active learning according to an embodiment of the present invention, wherein the on-board active learning is used to train a perception network of an autonomous driving vehicle.

[0045] Figure 2 2 is a diagram schematically illustrating an on-board active learning method for training a perception network of an autonomous driving vehicle according to an embodiment of the present invention.

[0046] Figure 3 2 is a diagram schematically illustrating a method for generating scenario codes in on-board active learning for training a perception network of an autonomous vehicle according to an embodiment of the present invention.

[0047] Figure 4 2 is a diagram schematically illustrating a method for determining data useful for learning in on-board active learning for training a perception network of an autonomous vehicle according to an embodiment of the present invention.

[0048] Figure 5 2 is a diagram schematically illustrating another method for determining data useful for learning in on-board active learning for training a perception network of an autonomous vehicle according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] Hereinafter, the objects, technical solutions and advantages of the present invention will be described in detail with reference to the accompanying drawings, which illustrate specific embodiments in which the present invention can be implemented. These embodiments are described in sufficient detail to enable those skilled in the art to implement the present invention.

[0050] In addition, in the present invention's summary and claims, the term "comprising" and its variations are not intended to exclude other technical features, additional components, components, or steps. Other objects, advantages, and features of the present invention will become apparent to those skilled in the art in part from this specification and in part from the embodiments of the present invention. The following examples and figures are provided as illustrative examples and are not intended to limit the present invention.

[0051] In addition, the present invention covers all combinations of the embodiments in this specification. It should be understood that the various embodiments of the present invention, although different, do not necessarily exclude each other. For example, the specific shapes, structures and characteristics described in this specification can be implemented in other embodiments for one embodiment without departing from the technical concept and scope of the present invention. In addition, it should be understood that the position or configuration of the various components of each disclosed embodiment can be changed without departing from the technical concept and scope of the present invention. Therefore, the following detailed description is not restrictive, and if properly described, the scope of the present invention is limited only by the contents of the claims and all scopes equivalent thereto and the claims. Similar reference numbers in the drawings refer to the same or similar functions in many aspects.

[0052] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings so that those skilled in the art to which the present invention pertains can easily implement the present invention.

[0053] Figure 1 FIG2 is a diagram schematically illustrating an on-board active learning device for on-board active learning according to an embodiment of the present invention, wherein the on-board active learning is used to train a perception network of an autonomous driving vehicle. Figure 1 The on-board active learning device 1000 may include a memory 1001 and a processor 1002, where instructions are stored in the memory 1001, and the instructions are used to perform on-board active learning on consecutive frames in a video obtained from an autonomous driving vehicle, and the processor 1002 performs on-board active learning operations according to the instructions stored in the memory 1001.

[0054] Specifically, the learning device 1000 can generally utilize a combination of a computing device and computer software (i.e., instructions that enable the computing device to function in a specific manner) to achieve the desired system performance. For example, the computing device can be a device including a computer processor, memory, storage, input devices and output devices, as well as other existing computing device components; electronic communication devices such as routers and switches; electronic information storage systems such as network attached storage (NAS) and storage area networks (SAN).

[0055] In addition, the processor of the computing device may include hardware configurations such as an MPU (Micro Processing Unit) or a CPU (Central Processing Unit), cache memory, and a data bus. In addition, the computing device may further include a software configuration of an operating system and application programs for executing specific purposes.

[0056] However, it is not excluded that the computing device does not include a form in which the medium, processor, and memory for implementing the present invention are integrated, ie, an integrated processor.

[0057] Below, reference Figure 2 A method for performing on-vehicle active learning using an on-vehicle active learning device 1000 according to an embodiment of the present invention is described. The on-vehicle active learning device 1000 is used for on-vehicle active learning to train a perception network of an autonomous driving vehicle.

[0058] First, during the operation of the autonomous driving vehicle, after obtaining sensing information from the driving video of the lens (e.g., image sensor) and the vehicle sensor, the on-board active learning device 1000 inputs the continuous frames and sensing information in the driving video into the scene code generation module 1200, so that the scene code generation module 1200 generates a scene code with reference to the frames and sensing information, wherein the scene code includes information for each frame scene and information for a driving event.

[0059] At this time, the scene code can be obtained by encoding information for each scene and information for the driving event.

[0060] For example, reference Figure 3 The scene code generation module 1200 can use a deep learning-based scene classifier 1210 to perform a learning operation on each frame to classify the scene of each frame into the categories of the driving environment and the driving road, thereby generating a scene category code for each frame. In addition, the scene classifier 1210 can extract features from each frame and classify the extracted features into categories corresponding to preset driving environments and driving roads, thereby generating a scene category code for each frame.

[0061] In this case, the driving environment may include, but is not limited to, weather information and time period information for the autonomous vehicle's driving area. It may also include various types of weather information for the autonomous vehicle's driving area or region. Furthermore, weather information may include sunny, rainy, snowy, and foggy weather conditions, while time period information may include daytime and nighttime weather conditions. Furthermore, the driving road may include the type of road the autonomous vehicle is traveling on, such as, for example, expressways, urban roads, and tunnels, but is not limited to, it may include various road environments the autonomous vehicle is traveling on.

[0062] In addition, the scene code generation module 1200 can detect driving events occurring during the driving process of the autonomous driving vehicle from the frames and the sensing information of each frame through the driving event detection module 1220, thereby generating a driving event code.

[0063] At this time, the driving event code may include a frame-based event code detected using continuous frames and a vehicle-based event code detected using sensing information.

[0064] For example, the scene code generation module 1200 can input consecutive frames into the scene change detector of the driving event detection module 1220, causing the scene change detector to detect whether the scene in each frame has changed and generate a frame-based event code for each frame based on whether the scene has changed. In this case, the frame-based event code may include a uniform sample, scene change, or other code based on whether the scene has changed. Furthermore, the scene code generation module 1200 can use sensory information to detect the operating state of the autonomous vehicle and detect events occurring during vehicle operation, thereby generating vehicle-based event codes. Vehicle-based event codes may include rapid steering, slam on the brakes, normal action, AEB activated, and other events. Furthermore, the scene code generation module 1200 can generate a scene code for each frame using the scene category code and driving event code for each frame.

[0065] At this point, the scene code corresponding to each frame can be shown in the following table.

[0066]

[0067] On the other hand, the above-mentioned scene codes are exemplary, and the present invention is not limited thereto. The scene codes corresponding to each continuous frame in the driving video can be generated in various forms.

[0068] On the other hand, re-referencing Figure 2, the driving video and sensing information can be input into the driving video and operation information analysis module 1110 of the autonomous driving vehicle.

[0069] The driving video and operation information analysis module 1110 can then use the perception network to learn from consecutive frames of the driving video to detect information about the autonomous vehicle's surrounding environment, such as objects such as vehicles and pedestrians, lane markings on the road, and traffic signals. The module can then use this information to detect the autonomous vehicle's operating status. Furthermore, the autonomous vehicle's surrounding environment information and operating status information can be transmitted to the autonomous driving control unit 1500, which can use this information to control the autonomous vehicle's driving operations.

[0070] For example, the driving video and operation information analysis module 1110 performs object detection on frames of the driving video using a deep learning-based object detector (e.g., a convolutional neural network (CNN)) to generate object detection information for each frame, or segments the frames of the driving video using a deep learning-based segmentation network to generate lane information for each frame. Furthermore, the driving video and operation information analysis module 1110 can reference the sensing information to output vehicle operation status information, such as the acceleration and deceleration status of the autonomous vehicle, steering wheel operation status, and whether the autonomous emergency braking system (AEB) is engaged.

[0071] The on-board active learning device 1000 can then use the scene code of each frame and the object detection information of each frame detected by the object detector to enable the frame selection module 1300 to select frames that are helpful for learning data, and store the frames selected as learning data in the frame storage unit 1400, wherein the learning data is used to train the perception network of the autonomous driving vehicle.

[0072] That is, the on-vehicle active learning device 1000 can enable the frame selection module 1300 to select frames, i.e., images, from continuous frames obtained through driving videos that are helpful for training a perception network based on deep learning of an autonomous driving vehicle.

[0073] At this time, the frame selection module 1300 can select frames that are helpful for training the perception network through various methods.

[0074] That is, the on-board active learning device 1000 can perform the following process, using the scene code of each frame and the object detection information of each frame detected by the object detector, so that the frame selection module 1300 selects the frame whose object detection information of the driving event in the frame meets the preset conditions as the specific frame for training the perception network of the autonomous driving vehicle, and matches the specific frame and the corresponding specific scene code and stores them in the frame storage unit 1400, that is, the memory with limited storage capacity installed on the autonomous driving vehicle.

[0075] In addition, the on-board active learning device 1000 can perform the following process, using the scene code and object detection information of each frame to enable the frame selection module 1300 to select the frame that matches the learning strategy of the autonomous driving vehicle perception network as a specific frame, and match the specific frame and the corresponding specific scene code and store them in the frame storage unit 1400.

[0076] For example, the in-vehicle active learning device 1000 can refer to the scene code and select frames in the collision zone of an accident event in which no object was detected as specific frames that facilitate perceptual network learning. In this case, the accident event can be a driving event executed when the autonomous vehicle is in a collision state or a collision is predicted, such as sudden braking, a sharp left turn, or a sharp right turn. Furthermore, the collision zone can be the area where the object is predicted to be located in a frame in which the autonomous vehicle collides with an object or a collision is predicted.

[0077] That is, when the event code of the autonomous driving vehicle is a sharp turn, a sharp left turn, a sharp right turn, etc., an object should be detected in the collision area, but when the object is not detected in the collision area in the frame of the driving video, it is suspected to be a false negative, so the corresponding frame can be selected as a frame that helps the perception network learn.

[0078] For example, the in-vehicle active learning device 1000 may refer to the scene code to select a frame in which an object is detected in a collision area of ​​a normal event as a specific frame that is helpful for perceptual network learning.

[0079] That is, under normal driving conditions, such as when no accident occurs, no object should be detected in the collision zone of an autonomous vehicle. However, when an object is detected in the collision zone in a frame of a driving video, it is suspected to be a functional false alarm. Therefore, the corresponding frame can be selected as a frame that helps the perception network learn.

[0080] In addition, the in-vehicle active learning device 1000 may select a frame in which an object exists and a confidence score about the object in the object detection information is lower than or equal to a preset value (ie, a threshold) as a specific frame that facilitates perceptual network learning.

[0081] In addition, for frames other than the above special cases, the perception network will make accurate judgments and determine that they are frames that are not helpful for the perception network to learn, and thus discard the corresponding frames.

[0082] On the other hand, the on-vehicle active learning device 1000 can select frames in which pedestrians are detected in rare driving environments as specific frames that are helpful for learning the perception network based on the learning strategy of the perception network and with reference to the scene code.

[0083] For example, when the scene code is rainy night, it can be judged that the frame without detected pedestrians is a hard case that helps the perception network learn, and thus the corresponding frame can be judged to help the perception network learn. When the scene code is daytime / sunny, it is possible to judge that the perception network has been fully learned, and in order to prevent overfitting, the corresponding frame can be judged to be unhelpful to the learning of the perception network.

[0084] However, the above method of determining whether a frame in a driving video is helpful for learning a perception network is exemplary, and the present invention is not limited thereto. Various methods may be used to determine whether a frame is helpful for learning a perception network according to setting conditions.

[0085] On the other hand, the frame selection module 1300 can determine whether the frames in the driving video are helpful for learning the perception network through a trained network (i.e., a trained deep learning network).

[0086] For example, reference Figure 4 The frame selection module 1300 can input frames into the automatic annotation network 1310 and the trained deep learning network 1320 respectively, and compare the output information output from the automatic annotation network 1310 and the output information output from the trained deep learning network 1320. When the output information of the automatic annotation network 1310 and the output information of the trained deep learning network 1320 are the same or similar, it is judged that the corresponding frame does not contribute to the learning of the perception network. When the difference between the two is greater than or equal to a preset threshold, it is judged as a hard example, thereby judging that the corresponding frame contributes to the learning of the perception network.

[0087] For example, refer to Figure 5, the frame selection module 1300 can modify the frame in various ways to generate multiple modified frames. In this case, the modification can be performed by various methods such as resizing the frame, changing the aspect ratio or color tone. Then, each of the multiple modified frames is input into the trained deep learning network 1320, and the variance of the output values ​​corresponding to each of the multiple modified frames output from the trained deep learning network 1320 is calculated. When the calculated variance is less than or equal to a preset threshold, the corresponding frame is judged to be unhelpful for the learning of the perception network. When the calculated variance is greater than the preset threshold, it is judged to be a difficult sample, thereby judging that the corresponding frame is helpful for the learning of the perception network.

[0088] Then, the on-board active learning device 1000 can sample specific frames stored in the frame storage unit 1400 through specific scene codes to generate learning data, and use the learning data to train the perception network of the autonomous driving vehicle on-board.

[0089] At this time, the on-board active learning device 1000 can perform the following process: when sampling a specific frame stored in the frame storage unit 1400, the reference scene code selects only a part of the majority category and uses as much data as possible for undersampling of the minority category, or oversampling the data of the minority category to generate the same number of copies as the majority category to generate learning data and use the sampled learning data to train the perception network.

[0090] In addition, the on-board active learning device 1000 can perform the following process: when training the perception network using a specific frame stored in the frame storage unit 1400, a weighted balanced loss of learning data about the scene code is generated through weight balancing, and the perception network is trained by back propagation using the weighted balanced loss.

[0091] In addition, the embodiments of the present invention as described above can be implemented in the form of program commands executed by various computer components and recorded to a computer-readable recording medium. The computer-readable recording medium can include program commands, data files, data structures, etc., alone or in combination. The program commands recorded to the computer-readable recording medium can be specially designed or configured for the present invention, or program commands known and used by technicians in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, and hardware devices such as ROMs, RAMs, and flash memories that are specially configured to store and execute program commands. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware device can be configured to operate as one or more software modules to perform the processing according to the present invention, and vice versa.

[0092] The present invention has been described above through specific matters such as specific components, limited embodiments, and accompanying drawings. However, these are intended to facilitate a more comprehensive understanding of the present invention. The present invention is not limited to the embodiments, and those skilled in the art to which the present invention belongs may attempt various modifications and changes based on these descriptions.

[0093] Therefore, the technical concept of the present invention should not be limited to the embodiments described above, and all contents of the claims of the present invention and their equivalents or equivalent transformations belong to the technical concept of the present invention.

Claims

1. An on-board active learning method for training a perception network of an autonomous vehicle, wherein: include: (a) During driving of the autonomous vehicle, after acquiring driving video from a camera and sensing information from vehicle sensors, the on-board active learning device inputs consecutive frames of the driving video and the sensing information into a scene code generation module, causing the scene code generation module to generate a scene code with reference to the frames and the sensing information, the scene code including information about the scene of each frame and information about the driving event; (b) the on-board active learning device performs at least one of the following processes: (i) using the scene code of each of the frames and the object detection information in each of the frames detected by the object detector, selecting frames whose object detection information in the driving event meets a preset condition as specific frames for training the perception network of the autonomous driving vehicle, and matching the specific frames with the specific scene codes corresponding thereto and storing them in a frame storage unit; and (ii) using the scene code and the object detection information, selecting frames that match the learning strategy of the perception network of the autonomous driving vehicle as specific frames, and matching the specific frames with the specific scene codes corresponding thereto and storing them in the frame storage unit; as well as (c) The on-board active learning device (c1) samples the specific frame stored in the frame storage unit through the specific scene code to generate learning data, and (c2) uses the learning data to train the perception network of the autonomous driving vehicle on-board.

2. The method according to claim 1, wherein In step (c): The on-board active learning device performs at least one of the following processes: (i) in step (c1), when sampling the specific frame, only a part of the majority class is selected with reference to the scene code and the minority class is undersampled using as much data as possible, or oversampling is performed to generate the same number of copies of the minority class as the majority class to generate the learning data and train the perception network, (ii) in step (c2), when training the perception network using the learning data, (ii-1) a weighted balanced loss is generated with respect to the learning data corresponding to the scene code by weighted balanced, and (ii-2) the perception network is trained by back propagation using the weighted balanced loss.

3. The method according to claim 1, wherein In step (a): The on-board active learning device enables the scene code generation module to (i) perform a learning operation on each of the frames through a deep learning-based scene classifier to classify the scene of each frame into the categories of driving environment and driving road, thereby generating a scene category code for each of the frames; (ii) enables the driving event detection module to detect driving events occurring during the driving of the autonomous driving vehicle from the frames and the sensing information of each of the frames to generate a driving event code; and (iii) utilize the scene category code and the driving event code of each of the frames to generate a scene code corresponding to each of the frames.

4. The method according to claim 3, wherein: The on-board active learning device enables the scene code generation module to (i) generate a frame-based event code by detecting scene changes in the frame through the driving event detection module, and (ii) generate a vehicle-based event code by detecting the operating state of the autonomous driving vehicle in the sensing information, thereby generating the driving event code.

5. The method according to claim 1, wherein In step (b): The on-board active learning device selects a frame in which no object is detected from a collision area in an accident event as the specific frame with reference to the scene code, wherein the collision area is an area in the frame where the predicted object is located when the autonomous driving vehicle collides with the object or predicts that a collision will occur.

6. The method according to claim 1, wherein In step (b): The on-board active learning device selects a frame in which an object is detected from a collision area in a normal event as the specific frame with reference to the scene code, wherein the collision area is an area in the frame where the predicted object is located when the autonomous driving vehicle collides with the object or predicts that a collision will occur.

7. The method according to claim 1, wherein In step (b): The vehicle-mounted active learning device selects a frame containing an object whose confidence score about the object in the object detection information is lower than or equal to a preset value as the specific frame.

8. The method according to claim 1, wherein In step (b): The on-vehicle active learning device refers to the scene code and selects a frame in which a pedestrian is detected in a rare driving environment as the specific frame.

9. An on-board active learning device for training a perception network of an autonomous vehicle, wherein: include: a memory storing instructions for performing on-board active learning on consecutive frames of a video acquired from an autonomous vehicle; as well as a processor that performs on-board active learning operations according to instructions stored in the memory, The processor performs at least one of the following processes: (I) during driving of the autonomous vehicle, after acquiring driving video from a camera and sensing information from a vehicle sensor, inputting consecutive frames in the driving video and the sensing information into a scene code generation module, so that the scene code generation module generates a scene code with reference to the frames and the sensing information, the scene code including information about a scene of each frame and information about a driving event; (II) (i) using the scene code of each of the frames and object detection information in each of the frames detected by the object detector, selecting a frame in which the object detection information in the driving event satisfies a preset condition as a specific frame for training a perception network of the autonomous driving vehicle, and matching the specific frame and the specific scene code corresponding thereto and storing them in a frame storage unit; and (ii) using the scene code and the object detection information, selecting a frame that matches the learning strategy of the perception network of the autonomous driving vehicle as a specific frame, and matching the specific frame and the specific scene code corresponding thereto and storing them in the frame storage unit; and (III) the processor further performs the following process: (III-1) sampling the specific frame stored in the frame storage unit through the specific scene code to generate learning data, and (III-2) using the learning data to train the perception network of the autonomous driving vehicle on-board.

10. The vehicle-mounted active learning device according to claim 9, wherein: In process (III): The processor performs at least one of the following processes: (i) in process (III-1), when sampling the specific frame, referring to the scene code, selecting only a portion of the majority class and performing undersampling using as much data as possible from the minority class, or performing oversampling by generating replicas of the minority class to the same number as that of the majority class, thereby generating the learning data and training the perception network; (ii) and in process (III-2), when training the perception network using the learning data, (ii-1) generating a weighted balanced loss of the learning data corresponding to the scene code through weighted balanced, and (ii-2) training the perception network by backpropagation using the weighted balanced loss.

11. The vehicle-mounted active learning device according to claim 9, wherein: In process (I), the processor causes the scene code generation module (i) to perform a learning operation on each of the frames using a deep learning-based scene classifier to classify the scene of each frame into a driving environment and a driving road category, thereby generating a scene category code for each of the frames; (ii) enabling the scene code generation module to detect, through a driving event detection module, a driving event occurring during the driving process of the autonomous driving vehicle from the frames and the sensing information of each of the frames, thereby generating a driving event code; (iii) enabling the scene code generation module to generate a scene code corresponding to each of the frames using the scene category code and the driving event code of each of the frames.

12. The vehicle-mounted active learning device according to claim 11, wherein: The processor (i) causes the scene code generation module to detect a scene change in the frame through the driving event detection module to generate a frame-based event code, and (ii) causes the scene code generation module to detect an operating state of the autonomous driving vehicle in the sensing information through the driving event to generate a vehicle-based event code, thereby generating the driving event code.

13. The vehicle-mounted active learning device according to claim 9, wherein: In process (II), the processor selects, with reference to the scenario code, a frame in which no object is detected in a collision zone during an accident event as the specific frame, wherein the collision zone is an area in the frame where the object is predicted to be located when the autonomous driving vehicle collides with the object or predicts that a collision will occur.

14. The vehicle-mounted active learning device according to claim 9, wherein: In process (II), the processor selects, with reference to the scenario code, a frame in which an object is detected from a collision zone in a normal event as the specific frame, wherein the collision zone is an area in the frame where the object is predicted to be located when the autonomous driving vehicle collides with the object or predicts that a collision will occur.

15. The vehicle-mounted active learning device according to claim 9, wherein: In process (II), the processor selects a frame in which an object having a confidence score of the object in the object detection information lower than or equal to a preset value exists as the specific frame.

16. The vehicle-mounted active learning device according to claim 9, wherein: In process (II), the processor selects a frame in which a pedestrian is detected in a rare driving environment as the specific frame with reference to the scene code.

Images