Server device and method of controlling server device

Abstract

The invention relates to a server apparatus and a method of controlling the server apparatus. The server apparatus includes: a communicator configured to communicate with a vehicle; a storage medium configured to store a machine learning model and training data; and one or more processors electrically connected to the communication circuitry and the storage medium. The one or more processors are configured to obtain training data including reference data and marker data corresponding to the reference data, train a machine learning model using the training data, receive, via the communicator, detection data and detection trajectories corresponding to the detection data from the vehicle, evaluate the trained machine learning model using the training data, and determine, via the communicator, detection trajectories corresponding to the detection data. The machine learning model is corrected using the detection data and the detection trajectory based on the evaluation result, and the corrected machine learning model is output to the vehicle.

Description

Technical Field

[0001] Some embodiments of the present invention generally relate to a server device for assisting vehicle driving and a method of controlling the server device. Background Art

[0002] Vehicles are the most common means of transportation in modern society, and the number of people using vehicles is increasing day by day. Although the development of vehicle technology has brought advantages such as easy long-distance travel and convenience of life, problems such as deteriorating road traffic conditions and serious traffic jams often occur in densely populated areas.

[0003] Recently, research has been actively conducted on vehicles equipped with advanced driver assistance systems (ADAS) for proactively providing information on a vehicle state, a driver state, and / or surroundings in order to reduce the burden on the driver, provide assistance in driving the vehicle, and improve convenience.

[0004] Examples of ADAS installed on vehicles include Lane Departure Warning (LDW), Lane Keeping Assist (Lane Keeping Assist, LKA), High Beam Assist (HBA), Autonomous Emergency Braking (AEB), Traffic Sign Recognition (TSR), Adaptive Cruise Control (ACC), Blind Spot Detection (BSD), etc.

[0005] ADAS can collect information or data about the external environment of the vehicle and process the collected information. In addition, ADAS can identify objects and design a route for the vehicle to travel based on the results of processing the collected information or data.

[0006] ADAS may include a sensor module such as a camera or a radar, and may acquire information about one or more objects around or outside the vehicle by fusing detection data collected by each of the camera and the radar.

[0007] Recently, to assist driving, ADAS may use light detection and ranging (LiDAR) as well as cameras and radars to fuse detection data collected by each of the camera, radar, and LiDAR. Summary of the invention

[0008] An aspect of the present invention is to provide a server device for driving assistance and a method of controlling the server device, the server device being capable of acquiring object information from Light Detection and Ranging (LiDAR) data using a machine learning model.

[0009] Another aspect of the present invention is to provide a server device for driving assistance and a method for controlling the server device, which can use image data, radar data, LiDAR data and / or sensor fusion data to train a machine learning model.

[0010] Another aspect of the present invention is to provide a server device for driving assistance and a method for controlling the server device, wherein the server device is capable of acquiring object information from LiDAR data using a machine learning model trained using image data, radar data, LiDAR data and / or sensor fusion data.

[0011] Additional aspects of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

[0012] According to one aspect of the present invention, there is provided a server device, comprising: a communication circuit configured to communicate with a vehicle; a storage medium configured to store a first machine learning model and a second machine learning model as well as first training data and second training data; and a processor electrically connected to the communication circuit and the storage medium. The processor acquires second training data including reference data and reference tag data corresponding to the reference data, trains the second machine learning model using the second training data, receives detection data and a detection trajectory corresponding to the detection data from the vehicle through the communication circuit, corrects the trained second machine learning model using the detection data and the detection trajectory, acquires first training data using the corrected second machine learning model, trains the first machine learning model using the first training data, and provides the trained first machine learning model to the vehicle.

[0013] The processor can input reference data into a second machine learning model, obtain first labeled data corresponding to the reference data from the second machine learning model, and train the second machine learning model to reduce the error between the first labeled data and the reference labeled data.

[0014] The processor can input reference data into a trained second machine learning model, obtain evaluation label data corresponding to the reference data from the trained second machine learning model, and correct the trained second machine learning model when an error between the evaluation label data and the reference label data is greater than a reference error.

[0015] When the error between the evaluation labeled data and the reference labeled data is less than or equal to the reference error, the processor may use the trained second machine learning model to obtain the first training data.

[0016] The processor can input the detection data into a trained second machine learning model, obtain correction label data corresponding to the detection data from the trained second machine learning model, and correct the trained second machine learning model to reduce the correction error between the correction label data and the detection trajectory.

[0017] The processor may adjust the correction error based on the evaluation error.

[0018] The processor may adjust the correction error such that the correction error increases as the evaluation error increases and such that the correction error decreases as the evaluation error decreases.

[0019] According to another aspect of the present invention, a method for controlling a server device is provided, the server device comprising: a communication circuit configured to communicate with a vehicle; and a storage medium configured to store a first machine learning model and a second machine learning model as well as first training data and second training data, the method for controlling the server device comprising the following steps: obtaining second training data comprising reference data and reference tag data corresponding to the reference data, training the second machine learning model using the second training data, receiving detection data and a detection trajectory corresponding to the detection data from the vehicle through the communication circuit, correcting the trained second machine learning model using the detection data and the detection trajectory, obtaining first training data using the corrected second machine learning model, training the first machine learning model using the first training data, and providing the trained first machine learning model to the vehicle.

[0020] The step of training the second machine learning model may include the following steps: inputting reference data into the second machine learning model, obtaining first labeled data corresponding to the reference data from the second machine learning model, and training the second machine learning model to reduce the error between the first labeled data and the reference labeled data.

[0021] The step of evaluating the second machine learning model may include the steps of inputting reference data into the trained second machine learning model, obtaining evaluation label data corresponding to the reference data from the trained second machine learning model, and correcting the trained second machine learning model when the error between the evaluation label data and the reference label data is greater than the reference error.

[0022] The method of controlling the server device may further include the step of acquiring first training data using a trained second machine learning model when an error between the evaluation label data and the reference label data is less than or equal to a reference error.

[0023] The step of correcting the second machine learning model may include the steps of inputting the detection data into a trained second machine learning model, obtaining correction label data corresponding to the detection data from the trained second machine learning model, and correcting the trained second machine learning model to reduce the correction error between the correction label data and the detection trajectory.

[0024] The step of correcting the second machine learning model may include the step of adjusting the correction error based on the evaluation error.

[0025] The step of adjusting the correction error may include the step of adjusting the correction error such that the correction error increases as the evaluation error increases, and such that the correction error decreases as the evaluation error decreases.

[0026] According to another aspect of the present invention, there is provided a server device, comprising: a communication circuit configured to communicate with a vehicle; a storage medium configured to store a machine learning model and training data; and a processor electrically connected to the communication circuit and the storage medium. The processor acquires training data including reference data and labeled data corresponding to the reference data, trains the machine learning model using the training data, receives detection data and a detection trajectory corresponding to the detection data from the vehicle through the communication circuit, evaluates the trained machine learning model using the training data, corrects the machine learning model using the detection data and the detection trajectory based on the evaluation result, and provides the corrected machine learning model to the vehicle.

[0027] The processor may input reference data into a machine learning model, obtain a training trajectory corresponding to the reference data from the machine learning model, and train the machine learning model to reduce an error between the training trajectory and the labeled data.

[0028] The processor can input reference data into a trained machine learning model, obtain an evaluation trajectory corresponding to the reference data from the trained machine learning model, and correct the trained machine learning model when the error between the evaluation trajectory and the labeled data is greater than the reference error.

[0029] The processor may input the detection data into a trained machine learning model, obtain a correction trajectory corresponding to the detection data from the trained machine learning model, and correct the trained machine learning model to reduce a correction error between the correction trajectory and the detection trajectory.

[0030] The processor may adjust the correction error based on the evaluation error.

[0031] The processor may adjust the correction error such that the correction error increases as the evaluation error increases and such that the correction error decreases as the evaluation error decreases. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] These and / or other aspects of the present invention will become apparent and more easily understood from the following description of the embodiments in conjunction with the following drawings:

[0033] Figure 1 is a schematic diagram for illustrating a driving assistance system according to an embodiment of the present invention;

[0034] Figure 2 is a block diagram for illustrating a configuration of a vehicle including a driving assistance device according to an embodiment of the present invention;

[0035] Figure 3 An example of a field of view of a sensor module included in a driving assistance device according to an embodiment of the present invention is shown;

[0036] Figure 4 is a block diagram for illustrating a configuration of a server device according to an embodiment of the present invention;

[0037] Figure 5 is a diagram showing an example in which light detection and ranging (LiDAR) data and fused data are matched;

[0038] Figure 6 is a diagram showing an example in which a server device according to an embodiment of the present invention adjusts an error between an output of a machine learning model and training data;

[0039] Figure 7 is a flowchart for illustrating a method for training a machine learning model according to an embodiment of the present invention;

[0040] Figure 8 is a flowchart for illustrating a method for labeling training data for training a machine learning model according to an embodiment of the present invention; and

[0041] Fig. 9 is a flowchart for illustrating a method of labeling training data for training a machine learning model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatuses, and / or systems described herein. Accordingly, various changes, modifications, and equivalents of the methods, apparatuses, and / or systems described herein will be suggested to those of ordinary skill in the art. The progression of processing steps described is an example; however, the order of steps and / or steps are not limited to the order of steps and / or steps set forth herein, and, except for steps that must occur in a particular order, may be varied as known in the art. In addition, for greater clarity and conciseness, corresponding descriptions of well-known functions and structures may be omitted.

[0043] In addition, exemplary embodiments will now be described more fully below with reference to the accompanying drawings. However, exemplary embodiments may be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein. These embodiments are provided to make the present invention detailed and complete and to fully convey the exemplary embodiments to those of ordinary skill in the art. The same reference numerals represent the same elements throughout.

[0044] It should be understood that although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. As used herein, the term "and / or" includes any and all combinations of one or more of the relevant listed items.

[0045] It should be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or intervening elements may exist. In contrast, when an element is referred to as being "directly connected" or "directly coupled" to another element, there are no intervening elements.

[0046] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.As used herein, the singular forms "a", "an", and "the" include the plural forms as well, unless the context clearly indicates otherwise.

[0047] The expression "at least one of a, b, and c" should be understood to include only a, only b, only c, both a and b, both a and c, both b and c, or all a, b, and c.

[0048] Reference will now be made in detail to the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout.

[0049] Figure 1 is a schematic diagram for illustrating a driving assistance system according to an embodiment of the present invention.

[0050] refer to Figure 1 , the driving assistance system 1 may include a driving assistance device 100 installed in or included in a vehicle 10 and a server device 200 .

[0051] The vehicle 10 may travel on roads or tracks using fossil fuels, electricity, or any power source. The vehicle 10 may include a driving assistance device 100 for assisting a driver in driving or assisting in driving the vehicle 10 .

[0052] The driving assistance device 100 may include a sensor module for detecting data about at least one object located outside or around the vehicle 10. The driving assistance device 100 may process the detection data from the sensor module and acquire a trajectory representing an object outside or around the vehicle based on the processed detection data. For example, the driving assistance device 100 may acquire an image trajectory by processing image data, acquire a radar trajectory by processing radar data, and acquire a light detection and ranging (LiDAR) trajectory by processing LiDAR data.

[0053] The image track, radar track, or LiDAR track may each represent at least one object outside or around the vehicle 10. Information about the image track, radar track, or LiDAR track may include, for example, but not limited to, the location (e.g., distance and azimuth) of at least one object outside or around the vehicle 10, the size of at least one object outside or around the vehicle 10, the speed of at least one object (e.g., relative speed with respect to the vehicle 10), or the classification or type of at least one object outside the vehicle 10 (e.g., a stationary object, a vehicle, a pedestrian, or a cyclist).

[0054] Information about the detection data (e.g., image data, radar data, or LiDAR data) and / or the trajectory (e.g., image trajectory, radar trajectory, or LiDAR trajectory) of the sensor module may be provided to the server device 200. For example, the information about the detection data and / or trajectory of the sensor module may be provided to the server device 200 in real time through the wireless communication system of the vehicle 10. As another example, the information about the detection data and / or trajectory of the sensor module may be provided to the server device 200 after the vehicle 10 stops (or immediately after the stop) or at a predetermined time. As yet another example, the information about the detection data and / or trajectory of the sensor module may be stored in a storage medium and provided to the server device 200 through the storage medium.

[0055] The server device 200 may include a first machine learning model for outputting an image track, a radar track, a LiDAR track, and / or a fused track from the image data, the radar data, and / or the LiDAR data.

[0056] The machine learning model may be, for example, an algorithm or model for obtaining a relationship or function between input and output data by learning (or training) a plurality of input and output data, and automatically outputting output data corresponding to the input data using the relationship or function between the input and output data obtained. The machine learning model may include, for example, a deep learning model, but is not limited thereto. The deep learning model may be a machine learning model that uses an artificial neural network structure (or an artificial neural network algorithm) that is a human logical structure. For example, the deep learning model may include, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GNN), reinforcement learning (RL), etc., but is not limited thereto.

[0057] For example, when at least one of image data, radar data, and / or LiDAR data is input to the server device 200, the first machine learning model of the server device 200 may output at least one of an image trajectory, a radar trajectory, a LiDAR trajectory, and / or a fused trajectory corresponding to the input detection data. For example, the first machine learning model may output a LiDAR trajectory when LiDAR data is input, or output an image trajectory when image data is input. In addition, the first machine learning model may output an image and radar fused trajectory when image data and radar data are input.

[0058] The server device 200 may also include first training data for training the first machine learning model. For example, the first training data may include labeled detection data. The labeled detection data may include labeled image data, labeled radar data, and / or labeled LiDAR data. The labeled detection data may include detection data (e.g., image data, radar data, or LiDAR data) and labeled data representing the location, size, and / or classification of the track.

[0059] For example, upon inputting the LiDAR data in the first training data, the first machine learning model may output a LiDAR trajectory corresponding to the LiDAR data. The output LiDAR trajectory may be compared to the labeled data, and the first machine learning model may be modified or updated based on the training error between the LiDAR trajectory and the labeled data. For example, the first machine learning model may be modified or updated to reduce the training error between the LiDAR trajectory output from the first machine learning model and the labeled data included in the first training data. The first machine learning model may be trained in this exemplary manner.

[0060] In the above description, a machine learning model, such as the first machine learning model, has been used to obtain an image trajectory, a radar trajectory, and / or a fused trajectory, but the present invention is not limited thereto. For example, in order to obtain an image trajectory, a radar trajectory, and / or a fused trajectory, the server device 200 may use an application (hereinafter referred to as a "conventional program") instead of a machine learning model. The conventional program may output output data corresponding to the input data according to a predetermined relationship or function between the input data and the output data, without using training data to obtain the relationship or function between the input data and the output data.

[0061] The server device 200 may further include a second machine learning model for generating the first training data.

[0062] The second machine learning model can output labeled data corresponding to the input detection data (e.g., image data, radar data, or LiDAR data).

[0063] The second machine learning model can be trained by second training data including reference detection data and reference label data. Conventionally, the labeling operation of obtaining reference label data from the detection data is performed manually, but the manual labeling operation takes a lot of time and cost.

[0064] To reduce cost and time, the server device 200 may use a second machine learning model to perform the labeling operation.

[0065] Server device 200 may also include second training data for training a second machine learning model.

[0066] The second training data may include reference detection data provided by a manual labeling operation and reference labeling data corresponding thereto.

[0067] The server device 200 may use the second training data to train the second machine learning model. For example, the server device 200 may input the reference detection data into the second machine learning model, and obtain the first label data corresponding to the reference detection data from the second machine learning model. The server device 200 may compare the first label data with the reference label data, and modify or update the second machine learning model according to the error generated by the comparison. For example, the second machine learning model may be modified or updated to reduce the error between the first label data and the reference label data. The second machine learning model may be trained in this exemplary manner.

[0068] In this case, since the second training data can be provided by a manual labeling operation, the number of the second training data may be insufficient. Therefore, the reliability and labeling accuracy of the trained second machine learning model are limited.

[0069] In order to improve the reliability and labeling accuracy of the trained second machine learning model, the server device 200 may use information about the detection data and / or trajectory collected from the vehicle 10 to calibrate the second machine learning model.

[0070] For example, the server device 200 may acquire LiDAR data and a fused trajectory (e.g., a trajectory obtained by fusing data received from multiple sensors, such as image data and radar data) from the vehicle 10. The server device 200 may input the LiDAR data into a second machine learning model, and the second machine learning model may output correction mark data corresponding to the input LiDAR data. The correction mark data may be compared with the fused trajectory acquired from the vehicle 10, and the trained second machine learning model may be corrected according to the error generated by the comparison. The corrected second machine learning model may further improve reliability and accuracy.

[0071] The first machine learning model trained by the first training data may be provided to or included in the vehicle 10. For example, the first machine learning model may be provided to or included in a vehicle that has not previously had a machine learning model installed (e.g., a new car that has not yet been shipped or sold). As another example, the first machine learning model may be provided to or included in a vehicle that has previously had a machine learning model installed (e.g., a vehicle that has already been sold or shipped). In other words, the machine learning model already installed in the vehicle may be updated to the newly trained first machine learning model.

[0072] In the above description, the server device 200 including the first machine learning model for generating a trajectory and the second machine learning model for labeling has been described. However, the server device 200 according to the present invention is not limited thereto.

[0073] For example, the server device 200 may include a machine learning model for generating a trajectory corresponding to the detection data and training data for training the machine learning model. The server device 200 may use the training data to train the machine learning model.

[0074] The server device 200 may receive detection data (e.g., image data, radar data, or LiDAR data) and a trajectory (e.g., an image trajectory, a radar trajectory, a LiDAR trajectory, or a fused trajectory) from the vehicle 10. For example, the server device 200 may receive LiDAR data and an image and radar fused trajectory (e.g., a trajectory obtained by fusing the image data and the radar data) from the vehicle 10.

[0075] The server device 200 may use the data and trajectory received from the vehicle 10 to calibrate the trained machine learning model.

[0076] The LiDAR data may be input to a trained machine learning model, and the trained machine learning model may output a LiDAR trajectory corresponding to the LiDAR data. The output LiDAR trajectory may be compared to the fused trajectory, and the machine learning model may be corrected based on the error between the LiDAR trajectory and the fused trajectory. For example, the trained machine learning model may be corrected to reduce the error between the LiDAR trajectory and the fused trajectory.

[0077] The trained machine learning model (or first machine learning model) may be provided to or included in the driving assistance device 100 of the vehicle 10. The driving assistance device 100 may use the trained machine learning model (or first machine learning model) to obtain a trajectory representing at least one object outside or around the vehicle 10 from the image data, radar data, and / or LiDAR data. In other words, when the image data, radar data, and / or LiDAR data are input, the trained machine learning model may output an image trajectory, a radar trajectory, and / or a LiDAR trajectory.

[0078] As described above, the driving assistance device 100 of the vehicle 10 may use the trained machine learning model to obtain a trajectory representing at least one object outside or around the vehicle 10 from at least one of the image data, radar data, and / or LiDAR data. At this time, at least one of the image data, radar data, and / or LiDAR data collected by the driving assistance device 100 and information about the trajectory corresponding thereto may be provided to the server device 200.

[0079] The server device 200 may acquire information about at least one of the image data, radar data, and / or LiDAR data and information about the trajectory corresponding thereto, and use the acquired data and the trajectory corresponding thereto to train a machine learning model. The newly trained machine learning model may be provided to the vehicle 10.

[0080] It has been described above that the driving assistance device 100 uses the trained machine learning model to obtain information about the object, and the server device 200 trains the machine learning model, but the present invention is not limited to this. For example, the driving assistance device 100 may transmit at least one of the image data, radar data, and / or LiDAR data to the server device 200, and the server device 200 may not only train the machine learning model, but also use the trained machine learning model to obtain information about the object, and transmit the information about the object to the driving assistance device 100. As another example, the driving assistance device 100 may not only use the trained machine learning model to obtain information about the object, but also train the machine learning model.

[0081] Figure 2is a block diagram for illustrating a configuration of a vehicle including a driving assist device according to an embodiment of the present invention. Figure 3 An example of a field of view of a sensor module included in the driving assistance device according to the embodiment of the present invention is shown.

[0082] refer to Figure 2 , the vehicle 10 may include a drive device 20, a brake device 30, a steering device 40 and / or a driving assistance device 100. Optionally, the vehicle 10 may also include a communication device 50. The drive device 20, the brake device 30, the steering device 40 and / or the driving assistance device 100 may communicate with each other via a vehicle communication network. For example, the electrical devices (drive device 20, brake device 30, steering device 40 and driving assistance device 100) included in the vehicle 10 may transmit or receive data via a communication network, such as Ethernet, media-oriented system transport (MOST), Flexray, controller area network (CAN), local interconnect network (LIN), etc.

[0083] The drive device 20 may supply power for driving the vehicle 10 to travel and, for example, move or accelerate the vehicle 10 in response to the driver's intention or a controller of the vehicle 10 to accelerate through an accelerator pedal or a request of the driving assist device 100 .

[0084] Braking device 30 may provide braking force to vehicle 10 to brake vehicle 10 and, for example, decelerate or stop vehicle 10 in response to a driver's intention or a controller of vehicle 10 , thereby performing braking through a request of the brake pedal and / or driving assist device 100 .

[0085] The steering device 40 may steer the vehicle 10 or change the driving direction of the vehicle 10 in response to a driver's steering intention and / or a signal from a controller of the vehicle 10 through the steering wheel and / or the driving assist device 100 .

[0086] The communication device 50 may be optionally provided or included in the vehicle 10. The communication device 50 may exchange or transfer data with the server device 200 in a wireless or wired manner. For example, the communication device 50 may transmit data to a base station or an access repeater in a wireless manner, and data from the base station or the access repeater may be transmitted to the server device 200 in a wired manner. In addition, the server device 200 may transmit data to a base station or an access repeater in a wired manner, and data from the base station or the access repeater may be transmitted to the communication device 50 in a wireless manner.

[0087] The driving assist device 100 can communicate with the drive device 20 , the brake device 30 , and the steering device 40 via the vehicle communication network.

[0088] The driving assistance device 100 can provide various functions for assisting driving and enhancing safety. For example, the driving assistance device 100 can be configured to provide one or more functions of lane departure warning (LDW), lane keeping assist (LKA), high beam assist (HBA), autonomous emergency braking (AEB), traffic sign recognition (TSR), adaptive cruise control (ACC), blind spot detection (BSD), etc.

[0089] The driving assistance device 100 may include or may be operably associated with a camera 110, a radar 120, a LiDAR 130, and a controller 140. The camera 110, the radar 120, the LiDAR 130, and the controller 140 may not be essential components of the driving assistance device 100. For example, at least one of the camera 110, the radar 120, and the LiDAR device 130 may be configured to be operable. Figure 2 Any detector or sensor capable of detecting or sensing an external object of the vehicle 10 may be omitted from or may not be included in the illustrated driving assist device 100 , or may be added to the vehicle 10 or the driving assist device 100 .

[0090] The camera 110, the radar 120, the LiDAR 130, and the controller 140 may be provided separately from each other. For example, the controller 140 may be installed in a housing separate from the housing of the camera 110, the housing of the radar 120, and the housing of the LiDAR 130. The controller 140 may exchange data with the camera 110, the radar 120, and / or the LiDAR 130 via a broadband network.

[0091] The camera 110 can capture the external field of view of the vehicle 10 and obtain image data of the exterior of the vehicle 10. For example, Figure 3 As shown, the camera 110 may be mounted on or near the front windshield of the vehicle 10 and may have a forward field of view 110 a of the vehicle 10 , although this is not required.

[0092] For example, the camera 110 may include a plurality of lenses and an image sensor. The image sensor may include a plurality of photodiodes for converting light into electrical signals, and the plurality of photodiodes may be arranged in a two-dimensional matrix.

[0093] The camera 110 may optionally include an image processor for processing image data, and based on the processing of the image data, identify objects outside or around the vehicle 10. The camera 110 may, for example, generate a track representing an object based on the processed image data, and classify the track. For example, the camera 110 may identify whether the track is another vehicle, a pedestrian, an animal, a cyclist, etc. The camera 110 may use, for example, a machine learning model to obtain an image track representing an external object from the image data.

[0094] The camera 110 may be electrically or communicatively connected to the controller 140. For example, the camera 110 may be connected to the controller 140 via a vehicle communication network, connected to the controller 140 via a hard wire, or connected to the controller 140 by wires of a printed circuit board (PCB). The camera 110 may provide the controller 140 with image data or information about an image track of an external field of view of the vehicle 10. The information about the image track may include information about the location of an object outside or around the vehicle 10 (e.g., information about the distance and / or angle of the object), information about the size of the object, and / or the classification of the object.

[0095] The radar 120 may transmit transmission radio waves to the outside of the vehicle 10 and recognize external objects of the vehicle 10 based on reflected radio waves reflected from the external objects. Figure 3 As shown, the radar 120 may be mounted on the grille or bumper of the vehicle 10 and may have a forward sensing area 120 a of the vehicle 10 , although this is not required.

[0096] The radar 120 may include one or more transmitting antennas (or transmitting antenna arrays) for transmitting or radiating transmitting radio waves to the outside of the vehicle 10 and one or more receiving antennas (or receiving antenna arrays) for receiving reflected radio waves reflected from one or more objects.

[0097] The radar 120 may acquire radar data from transmission radio waves transmitted by a transmission antenna and reflected radio waves received by a reception antenna.

[0098] The radar 120 may include a signal processor configured to process radar data and generate a track representing an object by clustering reflection points from reflected radio waves. The radar 120 may obtain a radar track representing an external object from the radar data using, for example, a machine learning model.

[0099] The radar 120 may be connected to the controller 140 via a vehicle communication network, hard wires, or wires of a PCB, and transmit information about radar data or radar tracks to the controller 140. The information about the radar tracks may include position information about objects outside or around the vehicle 10 (e.g., information about the distance and / or angle of the object) and / or information about the size of the object.

[0100] The LiDAR 130 may emit light (e.g., infrared rays) to the outside of the vehicle 10 and detect external objects of the vehicle 10 based on reflected light reflected from the external objects. Figure 3 As shown, the LiDAR 130 may be mounted on the roof or windshield of the vehicle 10 and may have a sensing area 130 a in all directions outside the vehicle 10 .

[0101] The LiDAR 130 may include a light source (e.g., a light emitting diode, a light emitting diode array, a laser diode, or a laser diode array) configured to emit light (e.g., infrared rays) and an optical sensor (e.g., a photodiode or a photodiode array) configured to receive light. In addition, the LiDAR 130 may further include a driving device configured to rotate or move the light source and / or the optical sensor as needed.

[0102] The LiDAR 130 may acquire LiDAR data by emitting light through or by a light source and receiving light reflected from an object through or by an optical sensor.

[0103] The LiDAR 130 may include a signal processor configured to process the LiDAR data and generate a track representing an object by clustering reflection points from reflected radio waves. The LiDAR 130 may use, for example, a machine learning model to obtain a LiDAR track representing an external object from the LiDAR data.

[0104] The LiDAR 130 may be connected to the controller 140 via a vehicle communication network, hardwire, or PCB, and transmit information about the LiDAR data or the LiDAR track to the controller 140. The information about the LiDAR track may include the location of an external object of the vehicle 10 (e.g., the distance and / or direction of the external object), the size of the external object, and / or the classification or type of the external object.

[0105] The controller 140 may be electrically or communicatively connected to the camera 110, the radar 120, and / or the LiDAR 130. In addition, the controller 140 may be connected to the driving device 20, the braking device 30, and the steering device 40 via a vehicle communication network.

[0106] The controller 140 may receive image data (or information about image trajectories) from the camera 110 , radar data (or information about radar trajectories) from the radar 120 , and LiDAR data (or information about LiDAR trajectories) from the LiDAR 130 , and provide control signals to the drive device 20 , the braking device 30 , and / or the steering device 40 .

[0107] The controller 140 may include a processor 141 and a memory 142 .

[0108] The processor 141 may process the image data (or image track) of the camera 110 , the radar data (or radar track) of the radar 120 , and / or the LiDAR data (or LiDAR track) of the LiDAR 130 .

[0109] For example, the processor 141 may include: an image processor configured to process image data of the camera 110; one or more signal processors configured to process radar data of the radar 120 and / or LiDAR data of the LiDAR 130; and / or a microcontroller unit (MCU) configured to generate control signals such as driving, braking, and steering signals. The image processor, the signal processor, and / or the MCU may not be essential components of the processor 141, and one or more of them may be omitted from the processor 141 or may not be included in the processor 141.

[0110] For example, the processor 141 may receive image data from the camera 110, radar data from the radar 120, and / or LiDAR data from the LiDAR 130. The processor 141 may process the image data, radar data, and / or LiDAR data using a trained machine learning model, and obtain information related to the object (e.g., the position, size, speed, classification, etc. of the external object) based on the result of the processed data. Based on the information about the object, the processor 141 may generate a control signal, such as, but not limited to, a drive signal, a brake signal, and / or a steering signal for controlling the drive device 20, the brake device 30, and / or the steering device 40, respectively.

[0111] As another example, the processor 141 may receive information about an image track of the camera 110, information about a radar track of the radar 120, and / or information about a LiDAR track of the LiDAR 130. The processor 141 may acquire information about an object by fusing two or more of the image track, the radar track, and / or the LiDAR track.

[0112] As another example, the processor 141 may receive information about an image track of the camera 110, information about a radar track of the radar 120, and / or information about LiDAR data of the LiDAR 130. The processor 141 may obtain a LiDAR track from the LiDAR data using a trained machine learning model. In addition, the processor 141 may obtain sensor fusion data by fusing the image track and the radar track, and obtain information about the object based on the sensor fusion data and the LiDAR track.

[0113] The processor 141 may generate a driving signal, a braking signal, and / or a steering signal for avoiding the object based on the information about the object.

[0114] The memory 142 may store programs, instructions, and / or data for the processor 141 to process image data (or image tracks), radar data (or radar tracks), and / or LiDAR data (or LiDAR tracks). The memory 142 may also store programs, instructions, and / or data for the processor 141 to generate driving, braking, and steering control signals.

[0115] In addition, the memory 142 can temporarily store image data (or image trajectories), radar data (or radar trajectories) and / or LiDAR data (or LiDAR trajectories), and temporarily store processed data as a result of the processor 141 processing the image data (or image trajectories), radar data (or radar trajectories) and / or LiDAR data (or LiDAR trajectories).

[0116] For example, the memory 142 may include not only volatile memories such as static random access memories (SRAM) and dynamic RAM (DRAM), but also nonvolatile memories such as read only memories (ROM) and erasable programmable ROM (EPROM).

[0117] The controller 140 may store at least one of the image data, the radar data, and / or the LiDAR data and the information about the object in the memory 142. For example, the controller 140 may store the LiDAR data and the information about the object (e.g., fused data obtained by fusing the image track and the radar track) in a non-volatile memory such as a flash memory.

[0118] In addition, the controller 140 may provide at least one of the image data, the radar data, and / or the LiDAR data and the information about the object to the server device 200 through the communication device 50 of the vehicle 10. For example, the controller 140 may transmit the LiDAR data and the information about the object (e.g., fused data obtained by fusing the image track and the radar track) to the server device 200 through the communication device 50 of the vehicle 10. The controller 140 may transmit the LiDAR data and the information about the object to the server device 200 in real time, or transmit the LiDAR data and the information about the object to the server device 200 after the vehicle 10 is parked or at a predetermined time.

[0119] As described above, the controller 140 may provide a driving signal, a braking signal, and / or a steering signal based on the image data (or image track), the radar data (or radar track), and / or the LiDAR data (or LiDAR track). In addition, the controller 140 may store at least one of the image data, the radar data, and / or the LiDAR data and information about the object, or provide at least one of the image data, the radar data, and / or the LiDAR data and information about the object to the server device 200.

[0120] Figure 4 is a block diagram for illustrating a configuration of a server device according to an embodiment of the present invention. Figure 5 is a view showing an example in which light detection and ranging (LiDAR) data and fused data are matched.

[0121] Figure 6 is a view showing an example in which a server device according to an embodiment of the present invention adjusts an error between an output of a machine learning model and training data.

[0122] The server device 200 may include a first machine learning model configured to output a trajectory (e.g., an image trajectory, a radar trajectory, a LiDAR trajectory, and / or a fused trajectory) corresponding to data (e.g., image data, radar data, and / or LiDAR data) detected by an image sensor, a radar, and / or a LiDAR, and first training data for training the first machine learning model. The first training data may include at least one of the image data, the radar data, and / or the LiDAR data and label data corresponding thereto.

[0123] In addition, the server device 200 may include a second machine learning model for generating the first training data and second training data for training the second machine learning model. The second training data may include detection data (e.g., image data, radar data, or LiDAR data) and reference marker data corresponding thereto. The reference marker data may include information about the trajectory obtained from the detection data (e.g., the position and / or size of the trajectory).

[0124] The server device 200 may calibrate the second machine learning model trained by the second training data using the detection data (e.g., image data, radar data, and / or LiDAR data) and / or the trajectory (e.g., image trajectory, radar trajectory, and / or fused trajectory) acquired from the vehicle 10. For example, the server device 200 may calibrate the trained second machine learning model using the LiDAR data acquired from the vehicle 10 and the image and radar fused trajectory (i.e., the trajectory obtained by fusing the image data and the radar data).

[0125] For example, Figure 4As shown, the server device 200 may include a receiver 210, a database 220, a training data generator or a training data generation module 230, a model trainer or a model training module 240, and a transmitter 250. The receiver 210, the database 220, the training data generator 230, the model trainer 240, and the transmitter 250 may not be essential components of the server device 200, and one or more of these components may be omitted from the server device 200 or may not be included in the server device 200. The receiver 210 and the transmitter 250 may be implemented as a transceiver, a communicator, or a communication circuit, and the database 220 may be implemented as a memory or a storage medium. The training data generator 230 and the model trainer 240 may be implemented as at least one processor or a software module stored in a memory and configured to be executable by at least one processor.

[0126] The receiver 210 may acquire detection data (image data, radar data, and / or LiDAR data) and a trajectory (image trajectory, radar trajectory, LiDAR trajectory, and / or fused trajectory) of the vehicle 10. For example, the receiver 210 may be implemented as a transceiver, a communicator, or a communication device, and may receive data and trajectories from the vehicle 10 via a broadband communication network. As another example, the receiver 210 may be implemented as a data reader, and may read the data and trajectories of the vehicle 10 from a storage medium storing the data and trajectories of the vehicle 10.

[0127] Database 220 may store machine learning models and training data. The machine learning models may include a first machine learning model and a second machine learning model, and the training data may include a first training data and a second training data.

[0128] When image data, radar data and / or LiDAR data are input into the first machine learning model, the first machine learning model may output an image trajectory, a radar trajectory, a LiDAR trajectory and / or a fused trajectory corresponding to the input detection data.

[0129] The first training data may be data for training the first machine learning model. The first training data may include labeled detection data. The labeled detection data may include, for example, but not limited to, labeled image data, labeled radar data, and / or labeled LiDAR data. The labeled image data may include image data and labeled data representing information about a track corresponding to the image data. The labeled radar data may include radar data and labeled data representing information about a track corresponding to the radar data. The labeled LiDAR data may include LiDAR data and labeled data representing information about a track corresponding to the LiDAR data.

[0130] The second machine learning model can generate first training data corresponding to the image data, radar data and / or LiDAR data. For example, when the image data, radar data and / or LiDAR data are input to the second machine learning model, the second machine learning model can output labeled data corresponding to the input detection data.

[0131] The second training data may be data for training the second machine learning model. The second training data may include reference data of at least one of the image data, radar data, and / or LiDAR data and reference marker data corresponding thereto.

[0132] The training data generator 230 may generate labeled detection data (eg, labeled image data, labeled radar data, or labeled LiDAR data) using a second machine learning model stored in the database 220 .

[0133] The training data generator 230 may input detection data (e.g., at least one of image data, radar data, and / or LiDAR data) into a second machine learning model and obtain labeled data from the second machine learning model. The training data generator 230 may obtain labeled detection data (e.g., labeled image data, labeled radar data, or labeled LiDAR data), which has the detection data input to the second machine learning model and the labeled data output from the second machine learning model. To this end, the training data generator 230 may combine the detection data input to the second machine learning model with the labeled data output from the second machine learning model. For example, the training data generator 230 may obtain labeled LiDAR data by combining LiDAR data with labeled data.

[0134] As described above, the training data generator 230 may obtain first training data including labeled image data, labeled radar data, and / or labeled LiDAR data from image data, radar data, and / or LiDAR data.

[0135] The model trainer 240 may train a first machine learning model and a second machine learning model.

[0136] The model trainer 240 may train the first machine learning model using the first training data. The model trainer 240 may input the image data, radar data, or LiDAR data to the first machine learning model, and obtain the image trajectory, radar trajectory, or LiDAR trajectory from the first machine learning model. The model trainer 240 may compare the trajectory output from the first machine learning model with the labeled data of the first training data. The model trainer 240 may modify or update the first machine learning model to reduce the training error between the trajectory output from the first machine learning model and the labeled data.

[0137] For example, the model trainer 240 may input the LiDAR data into the first machine learning model and obtain the LiDAR trajectory from the first machine learning model. The model trainer 240 may compare the LiDAR trajectory with the labeled data and modify or update the first machine learning model to reduce the training error between the LiDAR trajectory and the labeled data.

[0138] In addition, the model trainer 240 may use the second training data to train the second machine learning model. The model trainer 240 may input the image data, radar data, or LiDAR data to the second machine learning model and obtain the first labeled data from the second machine learning model. The model trainer 240 may modify or update the second machine learning model to reduce the training error between the first labeled data output from the second machine learning model and the reference labeled data of the second training data.

[0139] For example, the model trainer 240 may input the LiDAR data into the second machine learning model and obtain the first labeled data from the second machine learning model. The model trainer 240 may compare the first labeled data of the second machine learning model with the reference labeled data of the second training data, and modify or update the second machine learning model to reduce the training error between the first labeled data of the second machine learning model and the reference labeled data of the second training data.

[0140] The model trainer 240 may use the second training data to evaluate the trained second machine learning model. When the accuracy of the trained second machine learning model is low (e.g., if the accuracy of the trained second machine learning model is lower than a preset threshold or reference error), the model trainer 240 may use the data and trajectory acquired from the vehicle 10 to correct the trained second machine learning model.

[0141] For example, the model trainer 240 may input the LiDAR data included in the second training data into the trained second machine learning model and obtain the evaluation label data from the trained second machine learning model. The model trainer 240 may compare the evaluation label data of the trained second machine learning model with the reference label data of the second training data and obtain the evaluation error between the evaluation label data of the trained second machine learning model and the reference label data of the second training data.

[0142] The model trainer 240 may compare the obtained evaluation error with the reference error, and when the obtained evaluation error is less than or equal to the reference error, determine that the accuracy of the trained second machine learning model is high. In addition, when the obtained evaluation error is greater than the reference error, the model trainer 240 may determine that the accuracy of the trained second machine learning model is low.

[0143] When the acquired evaluation error is greater than the reference error, the model trainer 240 may correct or modify the trained second machine learning model using the detection data and / or trajectory acquired from the vehicle 10 .

[0144] For example, the model trainer 240 may use LiDAR data and image / radar fusion trajectories (eg, trajectories obtained by fusing image data and radar data) acquired from the vehicle 10 to calibrate the trained second machine learning model.

[0145] The model trainer 240 may input the LiDAR data acquired from the vehicle 10 into the trained second machine learning model, and receive or acquire the correction label data from the trained second machine learning model. The model trainer 240 may compare the label data of the trained second machine learning model with the fused trajectory acquired from the vehicle 10, and correct the trained second machine learning model to reduce the correction error between the correction label data and the fused trajectory.

[0146] For example, the model trainer 240 may acquire LiDAR data (e.g., Figure 5 image on the left) and image and radar fusion tracks (e.g. Figure 5 The server device 200 may acquire LiDAR data, image data, and radar data from the vehicle 10 and derive an image and radar fusion trajectory from the image data and radar data using a first machine learning model or a conventional procedure.

[0147] like Figure 5 As shown, the LiDAR data may include a plurality of points. The model trainer 240 may input the LiDAR data including the plurality of points into the second machine learning model and obtain the correction mark data indicating the position and / or size of the track. The model trainer 240 may compare the correction mark data of the second machine learning model with the Figure 5 For example, the model trainer 240 may compare the position and / or size of the trajectory according to the correction mark data with the position and / or size of the image and radar fusion trajectory, and obtain the correction error between the correction mark data and the image and radar fusion trajectory.

[0148] The model trainer 240 may calibrate the trained second machine learning model to reduce the calibration error.

[0149] As described above, the model trainer 240 may use data acquired from the vehicle 10 and the fused trajectory to compensate for errors or missing labeled detection data (eg, labeled image data, labeled radar data, or labeled LiDAR data).

[0150] In addition, the model trainer 240 can use the data and trajectory obtained from the vehicle 10 to change the degree of correction when correcting the trained second machine learning model according to the accuracy of the second machine learning model.

[0151] As described above, the model trainer 240 can evaluate the trained second machine learning model. Specifically, the model trainer 240 can obtain the evaluation error between the evaluation label data of the trained second machine learning model and the reference label data of the second training data.

[0152] The model trainer 240 may change the rate of correction using the data and trajectory acquired from the vehicle 10 based on the acquired evaluation error. As the acquired evaluation error increases, the model trainer 240 may increase the rate of correction using the detection data and / or trajectory acquired from the vehicle 10. However, as the acquired evaluation error decreases, the model trainer 240 may decrease the rate of correction using the detection data and / or trajectory acquired from the vehicle 10.

[0153] For example, as described above, the model trainer 240 may input the LiDAR data acquired from the vehicle 10 into the trained second machine learning model and acquire the correction label data from the trained second machine learning model. The model trainer 240 may correct the trained second machine learning model based on the correction error between the correction label data and the fused trajectory.

[0154] At this time, the model trainer 240 may adjust the correction error between the correction label data and the fused trajectory acquired from the vehicle 10 based on the evaluation error between the previously acquired evaluation label data and the reference label data. The model trainer 240 may adjust the correction error so that as the evaluation error between the evaluation label data and the reference label data increases, the correction error between the correction label data and the fused trajectory acquired from the vehicle 10 increases (e.g., so that the correction error becomes closer to the original error between the correction label data and the fused trajectory acquired from the vehicle 10). In addition, the model trainer 240 may adjust the correction error so that as the evaluation error between the evaluation label data and the reference label data decreases, the correction error between the correction label data and the fused trajectory acquired from the vehicle 10 decreases.

[0155] For example, Figure 6 As shown, the model trainer 240 can obtain the correction label data O1, O2 and O3 from the trained second machine learning model. In addition, the model trainer 240 can obtain the fused trajectories T1, T2 and T3 obtained from the vehicle 10.

[0156] The model trainer 240 may obtain the correction error by comparing the correction label data O1, O2 and O3 with the fusion trajectory T1, T2 and T3 respectively. At this time, the model trainer 240 may adjust the correction error based on the evaluation error between the evaluation label data and the reference label data.

[0157] For example, the model trainer 240 may change the data compared with the correction label data O1, O2, and O3 from the fused trajectories T1, T2, and T3 to the adjustment label data L1, L2, and L3. The adjustment label data L1, L2, and L3 may be located between one of the correction label data O1, O2, and O3 and one of the fused trajectories T1, T2, and T3, respectively. The smaller the evaluation error between the evaluation label data and the reference label data, the closer the adjustment label data L1, L2, and L3 may be to the correction label data O1, O2, and O3, respectively. In addition, the larger the evaluation error between the evaluation label data and the reference label data, the closer the adjustment label data L1, L2, and L3 may be to the fused trajectories T1, T2, and T3, respectively.

[0158] As described above, the reflectivity of the fused trajectory acquired from the vehicle 10, that is, the weight of the fused trajectory, can be reduced according to the evaluation error between the evaluation label data and the reference label data. This is because the accuracy of the fused trajectory is fixed despite the increase in the amount of data, while the accuracy of the leveling data of the second machine learning model increases according to the increase in the amount of data.

[0159] The transmitter 250 may provide the trained first machine learning model to the vehicle 10. For example, the transmitter 250 may be implemented as a transceiver, a communicator, or a communication device, and may provide the trained first machine learning model to the vehicle 10 through a broadband communication network. As another example, the transmitter 250 may be implemented as a data recorder, and may store the trained first machine learning model in a storage medium.

[0160] As described above, the server device 200 may include a first machine learning model, first training data, a second machine learning model, and second training data. For example, the first machine learning model may output a LiDAR trajectory corresponding to the input LiDAR data, and the first training data may include labeled LiDAR data for training the first machine learning model. The second machine learning model may output labeled LiDAR data corresponding to the input LiDAR data, and the second training data may include LiDAR data and labeled data for training the second machine learning model. The second machine learning model trained by the second training data may be corrected using the LiDAR data acquired from the vehicle 10 and the image and radar fusion trajectory.

[0161] However, the server device 200 is not limited thereto. The server device 200 may include only the first machine learning model and the first training data.

[0162] For example, the server device 200 may include a machine learning model for generating a trajectory corresponding to the detection data and training data for training the machine learning model. The server device 200 may use the training data to train the machine learning model. The training data may include reference detection data and labeled data.

[0163] In addition, in order to correct the trained machine learning model, the server device 200 can receive detection data (e.g., image data, radar data, or LiDAR data) and a trajectory (e.g., an image trajectory, a radar trajectory, a LiDAR trajectory, or a fused trajectory) from the vehicle 10.

[0164] The server device 200 may use the training data to evaluate and correct the trained machine learning model. The server device 200 may input the reference detection data into the trained machine learning model and obtain an evaluation trajectory from the trained machine learning model. The server device 200 may compare the evaluation trajectory with the labeled data of the training data and correct the trained machine learning model based on the evaluation error between the evaluation trajectory and the labeled data.

[0165] When the evaluation error between the evaluation trajectory and the labeled data is less than or equal to the reference error, the server apparatus 200 may determine that the accuracy of the trained machine learning model is high.

[0166] When the evaluation error between the evaluation trajectory and the labeled data is greater than the reference error, the server device 200 may correct the trained machine learning model using the detection data and trajectory acquired from the vehicle 10 .

[0167] The server device 200 may input the detection data of the vehicle 10 into the trained machine learning model and obtain a correction trajectory from the trained machine learning model. The server device 200 may compare the correction trajectory with the trajectory of the vehicle 10 and correct the trained machine learning model based on the correction error between the correction trajectory and the trajectory of the vehicle 10. For example, the server device 200 may correct the trained machine learning model to reduce the correction error between the correction trajectory and the trajectory of the vehicle 10.

[0168] In addition, the server device 200 can adjust the correction of the trained machine learning model based on the evaluation error caused by the training data. For example, the server device 200 can adjust the correction error between the correction trajectory and the trajectory of the vehicle 10 based on the evaluation error caused by the training data, and correct the trained machine learning model based on the adjusted correction error.

[0169] In addition, the server device 200 may provide the corrected machine learning model to the vehicle 10 .

[0170] Figure 7 is a flowchart for illustrating a method for training a machine learning model according to an embodiment of the present invention.

[0171] Reference Figure 7 Describe method 1000 of training a first machine learning model. Figure 7 Step 1010, step 1020 and step 1030 in method 1000 included in the method 1000 may not be necessary steps of method 1000, and one or some of step 1010, step 1020 and step 1030 may be omitted from method 1000 or may not be included in method 1000.

[0172] In step 1010, first training data may be acquired.

[0173] The first training data may include marker detection data. For example, the marker detection data may include marker image data, marker radar data, and / or marker LiDAR data.

[0174] The labeled detection data may include the detection data and the labeled data corresponding to the detection data. For example, the labeled image data may include the image data and the labeled data corresponding to the image data, and the labeled radar data may include the radar data and the labeled data corresponding to the radar data. In addition, the labeled LiDAR data may include the LiDAR data and the labeled data corresponding to the LiDAR data.

[0175] For example, the server device 200 may use the second machine learning model to obtain labeled LiDAR data. The second machine learning model may output labeled data corresponding to the input LiDAR data, and the server device 200 may obtain labeled LiDAR data as a combination of LiDAR data and labeled data or having LiDAR data and labeled data.

[0176] At step 1020, a first machine learning model may be trained.

[0177] The first machine learning model can be trained using the first training data.

[0178] Detection data (e.g., image data, radar data, or LiDAR data) may be input into a first machine learning model, and the first machine learning model may output a trajectory (e.g., an image trajectory, a radar trajectory, or a LiDAR trajectory) corresponding to the input detection data.

[0179] The first machine learning model may be trained by labeled detection data (e.g., labeled image data, labeled radar data, or labeled LiDAR data). For example, the server device 200 may input the detection data into the first machine learning model and compare the output trajectory with the labeled data of the labeled detection data. In addition, the server device 200 may modify or update the first machine learning model based on the error between the output trajectory and the labeled data.

[0180] At step 1030 , the trained first machine learning model may be provided or output to the vehicle 10 .

[0181] The trained first machine learning model may be provided or output from the server device 200 to the vehicle 10 in various ways.

[0182] For example, the trained first machine learning model may be output or provided to the vehicle 10 via a network such as a wired and / or wireless communication network. As another example, the trained first machine learning model may be provided to the vehicle 10 via a storage medium or memory.

[0183] Figure 8 is a flowchart for illustrating a method of labeling training data for training a machine learning model according to an embodiment of the present invention.

[0184] Reference Figure 8 Describe method 1100 for training a second machine learning model. Figure 8 Steps 1110 to 1170 in method 1100 included in the method 1100 may not be necessary configurations of method 1100 , and one or some of steps 1110 to 1170 may be omitted from method 1100 or may not be included in method 1100 .

[0185] In step 1110, second training data may be acquired.

[0186] The second training data may include detection data (eg, image data, radar data, or LiDAR data) and reference label data corresponding to the detection data. For example, the second training data may be acquired through a human labeling operation.

[0187] At step 1120 , detection data may be acquired from the vehicle 10 .

[0188] The server device 200 may receive or acquire data from the vehicle 10 through a transceiver, a communicator, a communication network, or a storage medium.

[0189] For example, the server device 200 may receive or obtain information about detection data (e.g., image data, radar data, or LiDAR data) collected by the vehicle 10 and a trajectory (e.g., image trajectory, radar trajectory, or LiDAR trajectory) corresponding to the detection data from the vehicle 10. The server device 200 may obtain LiDAR data and an image and radar fusion trajectory.

[0190] As another example, the server device 200 may obtain detection data collected by the vehicle 10 and obtain a trajectory by inputting the detection data into a first machine learning model or a conventional program. The server device 200 may receive image data, radar data, and LiDAR data and obtain an image and radar fusion trajectory by inputting the image data and radar data into a first machine learning model.

[0191] At step 1130, a second machine learning model may be trained.

[0192] The second machine learning model can be trained by the second training data. Detection data (e.g., image data, radar data, or LiDAR data) can be input to the second machine learning model, and the second machine learning model can output labeled data corresponding to the input detection data.

[0193] The second machine learning model can be trained by the detection data and the reference labeled data. For example, the server device 200 can input the detection data into the first machine learning model and compare the output first labeled data with the reference labeled data. In addition, the server device 200 can modify or update the second machine learning model based on the error between the first labeled data and the reference labeled data.

[0194] At step 1140, the trained second machine learning model may be evaluated.

[0195] The trained second machine learning model may be evaluated by the second training data. For example, the server device 200 may input the LiDAR data included in the second training data into the trained second machine learning model, and obtain evaluation label data from the trained second machine learning model. The server device 200 may compare the evaluation label data of the trained second machine learning model with the reference label data of the second training data, and obtain an evaluation error between the evaluation label data of the trained second machine learning model and the reference label data of the second training data.

[0196] In step 1150 , it may be confirmed whether the evaluation error is greater than the reference error.

[0197] The server device 200 may compare the acquired evaluation error with the reference error, and when the acquired evaluation error is less than or equal to the reference error, determine that the accuracy of the trained second machine learning model is high. However, when the acquired evaluation error is greater than the reference error, the server device 200 may determine that the accuracy of the trained second machine learning model is low.

[0198] When it is determined that the evaluation error is greater than the reference error (yes in step 1150), the trained second machine learning model can be corrected (step 1160).

[0199] In step 1160 , when the acquired evaluation error is greater than the reference error, the server device 200 may correct the trained second machine learning model using the detection data and / or trajectory acquired from the vehicle 10 .

[0200] For example, the server device 200 may use LiDAR data acquired from the vehicle 10 and a fused trajectory (eg, a trajectory obtained by fusing image data and radar data) to calibrate the trained second machine learning model.

[0201] The server device 200 may input the LiDAR data acquired from the vehicle 10 into the trained second machine learning model and acquire the correction label data from the trained second machine learning model. The server device 200 may compare the correction label data of the trained second machine learning model with the fused trajectory and correct the trained second machine learning model to reduce the correction error between the correction label data and the fused trajectory.

[0202] After the trained second machine learning model is calibrated or when it is determined that the evaluation error is not greater than the reference error (No in step 1150), the first training data may be provided (step 1170).

[0203] The server device 200 may input the detection data acquired from the vehicle 10 into the second machine learning model, and acquire the marker detection data corresponding to the detection data from the second machine learning model. The marker detection data may be used as the first training data for training the first machine learning model.

[0204] Fig. 9 is a flowchart for illustrating a method of labeling training data for training a machine learning model according to an embodiment of the present invention.

[0205] Reference Fig. 9Method 1200 for training a second machine learning model is described. Steps 1210, 1220, 1230, and 1240 included in method 1200 may not be necessary steps of method 1200, and one or some of steps 1210, 1220, 1230, and 1240 may be included in method 1200. Fig. 9 is omitted from method 1200 or may not be included in method 1200.

[0206] In step 1210 , a trajectory may be obtained based on data received or acquired from the vehicle 10 .

[0207] The server device 200 may acquire information about the detection data and the trajectory, or acquire the detection data from the vehicle 10 .

[0208] For example, the server device 200 may acquire information about LiDAR data and an image and radar fusion trajectory (eg, a trajectory obtained by fusing image data and radar data) from the vehicle 10 .

[0209] As another example, the server device 200 may acquire image data, radar data, and LiDAR data. The server device 200 may acquire an image and radar fusion trajectory by fusing the image data and the radar data.

[0210] At step 1220, the vehicle's data may be input into a trained second machine learning model.

[0211] The server device 200 may input the acquired detection data into the trained second machine learning model. The trained second machine learning model may output correction mark data corresponding to the input detection data.

[0212] The server device 200 may compare the corrected label data of the trained second machine learning model with the fused trajectory, and obtain a correction error between the corrected label data and the fused trajectory.

[0213] At step 1230 , the correction error may be adjusted based on the evaluation error.

[0214] Can refer to Figure 8 Described step 1140 obtains the estimation error.

[0215] The correction error for correcting the trained second machine learning model can be obtained by Figure 8 The evaluation error obtained in step 1140 is corrected.

[0216] For example, the server device 200 may adjust the correction error so that as the evaluation error between the evaluation label data and the reference label data increases, the correction error between the correction label data and the fused trajectory acquired from the vehicle 10 increases (e.g., so that the correction error becomes closer to the original error between the correction label data and the fused trajectory acquired from the vehicle 10). In addition, the server device 200 may adjust the correction error so that as the evaluation error between the evaluation label data and the reference label data decreases, the correction error between the correction label data and the fused trajectory acquired from the vehicle 10 decreases.

[0217] At step 1240, the trained second machine learning model may be corrected.

[0218] The server device 200 may correct the trained second machine learning model based on the corrected correction error. The server device 200 may correct the trained second machine learning model to reduce the corrected correction error.

[0219] Therefore, according to one aspect of the present invention, a server device for driving assistance for acquiring object information from LiDAR data and a method of controlling the server device can be provided.

[0220] According to another aspect of the present invention, a server device for driving assistance and a method for controlling the server device may be provided, wherein the server device may use image data, radar data, LiDAR data and / or sensor fusion data to train a machine learning model.

[0221] According to another aspect of the present invention, a server device for driving assistance and a method for controlling the server device can be provided, wherein the server device is used to obtain object information from LiDAR data using a machine learning model trained using image data, radar data, LiDAR data and / or sensor fusion data.

[0222] Exemplary embodiments of the present invention are described above. In the above exemplary embodiments, some components may be implemented as "modules". Here, the term "module" means but is not limited to software and / or hardware components that perform specific tasks, such as field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs). Modules may advantageously be configured to reside on an addressable storage medium and to be executed on one or more processors.

[0223] Thus, as examples, modules can include components, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays and variables. The operations provided in the components and modules can be combined into fewer components and modules, or further divided into additional components and modules. In addition, the components and modules can be implemented so that they execute one or more CPUs in the device.

[0224] That is, in addition to the above exemplary embodiments, the embodiments may therefore be implemented by computer-readable code / instructions in / on a medium (e.g., a computer-readable medium) to control at least one processing element to implement any of the above exemplary embodiments. The medium may correspond to any medium that allows storage and / or transmission of computer-readable code.

[0225] The computer readable code may be recorded on a medium or transmitted over the Internet. The medium may include a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical recording medium. In addition, the medium may be a non-temporary computer readable medium. The medium may also be a distributed network so that the computer readable code is stored or transmitted and executed in a distributed manner. In addition, by way of example only, the processing element may include at least one processor or at least one computer processor, and the processing element may be distributed and / or included in a single device.

[0226] While exemplary embodiments have been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this disclosure, will appreciate that other embodiments can be devised which do not depart from the scope disclosed herein. Accordingly, the scope should be limited only by the claims appended hereto.

Claims

1. A server device, wherein: include: a communicator configured to communicate with the vehicle, a memory configured to store a first machine learning model, a second machine learning model, first training data, and second training data, and One or more processors configured to: acquiring second training data including reference data and reference labeled data corresponding to the reference data, training the second machine learning model using the second training data comprising the reference data and the reference labeled data, receiving detection data and a detection trajectory corresponding to the detection data from the vehicle, calibrating a trained second machine learning model using the detection data and the detection trajectory received from the vehicle, using the corrected trained second machine learning model to obtain first training data, training the first machine learning model using the first training data, the first training data being obtained using a trained second machine learning model calibrated by using the detection data and the detection trajectory received from the vehicle, The trained first machine learning model is output to the vehicle.

2. The server device according to claim 1, wherein: The one or more processors are configured to, inputting the reference data of the second training data into the second machine learning model; Acquire, from the second machine learning model, first labeled data corresponding to the reference data of the second training data inputted, The second machine learning model is trained to reduce an error between the first labeled data obtained from the second machine learning model and the reference labeled data included in the second training data.

3. The server device according to claim 1, wherein: The one or more processors are configured to, inputting the reference data of the second training data into a second machine learning model trained to reduce an error between the first labeled data obtained from the second machine learning model and the reference labeled data included in the second training data, acquiring evaluation labeled data corresponding to the reference data of the second training data from a second machine learning model trained to reduce an error between the first labeled data acquired from the second machine learning model and the reference labeled data included in the second training data, When the error between the evaluation label data obtained from the trained second machine learning model and the reference label data included in the second training data is greater than a reference error, the trained second machine learning model is corrected.

4. The server device according to claim 3, wherein: The one or more processors are configured to, When the error between the evaluation label data obtained from the trained second machine learning model and the reference label data included in the second training data is less than or equal to the reference error, the first training data is obtained using a second machine learning model trained to reduce the error between the first label data obtained from the second machine learning model and the reference label data included in the second training data.

5. The server device according to claim 3, wherein: The one or more processors are configured to, inputting the detection data received from the vehicle into a second machine learning model trained to reduce an error between the first labeled data acquired from the second machine learning model and the reference labeled data included in the second training data, acquiring correction label data corresponding to the detection data received from the vehicle from a second machine learning model trained to reduce an error between the first label data acquired from the second machine learning model and the reference label data included in the second training data, Correcting a second machine learning model trained to reduce an error between the first labeled data obtained from the second machine learning model and the reference labeled data included in the second training data to reduce a correction error between the corrected labeled data obtained from the trained second machine learning model and the detection trajectory received from the vehicle.

6. The server device according to claim 5, wherein: The one or more processors are configured to, Based on the error between the evaluation label data obtained from the trained second machine learning model and the reference label data included in the second training data, a correction error between the correction label data obtained from the trained second machine learning model and the detection trajectory received from the vehicle is adjusted.

7. The server device according to claim 6, wherein: The one or more processors are configured to, Adjust a correction error between the correction label data obtained from the trained second machine learning model and the detection trajectory received from the vehicle so that the correction error between the correction label data obtained from the trained second machine learning model and the detection trajectory received from the vehicle increases as an error between the evaluation label data obtained from the trained second machine learning model and the reference label data included in the second training data increases, and so that the correction error between the correction label data obtained from the trained second machine learning model and the detection trajectory received from the vehicle decreases as an error between the evaluation label data obtained from the trained second machine learning model and the reference label data included in the second training data decreases.

8. A method for controlling a server device, wherein: The server device comprises: a communicator configured to communicate with the vehicle, and a memory configured to store a first machine learning model, a second machine learning model, first training data, and second training data; The method for controlling the server device comprises the following steps: acquiring second training data including reference data and reference labeled data corresponding to the reference data, training the second machine learning model using the second training data comprising the reference data and the reference labeled data, receiving detection data and a detection trajectory corresponding to the detection data from the vehicle, calibrating a trained second machine learning model using the detection data and the detection trajectory received from the vehicle, using the corrected trained second machine learning model to obtain first training data, training the first machine learning model using the first training data, the first training data being obtained using a trained second machine learning model calibrated by using the detection data and the detection trajectory received from the vehicle, and The trained first machine learning model is output to the vehicle.

9. The method for controlling a server device according to claim 8, wherein: The step of training the second machine learning model comprises the following steps: inputting the reference data of the second training data into the second machine learning model, acquiring, from the second machine learning model, first labeled data corresponding to the reference data of the input second training data, and The second machine learning model is trained to reduce an error between the first labeled data obtained from the second machine learning model and the reference labeled data included in the second training data.

10. The method for controlling a server device according to claim 8, wherein: The following steps are also included: evaluating the second machine learning model; The step of evaluating the second machine learning model comprises the following steps: inputting the reference data of the second training data into a second machine learning model trained to reduce an error between the first labeled data obtained from the second machine learning model and the reference labeled data included in the second training data, acquiring evaluation labeled data corresponding to the reference data of the second training data from a second machine learning model trained to reduce an error between the first labeled data acquired from the second machine learning model and the reference labeled data included in the second training data, and When the error between the evaluation label data obtained from the trained second machine learning model and the reference label data included in the second training data is greater than a reference error, the trained second machine learning model is corrected.

11. The method for controlling a server device according to claim 10, wherein: The following steps are also included: When the error between the evaluation label data obtained from the trained second machine learning model and the reference label data included in the second training data is less than or equal to the reference error, the first training data is obtained using a second machine learning model trained to reduce the error between the first label data obtained from the second machine learning model and the reference label data included in the second training data.

12. The method for controlling a server device according to claim 10, wherein: The step of correcting the trained second machine learning model comprises the following steps: inputting the detection data received from the vehicle into a second machine learning model trained to reduce an error between the first labeled data acquired from the second machine learning model and the reference labeled data included in the second training data, acquiring, from a second machine learning model trained to reduce an error between the first labeled data acquired from the second machine learning model and the reference labeled data included in the second training data, correction label data corresponding to the detection data received from the vehicle, and Correcting a second machine learning model trained to reduce an error between the first labeled data obtained from the second machine learning model and the reference labeled data included in the second training data to reduce a correction error between the corrected labeled data obtained from the trained second machine learning model and the detection trajectory received from the vehicle.

13. The method for controlling a server device according to claim 12, wherein: The step of correcting the trained second machine learning model comprises the following steps: Based on the error between the evaluation label data obtained from the trained second machine learning model and the reference label data included in the second training data, a correction error between the correction label data obtained from the trained second machine learning model and the detection trajectory received from the vehicle is adjusted.

14. The method for controlling a server device according to claim 13, wherein: The step of adjusting the correction error comprises the following steps: Adjust a correction error between the correction label data obtained from the trained second machine learning model and the detection trajectory received from the vehicle so that the correction error between the correction label data obtained from the trained second machine learning model and the detection trajectory received from the vehicle increases as an error between the evaluation label data obtained from the trained second machine learning model and the reference label data included in the second training data increases, and so that the correction error between the correction label data obtained from the trained second machine learning model and the detection trajectory received from the vehicle decreases as an error between the evaluation label data obtained from the trained second machine learning model and the reference label data included in the second training data decreases.

15. A server device, wherein: include: a communicator configured to communicate with the vehicle, a memory configured to store the machine learning model and the training data, and One or more processors configured to: acquiring the training data including reference data and labeled data corresponding to the reference data, training the machine learning model using the training data comprising the reference data and the labeled data, receiving detection data and a detection trajectory corresponding to the detection data from the vehicle, evaluating a trained machine learning model using the training data comprising the reference data and the labeled data, based on an evaluation of the trained machine learning model using the training data, correcting the machine learning model using the detection data and the detection trajectory received from the vehicle, The corrected machine learning model is output to the vehicle.

16. The server device according to claim 15, wherein: The one or more processors are configured to, inputting the reference data of the training data into the machine learning model, Acquire a training trajectory corresponding to the reference data of the training data from the machine learning model, The machine learning model is trained to reduce an error between the training trajectory obtained from the machine learning model and labeled data included in the training data.

17. The server device according to claim 16, wherein: The one or more processors are configured to, inputting the reference data of the training data into a machine learning model trained to reduce an error between the training trajectory obtained from the machine learning model and the labeled data included in the training data, acquiring, from a machine learning model trained to reduce an error between the training trajectory acquired from the machine learning model and labeled data included in the training data, an evaluation trajectory corresponding to the reference data of the training data, When an evaluation error between the evaluation trajectory obtained from the machine learning model and the labeled data included in the training data is greater than a reference error, correct the machine learning model trained to reduce the error between the training trajectory obtained from the machine learning model and the labeled data included in the training data.

18. The server device according to claim 17, wherein: The one or more processors are configured to, inputting the detection data received from the vehicle into a machine learning model trained to reduce an error between the training trajectory obtained from the machine learning model and the labeled data included in the training data, acquiring a correction trajectory corresponding to the detection data received from the vehicle from a machine learning model trained to reduce an error between the training trajectory acquired from the machine learning model and labeled data included in the training data, Correcting a machine learning model trained to reduce an error between the training trajectory obtained from the machine learning model and labeled data included in the training data to reduce a correction error between the correction trajectory obtained from the trained machine learning model and the detection trajectory received from the vehicle.

19. The server device according to claim 18, wherein: The one or more processors are configured to, Based on an evaluation error between the evaluation trajectory obtained from the machine learning model and the labeled data included in the training data, a correction error between the correction trajectory obtained from the trained machine learning model and the detection trajectory received from the vehicle is adjusted.

20. The server device according to claim 19, wherein: The one or more processors are configured to, Adjust a correction error between the correction trajectory obtained from the trained machine learning model and the detection trajectory received from the vehicle so that the correction error increases as an evaluation error between the evaluation trajectory obtained from the machine learning model and the labeled data included in the training data increases, and so that the correction error between the correction trajectory obtained from the trained machine learning model and the detection trajectory received from the vehicle decreases as an evaluation error between the evaluation trajectory obtained from the machine learning model and the labeled data included in the training data decreases.

Images