Method and device for estimating a driving lane and location of a vehicle based on a map

Abstract

A method for estimating the location of a means of transportation according to an embodiment of the present disclosure may include: a step of obtaining first-1 location information of the means of transportation at a first time point; a step of generating lane classification information, which is information classifying lanes around the means of transportation and identifying lanes included in an image based on an image taken of the surroundings of the means of transportation; a step of identifying map information of a road where the means of transportation is located based on the first-1 location information from map data; a step of determining a lane where the means of transportation is located within the road where the means of transportation is located based on the lane classification information and the map information; a step of generating first-2 location information by correcting the first-1 location information based on the lane where the means of transportation is located and the map information; and a step of generating estimated location information of the means of transportation based on the first-2 location information and the determined lane.

Description

Technology Field

[0001] The technical concept of the present disclosure relates to a method for estimating the location of a means of transportation and an apparatus for performing the same, and more specifically, to a method for estimating the driving lane and location of a means of transportation based on a map. Background Technology

[0002] In navigation systems that guide means of transportation, such as aircraft, ships, and vehicles, to a destination by providing location and route information, it is important to determine the precise location of the means of transportation.

[0003] Currently, most navigation systems utilize a Global Navigation Satellite System (GNSS) that uses a satellite network to accurately estimate the location of a means of transportation.

[0004] GNSS uses satellites to determine the location of a means of transportation, thereby enabling the easy acquisition of location, speed, and time information regardless of time and space. Although GNSS is classified as a relatively stable system compared to other navigation systems because it fundamentally has an error within a certain range, errors in location information may occur due to clock offset, the influence of the atmosphere or ionosphere, multipath, and receiver noise, or there may be cases where the location cannot be determined because satellite signals cannot be received due to obstacles.

[0005] Recently, in navigation systems, there is an increasing need to provide users with the driving lane of a means of transportation by estimating the position in a direction perpendicular to the road driving direction, in addition to accurately estimating the position in the road driving direction. The problem to be solved

[0006] The problem that the technical concept of the present disclosure aims to solve is to provide a method for estimating information about the lane in which a means of transportation is traveling based on image data acquired around the means of transportation, and for accurately adjusting the position of the means of transportation based on the acquired lane information. means of solving the problem

[0007] A method for estimating the location of a means of transportation according to an embodiment of the present disclosure may include: a step of obtaining first-1 location information of the means of transportation at a first time point; a step of generating lane classification information, which is information classifying lanes around the means of transportation and identifying lanes included in an image based on an image taken of the surroundings of the means of transportation; a step of identifying map information of a road where the means of transportation is located based on the first-1 location information from map data; a step of determining a lane where the means of transportation is located within the road where the means of transportation is located based on the lane classification information and the map information; a step of generating first-2 location information by correcting the first-1 location information based on the lane where the means of transportation is located and the map information; and a step of generating estimated location information of the means of transportation based on the first-2 location information and the determined lane.

[0008] According to one embodiment, the step of determining the lane where the means of transportation is located within the road where the means of transportation is located, based on the lane classification information and the map information, includes the step of matching the lanes in the image with the lanes included in the map information based on the lane classification information and the map information, and determining the lane where the means of transportation is located using the matching result; and the step of generating the first-2 location information by correcting the first-1 location information based on the lane where the means of transportation is located and the map information may include the step of correcting the first-1 location information by performing map matching on the first-1 location information using the map information.

[0009] According to one embodiment, the method further includes the step of generating lane-in-lane location information indicating the location of the means of transportation within the lane where the means of transportation is located, based on an image taken of the surroundings of the means of transportation, and the step of determining the lane where the means of transportation is located within the road where the means of transportation is located, based on the lane classification information and the map information, may include the step of determining the lane where the means of transportation is located based on the lane-in-lane location information.

[0010] According to one embodiment, the step of determining the lane in which the means of transportation is located based on the lane location information may include the step of acquiring a plurality of lane location information corresponding to each of a plurality of time points, the step of determining whether the means of transportation changes lanes based on the change over time of the plurality of lane location information, and the step of determining the lane in which the means of transportation is located based on whether the lanes change lanes.

[0011] According to one embodiment, the step of generating location information within the lane based on the image may include the step of determining three-dimensional coordinates of lanes surrounding the means of transportation based on the means of transportation, and the step of determining the position of the means of transportation within the lane where the means of transportation is located based on the determined three-dimensional coordinates of the surrounding lanes.

[0012] According to one embodiment, the step of determining the three-dimensional coordinates of lanes around the means of transportation based on the means of transportation may include the step of determining the vanishing point of the lanes around the means of transportation within the image, the step of determining the attitude of the image sensor that captured the image based on the vanishing point, and the step of determining the three-dimensional coordinates of the surrounding lanes using the attitude of the image sensor.

[0013] According to one embodiment, the method may further include the steps of: obtaining 2-1 position information of the means of transportation at a second time point after the first time point; correcting the 2-1 position information using GPS data to generate GPS corrected position information; and correcting the GPS corrected position information using the 1-2 position information to generate the estimated position information including the estimated position and the estimated lane of the means of transportation.

[0014] According to one embodiment, the step of generating GPS corrected location information by correcting the 2-1 location information using GPS data includes the step of correcting the 2-1 location information using GPS data and the reliability range of the GPS data, and the step of generating estimated location information including the estimated location and estimated lane of the means of transportation by correcting the GPS corrected location information using the 1-2 location information may include the step of adjusting the reliability range of the 1-2 location information based on the lane where the means of transportation is located and the step of correcting the GPS corrected location information using the 1-2 location information and the reliability range of the 1-2 location information.

[0015] According to one embodiment, the method further includes the step of generating lane-in-lane location information indicating the location of the means of transportation within the lane where the means of transportation is located, based on an image taken of the surroundings of the means of transportation, and the step of adjusting the reliability range of the first-2 location information based on the lane where the means of transportation is located may include the step of adjusting the reliability range of the first-2 location information based on the lane where the means of transportation is located and the lane-in-lane location information.

[0016] According to one embodiment, the first-1 position information at the first time point may include information estimated using an IMU (inertial measurement unit) regarding the position to which the moving means has moved from an initial position at an initial time point prior to the first time point. Effects of the invention

[0017] A method for estimating the location of a means of transportation according to an exemplary embodiment of the present disclosure can obtain an accurate location of a means of transportation by estimating the initial location of the means of transportation based on GPS, image, and IMU information, and by estimating the driving lane of the means of transportation based on image data obtained by photographing the surroundings of the means of transportation. In addition, the method for estimating the location of a means of transportation according to the present disclosure can improve the accuracy of the location by adjusting the reliability depending on whether the means of transportation changes lanes.

[0018] The effects obtainable from the exemplary embodiments of the present disclosure are not limited to those mentioned above, and other unmentioned effects can be clearly derived and understood by those skilled in the art to which the exemplary embodiments of the present disclosure belong from the description below. That is, unintended effects resulting from the implementation of the exemplary embodiments of the present disclosure can also be derived by those skilled in the art from the exemplary embodiments of the present disclosure. Brief explanation of the drawing

[0019] FIG. 1 is a flowchart illustrating a method for estimating the position of a moving means according to an embodiment of the present disclosure. FIG. 2 is a block diagram illustrating the configuration of an electronic device according to an embodiment of the present disclosure. FIG. 3 is a block diagram illustrating the configuration of a processor according to one embodiment. FIG. 4 is a block diagram illustrating the configuration of a first position acquisition unit according to an embodiment. FIG. 5 is a diagram illustrating a method for estimating the position of a means of transportation based on image data obtained according to one embodiment. FIG. 6 is a block diagram illustrating the configuration of an image processing unit according to one embodiment. FIG. 7 is a diagram illustrating image information obtained about the surroundings of a means of transportation according to one embodiment. FIG. 8 is a block diagram illustrating the configuration of a second position acquisition unit according to one embodiment. FIG. 9 is a drawing illustrating an example of map data according to one embodiment. FIG. 10 is a block diagram illustrating the configuration of a position correction unit according to one embodiment. FIG. 11 is a drawing illustrating an example of determining a reliability range according to one embodiment. FIG. 12 is a graph illustrating an example of determining a reliability range according to one embodiment. FIG. 13 is a block diagram illustrating in detail the image processing unit of FIG. 6 according to one embodiment of the present disclosure. FIG. 14 is a block diagram of a data learning unit according to one embodiment of the present disclosure. FIG. 15 is a block diagram of a data inference unit according to one embodiment of the present disclosure. Specific details for implementing the invention

[0020] The terms used in this disclosure have been selected to be as widely used and general as possible, taking into account their functions within this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the overall content of this disclosure.

[0021] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by the terms. The terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present disclosure, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of multiple related items or any one of the multiple related items.

[0022] When a part of a specification is described as "including" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0023] Additionally, the term “part” as used in the specification refers to a hardware component, such as software, FPGA, or ASIC, and the “part” performs certain roles. However, the meaning of “part” is not limited to software or hardware. The “part” may be configured to reside in an addressable storage medium or may be configured to run one or more processors.

[0024] Accordingly, as an example, "parts" include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts." Additionally, "part" may be a designation used to describe the operation of a "part" to distinguish it from other operations, and the operation of a "part" may be executed by one or more processors. Furthermore, "part" may refer to a set of instructions that perform the operation of a "part."

[0025] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present disclosure in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0026] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0027] FIG. 1 is a flowchart illustrating a method for estimating the position of a moving means according to an embodiment of the present disclosure.

[0028] Referring to FIG. 1, an embodiment of the present disclosure may estimate first-1 position information of a means of transportation at a first time point, estimate a driving lane and first-2 position information based on lane classification information identified from an image taken around the means of transportation, and determine an estimated position based on the first-2 position information and the determined lane. The means of transportation according to an embodiment of the present disclosure may be referred to as a vehicle or an automobile, and may refer to any means of transportation that is operated by being classified as a road or a railway.

[0030] According to one embodiment, the 1-1 position information may be information estimated based on an inertial measurement unit (IMU) regarding the position to which a means of transportation has moved from an initial position at an initial time point prior to the first time point. The initial position may be a position estimated based on at least one of Global Positioning System (GPS) information, IMU information, and image information at an initial time point, and may be an estimated position generated for an initial time point according to the position estimation method of the present disclosure.

[0031] The first-2 location information may be a location estimated from road information specified in map information based on the first-1 location information and driving lane information identified based on image data sensed around the vehicle. That is, the electronic device of the present disclosure can perform highly reliable location estimation by first estimating the first-1 location information using only information sensed from the IMU without using GPS data and image information, and by estimating the first-2 location information in detail based on map information and image data. However, the first-1 location information of the present disclosure is not limited to information generated using only IMU data, and may also include information generated using at least one of GPS data and image information together with IMU data.

[0032] The method for obtaining the 1-1 position information will be described later through FIGS. 4 and 5, and the method for obtaining the 1-2 position information will be described later through FIGS. 8 and 9.

[0033] In step (S110), the electronic device can obtain first-1 position information of the means of transportation. The information required for estimating the first-1 position information may be IMU information, and additionally, image information may be used, but lane information identified from the image information may be excluded from the first-1 position information estimation. That is, the first-1 position information estimation may be the position of the means of transportation estimated solely by real-time IMU data without performing lane information identification processing on the image information.

[0034] In step (S120), the electronic device can generate lane classification information classifying lanes around the vehicle based on an image taken around the vehicle.

[0035] In step (S130), the electronic device can identify map information of a road based on first-1 location information from map data. For example, the electronic device can specify a location range for identifying map information from map data based on first-1 location information, and can identify map information of a road within the location range. The loaded map data may include lane information, and it can determine whether the generated lane classification information corresponds to lane information extracted from the map data.

[0036] In step (S140), the electronic device can determine the lane in which the means of transportation is located within the road in which the means of transportation is located, based on lane classification information and map information. The lane classification information is lane information specified based on an image taken of the surroundings of the means of transportation, and the map information is map information specified based on map data included in a previously generated map database; thus, the electronic device of the present disclosure can determine the lane in which the means of transportation is located by considering both the image and the map data.

[0037] In step (S150), the electronic device can generate first-2 location information by correcting first-1 location information based on the lane and map information where the means of transportation is located. For example, the electronic device can estimate a location in a direction perpendicular to the direction of movement of the means of transportation from the lane where the means of transportation is located, and can estimate a location in a direction corresponding to the direction of movement of the means of transportation from the map information. In addition, the electronic device can more specifically estimate a location in a direction perpendicular to the direction of movement of the means of transportation by estimating a location within the driving lane based on the lanes closest to the means of transportation among the identified surrounding lanes.

[0038] In step (S160), the electronic device can generate estimated location information of the means of transport based on the first-2 location information and the determined lane. The electronic device can adjust the reliability of the first-2 location information based on the lane where the means of transport is located.

[0039] In this case, if the Kalman Filter prediction model follows a Gaussian distribution, the confidence level of the first and second location information may vary depending on the covariance value of the Gaussian distribution.

[0040] For example, if the electronic device determines that the driving lane has not changed, it may set a narrow reliability range for the first-2 position information in the vertical direction of the movement direction of the means of transportation, and if it falls within the reliability range, it may estimate the first-2 position information as the final position. However, the method for estimating the final position according to the present invention is not limited thereto, and the final position may also be estimated by summing the coordinates of the first-1 position information and the first-2 position information by a certain ratio. Furthermore, the electronic device may estimate the final position by additionally using position information from a second time point after a first time point, which will be described later. The estimated final position may be utilized as an initial position for estimating the next position.

[0041] FIG. 2 is a block diagram illustrating the configuration of an electronic device according to an embodiment of the present disclosure.

[0042] Referring to FIG. 2, the electronic device (10) may include a processor (100), RAM (Random Access memory) (200), storage (300), and a communication unit (400). The electronic device (10) may be integrally coupled to a means of transport, but the electronic device (10) of the present disclosure is not limited thereto and may be coupled to the means of transport in a detachable configuration, or may exist within the means of transport as a separate device that is not coupled to the means of transport (e.g., a mobile phone).

[0043] The processor (100) can control the overall operation of the electronic device (10). The processor (100) may include a single processor core or a CPU (Central Processing Unit) that includes multiple processor cores. The electronic device (10) may include one or more processors (100).

[0044] Additionally, the processor (100) may include a Neural Processing Unit (NPU) that receives input data, performs computations using an artificial neural network, trains the artificial neural network, and provides output data based on the computation results. The NPU is capable of processing computations based on various types of networks, such as a Deep Neural Network, a Convolutional Neural Network (CNN), a Region with Convolutional Neural Network (R-CNN), a Region Proposal Network (RPN), a Residual Network (Residual Network), a Recurrent Neural Network (RNN), a Long Short-Term Memory (LSTM) Network, and a Transformer Neural Network. However, the components and functions of the processor (100) are not limited to these, and the processor (100) is capable of processing various types of computations that mimic human neural networks. Furthermore, the processor (100) can perform machine learning other than using an artificial neural network and can perform computations based on machine learning. The processor (100) can learn a set of tasks through various methods such as supervised learning, unsupervised learning, semi-supervised learning, self-directed learning, and reinforcement learning, and the algorithm used for the learning of the processor (100) can be determined in various ways.

[0045] The processor (100) can process or execute programs, data, or instructions stored in the storage (300). For example, the processor (100) can generate lane classification information, identify map information, and generate estimated location information of a means of transportation by executing programs stored in the storage (300) and determining the lane in which the means of transportation is located.

[0046] RAM (200) can temporarily store programs, data, or instructions. For example, programs and / or data stored in storage (300) may be temporarily stored in RAM (200) under the control of the processor (100) or boot code. For example, RAM (200) includes DRAM (Dynamic RAM), SRAM (Static RAM), SDRAM (Synchronous DRAM), etc.

[0047] Storage (300) is a storage place for storing data and can store an OS (Operating System), various programs, and various data. For example, storage (300) can store a learning network model and at least one of each component included in the learning network model.

[0048] Storage (300) includes ROM (Read Only Memory), flash memory, PRAM (Phase-change RAM), MRAM (Magnetic RAM), RRAM (Resistive RAM), FRAM (Ferroelectric RAM), etc. In an embodiment, storage (300) may be implemented as HDD (Hard Disk Drive), SSD (Solid State Drive), etc.

[0049] The communication unit (400) can transmit and / or receive data from the electronic device (10). For example, the communication unit (400) can transmit and receive data by various communication methods. For example, the communication unit (400) can perform communication by, for example, 3G, LTE, Wi-Fi, Bluetooth, BLE (Bluetooth Low Energy), Zigbee, NFC (Near Field Communication), and communication methods using ultrasound, and the communication method can be determined in various ways, such as wired communication, wireless communication, short-range communication, and long-range communication.

[0050] The electronic device may further include an image sensor and an input / output device. The image sensor may include a camera, LIDAR (Light Detection And Ranging), or RADAR (Radio Detection And Ranging), and may transmit sensed image data to a processor, RAM, and / or storage. The input / output device may be composed of an input device that receives operation requests from a user and an output device that provides data to the user. That is, the electronic device may receive operation input from a user through the input / output device and provide data corresponding to the operation input to the user, such as by displaying it.

[0051] FIG. 3 is a block diagram illustrating the configuration of a processor (100) according to one embodiment.

[0052] Referring to FIG. 3, a processor (100) according to one embodiment may include a first-1 position information acquisition unit (110), an image processing unit (120), a first-2 position information acquisition unit (130), and a position correction unit (140). Each component of the processor (100) may be composed of different hardware, but the processor (100) of the present disclosure is not limited thereto and may be composed of software components that perform different functions on a single hardware.

[0053] The first-1 position information acquisition unit (110) can receive initial position information (LOC_INI) and IMU information (IMU) at an initial time point prior to the first time point and generate first-1 position information (LOC_1-1) at the first time point.

[0054] According to another embodiment, the first-1 location information acquisition unit (110) can receive at least one of GPS information (GPS), image information (IMG), and IMU information (IMU), and can output the first-1 location information (LOC_1-1) of the means of transportation based on at least one of the GPS information (GPS), image information (IMG), and IMU information (IMU). At this time, the first-1 location information acquisition unit (110) can directly generate initial location information (LOC_INI) and can generate the first-1 location information (LOC_1-1) from the initial location information (LOC_INI) and IMU information (IMU). The first-1 location information (LOC_1-1) may be a location coordinate value estimated as the current location of the means of transportation. Alternatively, the first-1 position information acquisition unit (110) receives initial position information (LOC_INI) and IMU information (IMU), and additionally receives at least one of GPS information (GPS) and image information (IMG), and can generate a position moved from the initial position to the first time point based on at least one of IMU information (IMU), GPS information (GPS), and image information (IMG).

[0055] IMU information (IMU) may be information obtained from an IMU sensor that is included in an electronic device (10) or in a means of transportation and measures the speed, direction, gravity, and acceleration of the means of transportation. The first-1 position information acquisition unit (110) can predict the first-1 position information (LOC_1-1) from the speed and position of the means of transportation. For example, the first-1 position information prediction unit (114) can predict the first-1 position information (LOC_1-1) using IMU information (IMU) including the speed and acceleration of the means of transportation obtained through the IMU sensor during the elapsed time from the initial position generated or received.

[0056] According to one embodiment, the 1-1 position information acquisition unit (110) can generate 2-1 position information (LOC_2-1) at a second time point, which is a time point after the first time point, and transmit it to the position correction unit (140). The 2-1 position information (LOC_2-1) differs from the 1-1 position information (LOC_1-1) only in the time point, but the method of acquisition is the same.

[0057] The image processing unit (120) can receive image information (IMG) sensed around the means of transportation and can identify lanes from the image information (IMG) to output lane-in-lane position information (EL) and lane classification information (LC) of the means of transportation. The image processing unit (120) can calculate lane centerlines based on the lanes closest to the means of transportation among the identified lanes and can estimate lane-in-lane position information (EL) of the means of transportation based on the lane centerlines.

[0058] The first-2 location information acquisition unit (130) can receive the first-1 location information (LOC_1-1) from the first-1 location information acquisition unit (110) and receive lane-in-lane location information (EL) and lane classification information (LC) from the image processing unit (120). In addition, the first-2 location information acquisition unit (130) can receive map data (MAP) from an external component of the processor (100). The external component of the processor (100) may be storage and / or RAM, but the processor (100) disclosed herein is not limited thereto and may receive map data (MAP) from an external device through a communication unit.

[0059] The first-2 location information acquisition unit (130) can output driving lane information (LANE) and first-2 location information (LOC_1-2) based on first-1 location information (LOC_1-1), lane-in-lane location information (EL), lane classification information (LC), and map data (MAP). Driving lane information (LANE) may be identification information for the lane in which a means of transportation is driving, and, for example, may be the sequence number of the lane in which the vehicle is driving among the total number of lanes. The first-2 location information (LOC_1-2) may be location information in which the driving direction and the position in the direction perpendicular to the driving direction are more accurately specified for the first-1 location information (LOC_1-1) based on driving lane information (LANE) and lane-in-lane location information (EL).

[0060] The position correction unit (140) can generate estimated position information (LOC_EST) of a means of transportation based on first-second position information (LOC_1-2) and driving lane information (LANE). The estimated position information (LOC_EST) may be position information that is finally provided to a user riding in a moving means of transportation, and, for example, may include display graphic information showing the first-second position information (LOC_1-2) displayed on a map and the driving lane information (LANE) of the means of transportation. Additionally, the position correction unit (140) can generate estimated position information (LOC_EST) by combining position information generated at a first time point and a second time point. For example, the position correction unit (140) can generate estimated position information (LOC_EST) based on second-first position information (LOC_2-1) generated from IMU information (IMU) at the second time point and first-second position information (LOC_1-2) obtained at the first time point. At this time, the position correction unit (140) can generate a GPS correction position from the 2-1 position information (LOC_2-1) using GPS data obtained at the second time point, and can generate an estimated position information (LOC_EST) by correcting the GPS correction position using the 1-2 position information (LOC_1-2) generated at the first time point, which is the time point prior to the second time point.

[0061] That is, the electronic device (10) of the present disclosure may use the second-1 position information (LOC_2-1) obtained from IMU information (IMU) at a second time point to estimate the accurate position of the means of transportation at a second time point, wherein the IMU information (IMU) is information obtained at a fast speed of about 100 Hz, and, for example, the n-1 position information (n is a natural number) may be obtained every 0.01 seconds. That is, the electronic device (10) is able to obtain the n-1 position information at a faster speed compared to obtaining position information using GPS information and image information obtained at a slower speed (usually about 1 Hz) than the IMU information (IMU).

[0062] After the electronic device (10) acquires the 2-1 location information (LOC_2-1), it can use GPS data and load only the 1-2 location information (LOC_1-2) that has already been generated before the 2-1 time point, so it can accurately and quickly generate the estimated location information (LOC_EST) at the 2-1 time point from the 2-1 location information (LOC_2-1).

[0063] FIG. 4 is a block diagram illustrating the configuration of a first-1 position information acquisition unit (110) according to one embodiment.

[0064] Referring to FIG. 4, the first-1 position information acquisition unit (110) may include a GPS position estimation unit (111), an image position estimation unit (112), an initial position setting unit (113), a first-1 position information prediction unit (114), and a first-1 position information correction unit (115). Each component of the first-1 position information acquisition unit (110) may be composed of different hardware, but the first-1 position information acquisition unit (110) of the present disclosure is not limited thereto and may be composed of software components that perform different functions on a single hardware.

[0065] The GPS position estimation unit (111) can estimate the current location of a means of transportation by receiving GPS information (GPS) from multiple satellites. The GPS position estimation unit (111) can obtain time information of the GPS information (GPS) transmitted from the satellites and obtain the distance between the satellites and the means of transportation by comparing the time information of the GPS information (GPS) with the time information at the time of reception. In addition, the GPS position estimation unit (111) can obtain the orbit information of the satellites and ephemeral data as auxiliary information for accurately estimating the GPS position information (LOC_GPS). The GPS position estimation unit (111) can generate the GPS position information (LOC_GPS) of the means of transportation based on the GPS information (GPS) received from the satellites.

[0066] The image location estimation unit (112) can generate image location information (LOC_IMG) based on image information (IMG) sensed from the surroundings of the vehicle and GPS location information (LOC_GPS). The image location information (LOC_IMG) may be location information of the vehicle estimated based on the sensed image information (IMG). The image location estimation unit (112) can compare the image information (IMG) with image information of the surrounding roads near the GPS location information (LOC_GPS) stored in a database, and can estimate the location information of the vehicle based on the comparison result. The image information of the surrounding roads stored in the database may be Road view image information, and may be information from which feature points are extracted from the Road view image information.

[0067] The initial location setting unit (113) can output initial location information (LOC_INI) of the means of transportation based on GPS location information (LOC_GPS) and image location information (LOC_IMG). According to one embodiment, the initial location information (LOC_INI) may be information obtained by weighted summing the location coordinates obtained from the GPS location information and the location coordinates obtained from the image location information (LOC_IMG).

[0068] The electronic device according to an embodiment of the present disclosure may directly generate GPS location information (LOC_GPS), image location information (LOC_IMG), and initial location information (LOC_INI), but is not limited thereto, and may also receive initial location information (LOC_INI) set by a server. Additionally, the initial location information (LOC_INI) may be estimated location information generated according to the method of the present disclosure for an initial time point that is prior to the first time point (i.e., LOC_EST of FIG. 3 for the initial time point).

[0069] For example, at least one of the GPS position estimation unit (111), image position estimation unit (112), and initial position setting unit (113) may be performed by a server. The electronic device may transmit GPS information (GPS) and / or GPS position information (LOC_GPS) and image information (IMG) to the server, receive image position information (LOC_IMG) and / or initial position information (LOC_INI) from the server, and further receive GPS position information (LOC_GPS). As described above, some of what is described as the configuration or operation of the electronic device in this disclosure may be performed by a server communicating with the electronic device without further explanation.

[0070] The first-1 position information prediction unit (114) can receive generated or received initial position information (LOC_INI) and IMU (Inertial Measurement Unit) information and output the first-1 position information (LOC_1-1). The IMU information (IMU) may be information obtained from an IMU sensor that is included in the electronic device (10) or the means of transportation and measures the speed, direction, gravity, and acceleration of the means of transportation. According to one embodiment, the first-1 position information prediction unit (114) can predict the first-1 position information (LOC_1-1) from the speed and position of the means of transportation. For example, the first-1 position information prediction unit (114) can predict the first-1 position information (LOC_1-1) using IMU information (IMU) including the speed and acceleration of the means of transportation obtained through the IMU sensor during the elapsed time from the initial position.

[0071] The first-1 position prediction unit (114) transmits the first-1 position information (LOC_1-1) to the first-2 position acquisition unit (130). Additionally, according to one embodiment, the first-1 position prediction unit (114) may generate the second-1 position information (LOC_2-1) at a second time point after the first time point and transmit it to the position correction unit (140). The second-1 position information (LOC_2-1) differs from the first-1 position information (LOC_1-1) only in the time point, but the method of acquisition is the same.

[0072] FIG. 5 is a diagram illustrating a method for estimating the position of a means of transportation based on image data obtained according to one embodiment.

[0073] Referring to FIG. 5, an image sensor attached to an electronic device (10) or a means of transportation according to one embodiment can sense the front of the means of transportation. The electronic device (10) can load image information of the road surrounding the GPS location information from a database in which image information of the road surrounding the road is stored, compare the sensed image information with the loaded image information of the road surrounding the road, and predict the location of the means of transportation based on the comparison result. The previously stored image information of the road surrounding the road may be information in which image data and location data where the image was taken are mapped and stored, and the electronic device (10) can load image information of the road surrounding the road surrounding the means of transportation that is mapped to location data of the location surrounding the means of transportation.

[0074] The location of the means of transportation to be used for database search may be predicted based on GPS data, predicted based on GPS data and IMU data, or a final corrected location according to an embodiment of the present disclosure. The electronic device (10) can identify road-around image information with high similarity by comparing the road-around image information around the location of the means of transportation loaded from a database storing road-around image information with the feature points of the sensed image information. For example, the feature points may be shape information (51) of structures on both sides of the road, and the road-around image information most similar to the shape information (51) of structures obtained from the sensed image can be identified. The electronic device (10) can extract location data mapped to the identified road-around image information and can generate the extracted location data as image location information.

[0075] The database storing road surrounding image information according to the embodiment of FIG. 5 may be stored on an external server and transmitted to the electronic device (10) through a communication unit, but is not limited thereto and may be stored in the storage of the electronic device (10). In addition, the feature points of the road surrounding image information are not limited only to the shape information (51) of the structures, but may include all information for distinguishing images taken at different locations.

[0076] FIG. 6 is a block diagram illustrating the configuration of an image processing unit (120) according to one embodiment.

[0077] Referring to FIG. 6, the image processing unit (120) may include a lane detection unit (121), a lane classification unit (122), and a lane position estimation unit (123). Each component of the image processing unit (120) may be composed of different hardware, but the image processing unit (120) of the present disclosure is not limited thereto and may be composed of software components that perform different functions on a single hardware.

[0078] An image sensor attached to an electronic device (10) or vehicle according to one embodiment can sense the front of the means of transportation and can provide the sensed image information (IMG) to an image processing unit (120). The image information (IMG) input to the image processing unit (120) and the image information (IMG) input to the first-1 position information acquisition unit (110) may be information obtained from the same image sensor, but the source of the image information (IMG) of the present disclosure is not limited thereto and may be information obtained from different image sensors.

[0079] According to one embodiment, the image processing unit (120) can identify lanes from image information (IMG) based on a learning network model and classify the identified lanes into multiple groups according to the shape of the lanes. For example, the image processing unit (120) can identify lanes based on a first inference model and output coordinate values ​​of the identified lanes, and can identify the shapes of the identified lanes based on a second inference model and classify the identified lanes into multiple groups.

[0080] The image processing unit (120) may further include a data acquisition unit, a preprocessing unit, a training data selection unit, a model training unit, and a model evaluation unit to train a learning network model based on training data, and may further include a data acquisition unit, a preprocessing unit, an inference data selection unit, an inference result provision unit, and a model update unit to generate inference data for input data using the learning network model. Components for the image processing unit (120) to perform operations based on the learning network model will be described later through FIGS. 13 to 15.

[0081] The lane detection unit (121) can output lane information (LINE) by identifying lanes based on input image information (IMG). The lane information (LINE) may consist of the number of lanes and the coordinate values ​​of each lane (e.g., 3D coordinate values), and the coordinate values ​​of each lane may be extracted as 3D coordinates in space centered on the means of transportation. The lane detection unit (121) can output lane information (LINE) by mapping 3D coordinates to each extracted lane.

[0082] The lane classification unit (122) can classify lane information (LINE) into lane groups based on the lane shape and color of the extracted lane. For example, the lane groups may be configured to include solid line groups, dotted line groups, center line groups, and double line groups by color, and may include all types of lane shapes and colors identifiable by the lane detection unit (121). The lane classification unit (122) can identify the shape and color of each of the extracted lanes to group identical lane shapes and output grouped lane classification information (LC).

[0083] The lane position estimation unit (123) can identify the lanes closest to the means of transport among the identified multiple lanes and estimate the position of the means of transport within the driving lane. According to one embodiment, the lane position estimation unit (123) can calculate the coordinate values ​​of adjacent lanes (e.g., 3D coordinate values) based on the means of transport based on the vanishing point calculated based on some lanes and the attitude information of the sensor that senses image information (IMG), and can calculate the position of the means of transport within the lane based on the coordinate values ​​of the adjacent lanes. Accordingly, the lane position estimation unit (123) can output the direction and degree of deviation of the means of transport within the driving lane as lane position information (EL). For example, the lane position estimation unit (123) can output numerical lane position information, determine the sign of the lane position information according to the direction of deviation, and output the degree of deviation as one of multiple levels.

[0084] The image processing unit (120) can estimate lane classification information (LC) and lane position information (EL) from the received image information (IMG) and output them to the first-second position information acquisition unit (130).

[0085] FIG. 7 is a diagram illustrating image information obtained about the surroundings of a means of transportation according to one embodiment.

[0086] Referring to FIG. 7, the electronic device (10) can obtain image information including lanes by taking a picture of the front direction of the means of transportation. The image information may further include information about the means of transportation driving around the means of transportation and information about the road surroundings, but in the embodiment of FIG. 7, the description of other means of transportation and information about the road surroundings is omitted in order to explain the lane information.

[0087] The electronic device (10) can recognize lanes from image information and can extract coordinate values ​​of the recognized lanes by mapping them to each lane. For example, the electronic device (10) can identify a lane when pixels specified by the same RGB value or grayscale value in the image information are connected adjacently for a certain number of pixels or more, and can map coordinate values ​​of the lane to each identified lane. However, the lane recognition method that can be used in the present disclosure is not limited thereto and may include any lane recognition method, including a method for extracting lane regions based on a learning network model.

[0088] According to one embodiment, the electronic device (10) can identify areas other than lanes among the image information as unusable areas and divide the remaining areas of the unusable areas into a plurality of image patches. The electronic device (10) can input the divided image patches into a learning network model and obtain shape information for each lane as an inference result for the divided image patches. Based on the shape information for each lane obtained, the electronic device (10) can generate lane classification information by grouping lanes classified as having the same lane shape into the same lane group.

[0089] According to FIG. 7, the electronic device (10) can identify the first to seventh lanes (71 to 77), and since the first lane (71) and the seventh lane (77) are recognized as solid lines, they are grouped as solid line lanes, and since the second lane (72), the third lane (73), the fifth lane (75), and the sixth lane (76) are recognized as dotted lines, they can be grouped as dotted line lanes. The electronic device (10) can recognize the fourth lane (74) as a double line and group it as a center line group.

[0090] According to one embodiment, the electronic device (10) can obtain vanishing point points of lanes from image information. For example, the vanishing point point may be a point where the parts near the means of transportation of each lane converge when extended in a straight line. For example, the electronic device (10) can obtain vanishing point points through various methods of image analysis or depth analysis of the image, such as virtually extending each identified lane or using divided image patches. The electronic device (10) can obtain three-dimensional coordinate values ​​of the lanes relative to the means of transportation from the vanishing point points. For example, the electronic device (10) can estimate the orientation of the image sensor using the vanishing point points and calculate three-dimensional coordinate values ​​of the lanes based on the estimated orientation. The orientation of the image sensor may be the relative orientation of the image sensor to the lanes. The three-dimensional coordinate values ​​may not simply be a uniform coordinate system applied to a two-dimensional image, but may be a three-dimensional coordinate system applied to an actual three-dimensional space relative to the means of transportation. The electronic device (10) can obtain a lane centerline from the coordinate values ​​of the nearest lanes, which consist of the nearest left lane and the nearest right lane of the means of transportation.

[0091] According to FIG. 7, the electronic device (10) identifies the fourth lane (74) as the nearest left lane based on the means of transport and identifies the fifth lane (75) as the nearest right lane based on the means of transport, and can obtain the center line of the fourth lane (74) and the fifth lane (75) in three-dimensional coordinates as the lane center line.

[0092] According to one embodiment, the electronic device (10) can estimate the position of a means of transportation within the lane based on the acquired lane centerline. For example, if the lane centerline is formed at the center of the image information, the electronic device (10) can recognize that the means of transportation is located in the center within the lane. That is, the electronic device (10) can acquire position information within the lane according to the direction and degree in which the lane centerline extends relative to the vanishing point.

[0093] According to one embodiment, the electronic device (10) can acquire attitude information of the sensor and correct lane centerline and lane position information based on the attitude information of the sensor. For example, the electronic device (10) may not correct lane centerline and lane position information when the sensor acquiring image information acquires attitude information by looking at the center of the road, but may correct lane centerline and lane position information by acquiring a tilting angle when the sensor is tilted toward one of the edges of the road relative to the center of the road.

[0094] FIG. 8 is a block diagram illustrating the configuration of a first-second position information acquisition unit (130) according to one embodiment.

[0095] Referring to FIG. 8, the first-second location information acquisition unit (130) may include a lane determination unit (131), a map information extraction unit (132), and a first-second location information determination unit (133). Each component of the first-second location information acquisition unit (130) may be composed of different hardware, but the first-second location information acquisition unit (130) of the present disclosure is not limited thereto and may be composed of software components that perform different functions on a single hardware.

[0096] The map information extraction unit (132) can receive map data (MAP) and first-1 location information (LOC_1-1), and can provide map information (MAP_INFO) specified from the map data (MAP) based on the first-1 location information (LOC_1-1) to the lane determination unit (131) and the first-2 location information determination unit (133). The map data (MAP) may be referred to as HD (High-Definition) Map information and may include the number of lanes of a road, link information, and node information. For example, the map information extraction unit (132) can extract road information (MAP_INFO) within a certain range from the map information (MAP) based on the first-1 location information, and the map information (MAP_INFO) within a certain range may include the number of lanes of a road included in the map, link information, and node information.

[0097] The lane determination unit (131) receives map information (MAP_INFO), lane location information (EL), and lane classification information (LC), and can determine the driving lane in which the vehicle is driving based on the lane location information (EL) and lane classification information (LC) and output driving lane information (LANE). The driving lane information (LANE) may consist of sequence information of driving lanes for the total number of lanes. The lane determination unit (131) can estimate the lane sequence of the driving lane based on the lane classification information (LC).

[0098] For example, the lane determination unit (131) can estimate the driving lane as lane 1 when the nearest left lane is extracted as a center line group, and can estimate the driving lane as the last number among all driving lane numbers when the nearest right lane is extracted as a solid line group. This is an exemplary driving lane estimation method, and the driving lane estimation method that can be used in the present disclosure is not limited thereto and may include a method that can estimate the driving lane based on lane information (LINE) estimated by the image processing unit (120).

[0099] According to one embodiment, the lane determination unit (131) can match lanes within an image with lanes included in the map information (MAP_INFO) based on lane classification information (LC) and map information (MAP_INFO). The lane determination unit (131) can determine the lane where the means of transportation is located using the matching result.

[0100] For example, the lane determination unit (131) may predict a driving lane based on lane classification information (LC) and correct the predicted driving lane based on map information (MAP_INFO) extracted from map data (MAP). Even if the nearest left lane is extracted as a center line group, the lane determination unit (131) may not estimate the lane as the first lane, but determine it as the pocket lane and estimate the lane located to the right of the vehicle as the first lane, if the lane is recognized as a pocket lane for waiting for a left turn signal based on map information (MAP_INFO).

[0101] The lane determination unit (131) acquires multiple lane position information (EL) corresponding to each of multiple time points, determines whether the means of transportation changes lanes based on changes in the multiple lane position information (EL) over time, and can determine the lane in which the means of transportation is located based on whether the lane changes lanes. For example, the lane position information (EL) can be determined with a (-) sign for the left position value and a (+) sign for the right position value based on the center of the lane (position value is 0). In this case, if the absolute value of the lane position information (EL) increases with a (-) sign over time (e.g., -3, -4, -5) and then changes to a larger value with a (+) sign (e.g., +5, +4), the lane determination unit (131) can determine that the means of transportation is located in the first lane when the lane position information (EL) value is (-) and has changed lanes to the second lane when the lane position information (EL) value becomes (+). That is, when the lane determination unit (131) receives digitized lane location information (EL), it can determine the lane in which the means of transportation is located by determining that the means of transportation has changed lanes when the code of the lane location information (EL) changes.

[0102] The first-2 location information determination unit (133) can output the first-2 location information (LOC_1-2) by receiving driving lane information (LANE) from the lane determination unit (131) and receiving map information (MAP_INFO) from the map information extraction unit (132). The first-2 location information determination unit (133) can determine the road and lane sequence in which the means of transportation is traveling based on the map information (MAP_INFO) and driving lane information (LANE). That is, the second location determination unit (133) can accurately determine the position in the vertical direction of the driving direction by first determining the lane among the roads in which the means of transportation is traveling, and secondarily determining the direction and degree of deviation within the lane.

[0103] The first-2 position information determination unit (133) can correct the first-1 position information by performing map matching using map information on the first-1 position information. Map matching is a method of correcting the path traveled by the means of transportation, and may mean correcting the continuous position information of the means of transportation to a path that is highly likely to be the path actually traveled by the means of transportation by using node information or road information included in the map information. Additionally, map matching may include correcting the position to a location that is highly likely to have actually traveled by the means of transportation by matching only one position information with node information or road information included in the map information, instead of utilizing multiple position information according to the path traveled by the means of transportation. For example, a hidden Markov model may be used for map matching, but the map matching method of the present disclosure is not limited thereto.

[0104] According to one embodiment, the first-2 position information determining unit (133) can generate the first-2 position information (LOC_1-2) by correcting the first-1 position information (LOC_1-1) by matching the estimated position information and map information estimated at previous time points of the first time point with the path. For example, if the first-2 position information determining unit (133) determines that the means of transportation has not changed lanes, it can generate the first-2 position information (LOC_1-2) from the first-1 position information (LOC_1-1) and any point on the extension of the estimated position information at previous time points. Conversely, if the first-2 position information determining unit (133) determines that the means of transportation has changed lanes, it can generate the first-2 position information (LOC_1-2) from the first-1 position information (LOC_1-1) and any one of the adjacent lanes of the driving lane information (LANE).

[0105] According to one embodiment, the first-2 position information determining unit (133) may additionally receive lane-in-lane position information (EL) and may generate first-2 position information (LOC_1-2) that reflects a detailed position within the lane based on the lane-in-lane position information (EL).

[0106] FIG. 9 is a drawing illustrating an example of map data according to one embodiment.

[0107] Referring to FIG. 9, map information loaded from map data may include the number of lanes of the road, link information (92), and node information (91). The map data may be composed of a data model that extends the node-link system for the roads of an existing navigation map into individual lanes.

[0108] Node information (91) is generated for stop lines on a roadway, start and end points of entry and exit lanes, tunnels, bridges, overpasses, start and end points of underpasses, etc. Each node information (91) may be linked to at least one of the different node information (91), and the information to which the node information (91) are linked may be referred to as link information (92).

[0109] When the electronic device (10) estimates that a means of transportation is located in any of the node information (91), it may estimate that the next location of the means of transportation corresponds to the node information (93) connected to the corresponding node information (91) and link information (92). When the electronic device (10) recognizes that the means of transportation has not moved to the node information (93) connected to the link information (92), it may determine that the lane has been changed. The electronic device (10) may also include the lane change status regarding the previous driving lane in the driving lane information and output it.

[0110] FIG. 10 is a block diagram illustrating the configuration of a position correction unit (140) according to one embodiment.

[0111] Referring to FIG. 10, the position correction unit (140) may include a reliability adjustment unit (141) and an estimated position generation unit (142). Each component of the position correction unit (140) may be composed of different hardware, but the position correction unit (140) of the present disclosure is not limited thereto and may be composed of software components that perform different functions on a single piece of hardware.

[0112] The reliability adjustment unit (141) receives the first-2 position information (LOC_1-2) and driving lane information (LANE) from the first-2 position information acquisition unit (130) and can set a reliability range (LOC_RG) for position estimation. The reliability range (LOC_RG) for position estimation may be an effective position range set according to the adjustment of the covariance value of the Kalman filter model.

[0113] According to one embodiment, the reliability adjustment unit (141) can adjust the reliability range of the first-second location information (LOC_1-2) based on the lane where the means of transport is located. For example, the reliability adjustment unit (141) can adjust the mean value and / or variance value of the Gaussian distribution constituting the reliability range corresponding to the first-second location information (LOC_1-2) based on the lane where the means of transport is located. For example, the reliability adjustment unit (141) can adjust the reliability range so that it is limited to within the lane where the means of transport is located. Additionally, the reliability adjustment unit (141) can further adjust the reliability range by utilizing the lane-in-lane location information (EL). For example, the reliability adjustment unit (141) can adjust the reliability range in the direction perpendicular to the lane (i.e., the direction perpendicular to the direction of travel of the means of transport) so that it is limited to within a range equal to the lane-in-lane spacing (i.e., lane spacing) on ​​both sides from the lane-in-lane location.

[0114] If the confidence range (LOC_RG) for location estimation is set narrowly, it may mean that the vehicle can be located within a narrow radius with a certain probability or higher based on the estimated first and second location information; conversely, if the confidence range (LOC_RG) is set wide, it may mean that the vehicle can be located within a wide radius with a certain probability or higher based on the estimated first and second location information. In other words, the narrower the confidence range (LOC_RG) is set or adjusted, the more accurately the location of the vehicle can be estimated.

[0115] The estimated location generation unit (142) can generate estimated location information of a means of transportation by receiving 2-1 location information (LOC_2-1), 1-2 location information (LOC_1-2), and a reliability range (LOC_RG). The 2-1 location information (LOC_2-1) may be location information estimated based on IMU information (IMU) at a second time point after a first time point.

[0116] According to one embodiment, the estimated location generation unit (142) may receive additional GPS data and may generate GPS corrected location information by correcting the 2-1 location information (LOC_2-1) using the GPS data. The estimated location generation unit (142) may correct the 2-1 location information (LOC_2-1) using the GPS data and the reliability range of the GPS data. At this time, the reliability range of the GPS data may be a predetermined range.

[0117] For example, the estimated location generation unit (142) can calculate a confidence range based on location information indicated by GPS data, and can generate a GPS corrected location by inputting the calculated confidence range and the 2-1 location information (LOC_2-1) into a Kalman filter.

[0118] The estimated location generation unit (142) can generate estimated location information (LOC_EST) by correcting GPS correction location information using the first-2 location information (LOC_1-2) and the reliability range (LOC_RG) of the first-2 location information (LOC_1-2).

[0119] For example, the estimated location generation unit (142) can generate virtual location information where the extension line of the initial location information and the first-second location information (LOC_1-2) meets the GPS corrected location information and the extension line of the second-first location information (LOC_2-1). The estimated location generation unit (142) can generate estimated location information (LOC_EST) based on the virtual location information, the first-second location information (LOC_1-2), and the reliability range (LOC_RG) of the first-second location information (LOC_1-2). Accordingly, the estimated location generation unit (142) of the present disclosure can generate the first-second location information (LOC_1-2) from a first time point, obtain the second-first location information (LOC_2-1) at a second time point, and correct the location error as time elapses while generating the estimated location, thereby generating accurate estimated location information (LOC_EST).

[0120] That is, the method for estimating the location of a means of transportation of transportation of transportation can accurately estimate the location of the means of transportation by adjusting the reliability range (LOC_RG) for location estimation based on lane and lane estimation information. The method for adjusting the reliability range (LOC_RG) in FIGS. 11 and 12 will be described in detail below.

[0121] According to one embodiment, the estimated location generation unit (142) can receive lane-in-lane location information (EL) at a first time point and can generate estimated location information (LOC_EST) by reflecting the lane-in-lane location information (EL) when correcting GPS correction location information using the first-2 location information (LOC_1-2). The estimated location generation unit (142) can generate estimated location information (LOC_EST) corresponding to one of the neighboring lanes when the lane-in-lane location information (EL) approaches a threshold value. The threshold value may be the maximum and minimum values ​​of the quantified lane-in-lane location information, and the case of approaching the threshold value may be within a certain value from the threshold value.

[0122] For example, if the maximum and minimum values ​​of the lane position information (EL) are +5 and -5 respectively, the threshold value may be +5 and -5. When a certain value is set to 1, the estimated position generation unit (142) can generate estimated position information (LOC_EST) corresponding to any one of the neighboring lanes when the lane position at the first time point is +4 or greater or -4 or less.

[0123] FIGS. 11 and FIGS. 12 are graphs illustrating an example of determining a reliability range according to an embodiment.

[0124] Referring to Fig. 12, the confidence range may be a range of locations where it is determined that the means of transportation may exist with a certain probability or higher, when the coordinate estimation model of the means of transportation follows a Gaussian distribution. The Gaussian distribution is symmetrical with respect to the mean, and the confidence range may be a range of locations where the cumulative probability with respect to the mean is determined to be greater than a certain probability.

[0125] In this case, when the variance of the Gaussian distribution is set high in (a), the shape of the graph of the Gaussian distribution may be widely spread out, and when the variance of the Gaussian distribution is set low in (b), the shape of the graph of the Gaussian distribution may be high around the mean. Therefore, when the variance of the Gaussian distribution is set high, the confidence range (REL_X) may be wide, and when the variance of the Gaussian distribution is set low, the confidence range (REL_Y) may be narrowed.

[0126] Referring to FIG. 11, in case (a) where driving lane estimation is not performed, the position estimation can be formed with the same or similar confidence ranges (REL_X, REL_Y) in the driving direction and the direction perpendicular to the driving direction. The driving direction can be understood as the x-axis direction, and the lane movement direction perpendicular to the driving direction can be understood as the y-axis direction.

[0127] The electronic device (10) according to the embodiment of the present disclosure can estimate the driving lane, determine whether the driving lane has changed, and also estimate the y-axis direction, which is the direction of lane movement. In case (b) where the driving lane estimation is performed, the reliability range of the driving direction (REL_X) may be the same or similar as in case (a), but the reliability range of the lane movement direction (REL_Y) may be narrower than in case (a) because the driving lane is estimated.

[0128] According to one embodiment, the electronic device (10) can estimate a reliability range for each of the driving direction and the lane movement direction, and can determine whether a change in the driving lane is performed. If a change in the driving lane is performed, the reliability range of the lane movement direction can be estimated to be the same as the reliability range when the driving lane estimation is not performed, and if a change in the driving lane is not performed, the reliability range of the lane movement direction can be set narrower than the reliability range when the driving lane estimation is not performed. If the electronic device (10) determines that the driving lane of the means of transport has not changed, it is highly likely that the position in the lane movement direction will remain the same, so the reliability range for the position estimation result can be set narrowly. Alternatively, even if the driving lane has changed, the electronic device (10) may form a narrow reliability range for the lane movement direction by estimating the position of the means of transport with high reliability according to the position estimation method of the present disclosure, such as by setting a narrow reliability range based on the changed driving lane.

[0129] Meanwhile, at least one step included in at least some of the embodiments described above may be implemented as a software module stored on a non-transitory computer-readable media. In addition, in this case, at least one software module may be provided by an operating system (OS) or by a specific application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and the remaining portion may be provided by a specific application.

[0130] FIG. 13 is a block diagram illustrating in detail the image processing unit of FIG. 6 according to one embodiment of the present disclosure.

[0131] Referring to FIG. 13, an image processing unit (120) according to some embodiments may include a data learning unit (1210) and a data inference unit (1220).

[0132] The data learning unit (1210) can sample learning image information and input the sampled data into a learning network model to update and train the weights of the learning network model for estimating lane identification information and / or lane classification information. In the following, the lane identification information may include the three-dimensional coordinates of the lane. Additionally, the estimation of lane identification information may include estimating the attitude of the image sensor that acquired the image and estimating the three-dimensional coordinates of the lane using the estimated attitude.

[0133] The data inference unit (1220) can extract lane information from image information sensed around the means of transportation and classify the lanes into multiple groups. The data inference unit (1220) can input image data into a learning network model to output lane classification information and lane location information.

[0134] At least one of the data learning unit (1210) or the data inference unit (1220) may be manufactured in the form of at least one hardware chip and mounted on the electronic device (10). For example, at least one of the data learning unit (1210) or the data inference unit (1220) may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as part of an existing general-purpose processor (e.g., CPU or application processor) or a graphics-dedicated processor (e.g., GPU) and mounted on the various devices described above.

[0135] In this case, the data learning unit (1210) and the data inference unit (1220) may be mounted on the electronic device (10) or may be mounted on separate devices. For example, one of the data learning unit (1210) and the data inference unit (1220) may be included in the electronic device (10), and the other may be included in a server. Additionally, the data learning unit (1210) and the data inference unit (1220) may provide model information built by the data learning unit (1210) to the data inference unit (1220) via wired or wireless means, and data input to the data inference unit (1220) may be provided to the data learning unit (1210) as additional training data.

[0136] Meanwhile, at least one of the data learning unit (1210) or the data inference unit (1220) may be implemented as a software module. When at least one of the data learning unit (1210) or the data inference unit (1220) is implemented as a software module (or a program module including instructions), the software module may be stored on a non-transitory computer-readable media. In addition, in this case, at least one software module may be provided by an operating system (OS) or by a specific application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and the remaining portion may be provided by a specific application.

[0137] FIG. 14 is a block diagram of a data learning unit according to one embodiment of the present disclosure.

[0138] Referring to FIG. 14, a data learning unit (1210) according to one embodiment may include a data acquisition unit (1211), a preprocessing unit (1212), a learning data selection unit (1213), a model learning unit (1214), and a model evaluation unit (1215). However, this is only one embodiment, and the data learning unit (1210) may include some of the aforementioned components or may include additional components other than the aforementioned components.

[0139] The data acquisition unit (1211) can acquire data for training a learning network model. For example, the data acquisition unit (1211) can acquire image information and lane information included in the image information. According to another example, the data acquisition unit (1211) can acquire data from an external server, such as a social network server, a cloud server, or a provider server.

[0140] The preprocessing unit (1212) can preprocess image information and lane information used for learning. The preprocessing unit (1212) can process the acquired data into a pre-set format so that the model learning unit (1214), which will be described later, can use the acquired data for learning to generate numerical data. According to one embodiment, the preprocessing unit (1212) can process the lane coordinate values ​​and / or lane shape information for each lane included in the image information into an annotated form.

[0141] The training data selection unit (1213) can sample data required for training from among the preprocessed data. The sampled data can be provided to the model training unit (1214). The training data selection unit (1213) can sample data required for training from among the preprocessed data according to pre-set selection criteria. Additionally, the training data selection unit (1213) may select data according to pre-set selection criteria through training by the model training unit (1214), which will be described later.

[0142] The model learning unit (1214) can learn the learning network model to identify lanes or classify lane shapes within image information based on the learning data.

[0143] A learning network model can be constructed considering the application field of the learning network model, the purpose of learning, or the computing performance of the device. For example, the learning network model may be a model based on a neural network, but is not limited thereto.

[0144] According to various embodiments, if there are multiple pre-built learning network models, the model learning unit (1214) may determine the learning network model to be learned as the one with a high correlation between the input learning data and the basic learning data. In this case, the basic learning data may be pre-classified by data type, and the learning network model may be pre-built by data type. For example, the basic learning data may be pre-classified based on various criteria such as the region where the learning data was generated, the time the learning data was generated, the size of the learning data, the genre of the learning data, the creator of the learning data, and the type of object within the learning data.

[0145] Additionally, the model learning unit (1214) can train the learning network model using at least one of various learning algorithms, such as, for example, error back-propagation or gradient descent.

[0146] Additionally, the model learning unit (1214) can train the learning network model through supervised learning, for example, using learning data for learning judgment criteria as input values. Additionally, the model learning unit (1214) can train the learning network model through unsupervised learning, for example, by learning on its own using data necessary to generate lane classification information and lane location information from images without a separate map, thereby discovering criteria for classifying lanes from images and criteria for generating lane location information.

[0147] Additionally, the model learning unit (1214) can train the learning network model through reinforcement learning, for example, using feedback on whether the lane classification information and lane location information generated according to the learning are correct.

[0148] Additionally, when the learning network model is trained, the model learning unit (1214) can store the trained learning network model. In this case, the model learning unit (1214) can store the trained learning network model in the storage of the electronic device (10) including the data learning unit (1210). Additionally, the model learning unit (1214) can store the trained learning network model in the storage of the electronic device (10) including the data inference unit (1220) to be described later. Additionally, the model learning unit (1214) can store the trained learning network model in the storage of a server connected to the electronic device (10) via a wired or wireless network.

[0149] In this case, the storage in which the learned learning network model is stored may also store commands or data related to at least one other component of the electronic device (10), for example. Additionally, the storage may store software and / or programs. The programs may include, for example, a kernel, middleware, an application programming interface (API) and / or an application program (or "application").

[0150] The model evaluation unit (1215) inputs evaluation data into the learning network model, and if the lane identification result and / or lane classification result output from the evaluation data does not satisfy a predetermined standard, it may cause the model learning unit (1214) to learn again. In this case, the evaluation data may be pre-configured data for evaluating the learning network model. For example, the evaluation data may include the matching ratio between lane information identified based on the learning network model and actual lane information.

[0151] For example, the model evaluation unit (1215) may evaluate that the number or ratio of evaluation data for which the identification result is inaccurate among the identification results of the learned learning network model for the evaluation data exceeds a preset threshold, and that the model does not satisfy a predetermined criterion. For example, if the predetermined criterion is defined as a ratio of 2%, and the learned learning network model outputs an incorrect identification result for more than 20 evaluation data out of a total of 1000 evaluation data, the model evaluation unit (1215) may evaluate that the learned learning network model is not suitable.

[0152] Meanwhile, if there are multiple learned network models, the model evaluation unit (1215) evaluates whether each learned network model satisfies a predetermined criterion and can determine the model that satisfies the predetermined criterion as the final learning network model. In this case, if there are multiple models that satisfy the predetermined criterion, the model evaluation unit (1215) can determine any one of the pre-set models or a predetermined number of models in order of highest evaluation score as the final learning network model.

[0153] Meanwhile, at least one of the data acquisition unit (1211), preprocessing unit (1212), training data selection unit (1213), model training unit (1214), or model evaluation unit (1215) within the data learning unit (1210) may be manufactured in the form of at least one hardware chip and mounted on the electronic device (10). For example, at least one of the data acquisition unit (1211), preprocessing unit (1212), training data selection unit (1213), model training unit (1214), or model evaluation unit (1215) may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as part of an existing general-purpose processor (e.g., CPU or application processor) or a graphics-dedicated processor (e.g., GPU) and mounted on the aforementioned various electronic devices (10).

[0154] Additionally, the data acquisition unit (1211), preprocessing unit (1212), training data selection unit (1213), model training unit (1214), and model evaluation unit (1215) may be mounted on a single electronic device (10), or they may be mounted on separate electronic devices (10). For example, some of the data acquisition unit (1211), preprocessing unit (1212), training data selection unit (1213), model training unit (1214), and model evaluation unit (1215) may be included in the electronic device (10), and the remaining parts may be included in the server.

[0155] Additionally, at least one of the data acquisition unit (1211), preprocessing unit (1212), training data selection unit (1213), model training unit (1214), or model evaluation unit (1215) may be implemented as a software module. When at least one of the data acquisition unit (1211), preprocessing unit (1212), training data selection unit (1213), model training unit (1214), or model evaluation unit (1215) is implemented as a software module (or a program module including instructions), the software module may be stored on a non-transitory computer-readable media. Additionally, in this case, at least one software module may be provided by an operating system (OS) or provided by a specific application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and the remaining portion may be provided by a specific application.

[0156] FIG. 15 is a block diagram of a data inference unit according to one embodiment of the present disclosure.

[0157] Referring to FIG. 15, a data inference unit (1220) according to one embodiment may include a data acquisition unit (1221), a preprocessing unit (1222), an inference data selection unit (1223), an inference result providing unit (1224), and a model update unit (1225). However, this is only one embodiment, and the data inference unit (1220) may include some of the aforementioned components or may include additional components other than the aforementioned components.

[0158] The data acquisition unit (1221) can acquire image information by sensing the surroundings of the means of transport, and the preprocessing unit (1222) can preprocess the acquired information so that the image information can be used. The preprocessing unit (1222) can process the acquired data into a pre-set format so that the inference result providing unit (1224), which will be described later, can use the acquired data.

[0159] The inference data selection unit (1223) can select data necessary for lane identification and lane type classification from among the preprocessed data. The selected data can be provided to the inference result providing unit (1224). The inference data selection unit (1223) can select some or all of the preprocessed data according to pre-set selection criteria for quantifying spatial characteristics.

[0160] The inference result providing unit (1224) can determine the situation by applying the selected data to a learning network model and can generate lane classification information and lane location information. The inference result providing unit (1224) can provide result data according to the purpose of data generation. The inference result providing unit (1224) can apply the selected data to a learning network model by using the data selected by the inference data selection unit (1223) as an input value. Additionally, the result data can be determined by the learning network model.

[0161] The model update unit (1225) can control the learning network model to be updated based on an evaluation of the result data provided by the inference result providing unit (1224). For example, the model update unit (1225) can control the model learning unit (1214) to update the learning network model by providing the result data provided by the inference result providing unit (1224) to the model learning unit (1214).

[0162] Meanwhile, at least one of the data acquisition unit (1221), preprocessing unit (1222), inference data selection unit (1223), inference result providing unit (1224), or model update unit (1225) within the data inference unit (1220) may be manufactured in the form of at least one hardware chip and mounted on the electronic device (10). For example, at least one of the data acquisition unit (1221), preprocessing unit (1222), inference data selection unit (1223), inference result providing unit (1224), or model update unit (1225) may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as part of an existing general-purpose processor (e.g., CPU or application processor) or a graphics-dedicated processor (e.g., GPU) and mounted on the aforementioned various electronic devices (10).

[0163] Additionally, the data acquisition unit (1221), preprocessing unit (1222), inference data selection unit (1223), inference result providing unit (1224), and model update unit (1225) may be mounted on a single electronic device (10), or they may be mounted on separate electronic devices (10). For example, some of the data acquisition unit (1221), preprocessing unit (1222), inference data selection unit (1223), inference result providing unit (1224), and model update unit (1225) may be included in the electronic device (10), and the remaining parts may be included in the server.

[0164] Additionally, at least one of the data acquisition unit (1221), preprocessing unit (1222), inference data selection unit (1223), inference result providing unit (1224), or model update unit (1225) may be implemented as a software module. When at least one of the data acquisition unit (1221), preprocessing unit (1222), inference data selection unit (1223), inference result providing unit (1224), or model update unit (1225) is implemented as a software module (or a program module including instructions), the software module may be stored on a non-transitory computer-readable media. Additionally, in this case, at least one software module may be provided by an operating system (OS) or provided by a specific application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and the remaining portion may be provided by a specific application.

[0166] As described above, exemplary embodiments have been disclosed in the drawings and specification. Although specific terms have been used to describe the embodiments in this specification, they are used only for the purpose of explaining the technical concept of this disclosure and are not intended to limit the meaning or the scope of this disclosure as defined in the claims. Therefore, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of this disclosure should be determined by the technical concept of the appended claims.

Claims

Claim 1 A method for estimating the location of a means of transportation comprises: a step of obtaining first-1 location information of the means of transportation at a first time point; a step of generating lane classification information, which is information classifying lanes around the means of transportation and identifying lanes included in an image based on an image taken of the surroundings of the means of transportation; a step of identifying map information of a road where the means of transportation is located from map data based on the first-1 location information; a step of determining the lane where the means of transportation is located within the road where the means of transportation is located based on the lane classification information and the map information; a step of generating first-2 location information by correcting the first-1 location information based on the lane where the means of transportation is located and the map information; a step of generating estimated location information of the means of transportation based on the first-2 location information and the determined lane; a step of obtaining second-1 location information of the means of transportation at a second time point after the first time point; and a step of generating GPS corrected location information by correcting the second-1 location information using GPS data. A method comprising the step of correcting the above GPS correction location information using the above first-second location information to generate the above estimated location information including the estimated location and estimated lane of the above means of transportation. Claim 2 ◈Claim 2 was abandoned upon payment of the registration fee.◈ Method according to Claim 1, wherein the step of determining the lane in which the means of transportation is located within the road in which the means of transportation is located, based on the lane classification information and the map information, includes the step of matching the lanes in the image with the lanes included in the map information based on the lane classification information and the map information, and determining the lane in which the means of transportation is located using the matching result; and the step of generating the first-2 location information by correcting the first-1 location information based on the lane in which the means of transportation is located and the map information includes the step of correcting the first-1 location information by performing map matching using the map information on the first-1 location information. Claim 3 ◈Claim 3 was abandoned upon payment of the registration fee.◈ The method of Claim 1 further comprises the step of generating lane-in-lane location information indicating the location of the means of transportation within the lane where the means of transportation is located, based on an image taken of the surroundings of the means of transportation, and the step of determining the lane where the means of transportation is located within the road where the means of transportation is located, based on the lane classification information and the map information, comprises the step of determining the lane where the means of transportation is located based on the lane-in-lane location information. Claim 4 ◈Claim 4 was abandoned upon payment of the registration fee.◈ The method according to Claim 3, wherein the step of determining the lane in which the means of transportation is located based on the lane location information comprises: the step of acquiring a plurality of lane location information corresponding to each of a plurality of time points; the step of determining whether the means of transportation changes lanes based on the change over time of the plurality of lane location information; and the step of determining the lane in which the means of transportation is located based on whether the lanes change lanes. Claim 5 ◈Claim 5 was abandoned upon payment of the registration fee.◈ Method according to Claim 3, wherein the step of generating location information within the lane based on the image comprises: a step of determining three-dimensional coordinates of lanes surrounding the means of transportation based on the means of transportation; and a step of determining the position of the means of transportation within the lane where the means of transportation is located based on the determined three-dimensional coordinates of the surrounding lanes. Claim 6 ◈Claim 6 was abandoned upon payment of the registration fee.◈ Method according to Claim 5, wherein the step of determining the three-dimensional coordinates of lanes surrounding the means of transportation based on the means of transportation comprises: determining the vanishing point of the lanes surrounding the means of transportation within the image; determining the attitude of the image sensor that captured the image based on the vanishing point; and determining the three-dimensional coordinates of the surrounding lanes using the attitude of the image sensor. Claim 7 delete Claim 8 ◈Claim 8 was abandoned upon payment of the registration fee.◈ Method according to Claim 1, wherein the step of generating GPS corrected location information by correcting the 2-1 location information using GPS data includes the step of correcting the 2-1 location information using GPS data and the reliability range of the GPS data, and the step of generating estimated location information including the estimated location and estimated lane of the means of transportation by correcting the GPS corrected location information using the 1-2 location information includes the step of adjusting the reliability range of the 1-2 location information based on the lane where the means of transportation is located; and the step of correcting the GPS corrected location information using the 1-2 location information and the reliability range of the 1-2 location information. Claim 9 ◈Claim 9 was abandoned upon payment of the registration fee.◈ The method of claim 8 further comprises the step of generating lane-in-lane location information indicating the location of the means of transportation within the lane where the means of transportation is located, based on an image taken of the surroundings of the means of transportation, and the step of adjusting the reliability range of the first-2 location information based on the lane where the means of transportation is located comprises the step of adjusting the reliability range of the first-2 location information based on the lane where the means of transportation is located and the lane-in-lane location information. Claim 10 ◈Claim 10 was abandoned upon payment of the registration fee.◈ The method of claim 1, wherein the 1-1 position information of the first time point includes information estimated using an IMU (inertial measurement unit) of the position to which the means of movement has moved from the initial position of an initial time point prior to the first time point. Claim 11 A first-1 position acquisition unit for acquiring first-1 information at a first point in time of a means of transportation; an image processing unit for identifying lanes included in an image based on an image taken of the surroundings of the means of transportation and generating lane classification information, which is information classifying the lanes around the means of transportation; a first-2 position acquisition unit for identifying map information of a road where the means of transportation is located based on the first-1 position information from map data, determining the lane where the means of transportation is located within the road where the means of transportation is located based on the lane classification information and the map information, and correcting the first-1 position information based on the lane where the means of transportation is located and the map information to generate first-2 position information. A vehicle position estimation device comprising a position correction unit that generates estimated position information of the vehicle based on the first-2 position information and the determined lane, wherein the position correction unit acquires second-1 position information of the vehicle at a second time point after the first time point, corrects the second-1 position information using GPS data to generate GPS corrected position information, and corrects the GPS corrected position information using the first-2 position information to generate the estimated position information including the estimated position and the estimated lane of the vehicle. Claim 12 ◈Claim 12 was abandoned upon payment of the registration fee.◈ In claim 11, the first-2 position acquisition unit comprises: a lane determination unit that matches lanes within the image with lanes included in the map information based on the lane classification information and the map information, and determines the lane where the means of transportation is located using the matching result; and a first-2 position determination unit that performs map matching on the first-1 position information using the map information to correct the first-1 position information, characterized in that it comprises a means of transportation position estimation device. Claim 13 ◈Claim 13 was abandoned upon payment of the registration fee.◈ A vehicle position estimation device according to Claim 11, wherein the image processing unit comprises a lane-in-lane position estimation unit that generates lane-in-lane position information indicating the position of the vehicle within the lane where the vehicle is located, based on an image taken of the surroundings of the vehicle, and the first-second position acquisition unit comprises a lane determination unit that determines the lane where the vehicle is located based on the lane-in-lane position information. Claim 14 ◈Claim 14 was abandoned upon payment of the registration fee.◈ A vehicle position estimation device according to Claim 13, wherein the lane determination unit acquires multiple lane position information corresponding to each of multiple time points, determines whether the vehicle changes lanes based on changes over time of the multiple lane position information, and determines the lane in which the vehicle is located based on whether the lane changes lanes. Claim 15 ◈Claim 15 was abandoned upon payment of the registration fee.◈ A vehicle position estimation device according to Claim 13, wherein the lane position estimation unit determines the three-dimensional coordinates of lanes surrounding the vehicle based on the vehicle, and determines the position of the vehicle within the lane where the vehicle is located based on the determined three-dimensional coordinates of the surrounding lanes. Claim 16 ◈Claim 16 was abandoned upon payment of the registration fee.◈ A vehicle position estimation device according to Claim 15, wherein the lane position estimation unit determines the vanishing point of the lanes surrounding the vehicle within the image, determines the attitude of the image sensor that captured the image based on the vanishing point, and determines the three-dimensional coordinates of the surrounding lanes using the attitude of the image sensor. Claim 17 delete Claim 18 A vehicle position estimation device according to claim 11, wherein the position correction unit comprises: a reliability adjustment unit that adjusts the reliability range of the first-2 position information based on the lane where the vehicle is located; and an estimated position generation unit that corrects the second-1 position information using GPS data and the reliability range of the GPS data, and corrects the GPS corrected position information using the first-2 position information and the reliability range of the first-2 position information. Claim 19 A vehicle position estimation device according to claim 18, wherein the lane-in-lane position estimation unit generates lane-in-lane position information indicating the position of the vehicle within the lane where the vehicle is located, based on an image taken of the surroundings of the vehicle, and the reliability adjustment unit adjusts the reliability range of the first and second position information based on the lane where the vehicle is located and the lane-in-lane position information. Claim 20 ◈Claim 20 was abandoned upon payment of the registration fee.◈ A device for estimating the position of a means of transportation, characterized in that the 1-1 position information at the first point in time above includes information estimated using an IMU (inertial measurement unit) of the position to which the means of transportation has moved from the initial position at an initial point in time prior to the first point in time above.

Images