Method and apparatus for determining the location of an object on a lane map - Patents.com

Abstract

The present disclosure relates to a method and apparatus for determining a position of an object on a lane map. The method according to an embodiment of the present disclosure can acquire image data captured by a camera mounted on a vehicle. The method can also generate feature information of a plurality of features included in the image data by performing a predetermined process on the image data. Furthermore, the method can determine a plurality of features corresponding to the same object in the plurality of image data based on a correlation degree between the plurality of features included in the image data. Furthermore, the method can determine the position of the object on the lane map based on the position information on the lane map of the vehicle and the feature position information on the plurality of image data.

Description

[Technical field]

[0001] The present invention relates to a method and apparatus for determining the location of an object on a lane map. [Background technology]

[0002] A vehicle traveling on a road can travel on any one of multiple lanes included in the road. A vehicle can frequently change lanes while traveling, and situations in which the number of lanes on a road changes frequently occur.

[0003] The fusion of information and communication technology and the automotive industry is rapidly making vehicles smarter. As vehicles become smarter, they are evolving from simple mechanical devices to smart cars, and autonomous driving in particular is attracting attention as a core technology of smart cars. Autonomous driving is a technology that allows a vehicle to reach its destination by itself without the driver having to operate the steering wheel, accelerator pedal, brakes, etc.

[0004] Various additional functions related to autonomous driving are being continuously developed, and there is a demand for research into methods to provide passengers with a safe autonomous driving experience by using various data to recognize and judge the driving environment and control the vehicle.

[0005] In recent years, there has been a need for research into how to display various objects (e.g., traffic lights, road signs, humps, etc.) located on or around the roads on which vehicles are traveling on maps more accurately with less computational effort.

[0006] The above-mentioned background art is technical information possessed by the inventor for the purpose of deriving the present invention or acquired in the process of deriving the present invention, and is not necessarily publicly known art that was disclosed to the general public prior to the filing of the application of the present invention. Summary of the Invention [Problem to be solved by the invention]

[0007] The present invention provides a method and apparatus for determining the position of an object on a lane map. The problem to be solved by the present invention is not limited to the problem described above, and other problems and advantages of the present invention not mentioned can be understood from the following description and will be more clearly understood in the embodiments of the present invention. Furthermore, it will be understood that the problem and advantages to be solved by the present invention can be realized by the means and combinations thereof shown in the claims. [Means for solving the problem]

[0008] As a technical means for achieving the above-mentioned technical problem, a first aspect of the present disclosure can provide a method for determining a position of an object on a lane map, the method including the steps of acquiring image data captured by a camera mounted on a vehicle, generating feature information of a plurality of features contained in the image data by performing a predetermined processing on the image data, determining a plurality of features corresponding to the same object in the plurality of image data based on a correlation degree between the plurality of features contained in the image data, and determining a position of the object on the lane map based on position information on the lane map of the vehicle and feature position information on the plurality of image data.

[0009] A second aspect of the present disclosure can provide an apparatus for determining a position of an object on a lane map, the apparatus including a memory having at least one program stored therein and a processor that performs calculations by executing the at least one program, wherein the processor acquires image data captured by a camera mounted on a vehicle, and performs a predetermined process on the image data to generate feature information of a plurality of features contained in the image data, determines a plurality of features corresponding to the same object in the plurality of image data based on a correlation degree between the plurality of features contained in the image data, and determines the position of the object on the lane map based on position information on the lane map of the vehicle and feature position information in the plurality of image data.

[0010] A third aspect of the present disclosure can provide a computer-readable recording medium having recorded thereon a program for causing a computer to execute the method according to the first aspect.

[0011] Additionally, other methods and systems for implementing the present invention, and a computer readable recording medium having a computer program stored thereon for carrying out the methods may also be provided.

[0012] Other aspects, features and advantages in addition to those described above will become apparent from the following drawings, claims and detailed description of the invention. Effect of the Invention

[0013] According to the problem-solving means of the present disclosure described above, data can be reduced by utilizing feature information corresponding to metadata for an object rather than using the entire image data, and determining what the same object is within multiple image data. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram for explaining an automatic driving method according to an embodiment. [Diagram 2] FIG. 2 is a diagram for explaining an automatic driving method according to an embodiment. [Diagram 3] FIG. 3 is a diagram for explaining an automatic driving method according to an embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a plurality of objects included in image data according to an embodiment. [Figure 5a] FIG. 5a is a diagram for explaining feature information included in image data according to one embodiment. [Figure 5b] FIG. 5b is a diagram for explaining feature information included in image data according to an embodiment. [Figure 6a]FIG. 6a is an exemplary diagram illustrating feature aggregation in image data according to one embodiment. [Figure 6b] FIG. 6b is an exemplary diagram illustrating feature aggregation in image data according to one embodiment. [Figure 7] FIG. 7 is an exemplary diagram for explaining a method for determining a position of an object on a lane map according to an embodiment. [Figure 8] FIG. 8 is an exemplary diagram for explaining a method for determining a position of an object based on vehicle position information and feature positions on an image according to an embodiment. [Figure 9] FIG. 9 is an exemplary diagram for explaining a method for determining a position of an object on a lane map using multiple driving trajectories according to an embodiment. [Figure 10] FIG. 10 is an exemplary diagram illustrating a lane fitting and object fitting method according to an embodiment. [Figure 11] FIG. 11 is a flow chart of a method for determining a location of an object on a lane map according to one embodiment. [Figure 12] FIG. 12 is a block diagram of an object position determining apparatus according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The present disclosure relates to a method and an apparatus for determining a position of an object on a lane map. The method according to an embodiment of the present disclosure can acquire image data captured by a camera mounted on a vehicle. The method can also generate feature information of a plurality of features included in the image data by performing a predetermined process on the image data. Furthermore, the method can determine a plurality of features corresponding to the same object in the plurality of image data based on a correlation degree between the plurality of features included in the image data. Furthermore, the method can determine the position of the object on the lane map based on the position information on the lane map of the vehicle and the feature position information on the plurality of image data.

[0016] The advantages and features of the present invention, as well as the methods for achieving them, will become clear with reference to the embodiments described in detail with the accompanying drawings. However, the present invention is not limited to the embodiments presented below, and may be implemented in various different forms, and should be understood to include all modifications, equivalents, and alternatives within the spirit and technical scope of the present invention. The embodiments presented below are provided to fully disclose the present invention and to fully inform those skilled in the art to which the present invention pertains of the scope of the invention. In describing the present invention, if it is determined that a detailed description of related known technology may hinder the gist of the present invention, the detailed description will be omitted.

[0017] The terms used in this application are merely used to describe certain embodiments and are not intended to limit the present invention. A singular expression includes a plural expression unless the context clearly indicates otherwise. In this application, terms such as "include" or "have" specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described herein, and should be understood not to preclude the presence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0018] Some embodiments of the present disclosure may be expressed in terms of functional block configurations and various processing steps. Some or all of such functional blocks may be embodied in various numbers of hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be embodied by one or more microprocessors or by circuit configurations for a given function. Further, for example, the functional blocks of the present disclosure may be embodied in various programming or scripting languages. The functional blocks may be embodied by algorithms executed by one or more processors. Furthermore, the present disclosure may employ conventional techniques for electronic configuration, signal processing, and / or data processing, etc. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations.

[0019] Furthermore, the connecting lines or connecting members between components shown in the figures are merely exemplary of functional and / or physical or circuit connections, and in an actual device the connections between components may be represented by various alternative or additional functional, physical or circuit connections.

[0020] In the following, "vehicle" may mean any kind of transport means having a locomotive and used to move people or goods, such as a car, a bus, a motorcycle, a scooter or a truck.

[0021] The present disclosure will now be described in detail with reference to the accompanying drawings.

[0022] 1 to 3 are diagrams for explaining an automatic driving method according to an embodiment.

[0023] Referring to FIG. 1, an autonomous driving device according to an embodiment of the present invention may be installed in a vehicle to realize an autonomous vehicle 10. The autonomous driving device installed in the autonomous vehicle 10 may include various sensors for collecting surrounding situation information. As an example, the autonomous driving device may detect the movement of a leading vehicle 20 traveling ahead through an image sensor and / or an event sensor installed in the front of the autonomous vehicle 10. The autonomous driving device may further include sensors for detecting other vehicles 30 traveling in adjacent lanes as well as the front of the autonomous vehicle 10, and pedestrians around the autonomous vehicle 10.

[0024] At least one of the sensors for collecting situation information around the autonomous vehicle may have a predetermined angle of view FoV, as shown in Fig. 1. As an example, when a sensor attached to the front of the autonomous vehicle 10 has an angle of view FoV as shown in Fig. 1, information detected from the center of the sensor may have a relatively high importance. This is because the information detected from the center of the sensor may include most of the information corresponding to the movement of the leading vehicle 20.

[0025] The autonomous driving device processes information collected by sensors of the autonomous vehicle 10 in real time to control the movement of the autonomous vehicle 10, while storing at least a portion of the information collected by the sensors in a memory device.

[0026] 2, the autonomous driving device 40 may include a sensor unit 41, a processor 46, a memory system 47, and a vehicle body control module 48. The sensor unit 41 may include a plurality of sensors 42 to 45, which may include an image sensor, an event sensor, an illuminance sensor, a GPS device, and an acceleration sensor.

[0027] The data collected by the sensors 42-45 may be transmitted to the processor 46. The processor 46 may store the data collected by the sensors 42-45 in the memory system 47 and control the body control module 48 to determine the movement of the vehicle based on the data collected by the sensors 42-45. The memory system 47 may include two or more memory devices and a system controller for controlling the memory devices. Each of the memory devices may be provided by a single semiconductor chip.

[0028] In addition to the system controller of memory system 47, each memory device included in memory system 47 may include a memory controller, and the memory controller may include an artificial intelligence (AI) calculation circuit such as a neural network. The memory controller may generate calculation data by applying a predetermined weight to data received from sensors 42 to 45 or processor 46, and store the calculation data in a memory chip.

[0029] Fig. 3 is a diagram showing an example of video data acquired by a sensor of an autonomous vehicle equipped with an autonomous driving device. Referring to Fig. 3, video data 50 may be data acquired by a sensor attached to the front of the autonomous vehicle. Therefore, the video data 50 may include a front part 51 of the autonomous vehicle, a preceding vehicle 52 in the same lane as the autonomous vehicle, a traveling vehicle 53 around the autonomous vehicle, and a background 54.

[0030] 3, data of an area in which the front part 51 of the autonomous vehicle and the background 54 appear may be data that is unlikely to affect the traveling of the autonomous vehicle. In other words, the front part 51 of the autonomous vehicle and the background 54 can be considered to be data of relatively low importance.

[0031] Meanwhile, the distance to the preceding vehicle 52 and the lane-changing behavior of the moving vehicle 53 may be very important factors in the safe driving of the autonomous vehicle. Therefore, data of an area including the preceding vehicle 52 and the moving vehicle 53 in the video data 50 may have a relatively high importance in the driving of the autonomous vehicle.

[0032] The memory device of the autonomous driving device may assign different weights to different areas of the image data 50 received from the sensor and store the data. For example, a high weight may be assigned to data of an area including a leading vehicle 52 and a moving vehicle 53, and a low weight may be assigned to data of an area including a front part 51 of the autonomous driving vehicle and a background 54.

[0033] FIG. 4 is a diagram illustrating an example of a plurality of objects included in image data according to an embodiment.

[0034] Image data acquired from a vehicle-mounted camera at a particular time may include multiple objects.

[0035] The information on the object includes object type information and object attribute information. Here, the object type information is index information indicating the type of the object and is composed of a broad range of groups and a detailed range of classes. Also, the object attribute information indicates attribute information on the current state of the object and includes movement information, rotation information, traffic information, color information, and visibility information.

[0036] In one embodiment, the groups and classes included in the object type information may be set as shown in Table 1 below, but are not limited thereto.

[0037] [Table 1]

[0038] Furthermore, the information included in the object attribute information may include Action, Rotate, Traffic information, color, and Visibility information.

[0039] Action information represents movement information of an object, and may be defined as stopping, parking, moving, etc. In the case of a vehicle, stopping, parking, and moving may be determined as object attribute information, and in the case of an object that cannot move, such as a traffic light, the object attribute information may be determined as a default value of stopping.

[0040] Rotate information indicates rotation information of an object, and may be defined as front, back, horizontal, vertical, side, etc. In the case of a vehicle, object attribute information may be defined as front, back, and side, and object attribute information may be defined as horizontal or vertical for a horizontal or vertical traffic light, respectively.

[0041] Traffic info means traffic information of an object, and can be defined as traffic sign instructions, cautions, regulations, auxiliary signs, etc. Color means color information of an object, and can represent the color of an object, traffic lights, and traffic signs.

[0042] For example, the object attribute information for a particular vehicle may be determined to be stopped, front, red, and visibility level 3.

[0043] Referring to FIG. 4, objects included in an image 400 captured from a camera may be, but are not limited to, traffic lights, signs, the current ego-lane, intersections, road markings or humps, etc.

[0044] 5a and 5b are diagrams for explaining feature information included in image data according to one embodiment.

[0045] As the vehicle travels along a road, multiple image data may be acquired from a camera mounted on the vehicle.

[0046] Image data 510 acquired from a vehicle-mounted camera may include an object 511. Referring to Figure 5a, the object 511 may be a traffic light.

[0047] The image data may have a predetermined size M×N (M and N are natural numbers). The multiple image data may contain the same object 511, but as the vehicle travels along the road, the relative position of the vehicle and the object 511 keeps changing, which causes the same object 511 to be in a different position in each image data.

[0048] If the entire image data is used to determine what the same object is in each image data, the amount of data transmission and calculations becomes very large, which makes processing through edge computing in a device mounted on a vehicle difficult and makes real-time analysis difficult.

[0049] 5b, features 521 included in image data 520 are shown. Features 521 are metadata of object 511, and feature information may include object type information (group, class, etc.), position information of object 511 on image data 520, size information of object 511, etc. Referring to FIG. 5b, feature information may include information that object 511 corresponds to a traffic light class, information that object 511 has a rectangular shape, information that the lower left vertex of object 511 is located at (m, n) on image data, and information that object 511 has a size of a×b.

[0050] In the present disclosure, it is possible to determine what the same object is in multiple pieces of image data by utilizing feature information that corresponds to the object's metadata, without using the entire image data.

[0051] 6a-6b are exemplary diagrams for explaining feature aggregation in image data according to an embodiment.

[0052] 6a, image data captured from a camera mounted on a vehicle at a particular time point is shown, where first image data 610 is image data captured at the first time point, and second image data 620 is image data captured at a second time point after a predetermined time has elapsed from the first time point.

[0053] On the other hand, a first image 601 and a second image 602 are provided to supplement the explanation, and show the actual state of a road captured by a camera.

[0054] The first image data 610 and the second image data 620 include the same object.

[0055] A device for determining the location of an object on a lane map (hereinafter, an "object location device") can determine multiple features indicative of the same object in multiple image data.

[0056] 6a, a first feature group 611 and a second feature group 612 may be located adjacent to each other in the first image data 610. That is, when the distance between the objects corresponding to the first feature group 611 and the second feature group 612 and the vehicle is large, the actual distance difference between the objects corresponding to the first feature group 611 and the second feature group 612 is not reflected, and the objects may be located adjacent to each other on the first image data 610.

[0057] Meanwhile, the first feature group 621 and the second feature group 622 may be positioned at positions that are somewhat separated from each other in the second image data 620. That is, when the distance between the vehicle and the objects corresponding to the first feature group 611 and the second feature group 612 becomes closer compared to the position of the vehicle when the first image data 610 was captured, the actual distance difference between the objects corresponding to the first feature group 611 and the second feature group 612 is further reflected, and the objects may be positioned at positions that are somewhat separated from each other on the second image data 620.

[0058] Meanwhile, in the first image data 610 and the second image data 620, the first feature groups 611 and 621 represent the same object, and the second feature groups 612 and 622 represent the same object, but the size may differ depending on the image data.

[0059] As in the examples of the first image data 610 and the second image data 620, image data captured while a vehicle is traveling may contain multiple features indicating the same object, but as the position between the vehicle and the object changes over time while the vehicle is traveling, the positions, sizes, positional relationships, and layout relationships of the multiple features will differ from one another in each image data.

[0060] The object location determination device can determine multiple features corresponding to the same object in the multiple image data by aggregation of features in the multiple image data.

[0061] The object position determination device can determine a plurality of features corresponding to the same object in a plurality of image data based on the degree of correlation between the plurality of features included in the image data. The degree of correlation can be determined based on the positional relationship and size relationship between the plurality of features, but factors for determining the degree of correlation are not limited thereto.

[0062] In one embodiment, the object location determination device can use a graph-matching method to determine multiple features indicative of the same object in multiple image data.

[0063] The object location determination device can register a graph consisting of nodes and links by designating a plurality of features including a key feature in the image data as nodes and setting the connection relationships between the plurality of features as links. Furthermore, the object location determination device can determine whether the key features in each image data indicate the same object through a comparison between the graphs registered from different image data.

[0064] Specifically, the object position determination device may select a first feature 611a of the first image data and a second feature 621a of the second image data 620, and search for other features around the first feature 611a and the second feature 621a. When a second neighbor feature 621b having an attribute similar to a first neighbor feature 611b around the first feature 611a exists around the second feature 621a, the object position determination device may register the first neighbor feature 611b and the second neighbor feature 621b as a graph with the first feature 611a and the second feature 621a as key features, respectively. Here, the attribute may include at least one of a class, an aspect ratio of the feature, a distance, and an angle.

[0065] The object positioning device can repeat the steps of searching and registering all features around the first feature 611a. After attempting to construct a graph around all features around the first feature 611a, the object positioning device can calculate the similarity between the first feature 611a and the second feature 621a based on the number of features and the similarity of the registered key features (e.g., size or size relationship with adjacent features, etc.). The object positioning device can determine that the second feature 621a of the second image data 620 that is most similar to the first feature 611a of the first image data 610 indicates the same object by executing the steps of searching, registering as a graph, repeating, and calculating the similarity for the remaining features except for the second feature 621a of the second image data 620.

[0066] In the present disclosure, feature information corresponding to metadata of an object can be utilized without using the entire image data, and thus, in the present disclosure, even if the positions, sizes, positional relationships, and layout relationships of multiple features differ from one another in each image data as the positions of the vehicle and the object change over time while the vehicle is traveling, it is possible to determine what the same object is in multiple image data.

[0067] Alternatively, the object positioning device may determine that features included in the graph have the same depth, which may mean that their lateral (vehicle travel direction) position on the road is the same or within a certain range (e.g., 1 m).

[0068] When the object location device performs post-processing on a given feature, it may perform the same post-processing on the remaining features in the graph that have the same depth.

[0069] 6a, the object localization device may determine that the features in the first feature groups 611, 621 have the same depth as each other, and that the features in the second feature groups 612, 622 have the same depth as each other. In this case, when the object localization device corrects the position of a given feature in the first feature groups 611, 621, it may also correct the positions of the remaining features in the first feature groups 611, 621.

[0070] Referring to Fig. 6b, image data obtained from a camera mounted on a vehicle at a particular time point is shown. During the vehicle's travel, the vehicle's longitudinal position on the road may be changed (e.g., lane change). First image data 630 is image data taken from a vehicle traveling in a first travel lane of the road, and second image data 640 is image data taken from a vehicle traveling in an overtaking lane of the road.

[0071] On the other hand, a first image 603 and a second image 604 are provided to supplement the explanation, and show the actual state of the road as captured by a camera.

[0072] The first image data 630 and the second image data 640 include the same object.

[0073] The object location determining device can determine multiple features indicative of the same object in the multiple image data.

[0074] 6b, the first feature group 631 and the second feature group 632 may be located adjacent to each other in the first image data 630. Also, the first feature group 631 may be located to the right of the second feature group 632. That is, in the first image data 630 captured while the vehicle is traveling in the first driving lane, the positional relationship between the first feature group 631 and the second feature group 632 may be determined as described above depending on the relative positional relationship between the vehicle and multiple objects.

[0075] Meanwhile, in the second image data 640, the first feature group 641 and the second feature group 642 may be located at positions separated from each other to a certain extent. Also, the first feature group 641 may be located to the left of the second feature group 642. That is, in the second image data 640 captured while the vehicle is traveling in the passing lane, the positional relationship between the first feature group 641 and the second feature group 642 may be determined as described above depending on the relative positional relationship between the vehicle and multiple objects.

[0076] Meanwhile, in the first image data 630 and the second image data 640, the first feature groups 631 and 641 indicate the same object, and the second feature groups 632 and 642 indicate the same object, but the size may differ depending on the image data.

[0077] As in the examples of the first image data 630 and the second image data 640, image data captured while a vehicle is traveling may contain multiple features indicating the same object, but since the position between the vehicle and the object changes depending on which lane the vehicle is traveling in, the positions, sizes, positional relationships, and layout relationships of the multiple features will differ from one another in each image data.

[0078] The object location device can determine multiple features corresponding to the same object in the multiple image data by aggregating features in the multiple image data.

[0079] The object position determining device can determine a plurality of features corresponding to the same object in a plurality of image data based on a degree of correlation between a plurality of features contained in the image data.

[0080] Using the method described above in FIG. 6b, the object localization device can determine multiple features corresponding to the same object in multiple image data.

[0081] In the present disclosure, instead of using the entire image data, feature information corresponding to metadata of the object can be utilized, so that the present disclosure can determine what the same object is in multiple image data even if the positions, sizes, positional relationships, and layout relationships of multiple features are different in each image data due to the vehicles driving in different lanes and the positions of the vehicles and the object changing.

[0082] FIG. 7 is an exemplary diagram for explaining a method for determining a position of an object on a lane map according to an embodiment.

[0083] The object position determination device can obtain the vehicle position information using a GPS installed in the vehicle. The vehicle position information may be position information on a lane map.

[0084] The object position determination device can also determine the position of the object on the lane map using position information in the image data of multiple features that are determined to correspond to the same object in the multiple image data.

[0085] Referring to FIG. 7, as described in FIG. 6a to FIG. 6b, in the first image data 630 and the second image data 640, the first feature groups 631, 641 may indicate the same object, and the second feature groups 632, 642 may indicate the same object.

[0086] In one embodiment, the object position determining device can determine the position of an object on the lane map using the position information on the lane map obtained via a position confirmation device connected to the vehicle, and the first image data 630 and the second image data 640. Meanwhile, the object position determining device can also use one or more image data to determine the position of an object on the lane map.

[0087] Referring to FIG. 7, the object position determination device can determine the position of a first object group 710 corresponding to a first feature group 631, 641 and the position of a second object group 720 corresponding to a second feature group 632, 642 and display them on a lane map 700.

[0088] FIG. 8 is an exemplary diagram for explaining a method for determining a position of an object based on vehicle position information and feature positions on an image according to an embodiment.

[0089] Referring to FIG. 8, a vehicle 800 is moving from a first location to a second location along a travel path.

[0090] The object position determination device can acquire position information of the vehicle 800 using a GPS mounted on the vehicle 800 or a position confirmation device connected to the vehicle. The object position determination device can acquire position information of the vehicle 800 on a lane map using the position information of the vehicle 800.

[0091] The object position determination device can acquire image data 810 a , 810 b of the surroundings (front, sides, rear, etc.) of the vehicle 800 from a camera 830 mounted on the vehicle 800 .

[0092] The first image data 810a is image data captured by the camera 830 when the vehicle 800 is located at a first position, and the second image data 810b is image data captured by the camera 830 when the vehicle 800 is located at a second position.

[0093] The first image data 810a may include a first feature 811, and the second image data 810b may include a second feature 812. The first feature 811 and the second feature 812 represent the same object, a traffic light 820.

[0094] The first feature 811 and the second feature 812 indicate the same object, but when the vehicle 800 moves from the first position to the second position, the relative distance between the vehicle 800 and the traffic light 820 changes, and as a result, the position and size of the features 811, 812 indicating the traffic light 820 change in each image data 810a, 810b captured by the camera 830 of the vehicle 800.

[0095] The object position determination device can determine the position of the object 820 on the lane map based on the position information of the vehicle 800 on the lane map and the position information of the features 811, 812 on the image data 810a, 810b.

[0096] In one embodiment, the object position determining device can determine the position of the object 820 on the lane map using a distance measurement method, but the method of determining the position of the object 820 on the lane map is not limited to this.

[0097] In one embodiment, the object position determining device can obtain an attitude value of the camera 830. The attitude value of the camera 830 can include factors related to the installation position, installation direction, and installation angle of the camera 830.

[0098] Referring to FIG. 8, the object position determination device can take into account the pose value of the camera 830 and determine the position of the intersection point of virtual rays starting from the camera 830 and passing through features 811, 812 on the image data 810a, 810b.

[0099] Specifically, a vehicle 800 may be traveling from a first position to a second position. A feature 811 indicating an object 820 may be displayed in first image data 810a captured by a camera 830 of the vehicle 800 located at the first position, and a feature 812 indicating the object 820 may be displayed in second image data 810b captured by a camera 830 of the vehicle 800 located at the second position. The object position determination device may determine a position 850 of a point where a virtual first light ray 841 that starts from the camera 830 of the vehicle 800 at the first position and passes through the feature 811 on the first image data 810a and a virtual second light ray 842 that starts from the camera 830 of the vehicle 800 at the second position and passes through the feature 812 on the second image data 810b intersect with each other.

[0100] The object locator can determine the position of the object 820 on the lane map based on the distance traveled by the vehicle 800 and the location of the point where the imaginary rays intersect.

[0101] Specifically, the object position determination device can determine the position of object 820 on the lane map using the position 850 of the point where the first light ray 841 and the second light ray 842 intersect and the distance traveled by vehicle 800 from the first position to the second position.

[0102] In one embodiment, the object positioning device can further obtain an attitude value of the vehicle 800. The attitude value of the vehicle 800 can include factors related to the slope of the road on which the vehicle 800 is traveling, the air pressure difference between the tires of the vehicle 800, and the steering angle of the vehicle 800.

[0103] The object position determination device can take into account the pose of the camera 830 and the pose of the vehicle 800 and determine the position of the point of intersection of virtual rays starting from the camera 830 and passing through the features 811, 812 on the image data 810a, 810b.

[0104] FIG. 9 is an exemplary diagram for explaining a method for determining a position of an object on a lane map using multiple driving trajectories according to an embodiment.

[0105] 9, a first trajectory 910 and a second trajectory 920 of a vehicle that has traveled along a predetermined road are shown. In FIG. 9, the positions of a 1-1 object 911 and a 1-2 object 921 are positions determined based on a plurality of image data captured while the vehicle is traveling along the first trajectory 910. In addition, the positions of a 2-1 object 912 and a 2-2 object 922 are positions determined based on a plurality of image data captured while the vehicle is traveling along the second trajectory 920.

[0106] The 1-1 object 911 and the 1-2 object 921 are referred to as a first group of objects, and the 2-1 object 912 and the 2-2 object 922 are referred to as a second group of objects.

[0107] The object position determination device can construct a first graph using a first group of objects, construct a second graph using a second group of objects, and determine which objects are the same in the first group of objects and the second group of objects through a similarity comparison between the first graph and the second graph.

[0108] Specifically, in order to calculate the degree of correlation between objects, the object positioning device targets a specific object generated by vehicle or time, and if there is an object of the same class around the specific object (e.g., at a similar distance and direction from the lane), it registers it in a graph, calculates the similarity based on the position, etc., and determines that the most similar object is the same object.

[0109] When a 1-2 object 921 of the same class exists around a 1-1 object 911, the object position determination device can form a first graph with the 1-1 object 911 and the 1-2 object 921, and can form a second graph with the 2-1 object 912 and the 2-2 object 922. The object position determination device can determine which objects are the same in each graph by comparing the similarity of each object in the graph as a key object.

[0110] In FIG. 9, the object positioning device can determine that 1-1 object 911 and 2-1 object 912 refer to the same object, and that 1-2 object 921 and 2-2 object 922 refer to the same object.

[0111] The object locator may aggregate the positions of objects determined to be the same object and display the final position of each object on the lane map 900 .

[0112] 9, the object position determination device can aggregate the positions of the 1-1 object 911 and the 1-2 object 921, and determine the position of the 1-3 object 931 as the final position. Also, the object position determination device can aggregate the positions of the 2-1 object 912 and the 2-2 object 922, and determine the position of the 2-3 object 932 as the final position.

[0113] FIG. 10 is an exemplary diagram illustrating a lane fitting and object fitting method according to an embodiment.

[0114] The object position determination device can obtain the travel trajectory of the vehicle using a position confirmation device connected to the vehicle. The position confirmation device may be a GPS, and in the following, it is assumed that a GPS is used as the position confirmation device. In addition, the object position determination device can map the obtained travel trajectory on a lane map 1000 and display the travel trajectory on the lane map 1000.

[0115] Referring to FIG. 10, the object position determination device can obtain a first driving trajectory 1011 using a GPS mounted on the vehicle and display it on a lane map 1000.

[0116] On the other hand, the GPS signal received by the GPS installed in the vehicle may have errors due to various reasons. For example, when the vehicle passes through an alley with many buildings, the reception of the GPS signal may become temporarily unstable. Or, the vehicle may receive an error-containing GPS signal due to radio waves reflected by the glass surface of a building.

[0117] The object position determining device can determine the current lane that the vehicle is traveling in. The information about the current lane can be information included in the feature information.

[0118] The object positioning device can determine that the current driving lane of the vehicle is the predetermined lane 1030. In this case, the object positioning device can confirm that the first driving trajectory 1011 acquired from the GPS signal does not match the predetermined lane 1030. Specifically, the object positioning device can calculate a separation distance between the first driving trajectory 1011 acquired from the GPS signal and the predetermined lane 1030, and determine that the first driving trajectory 1011 does not match the predetermined lane 1030 if the separation distance exceeds a threshold. This may be the case when the first driving trajectory 1011 and the predetermined lane 1030 do not match due to an error in the GPS signal even though the vehicle is currently driving in the predetermined lane 1030.

[0119] The object position determination device can perform lane fitting so that the vehicle's travel trajectory is fitted to the lanes on the lane map based on the current travel lane included in the feature information.

[0120] Specifically, the object position determination device can perform lane fitting by taking into consideration the distance and curvature between the vehicle's travel trajectory and the lane marking lines on the lane map, etc. The object position determination device can perform lane fitting by moving at least a part of the vehicle's travel trajectory in a straight line and adjusting at least a part of the curvature, so that the vehicle's travel trajectory matches the lane marking lines on the lane map that correspond to the current travel lane.

[0121] Referring to FIG. 10, the object position determination device can perform lane fitting by moving at least a portion of a first driving trajectory 1011 of the vehicle in a straight line and adjusting at least a portion of the curvature to correct it as a second driving trajectory 1012 so that it fits a specified lane 1030 on the lane map.

[0122] In one embodiment, an object 1021 can be displayed on the lane map 1000. The object position determination device can perform object fitting such that the position of the object on the lane map 1000 is fitted in conjunction with the lane fitting results.

[0123] Referring to FIG. 10, the object position determination device can perform object fitting by correcting the position of a first object 1021 to the position of a second object 1022 in the same way that a first driving trajectory 1011 of a vehicle is corrected to a second driving trajectory 1012.

[0124] Meanwhile, the lane fitting may be performed before and / or after the image integration described above in FIG. 6-FIG. 7 is completed.

[0125] FIG. 11 is a flow chart of a method for determining a location of an object on a lane map according to one embodiment.

[0126] The method for determining the current lane of a vehicle shown in FIG. 11 is related to the embodiments described in the previous drawings, so even if the content is omitted below, the content described in the previous drawings can also be applied to the method of FIG. 11.

[0127] Referring to FIG. 11, in step 1110, a processor may acquire image data captured by a camera mounted on a vehicle.

[0128] In step 1120, a predetermined process is performed on the image data, thereby generating feature information of a plurality of features contained in the image data.

[0129] The predetermined process may correspond to any processing method capable of generating feature information from image data without limitation. Specifically, the predetermined process may include a processing method capable of generating object type information (group, class, etc.) from image data, object position information on the image data, object size information, etc. For example, the predetermined process may be embodied by a module for traffic lights detection (DLD), traffic signs detection (TSD), ego-lane recognition (ELR), intersection recognition (ELR), road markings detection (RMD), bump recognition (BR), etc.

[0130] In step 1130, a plurality of features corresponding to the same object in the plurality of image data can be determined based on the degree of correlation between the plurality of features included in the image data.

[0131] The processor may determine multiple features corresponding to the same object in the multiple image data by aggregation of the multiple image data.

[0132] The processor may use a graph-matching method to determine multiple features indicative of the same object in the multiple image data.

[0133] The processor may select a first feature of the first image data and a second feature of the second image data, and search for other features around the first feature and the second feature. When a second neighbor feature having an attribute similar to a first neighbor feature around the first feature exists around the second feature, the processor may register the first neighbor feature and the second neighbor feature as a graph with the first feature and the second feature as key features, respectively. Here, the attribute may include at least one of a class, an aspect ratio of the feature, a distance, and an angle.

[0134] The processor can repeat the steps of searching and registering for all features around the first feature. After attempting to construct a graph for all features around the first feature, the processor can calculate the similarity between the first feature and the second feature based on the number of features and the similarity of the registered key features (e.g., size, size relationship with adjacent features, etc.). The processor can determine that the second feature of the second image data that is most similar to the first feature of the first image data indicates the same object by executing the steps of searching, registering as a graph, repeating, and calculating the similarity for the remaining features except for the second feature of the second image data.

[0135] The processor may determine that features included in the graph have the same depth.

[0136] When the processor performs post-processing on a given feature, it may perform the same post-processing on the remaining features in the graph that have the same depth.

[0137] In step 1140, the processor may determine a location of the object on the lane map based on the vehicle's location information on the lane map and feature location information on the plurality of image data.

[0138] The processor may obtain the vehicle's location on the lane map using a location determination device connected to the vehicle.

[0139] The processor may determine the location of the object based on location information on a lane map of the vehicle and feature location information on the image data.

[0140] The processor may determine the location of the object using location information of multiple features determined to correspond to the same object in the multiple image data.

[0141] In one embodiment, the object may include at least one of a traffic light, a road sign, a road marking, a current lane of travel, an intersection, and a road hump.

[0142] In one embodiment, the processor can obtain a driving trajectory of the vehicle using a position confirmation device connected to the vehicle, and can perform lane fitting so that the driving trajectory of the vehicle is fitted to a lane on the lane map based on a current driving lane included in the characteristic information.

[0143] In one embodiment, the processor may perform object fitting such that the positions of objects on the lane map are fitted in conjunction with the lane fitting results.

[0144] In one embodiment, the processor may display on the lane map a first position of a first group object determined from a first trajectory of a vehicle traveling on a predetermined road. The processor may also display on the lane map a second position of a second group object determined from a second trajectory of a vehicle traveling on a predetermined road. The processor may also determine objects that are the same between the first group object and the second group object, aggregate the positions of the objects determined to be the same object, and display the final position of each object on the lane map.

[0145] In one embodiment, the processor may construct a first graph using the first group objects and a second graph using the second group objects, and determine which objects in the first group objects and the second group objects are the same through a similarity comparison between the first graph and the second graph.

[0146] FIG. 12 is a block diagram of an object position determining apparatus according to an embodiment.

[0147] 12, the object positioning device 1200 may include a communication unit 1210, a processor 1220, and a DB 1230. In the object positioning device 1200 of FIG. 12, only components related to the embodiment are illustrated. Therefore, a person skilled in the art can understand that other general components may be further included in addition to the components illustrated in FIG. 12. In addition, in one embodiment, the object positioning device 1200 may not include the communication unit 1210.

[0148] The communication unit 1210 may include one or more components that enable wired / wireless communication with an external server or device, For example, the communication unit 1210 may include at least one of a short-range communication unit (not shown), a mobile communication unit (not shown), and a broadcast receiving unit (not shown).

[0149] The DB 1230 is hardware that stores various data processed in the object position determination device 1200, and can store programs for processing and controlling the processor 1220. The DB 1230 can store payment information, user information, and the like.

[0150] DB1230 may include dynamic random access memory (DRAM), random access memory (RAM), such as static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray or other optical disk storage, hard disk drive (HDD), solid state drive (SSD), or flash memory.

[0151] The processor 1220 controls the overall operation of the object position determination device 1200. For example, the processor 1220 can generally control an input unit (not shown), a display (not shown), a communication unit 1210, a DB 1230, etc. by executing a program stored in the DB 1230. The processor 1220 can generally control the operation of the object position determination device 1200 by executing a program stored in the DB 1230.

[0152] The processor 1220 may control at least a portion of the operation of the object position determining device described above in FIGS.

[0153] The processor 1220 may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, and other electrical units for performing functions.

[0154] In one embodiment, the object position determining device 1200 may be a mobile electronic device. For example, the traffic information providing device 800 may be implemented as a smartphone, a tablet PC, a PC, a smart TV, a personal digital assistant (PDA), a laptop, a media player, a navigation system, a device equipped with a camera, or other mobile electronic devices. In addition, the object position determining device 1200 may be implemented as a wearable device, such as a watch, glasses, a hair band, or a ring, that has a communication function and a data processing function.

[0155] In another embodiment, the object positioning device 1200 may be an electronic device embedded in a vehicle, for example, the object positioning device 1200 may be an electronic device that is inserted into a vehicle after the production process through tuning.

[0156] In another embodiment, the object position determination device 1200 may be a server located outside the vehicle. The server may be embodied as a computer device or multiple computer devices that communicate over a network and provide instructions, codes, files, content, services, etc. The server may receive data required to determine the vehicle's route from a device mounted on the vehicle, and may determine the vehicle's route based on the received data.

[0157] In yet another embodiment, the processes performed by the object position determining device 1200 may be executed at least in part by a mobile electronic device, an electronic device embedded within a vehicle, and a server located outside the vehicle.

[0158] The embodiments according to the present invention may be implemented in the form of a computer program that can be executed by various components on a computer, and such a computer program may be recorded on a computer-readable medium, which may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROMs, RAMs, flash memories, etc.

[0159] On the other hand, the computer program may be one specially designed and constructed for the present invention, or it may be one that is well known and available to those skilled in the art of computer software. Examples of computer programs include not only machine code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc.

[0160] According to an embodiment, the method according to various embodiments of the present disclosure may be provided in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read only memory (CD-ROM)) or distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices. In the case of online distribution, at least a part of the computer program product may be at least temporarily stored or temporarily generated in a device-readable storage medium such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0161] Unless otherwise clearly stated or contrary to the order of steps constituting the method of the present invention, the steps may be performed in any suitable order. The present invention is not necessarily limited to the order of steps. In the present invention, the use of all examples or exemplary terms (such as, for example, etc.) is merely for the purpose of explaining the present invention in detail, and the scope of the present invention is not limited by the examples or exemplary terms unless limited by the claims. Furthermore, those skilled in the art will understand that various modifications, combinations and changes can be made according to design conditions and factors within the scope of the appended claims or their equivalents.

[0162] Therefore, the concept of the present invention should not be limited to the above-described embodiments, and it can be said that not only the scope of the claims described below, but also all scopes equivalent to or equivalently modified from the scope of the claims belong to the scope of the concept of the present invention.

Claims

1. 1. A method of operating an apparatus having a processor for determining a position of an object on a lane map, the method being performed by the processor and comprising: acquiring image data captured by a camera mounted on the vehicle; generating feature information for a plurality of features included in the image data by performing a predetermined process on the image data; determining a plurality of features corresponding to the same object in each of the plurality of image data based on a degree of correlation between the plurality of features; determining a position of an object on the lane map based on position information of the vehicle on the lane map and feature position information on the plurality of image data; Including, The step of determining a plurality of features corresponding to the same object includes: searching for neighbor features present around a first feature of a first image data and a second feature of a second image data among the plurality of image data; registering a graph including the first feature and the second feature as key features for the first adjacent feature and the second adjacent feature based on a search result of a first adjacent feature around the first feature and a second adjacent feature around the second feature; determining that the first feature of the first image data and the second feature of the second image data represent the same object based on a similarity between the registered key features; A method comprising:

2. The method of claim 1 , wherein the object comprises at least one of a traffic light, a road sign, a road marking, and a hump.

3. The step of determining a plurality of features corresponding to the same object includes: determining a plurality of features corresponding to the same object in the plurality of image data by aggregating features of one image data with features of other image data among the plurality of image data; The method of claim 1 , comprising:

4. The step of determining a plurality of features corresponding to the same object includes: Determining a plurality of features indicative of the same object in the plurality of image data using a graph-matching method. The method of claim 1 , comprising:

5. A step of selecting the first feature and the second feature; searching for other features around the first feature and the second feature; When the second adjacent feature having an attribute similar to the first adjacent feature around the first feature exists around the second feature, registering the first adjacent feature and the second adjacent feature as the graph having the first feature and the second feature as the key feature for each of the first adjacent feature and the second adjacent feature; repeating the searching step and the registering step for all features around the first feature; A step of calculating a similarity between the first feature and the second feature based on the number of features and the similarity of registered key features after attempting to construct a graph for all features around the first feature; determining that the second feature of the second image data, which is most similar to the first feature of the first image data, indicates the same object by executing the searching step, the registering step as a graph, the repeating step, and the calculating step of the similarity for the remaining features of the second image data excluding the second feature of the first image data; The method of claim 4 , comprising:

6. The method comprises: determining that features contained in the graph have the same depth; The method of claim 5 further comprising:

7. The method comprises: when performing post-processing on a given feature, performing the same post-processing on the remaining features in the graph having the same depth. The method of claim 6, comprising:

8. The method of claim 5 , wherein the attributes include at least one of a class, an aspect ratio of a feature, a distance, and an angle.

9. The method comprises: obtaining position information of the vehicle on the lane map using a position determination device connected to the vehicle; The method of claim 1 further comprising:

10. The method comprises: displaying on the lane map the positions of a first group of objects determined from a first trajectory of a vehicle traveling on a predetermined road; displaying on the lane map the positions of a second group of objects determined from a second trajectory of the vehicle that has traveled on the predetermined road; determining which objects are the same among the first group of objects and the second group of objects, aggregating positions of the objects determined to be the same objects, and displaying a final position of each object on the lane map; The method of claim 1 further comprising:

11. The step of displaying a final position of each of the objects on the lane map includes: constructing a first graph using the first group of objects, constructing a second graph using the second group of objects, and determining identical objects in the first group of objects and the second group of objects through a similarity comparison between the first graph and the second graph. The method of claim 10, comprising:

12. 1. An apparatus for determining a location of an object on a lane map, comprising: a memory having at least one program stored therein; a processor for executing the at least one program to perform operations; The processor, Acquire image data captured by a camera mounted on the vehicle, generating feature information for a plurality of features included in the image data by performing a predetermined process on the image data; determining a plurality of features corresponding to the same object in the plurality of image data based on a degree of correlation between the plurality of features included in the image data; determining a position of an object on the lane map based on position information of the vehicle on the lane map and feature position information on the plurality of image data; In determining a plurality of features corresponding to the same object, the processor further comprises: Searching for neighbor features present around a first feature of a first image data set and a second feature of a second image data set among the plurality of image data sets; registering a graph including the first feature and the second feature as a key feature for each of the first adjacent feature and the second adjacent feature based on a search result of a first adjacent feature around the first feature and a second adjacent feature around the second feature; determining that the first feature of the first image data and the second feature of the second image data represent the same object based on a similarity between the registered key features; Device.

13. A computer-readable recording medium having recorded thereon a program for executing the method according to claim 1 on a computer.

Images