Augmented reality processing device, route guiding method based on augmented reality and electronic device

Abstract

Disclosed is an augmented reality-based route guidance method using a processor. The method comprises: a step of acquiring a traveling image captured by an imaging device of a vehicle while traveling; acquiring route data of a destination of the vehicle; recognizing lane lines on both sides of a lane in which the vehicle is traveling from the acquired traveling image; a step for generating first route guidance linear data on the basis of the recognized lane lines on both sides with respect to a region in which the lane lines on both sides are recognized in the traveling image; a step for generating second route guidance linear data by using link linear data of the route data for a region in which lane lines on both sides are not recognized in the traveling image; a step of combining the first route guidance linear data and the second route guidance linear data to generate combined route guidance linear data; and a step of displaying a route guidance object on augmented reality using the generated combined route guidance linear data.

Description

[0001] This application is a divisional application of the invention patent application with the application date of August 6, 2021, application number "202110902611.5", and invention name "Augmented reality processing device, route guidance method based on augmented reality, and electronic device". Technical Field

[0002] The present invention relates to a technology for displaying various augmented reality objects related to vehicle driving through augmented reality. Background Art

[0003] Augmented Reality (AR) refers to a technology that synthesizes virtual objects or information in the real world to make them look like objects that exist in the real environment.

[0004] Unlike virtual reality (VR) in which virtual objects are synthesized in a virtual world and displayed, augmented reality synthesizes virtual objects in the real world, and thus can provide users with additional information that cannot be obtained from the real world alone.

[0005] Due to these characteristics, augmented reality is being applied in various fields.

[0006] As an example, there is augmented reality navigation that provides guidance related to vehicle driving based on augmented reality. This augmented reality navigation improves user convenience by visually overlapping additional information on a screen that projects the display world that the user actually sees. The additional information includes, for example, points of interest (POIs) and graphic elements that indicate the route to the destination.

[0007] However, traditional augmented reality navigation has problems such as difficulty displaying information far away from the vehicle and difficulty displaying information in high-speed driving environments due to limitations of the camera environment.

[0008] Due to these shortcomings, the route guidance objects provided by the conventional augmented reality navigation cannot be accurately expressed in the form and shape of the composition, so there is a problem that it is difficult for the driver to accurately understand the route through the objects. In addition, in the conventional augmented reality navigation, various ADAS (Advanced Driver Assistance System) objects are also simply expressed in the form of icons, which reduces the realism of the advantages of augmented reality and leads to the problem of reduced driving convenience for the driver. Summary of the invention

[0009] One of various objects of the present invention is to provide a method for realizing a route guidance object displayed through augmented reality by combining lane lines recognized during vehicle travel with link linear data of route data.

[0010] One of the various objects of the present invention is to provide a method for realizing a route guidance object, which method displays an augmented reality that is intuitive, dynamic, and has improved realism to a user.

[0011] One of various objects of the present invention is to provide a method for displaying a lane departure warning system (LDWS) to a route guidance object when a vehicle deviates from a lane line during driving.

[0012] To achieve the above object, a route guidance method based on augmented reality using a processor according to an embodiment of the present invention includes: a step of acquiring a driving image captured by a camera of a vehicle while driving; a step of acquiring route data to the destination of the vehicle; a step of identifying lane lines on both sides of a lane in which the vehicle is driving from the acquired driving image; a step of generating first route guidance linear data based on the identified lane lines on both sides for an area where lane lines on both sides are identified in the driving image; a step of generating second route guidance linear data using link linear data of the route data for an area where lane lines on both sides are not identified in the driving image; a step of generating combined route guidance linear data by combining the first route guidance linear data and the second route guidance linear data; and a step of displaying a route guidance object on augmented reality using the generated combined route guidance linear data.

[0013] Moreover, the step of generating the second route guidance linear data may also include: the step of calculating the movement values ​​of multiple points constituting the link linear data based on the position of the first point constituting the first route guidance linear data; and the step of moving the multiple points constituting the link linear data based on the calculated movement values.

[0014] Moreover, the step of generating the second route guidance linear data may also include: a step of calculating the rotation angle of the link linear data based on the angle difference between the driving direction vector of the vehicle and the route direction vector of the vehicle; and a step of rotating the link linear data generated by the movement based on the calculated rotation angle.

[0015] Furthermore, the method may further include generating a virtual lane line for the lane line on the other side based on the lane line width data when only the lane line on one side is identified in the driving image.

[0016] Moreover, the route guidance object may include: a first route guidance object in the shape of an arrow, having a center in the combined route guidance linear data and pointing to the route on which the vehicle should travel; a second route guidance object, displayed between the left side of the first route guidance object and the left lane line; and a third route guidance object, displayed between the right side of the first route guidance object and the right lane line.

[0017] Also, when the vehicle travels forward, the route guide object is displayed by moving from front to rear, and the moving speed of the route guide object may be changed according to the speed of the vehicle.

[0018] Furthermore, the method may further include displaying a curve location guidance object instead of displaying the route guidance object when the vehicle approaches a curve location.

[0019] Furthermore, the method may further include a step of changing a display state of at least one of the second route guidance object and the third route guidance object to display the object when the vehicle deviates from a lane line.

[0020] In addition, to achieve the above purpose, an electronic device for executing route guidance based on augmented reality according to an embodiment of the present invention includes: a display unit for displaying a screen; a driving image acquisition unit for acquiring a driving image captured by a camera of a driving vehicle; a route data acquisition unit for acquiring route data to the destination of the vehicle; a lane line recognition unit for identifying lane lines on both sides of a lane in which the vehicle is driving from the acquired driving image; a route guidance linear data generation unit for generating first route guidance linear data based on the identified lane lines on both sides for an area where lane lines on both sides are recognized in the driving image, and for an area where lane lines on both sides are not recognized in the driving image, generating second route guidance linear data using link linear data of the route data, and generating combined route guidance linear data by combining the first route guidance linear data and the second route guidance linear data; and a control unit for controlling the display unit so that the generated combined route guidance linear data is used to display the route guidance object in the augmented reality image.

[0021] Moreover, the route guidance object may include: a first route guidance object in the shape of an arrow, having a center in the combined route guidance linear data and pointing to the route on which the vehicle should travel; a second route guidance object, displayed between the left side of the first route guidance object and the left lane line; and a third route guidance object, displayed between the right side of the first route guidance object and the right lane line.

[0022] Furthermore, when the vehicle travels forward, the route guidance object moves from front to rear and is displayed in the augmented reality image, and the moving speed of the route guidance object may be changed according to the speed of the vehicle.

[0023] Furthermore, when the vehicle approaches a curve point, the control portion may control the display portion so that a curve point guidance object is displayed instead of the route guidance object.

[0024] Furthermore, when the vehicle deviates from a lane line, the control unit may control the display unit so as to change a display state of at least one of the second route guidance object and the third route guidance object and display it.

[0025] In addition, to achieve the above purpose, an augmented reality processing device according to an embodiment of the present invention includes: a driving image acquisition unit, which is used to acquire a driving image captured by a camera of a vehicle while driving; a route data acquisition unit, which is used to acquire route data to the destination of the vehicle; a lane line recognition unit, which recognizes lane lines on both sides of a lane in which the vehicle is traveling from the acquired driving image; a route guidance linear data generation unit, which generates first route guidance linear data based on the recognized lane lines on both sides for an area where lane lines on both sides are recognized in the driving image, and generates second route guidance linear data using link linear data of the route data for an area where lane lines on both sides are not recognized in the driving image, and generates combined route guidance linear data by combining the first route guidance linear data and the second route guidance linear data; and an augmented reality image generation unit, which generates an augmented reality image that maps a route guidance object using the generated combined route guidance linear data.

[0026] Moreover, the route guidance linear data generating unit can calculate the movement values ​​of multiple points constituting the link linear data based on the position of the first point constituting the first route guidance linear data, and can move the multiple points constituting the link linear data based on the calculated movement values.

[0027] Moreover, the route guidance linear data generating unit can calculate the rotation angle of the link linear data based on the angle difference between the vehicle's driving direction vector and the vehicle's route direction vector, and can rotate the link linear data generated by the movement based on the calculated rotation angle.

[0028] Furthermore, the route guidance linear data generating unit may further include a step of generating a virtual lane line for the lane line on the other side based on the lane line width data when only the lane line on one side is identified in the driving image.

[0029] In addition, to achieve the above object, according to an embodiment of the present invention, a program may be provided, which is stored in a computer-readable recording medium including a program code for executing a route guidance method based on augmented reality.

[0030] In addition, to achieve the above object, according to an embodiment of the present invention, a computer-readable recording medium may be provided, in which a program for executing a route guidance method based on augmented reality is recorded.

[0031] Effects of the Invention

[0032] According to various embodiments of the present invention, a route guidance object is represented based on the result of identifying lane markings on both sides of the vehicle's driving lane (Lane). At a long distance where the lane markings are erased or invisible, the route guidance object is displayed using route data stored in a storage unit, thereby allowing a more accurate route guidance object to be displayed on augmented reality.

[0033] According to various embodiments of the present invention, a dynamic route guidance object that a driver can intuitively recognize may be displayed on augmented reality.

[0034] According to various embodiments of the present invention, when lane deviation occurs during vehicle driving, a route guidance object and a lane departure guidance object may be intuitively displayed on augmented reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a block diagram showing an augmented reality processing device according to an embodiment of the present invention.

[0036] Figure 2 Detailed description of the block diagram of the augmented reality image generating unit according to one embodiment of the present invention.

[0037] Figure 3 FIG. 1 is a diagram showing a driving environment of a vehicle according to an embodiment of the present invention.

[0038] Figure 4 It is a diagram for explaining a method for generating route guidance linear data according to an embodiment of the present invention.

[0039] Figure 5 FIG. 1 is a diagram for explaining error correction of second route guidance linear data when a vehicle changes lanes according to an embodiment of the present invention.

[0040] Figure 6 FIG. 1 is a diagram for explaining error correction of second route guidance linear data when a vehicle travels along a curve according to an embodiment of the present invention.

[0041] Figure 72 is a diagram showing a transformation relationship between a captured image and a virtual 3D space according to an embodiment of the present invention.

[0042] Figure 8 FIG. 1 is a diagram for explaining a route guidance object according to an embodiment of the present invention.

[0043] Fig. 9 is a block diagram showing an augmented reality processing device according to another embodiment of the present invention.

[0044] Fig.10 is a block diagram showing an electronic device according to an embodiment of the present invention.

[0045] Fig.11 FIG. 1 is a diagram for explaining a system network connected to an electronic device according to an embodiment of the present invention.

[0046] Fig.12 is a flow chart showing a method for generating an augmented reality image according to an embodiment of the present invention.

[0047] Fig.13 Detailed description of the process of generating linear data according to an embodiment of the present invention.

[0048] Fig.14 is a diagram showing an augmented reality image displayed by an electronic device when navigation to a destination is performed.

[0049] Fig.15 FIG. 1 is a diagram showing an augmented reality image displayed by an electronic device when a vehicle deviates from a lane.

[0050] Figure 16 to Figure 17 FIG. 1 is a diagram showing an augmented reality image displayed by an electronic device when a vehicle approaches a curve.

[0051] Fig.18 FIG. 1 is a diagram showing an embodiment of an electronic device according to an embodiment of the present invention when the electronic device does not include a camera.

[0052] Fig.19 FIG. 1 is a diagram showing an embodiment of an electronic device including a camera according to an embodiment of the present invention.

[0053] Fig. 20 FIG. 1 is a diagram showing an embodiment using a head-up display (HUD) according to an embodiment of the present invention.

[0054] Fig.21 is a block diagram showing an automatic driving system according to an embodiment of the present invention.

[0055] Fig. 22is a block diagram showing the structure of an autonomous driving vehicle according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following content is only an example of the principle of the invention. Therefore, a person skilled in the art of the invention can invent various devices that are not explicitly described or illustrated in the specification but embody the principle of the invention and are included in the concept of the invention. In addition, all conditional terms and embodiments listed in this specification are only used to clearly understand the concept of the invention, and are not limited to the specifically listed embodiments and states as described above.

[0057] Furthermore, it should be understood that not only the principles, viewpoints and embodiments of the present invention, but also all detailed descriptions listing specific embodiments include structural and functional equivalents of the matters described. Furthermore, it should be understood that the equivalents include not only currently known equivalents, but also equivalents to be developed in the future, that is, all elements invented in a way to perform the same function regardless of the structure.

[0058] Thus, for example, it should be understood that the block diagrams in this specification show conceptual views of exemplary circuits that embody the principles of the present invention. Similarly, it should be understood that all flow charts, state transition diagrams, pseudo codes, etc. can be substantially embodied by computer-readable media, whether or not a computer or processor is explicitly shown, and represent various processing that can be executed by a computer or processor.

[0059] The functions of the various elements shown in the figure, including the functional modules represented by the processor or the concept similar thereto, are not only related to dedicated hardware, but also to appropriate software, so that hardware with the ability to execute software can be used. When provided by a processor, the functions can be provided by a single dedicated processor, a single shared processor, or multiple individual processors, some of which can be shared.

[0060] Furthermore, the explicit use of terms such as processing, control, or concepts similar to processing and control cannot be interpreted as exclusively referring to hardware capable of executing software, but should be understood to implicitly include digital signal processor (DSP) hardware, read-only memory (ROM) for storing software, random access memory (RAM), and non-volatile memory without limitation. Other conventional hardware may also be included.

[0061] In the scope of the invention claims of this specification, the structural elements that are expressed as means for executing the functions described in the detailed description are intended to include, for example, all methods of the functions of software in all forms including a combination of circuit elements that execute the functions or firmware / microcode, etc., in a manner to execute the functions in combination with appropriate circuits for executing the software. In the present invention defined by the scope of the invention claims, the functions provided by a plurality of means are combined and combined in the manner required by the scope of the invention claims, and therefore, it should be understood that any means that can provide the functions are also equivalent to the functions grasped from this specification.

[0062] The above objects, features and advantages will become more clearly understood through the following detailed description in conjunction with the accompanying drawings, so that a person skilled in the art can easily implement the technical concept of the present invention. In addition, in the process of describing the present invention, if it is determined that the specific description of the known technology related to the present invention will make the main idea of ​​the present invention unclear, the detailed description thereof will be omitted.

[0063] Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0064] Figure 1 is a block diagram showing an augmented reality processing device according to an embodiment of the present invention. Figure 2 Detailed description of the block diagram of the augmented reality image generation unit according to one embodiment of the present invention. Figure 1 to Figure 2 The augmented reality processing device 10 may include: an image acquisition unit 11, a route data acquisition unit 12, a lane line recognition unit 13, a route guidance linear data generation unit 14 and an augmented reality image generation unit 15.

[0065] Here, the augmented reality image generating unit 15 may include all or part of a calibration unit 15 - 1 , a virtual 3D space generating unit 15 - 2 , an object generating unit 15 - 3 , and a mapping unit 15 - 4 .

[0066] The augmented reality processing device 10 may generate an augmented reality image for guiding a car user based on a driving image captured by a camera of the vehicle.

[0067] Here, the driving of the vehicle refers to a state in which the vehicle is driven by a driver and is located on a road, and may be a concept including a driving state in which the vehicle is driving on a road, a parked state in which the vehicle is parked on the road, and a parked state in which the vehicle is parked.

[0068] In addition, the car user may be a concept including a car driver and a car non-driver (eg, a co-driver, a passenger, etc.).

[0069] In addition, the driving image captured by the camera of the vehicle may include a front driving image captured in front of the vehicle and a rear driving image captured in rear of the vehicle.

[0070] The augmented reality processing device 10 can be implemented using software, hardware or a combination thereof. As an example, according to hardware implementation, it can be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, and micro-processors for performing other functions.

[0071] For the convenience of explanation, the following will take the case where the driving image is a front driving image as an example to explain in more detail each component module constituting the augmented reality processing device 10 .

[0072] The image acquisition unit 11 can acquire the driving image captured by the camera of the vehicle. Specifically, the image acquisition unit 11 can acquire the driving image captured by the camera installed in the vehicle in real time when the vehicle is driving.

[0073] Here, the acquired driving image may include a plurality of lanes distinguished by lane markings, a road consisting of the plurality of lanes, and a plurality of vehicles traveling on the road.

[0074] Lane markings may refer to lines on both sides of a lane that a vehicle is in. In addition, lanes may refer to a road such as a first lane, a second lane, ... an Nth lane, etc., formed by lane markings on which a vehicle is traveling.

[0075] The route data acquisition unit 12 can acquire route data to the vehicle destination from the map data. Here, the map data is acquired from the storage unit 110 of the electronic device 100 provided with the augmented reality processing device 10, or acquired through wired / wireless communication with the electronic device 100 and other external map databases, or can be acquired from other electronic devices.

[0076] Additionally, the route data may include link data and node data representing a route to a vehicle destination.

[0077] Here, the link data may be composed of link linear data and link attribute data.

[0078] That is, in the map data, a road or each lane constituting a road may be constituted by connecting a plurality of links, and link linear data is data representing the linear characteristics of the link, and the link linear data may be assigned position data of at least one point constituting the line. Here, the position data may be GPS data. Such link linear data may be used to generate route guidance linear data, and the route guidance linear data is used to display a route guidance object in augmented reality.

[0079] In addition, the link attribute data may include at least one of an identifier of the link, data indicating whether the link is a bidirectional link or a unidirectional link, the starting point and end point of the link based on the vehicle's driving direction, a road number, a road name, a road length, road grade (Road Rank) information, road type (Road Type) information, road width data, road lane number data and road inclination data, and guide code data (for example, data guiding speed limits, control areas, etc.).

[0080] The node data may include an identifier of the node, attribute data of the node, and location data of the node. Here, the location data may be GPS data, and the node attribute data may include a direction attribute of the node.

[0081] The lane line recognition unit 13 can recognize the lane line of the lane in which the vehicle is traveling from the driving image acquired by the image acquisition unit 11. Specifically, the lane line recognition unit 13 can detect the edge corresponding to the lane line in the driving image acquired by the image acquisition unit 11, and generate an edge image based on the detected edge. In addition, the lane line recognition unit 13 can detect the straight line component based on the detected edge, and generate an edge straight line processed image based on the detected straight line component. In addition, the lane line recognition unit 13 can use the generated edge image and the edge straight line processed image to detect the lane line point corresponding to the lane line. In addition, the lane line recognition unit 13 can recognize the lane line based on the detected lane line point.

[0082] In this case, the lane line recognition unit 13 can recognize lane lines on both sides where the vehicle is traveling by respectively detecting a lane line point corresponding to the left lane line and a lane line point corresponding to the right lane line.

[0083] The route guidance linear data generating unit 14 can generate route guidance linear data for displaying the route guidance object in augmented reality. Specifically, the route guidance linear data generating unit 14 generates first route guidance linear data based on the identified lane lines on both sides for the area where the lane lines on both sides are identified in the driving image, and generates second route guidance linear data using link linear data of the route data for the area where the lane lines on both sides are not identified in the driving image. In addition, the route guidance linear data generating unit 14 can generate combined route guidance linear data by combining the generated first route guidance linear data and the second route guidance linear data.

[0084] Reference Figure 3 to Figure 4 The operation of the route guidance linear data generating unit 14 will be described in more detail.

[0085] Figure 3 FIG. 1 is a diagram showing a driving environment of a vehicle according to an embodiment of the present invention. Figure 4 It is a diagram for explaining a route guide line generating method according to an embodiment of the present invention.

[0086] During the driving process of the vehicle, there may be a lane area 31-2 where a portion of the lane line is erased (31) or a lane area 32-2 where a portion of the lane line is blocked by the vehicle ahead (32). In this case, the lane line recognition unit 13 can recognize the lane line in the partial lane areas 31-1 and 32-1 located in front of the vehicle, but cannot recognize the lane line in the remaining partial lane areas 31-2 and 32-2.

[0087] Alternatively, there may be a situation (33) where the lane line cannot be recognized by the lane line recognition unit 13 because the lane line is blurred or at a long distance. Even in this case, the lane line recognition unit 13 can recognize the lane line in the partial lane area 33-1 located in front of the vehicle, but cannot recognize the lane in the remaining partial lane area 33-2.

[0088] Therefore, for the area 41-1 where lane lines on both sides are identified in the driving image, the first route guidance linear data can be generated based on the identified lane lines on both sides. Specifically, the route guidance linear data generating unit 14 can determine the virtual point 41-4 located in the middle based on the multiple points 41-2 constituting the left lane line and the multiple points 41-3 constituting the right lane line detected by the lane line recognition unit 13, and can connect the multiple virtual points to generate the first route guidance linear data 41-5.

[0089] However, for the area 42-2 where the lane lines on both sides are not recognized in the driving image, the second route guidance linear data can be generated using the link linear data of the route data. Specifically, the route guidance linear data generation unit 14 can detect the link linear data 43-1 and 43-2 corresponding to the area 41-1 where the lane lines on both sides are not recognized based on the current position of the vehicle in the plurality of link linear data 43, and generate the second route guidance linear data 42-5 based on the detected link linear data 43-1 and 43-2.

[0090] On the one hand, the route guidance linear data generating unit 14 may generate the combined route guidance linear data 44 by combining the generated first route guidance linear data 41 - 5 and the generated second route guidance linear data 42 - 5 .

[0091] On the other hand, when generating the combined route guidance linear data 44 , the route guidance linear data generating section 14 may perform error correction of the second route guidance linear data 42 - 5 .

[0092] Specifically, the route guidance linear data generating unit 14 may calculate the movement values ​​of the plurality of points constituting the link linear data based on the position of the first point constituting the first route guidance linear data. In addition, the route guidance linear data generating unit 14 may translate the plurality of points constituting the link linear data based on the calculated movement values.

[0093] In addition, the route guidance linear data generating unit 14 can calculate the rotation angle of the link linear data based on the angle difference between the vehicle's driving direction vector and the vehicle's route direction vector. In addition, the route guidance linear data generating unit 14 can rotate the link linear data generated according to the movement (translation) based on the calculated rotation angle.

[0094] Will refer to Figure 5 to Figure 6 The error correction of the route guidance linear data generating unit 14 will be described in more detail.

[0095] Figure 5 FIG. 1 is a diagram for explaining error correction of second route guidance linear data when a vehicle changes lanes according to an embodiment of the present invention.

[0096] Reference numeral 71 is a diagram of the process of changing lanes from the second lane to the first lane while the vehicle is traveling. The image taken on the vehicle 71-1 traveling in the second lane may be the same as reference numeral 72, and the image taken on the vehicle 71-2 changing to the first lane may be the same as reference numeral 73.

[0097] In this case, the route guidance linear data generating unit 14 may generate the first route guidance linear data based on the recognized lane lines on both sides for the areas 72 - 1 and 73 - 1 where the lane lines on both sides are recognized in the driving image.

[0098] However, for the areas 72 - 3 and 73 - 3 where the lane lines on both sides are not recognized in the driving image, the second route guidance linear data can be generated using the link linear data 75 and 76 of the route data.

[0099] At this time, there will be a position error between the position data of the vehicle and the position data assigned to the link linear data. Therefore, the route guidance linear data generation unit 14 according to an embodiment of the present invention can determine the position of the first point 72-2, 73-2 constituting the first route guidance linear data based on the position data of the moving vehicles 71-1, 71-2, and determine the initial positions 75-1, 76-1 of the plurality of points constituting the link linear data based on the position data of the link linear data 75, 76. Here, the first point 72-2, 73-2 can be the last point of the first route guidance linear data of the identification area 72-1, 73-1.

[0100] On the other hand, the route guidance linear data generation unit 14 can calculate the movement value d1 of the initial positions 75-1 and 76-1 of the plurality of points constituting the link linear data based on the positions of the first points 72-2 and 73-2 constituting the first route guidance linear data. Furthermore, the route guidance linear data generation unit 14 can generate the second route guidance linear data 75-2 and 76-2 by moving (translating) the plurality of points constituting the link linear data based on the calculated movement value d1.

[0101] In addition, when the optical axis vector V1 of the camera set on the vehicle and the route direction vector V2 of the vehicle are different, such as when the vehicle 71-2 changes to the first lane, there will be an angle error. Here, the meaning of the optical axis vector V1 of the camera set on the vehicle is the same as the meaning of the driving direction vector toward the driving direction of the vehicle, and can be regarded as the same concept.

[0102] Therefore, the route guidance linear data generating unit 14 can calculate the rotation angle of the link linear data based on the angle difference θ between the optical axis vector V1 of the camera and the vehicle route direction vector V2. As an example, the rotation angle can be composed of the rotation direction and the size of the rotation angle of the vehicle route direction vector V2 based on the optical axis vector V1 of the camera.

[0103] Furthermore, the route guidance linear data generating unit 14 may rotate the link linear data generated by the movement (translation) by the same rotation angle in the same rotation direction based on the calculated rotation angle.

[0104] Therefore, the route guidance linear data generating section 14 can generate the second route guidance linear data 76 - 3 in which the position error and the angle error are corrected.

[0105] Figure 6 FIG. 1 is a diagram for explaining error correction of second route guidance linear data when a vehicle travels along a curve according to an embodiment of the present invention.

[0106] Reference numeral 81 is a diagram of a vehicle traveling on a curved section, reference numeral 82 is an image taken from a vehicle 81 - 1 before entering the curved section, and reference numeral 83 is an image taken from a vehicle 81 - 2 traveling on the curved section.

[0107] At this time, the route guidance linear data generating unit 14 may generate the first route guidance linear data based on the recognized lane lines on both sides for the regions 82 - 1 and 83 - 1 where the lane lines on both sides are recognized in the driving image.

[0108] However, for the areas 82 - 3 and 83 - 3 where the lane lines on both sides are not recognized in the driving image, the link linear data 85 and 86 of the route data can be used to generate the second route guidance linear data.

[0109] At this time, there will be a position error between the position data of the vehicle and the position data assigned to the link linear data. Therefore, the route guidance linear data generation unit 14 according to an embodiment of the present invention can determine the position of the first point 82-2, 83-2 constituting the first route guidance linear data based on the position data of the moving vehicles 81-1, 81-2, and can determine the initial positions 85-1, 86-1 of the multiple points constituting the link linear data based on the position data of the link linear data 85, 86. Here, the first point 82-2, 83-2 can be the last point of the first route guidance linear data of the identification area 82-1, 83-1.

[0110] On the other hand, the route guidance linear data generation unit 14 can calculate the movement value d1 of the initial positions 85-1 and 86-1 of the plurality of points constituting the link linear data based on the positions of the first points 82-2 and 83-2 constituting the first route guidance linear data. Furthermore, the route guidance linear data generation unit 14 can generate the second route guidance linear data 85-2 and 86-2 by moving (translating) the plurality of points constituting the link linear data based on the calculated movement value d1.

[0111] In addition, when the vehicle is traveling on a curved road, there will be an angle error because the optical axis vector V1 of the camera and the route direction vector V2 of the vehicle are different. Therefore, the route guidance linear data generating unit 14 can calculate the rotation angle of the link linear data based on the angle difference θ2 between the optical axis vector V1 of the camera and the route direction vector V2 of the vehicle.

[0112] As an example, the rotation angle is based on the optical axis vector V1 of the camera and can be composed of the rotation direction and rotation angle of the vehicle's route direction vector V2.

[0113] Furthermore, the route guidance linear data generating unit 14 may rotate the link linear data generated by the movement in the same rotation direction by the same rotation angle based on the calculated rotation angle. Thus, the route guidance linear data generating unit 14 may generate the second route guidance linear data 86-3 in which the position error and the angle error have been corrected.

[0114] However, before entering the curved section, there is no angular difference between the optical axis vector V1 of the camera and the course direction vector V2 of the vehicle, and therefore, rotation correction is not required.

[0115] On the other hand, when only one lane line is identified in the driving image, the route guidance linear data generation unit 14 can generate a virtual lane line for the other lane line based on the lane line width data. In addition, the route guidance linear data generation unit 14 can determine a virtual point located in the middle based on the lane line on one side identified by the lane line recognition unit 13 and the virtual lane line on the other side generated virtually, and can connect the virtual points to generate route guidance linear data. In addition, the route guidance linear data generation unit 14 can combine the route guidance linear data generated by the virtual point with the route guidance linear data of the area where the lane lines on both sides are not identified to generate combined route guidance linear data.

[0116] On the other hand, the augmented reality image generation unit 15 may generate an augmented reality image that displays various objects in augmented reality using the combined route guidance linear data generated by the route guidance linear data generation unit 14. Figure 2 The augmented reality image generating unit 15 will be described in more detail.

[0117] The calibration unit 15-1 may perform calibration to estimate camera parameters corresponding to the camera from the image captured by the camera. The camera parameters may be information about the relationship between the image of the photo and the actual space (or the real world), that is, parameters constituting the camera matrix.

[0118] Here, parameters such as Figure 7As shown in (a), the camera extrinsic parameters 601 and the camera intrinsic parameters 602 may be included. Here, fx and fy of the camera intrinsic parameters 602 may be focal lengths, cx and cy may be principal points, and skew_c=tanα may be a skew coefficient. In addition, the camera extrinsic parameters 601 may be a rotation / translation transformation matrix that transforms the coordinates X, Y, and Z of a 3D point on a world coordinate system 605 into the coordinates Xc, Yc, and Zc of a 3D point on a camera coordinate system 604. Since the camera extrinsic parameters are not inherent parameters of the camera, they may vary according to the installation position and installation direction of the camera, and may vary according to the definition of the world coordinate system.

[0119] That is, the calibration unit 15 - 1 according to the present invention can execute calibration for estimating camera parameters corresponding to the camera from the captured image.

[0120] The virtual 3D space generation unit 15-2 can generate a virtual 3D space based on the image captured by the camera. Specifically, the virtual 3D space generation unit 15-2 can obtain depth information from the image captured by the camera based on the camera parameters estimated by the calibration unit 15-1, and generate a virtual 3D space based on the obtained depth information and the captured image. Specifically, refer to Figure 7 b, the image captured by the camera is obtained by perspective projection of multiple points in the three-dimensional space of the world coordinate system 605 to the two-dimensional image plane of the image coordinate system 603. Therefore, the 3D space generation unit 15-2 can generate a virtual 3D (3-Dimensional) space of the world coordinate system 605 for the image captured by the camera by performing the reverse process of the above operation based on the camera parameters.

[0121] That is, the virtual 3D space generating unit 15 - 2 according to the present invention can obtain depth information from the driving image based on the camera parameters estimated by the calibration unit 15 - 1 , and generate a 3D space based on the obtained depth information and the driving image.

[0122] The object generation unit 15-3 can generate various objects for guidance on the augmented reality. In the present invention, the object will be described to refer to various graphic elements displayed through the augmented reality. Here, the object can include various graphic elements that can guide the vehicle when driving, such as the front vehicle collision prevention guidance object, the route guidance object, the lane departure guidance object, the curve guidance object, the curve location guidance object, etc.

[0123] As an example, when a user inputs a route guidance request to a destination, the object generation unit 15-3 may generate a route guidance object that displays a route from the current position of the vehicle to the destination in a captured image corresponding to the real world. Figure 8 The route guidance object will be described in more detail.

[0124] Reference Figure 8 The route guidance object generated by the object generation unit 15 may include a first route guidance object 611 having an arrow shape, a second route guidance object 612 having a rectangular shape, and a third route guidance object 613. Here, the first route guidance object 611 refers to an arrow shape of a route that the vehicle should travel, and is configured with the combined route guidance linear data generated by the route guidance linear data generation unit 14 as the center, and can be displayed through augmented reality. In addition, the second route guidance object 612 is configured between the left side of the first route guidance object 611 and the left lane line, and can be displayed through augmented reality, and the third route guidance object 613 is configured between the right side of the first route guidance object 61 and the right lane line, and can be displayed through augmented reality.

[0125] Here, the colors of the route guide objects 611, 612, and 613 may be different according to the driving environment of the vehicle. In addition, the second route guide object 612 and the third route guide object 613 may be embodied as gradually increasing transparency from the outside to the inside.

[0126] In addition, when the vehicle deviates from the lane line of the lane in which the vehicle is traveling, the object generation unit 15 - 3 may generate a lane line deviation guidance object for guiding the lane line deviation in a captured image corresponding to the real world.

[0127] Furthermore, when the vehicle approaches a curve point while traveling, the object generation unit 15 - 3 may generate a curve point guidance object for guiding the vehicle to approach the curve point in a captured image corresponding to the real world.

[0128] Furthermore, when the vehicle approaches a leading vehicle while traveling, the object generation unit 15 - 3 may generate a leading vehicle collision avoidance guidance object in a captured image corresponding to the real world.

[0129] Here, the object may be embodied as a 3D object, a texture image, or a line art.

[0130] The mapping unit 15-4 can determine the mapping position of the object generated by the object generating unit 15-3 in the virtual 3D space generated by the virtual 3D space generating unit 15-2, and map the object at the determined mapping position. That is, the mapping unit 16-4 according to the present invention can determine the mapping position in the virtual 3D space according to the type of the guide object generated by the object generating unit 15-3, and generate virtual 3D space data mapped at the determined mapping position.

[0131] In addition, the augmented reality image generation unit 15 may generate an augmented reality image by converting the virtual 3D space data of the mapped object into a 2D image using a camera matrix.

[0132] When the object generation unit 15-3 generates the route guidance object, the mapping unit 15-4 may calculate the position in the virtual 3D space of the combined route guidance linear data, the position of the left lane line in the virtual 3D space, and the position of the right lane line in the virtual 3D space generated by the route guidance linear data generation unit 14. In addition, the mapping unit 15-4 may determine the mapping position of the route guidance object based on the position calculated in the virtual 3D space, and generate virtual 3D space data mapped to the determined mapping position.

[0133] In this case, the mapping unit 15-4 can determine the mapping position of the route guidance object so that the center of the first route guidance object 611 is located at the position in the virtual 3D space combined with the route guidance linear data. In addition, the mapping unit 15-4 can determine the mapping position of the second route guidance object 612 between the position of the left lane line in the virtual 3D space and the position of the first route guidance object 611 in the virtual 3D space. In addition, the mapping unit 15-4 can determine the mapping position of the third route guidance object 613 between the position of the right lane line in the virtual 3D space and the position of the first route guidance object 611 in the virtual 3D space. In addition, the mapping unit 15-4 can generate virtual 3D space data mapped to the determined mapping positions of the first route guidance object 611, the second route guidance object 612, and the third route guidance object 613, respectively. In addition, the augmented reality image generation unit 15 can use the camera matrix to convert the virtual 3D space data of the mapped route guidance objects 611, 612, and 613 into 2D images to generate augmented reality images. Therefore, the image captured by the camera and the route guidance objects 611 , 612 , and 613 may be displayed overlappingly in the augmented reality image.

[0134] Here, when the vehicle is traveling forward, the route guidance object 611 may be displayed moving from front to back. That is, when the vehicle is traveling forward, the route guidance object 611 displayed in front of the vehicle may be displayed gradually approaching the vehicle.

[0135] In addition, the moving speed of the first route guidance object 611 may vary according to the driving speed of the vehicle. That is, when the forward driving speed of the vehicle increases, the speed at which the first route guidance object 611 moves from front to back also increases. In addition, when the forward driving speed of the vehicle decreases, the speed at which the first route guidance object 611 moves from front to back also decreases. In addition, when the forward driving speed of the vehicle is "0", i.e., in a stopped state, the moving speed of the first route guidance object 611 is also "0", and it may not move.

[0136] When the object generation unit 15-3 generates at least one of a curve location guidance object, a front vehicle collision prevention guidance object, and a lane line deviation prevention guidance object, the mapping unit 15-4 can calculate the position of the generated object in the virtual 3D space. In addition, the mapping unit 15-4 can determine the mapping position of the object based on the position calculated in the virtual 3D space, and generate virtual 3D space data mapped to the determined mapping position. In addition, the augmented reality image generation unit 15 can use a camera matrix to convert the virtual 3D space data of the mapping object into a 2D image to generate an augmented reality image. Therefore, at least one of the image captured by the camera, the curve location guidance object, the front vehicle collision prevention guidance object, and the lane line deviation prevention guidance object can be displayed on the augmented reality image.

[0137] On the other hand, when the vehicle deviates from the lane line, the augmented reality image generation unit 15 can change the display state of at least one of the second route guidance object 612 and the third route guidance object 613 to display the augmented reality image. As an example, the augmented reality image generation unit 15 can generate an augmented reality image that differently represents the colors of the second route guidance object 612 and the third route guidance object 613 before the vehicle deviates from the lane line and the colors of the second route guidance object 612 and the third route guidance object 613 after the vehicle deviates from the lane line. As another example, the augmented reality image generation unit 15 can generate an augmented reality image that distinguishes and represents the route guidance object corresponding to the vehicle deviating from the lane line (for example, the second route guidance object 612 when deviating from the left lane line and the third route guidance object 613 when deviating from the right lane line).

[0138] In addition, when the vehicle approaches a curve, the augmented reality image generation unit 15 may generate an augmented reality image displaying the curve guide object instead of displaying the first route guidance object 611 , the second route guidance object 612 , and the third route guidance object 613 .

[0139] Fig. 9 is a block diagram showing an augmented reality processing device according to another embodiment of the present invention. Fig. 9According to other embodiments of the present invention, the augmented reality processing device 20 can also be Figure 1 The curve forming unit 16 and the filter unit 17 in the augmented reality processing device 10 are shown.

[0140] The curvilinear unit 16 may curve the lane lines on both sides identified by the lane line recognition unit 13. Specifically, the curvilinear unit 16 detects a plurality of lane line points from the identified lanes on both sides, projects the detected lane line points to a virtual 3D space through camera perspective projection, and calculates the curved spatial line shapes corresponding to the lane line points in the virtual 3D space to perform the curvilinearization of the lane lines on both sides.

[0141] In addition, as described above, the curve forming unit 16 may interpret the first route guidance linear data and the second route guidance linear data generated by the route guidance linear data generating unit 14 in the virtual 3D space and form curves.

[0142] Here, the curve forming unit 16 may form a curve using, for example, a Bezier curve.

[0143] Through the curvilinear process of the curvilinear unit 16 , the irregular lane line recognition result can be calculated as a smooth lane line.

[0144] In addition, the filter unit 17 may perform filtering processing on the spatial coordinates calculated during the operation of the device.

[0145] Specifically, the filtering unit 17 may perform filtering to remove noise points from the spatial coordinates of the virtual 3D space corresponding to the plurality of lane line points of the lane lines on both sides recognized by the lane line recognition unit 13 .

[0146] In addition, as described above, the filtering unit 17 may interpret the first route guidance linear data and the second route guidance linear data generated by the route guidance linear data generating unit 14 in the 3D space and perform filtering.

[0147] Here, the filter unit 17 may perform filtering using a mean filter, for example. By using such a mean filter, when the vehicle changes lanes, the route guidance object may be controlled to move smoothly by weighted averaging each currently acquired spatial coordinate and the past spatial coordinate.

[0148] Fig.10 is a block diagram showing an electronic device according to an embodiment of the present invention. Fig.10 The electronic device 100 includes all or part of a storage unit 110 , an input unit 120 , an output unit 130 , an augmented reality processing unit 160 , a control unit 170 , a communication unit 180 , a sensing unit 190 , and a power supply unit 195 .

[0149] Among them, the electronic device 100 can be embodied by a variety of devices such as a smart phone, a tablet computer, a laptop computer, a personal digital assistant (PDA), a portable multimedia player (PMP), smart glass, engineering glass, navigation, a car dash cam and a car video recorder as a vehicle image shooting device that provide driving-related guidance to the driver of the vehicle, and can be installed in the vehicle.

[0150] Driving-related guidance may include various guidance such as route guidance, lane departure guidance, lane maintaining guidance, front vehicle departure guidance, signal light change guidance, front vehicle collision prevention guidance, lane change guidance, lane guidance, curve guidance, etc. to assist vehicle drivers in driving.

[0151] Among them, route guidance may include: augmented reality route guidance, which performs route guidance by combining the image in front of the moving vehicle with various information such as the user's location and direction; and two-dimensional or three-dimensional route guidance, which performs route guidance by combining two-dimensional (2D, 2-Dimensional) or three-dimensional (3D, 3-Dimensional) map data with various information such as the user's location and direction.

[0152] Furthermore, the route guidance may include an aerial map route guidance that performs route guidance by combining the aerial map data with various information such as the user's location, direction, etc. The route guidance includes route guidance when the user is driving in a vehicle and when the user is walking or running.

[0153] In addition, lane marking deviation guidance can be used to guide whether a moving vehicle deviates from a lane marking.

[0154] In addition, lane maintenance guidance can be used to guide the vehicle back to the lane it was originally traveling in.

[0155] Furthermore, the front vehicle departure guidance may be used to guide whether the vehicle in front of the parked vehicle should depart.

[0156] And, the signal light change guidance can be used to guide whether the signal of the signal light in front of the vehicle in the parking state is changed. As an example, if the red signal light representing the stop signal is turned on and then becomes the green signal light representing the departure signal, it is guided.

[0157] Furthermore, the front vehicle collision prevention guidance provides guidance to prevent a collision with the front vehicle if the distance to the vehicle in front of the parked or moving vehicle is within a predetermined distance.

[0158] The lane change guidance is to guide the vehicle to change its lane from the lane it is in to another lane in order to guide the route to the destination.

[0159] And, lane guidance can be used to guide the lane where the current vehicle is located.

[0160] Furthermore, the curve guidance may be used to guide the vehicle to a curve on a road after a predetermined time.

[0161] As described above, driving-related images such as the front image of the vehicle that can provide various guidance can be captured by a camera disposed on the vehicle or a camera of a smart phone. The camera can be a camera that is formed as a whole with the electronic device 100 disposed on the vehicle to capture the front of the vehicle.

[0162] As another example, the camera may be a camera that is separately installed in the vehicle with the electronic device 100 to shoot the front of the vehicle. In this case, the camera may be a separate vehicle image shooting device that is installed toward the front of the vehicle, and the electronic device 100 may receive the captured image through wired or wireless communication with the separate vehicle image shooting device, or if a storage medium for storing the captured image of the vehicle image shooting device is inserted into the electronic device 100, the electronic device 100 may receive the captured image.

[0163] Based on the above content, the electronic device 100 according to an embodiment of the present invention is described in more detail below.

[0164] The storage unit 110 is used to store various data and applications required for the operation of the electronic device 100. In particular, the storage unit 110 can store data required for the operation of the electronic device 100, such as an operating system, a route search application, map data, and the like.

[0165] Furthermore, the storage unit 110 may store data generated by the operation of the electronic device 100 , for example, route data searched from a departure point to a destination, received images, and the like.

[0166] The storage unit 110 can be embodied by a built-in storage device such as a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electronically erasable and programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a memory card, a universal subscriber identity module (USIM), etc., and can also be embodied by a removable storage device such as a USB memory.

[0167] The input unit 120 has a function of converting a physical input from the outside of the electronic device 100 into a specific electrical signal. The input unit 120 may include all or part of the user input unit 121 and the microphone unit 123 .

[0168] The user input unit 121 can receive user input such as touch, push action, etc. The user input unit 121 can be implemented by at least one of various button forms, a touch sensor for receiving touch input, and a proximity sensor for receiving proximity motion.

[0169] The microphone unit 123 can receive the user's voice and sounds generated inside and outside the vehicle.

[0170] The output unit 130 is a device that outputs data of the electronic device 100 to the user in the form of images and / or voices. The output unit 130 may include all or part of the display unit 131 and the audio output unit 133 .

[0171] The display unit 131 is a device that outputs visually recognizable data to the user. The display unit 131 can be embodied by a display unit disposed on the front of the housing of the electronic device 100. In addition, the display unit 131 is formed as a whole with the electronic device 100 and can output visually recognizable data, such as a head up display (Head Up Display), which is separately provided with the electronic device 100 and can output visually recognizable data.

[0172] The audio output unit 133 is a device that outputs data that can be auditorily recognized by the electronic device 100. The audio output unit 133 may be implemented by a speaker that expresses the data that the electronic device 100 needs to remind the user in the form of sound.

[0173] The lane detection unit 140 can perform the function of the lane recognition unit 13. The augmented reality processing unit 160 can perform the functions of the augmented reality processing devices 10 and 20. That is, the augmented reality processing unit 160 can generate an augmented reality image that overlaps and displays the driving image captured by the camera and various objects.

[0174] On the one hand, the communication unit 180 can be used to enable the electronic device 100 to communicate with other devices. The communication unit 180 can include all or part of the location data unit 181, the wireless network unit 183, the broadcast transceiver unit 185, the mobile communication unit 186, the short-range communication unit 187, and the wired communication unit 189.

[0175] The location data unit 181 is a device for obtaining location data through a global navigation satellite system (GNSS). The global navigation satellite system is a navigation system that uses radio wave signals received from artificial satellites to calculate the location of a receiving terminal. As specific examples of global navigation satellite systems, according to their operating systems, there can be cited global positioning systems (GPS), Galileo, global orbiting navigation satellite systems (GLONASS), compass (COMPASS), Indian regional navigation satellite systems (IRNSS), quasi-zenith satellite systems (QZSS), etc. The data unit 181 of the system according to one embodiment of the present invention receives global navigation satellite system signals serving in the area where the electronic device 100 is used to obtain location data. Alternatively, in addition to the global navigation satellite system, the location data unit 181 can obtain location data through communication with a base station or an access point (AP).

[0176] The wireless network unit 183 connects to a wireless network to obtain or send data. The wireless network unit 183 can connect to the network through a variety of communication protocols defined as wireless data transmission and reception for wireless local area network (WLAN), wireless broadband (Wibro), world interoperability for microwave access (Wimax), and high speed downlink packet access (HSDPA).

[0177] The broadcast transceiver 185 transmits and receives broadcast signals through various broadcast systems. The broadcast systems that can be transmitted and received by the broadcast transceiver 185 may be terrestrial digital multimedia broadcasting (DMBT), digital multimedia broadcasting satellite (DMBS), forward link media (MediaFLO), handheld digital video broadcast (DVBH), terrestrial integrated services digital broadcast (ISDBT), etc. The broadcast signals transmitted and received by the broadcast transceiver 185 may include traffic data, life data, etc.

[0178] The mobile communication unit 186 can connect to a mobile network to perform voice and data communications according to various mobile communication standards such as the third generation (3G), the third generation partnership project (3GPP), and the long term evolution (LTE).

[0179] The short-distance communication unit 187 is a device for short-distance communication. As described above, the short-distance communication unit 187 can communicate through Bluetooth (Bluetooth), Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra WideBand (UWB), ZigBee, Near Field Communication (NFC), Wireless-Fidelity (Wi-Fi), etc.

[0180] The wired communication unit 189 is an interface device that can connect the electronic device 100 to other devices by wire. The wired communication unit 189 can be a USB module that communicates through a USB port.

[0181] As described above, the communication unit 180 can communicate with other devices using at least one of the location data unit 181, the wireless network unit 183, the broadcast transceiver unit 185, the mobile communication unit 186, the short-range communication unit 187, and the wired communication unit 189.

[0182] As an example, when the electronic device 100 does not include a camera function, at least one of the short-range communication unit 187 and the wired communication unit 189 can be used to receive images captured by a vehicle imaging device such as a car dash cam or a car video recorder.

[0183] As another example, when communicating with a plurality of devices, one may communicate via the short-range communication unit 187 , and another may communicate via the wired communication unit 119 .

[0184] The sensing portion 190 is a device that can sense the current state of the electronic device 100. The sensing portion 190 may include all or part of the motion sensing portion 191 and the light sensing portion 193.

[0185] The motion sensing unit 191 can sense the motion of the electronic device 100 in three-dimensional space. The motion sensing unit 191 can include a three-axis geomagnetic sensor and a three-axis acceleration sensor. The motion data obtained by the motion sensing unit 191 is combined with the position data obtained by the position data unit 181 to more accurately calculate the trajectory of the vehicle to which the electronic device 100 is attached.

[0186] The light sensor 193 is a device for measuring the ambient illuminance of the electronic device 100. The illuminance data acquired by the light sensor 193 can be used to change the brightness of the display 131 to correspond to the ambient brightness.

[0187] The power supply unit 195 is a device that supplies power required for the operation of the electronic device 100 or the operation of other devices connected to the electronic device 100. The power supply unit 195 may be a device that receives power from a battery built into the electronic device 100 or an external power supply such as a vehicle. In addition, the power supply unit 195 is embodied by the wired communication module 119 or by a wireless receiving device according to the form of receiving power.

[0188] The control unit 170 controls the overall operation of the electronic device 100. Specifically, the control unit 170 may control all or part of the storage unit 110, the input unit 120, the output unit 130, the augmented reality processing unit 160, the communication unit 180, the sensing unit 190, and the power supply unit 195.

[0189] That is, the control unit 170 can control the augmented reality processing unit 160 based on predetermined conditions (for example, the vehicle's driving environment and / or user input through the input unit 120, etc.) to display an augmented reality image that meets the predetermined conditions on the display unit 131.

[0190] As an example, when a user inputs a route guidance request to a destination through the input unit 120, the control unit 170 can control the augmented reality processing unit 160 to generate an augmented reality image in which the captured driving image and the route guidance object are overlapped, and can control the display unit 131 to display the generated augmented reality image.

[0191] Fig.11 FIG. 2 is a diagram for explaining a system network connected to an electronic device according to an embodiment of the present invention. Fig.11 According to an embodiment of the present invention, the electronic device 100 can be embodied by various devices installed in the vehicle, such as a navigation device, a vehicle image capturing device, a smart phone or other vehicle augmented reality interface providing device, and can be connected to a variety of communication networks and other electronic devices 61, 62, 63, 64.

[0192] Furthermore, the electronic device 100 may be linked with the GPG module according to the radio wave signal received from the artificial satellite 20 to calculate the current location and the current time period.

[0193] Each artificial satellite 20 can transmit L-band frequencies of different frequency bands. The electronic device 100 calculates the current position based on the time required for the L-band frequencies transmitted from each artificial satellite 20 to reach the electronic device 100.

[0194] On the other hand, the electronic device 100 can be wirelessly connected to the network 30 through the communication unit 180, the control station 50 (ACR), the base station 50 (RAS), the access point (AP, Access Point), etc. If the electronic device 100 is connected to the network 30, it is also indirectly connected to other electronic devices 61 and 62 connected to the network 30 to exchange data.

[0195] On the other hand, the electronic device 100 can be indirectly connected to the network 30 through other devices 63 with communication functions. For example, if the electronic device 100 does not have a module that can connect to the network 30, it can communicate with other devices 63 with communication functions through a short-range communication module or the like.

[0196] Below, refer to Figure 12 to Figure 13 The augmented reality processing method according to an embodiment of the present invention is described in more detail.

[0197] Fig.12 FIG. 1 is a flow chart showing a method for generating an augmented reality image according to an embodiment of the present invention. Fig.12 , a driving image captured by a camera of the vehicle while driving can be acquired (S110). Specifically, the acquisition step (S110) can acquire in real time a driving image captured by a camera set on the vehicle during the driving process of the vehicle.

[0198] Then, route data to the destination of the vehicle may be acquired (S120). Specifically, the acquisition step (S120) may acquire route data to the destination of the vehicle from map data. Here, the route data may include link data and node data for representing the route to the destination of the vehicle, and the link data may be composed of link linear data and link attribute data.

[0199] Then, the lane lines on both sides of the lane in which the vehicle is traveling can be identified from the acquired driving image (S130). Specifically, the identification step (S130) may include: a step of detecting an edge corresponding to the lane line in the acquired driving image; a step of generating an edge image based on the detected edge; a step of detecting a straight line component based on the detected edge; a step of generating an edge straight line processed image based on the detected straight line component; a step of detecting lane line points corresponding to the lane line using the generated edge image and edge straight line processed image; and a step of identifying the lane line according to the detected lane line points. That is, in the identification step (S130), by respectively detecting the lane line points corresponding to the left lane line and the lane line points corresponding to the right lane line, the lane lines on both sides of the vehicle traveling can be identified.

[0200] In addition, for the area where lane lines on both sides are identified in the driving image, first route guidance linear data can be generated based on the identified lane lines on both sides (S140). Specifically, in the step of generating the first route guidance linear data, a virtual point located in the middle can be determined based on the detected multiple points constituting the left lane line and the multiple points constituting the right lane line, and the virtual points can be connected to generate the first route guidance linear data.

[0201] In addition, for the area where the lane lines on both sides are not identified in the driving image, the link linear data of the route data can be used to generate the second route guidance linear data (S150). Specifically, in the step of generating the second route guidance linear data (S150), the link linear data corresponding to the area where the lane lines on both sides are not identified can be detected from the multiple link linear data based on the current position of the vehicle, and the second route guidance linear data is generated based on the detected link linear data.

[0202] Here, the step of generating the second route guide linear data (S150) may include the step of correcting the position error and the angle error.

[0203] Specifically, the step of generating the second route guidance linear data (S150) may include: a step of calculating the movement values ​​of multiple points constituting the link linear data based on the position of the first point constituting the first route guidance linear data; and a step of moving the multiple points constituting the link linear data based on the calculated movement values.

[0204] In addition, the step of generating second route guidance linear data (S150) may also include: a step of calculating the rotation angle of the link linear data based on the angle difference between the vehicle's driving direction vector and the vehicle's route direction vector; and a step of rotating the link linear data generated by movement based on the calculated rotation angle.

[0205] According to the step of generating the second route guide linear data (S150), the second route guide linear data in which the position error and the angle error have been corrected can be generated.

[0206] On the other hand, the method may further include: when only one lane line is identified in the lane line identification step (S130), a step of generating a virtual lane line for the other lane line based on the lane line width data; and a step of determining a virtual point located in the middle based on the identified lane line on one side and the virtually generated virtual lane line on the other side, and generating route guidance linear data by connecting the virtual points.

[0207] Furthermore, combined route guide linear data may be generated by combining the first route guide linear data and the second route guide linear data ( S160 ).

[0208] Furthermore, an augmented reality image mapping a route guidance object may be generated based on the generated combined route guidance linear data (S170). Here, the route guidance object may include a first route guidance object in an arrow shape, a second route guidance object in a carpet shape, and a third route guidance object. Fig.13 The step of generating the augmented reality image ( S170 ) will be described in more detail.

[0209] In addition, the electronic device 100 may also execute the step of displaying the generated augmented reality image.

[0210] Fig.13 Detailed description of the process of generating linear data according to an embodiment of the present invention. Fig.13 , the position of the combined route guidance linear data generated in the generating step S160 in the virtual 3D space, the position of the left lane line in the virtual 3D space, and the position of the right lane line in the virtual 3D space can be calculated (S210).

[0211] And, a mapping position of the first route guide object may be determined such that a center of the first route guide object is located at a position combined with the route guide linear data in the virtual 3D space ( S220 ).

[0212] Also, a mapping position of the second route guide object may be determined between the position of the left lane line in the virtual 3D space and the position of the first route guide object in the virtual 3D space ( S230 ).

[0213] Also, a mapping position of the third route guide object may be determined between the position of the right lane line in the virtual 3D space and the position of the first route guide object in the virtual 3D space ( S240 ).

[0214] Also, virtual 3D space data mapped to each of the determined mapping positions of the first route guide object, the second route guide object, and the third route guide object may be generated ( S250 ).

[0215] Furthermore, the virtual 3D space data of the route guidance object can be converted into a 2D image using the camera matrix to generate an augmented reality image (S260). Therefore, the camera image and the route guidance object can be displayed in an overlapping manner in the augmented reality image.

[0216] In addition, the augmented reality image generation method according to an embodiment of the present invention may further include a step of curving at least one of the lane lines on both sides, the first route guidance linear data, and the second route guidance linear data identified by the lane line recognition unit 13 .

[0217] In addition, the augmented reality image generation method according to an embodiment of the present invention may further include a step of filtering the spatial coordinates in the 3D space calculated during the operation of the device.

[0218] Fig.14 2 is a diagram showing an augmented reality image displayed by an electronic device when performing route guidance to a destination. Fig.14 The display unit 131 of the electronic device 100 can display an augmented reality image 1300 of a driving image 1301 and route guidance objects 1302 , 1303 , and 1304 captured by an overlapping camera.

[0219] The augmented reality image 1300 is an augmented reality image at a first time point, and the augmented reality image 1320 is an augmented reality image at a second time point after a predetermined time has passed from the first time point.

[0220] The augmented reality images 1300 , 1320 may include an image of a preceding vehicle traveling in front of the vehicle.

[0221] In addition, the route guidance object may include: a first route guidance object 1302 displayed in an arrow shape on the lane where the vehicle is traveling, a second route guidance object 1303 displayed in a rectangle on the lane where the vehicle is traveling, and a third route guidance object 1304 displayed in a rectangle on the lane where the vehicle is traveling.

[0222] Here, the first route guidance object 1302 may be arranged centered on the combined route guidance linear data generated by the route guidance linear data generating unit 14 and displayed on the augmented reality image 1300. In addition, the second route guidance object 1303 may be arranged between the left side of the first route guidance object 1302 and the left lane line and displayed through augmented reality, and the third route guidance object 1304 may be arranged between the right side of the first route guidance object 1302 and the right lane line and displayed through augmented reality.

[0223] Here, the second route guide object 1303 and the third route guide object 1304 may gradually have transparency from the outside to the inside.

[0224] In addition, when the vehicle is moving forward, the route guidance objects 1302, 1303, 1304 can be displayed by moving from front to back. That is, the first route guidance object 1310-1 displayed in front of the vehicle in the augmented reality image 1300 at a first time point is displayed closer to the vehicle in the augmented reality image 1320 at a second time point when the vehicle is moving forward for a predetermined time (1310-1). In addition, the moving speed of the route guidance objects 1302, 1303, 1304 can be changed according to the driving speed of the vehicle.

[0225] Route guidance objects 1302, 1303, 1304 may be displayed in blue, and the color may change or disappear under a predetermined condition. For example, the predetermined condition for the color change of route guidance objects 1302, 1303, 1304 may be a condition that the vehicle deviates from the lane line, and the predetermined condition for the disappearance of route guidance objects 1302, 1303, 1304 may be a condition that the vehicle approaches a curve.

[0226] According to the present invention, the first route guidance object 1302 in the center of the lane is moved by animation processing according to the vehicle speed, so that the object is naturally and dynamically expressed as if drawn on the lane.

[0227] Fig.15 14 is a diagram showing an augmented reality image displayed by an electronic device when a vehicle deviates from a lane line. Augmented reality image 1410 is an image displayed on electronic device 100 when the distance to a preceding vehicle located in front of the vehicle decreases to within a predetermined distance during the execution of route guidance, and augmented reality image 1420 is an image displayed on electronic device 100 when the vehicle deviates from the lane line after a predetermined time of image 1410.

[0228] The display unit 131 of the electronic device 100 may display augmented reality images 1410 and 1420 that overlap the driving image 1401 captured by the camera and the route guidance objects 1402 , 1403 , and 1404 .

[0229] First, during the process of executing route guidance, when the distance to the front vehicle located in front of the vehicle decreases to within a predetermined distance, the electronic device 100 may display an augmented reality image 1410 including route guidance objects 1402, 1403, 1404 and a front vehicle collision prevention guidance object 1405. Here, the front vehicle collision prevention guidance object 1405 may be displayed in the image area of ​​the front vehicle, and the collision time with the front vehicle may be displayed.

[0230] Then, when the vehicle deviates from the lane line, the electronic device 100 may change the display state of at least one of the second route guidance object 1403 and the third route guidance object 1404 and display it. Specifically, the electronic device 100 changes the color of the second route guidance object 1403 and the third route guidance object 1404 from blue to red, and the third route guidance object 1404 corresponding to the lane line deviated by the vehicle may be displayed in a flashing state. The third route guidance object 1404 is not displayed in the augmented reality image 1420 because it is an image captured in a flashing state.

[0231] Additionally, in this case, the color of the first route guide object 1402 may remain blue.

[0232] In addition, the electronic device 100 may display an augmented reality image 1420 including a lane line deviation guide object 1406 corresponding to the lane line from which the vehicle deviates.

[0233] Figure 16 to Figure 17 FIG. 2 is a diagram showing an augmented reality image displayed by an electronic device when a vehicle approaches a curve. Figure 16 to Figure 17 When the vehicle approaches a curve location where the driving direction needs to be changed (U-turn, left turn, right turn, etc.), the electronic device 100 can delete the route guidance object and display an augmented reality image containing only the curve location guidance object. Here, the curve location guidance object may include a first curve location guidance object 1501 that guides the driving direction of the curve location with vertically arranged arrows at the curve location; and a second curve location guidance object 1502 that guides the driving direction of the curve location with an up and down jumping animation.

[0234] For example, when the vehicle approaches a U-turn location, the electronic device 100 may display an augmented reality image 1510. In addition, when the vehicle approaches a left turn location, the electronic device 100 may display an augmented reality image 1520, 1530, or 1540 according to the left turn direction. In addition, when the vehicle approaches a right turn location, the electronic device 100 may display an augmented reality image 1610, 1620, or 1630 according to the left turn direction.

[0235] According to the present invention, when approaching a curve point, the user is allowed to clearly recognize curve point information by deleting the route guidance object and displaying the curve point guidance object.

[0236] Fig.18 FIG. 2 is a diagram showing an embodiment of an electronic device according to an embodiment of the present invention without a camera unit. Fig.18 The vehicle image recording device 200 which is separately provided with the vehicle electronic device 100 can constitute a system according to an embodiment of the present invention through wired or wireless communication.

[0237] The vehicle electronic device 100 may include a display unit 131 , operation keys 121 , and a microphone 123 provided on the front of a housing 191 .

[0238] The vehicle image recording device 200 may include an image acquisition device, namely a camera 222 , a microphone 224 , and an attachment portion 281 .

[0239] Fig.19 FIG. 2 is a diagram showing an embodiment of an electronic device including a camera unit according to an embodiment of the present invention. Fig.19When the electronic device 100 includes the photographing unit 150, the photographing unit 150 of the electronic device 100 photographs the front of the vehicle and allows the user to recognize the display portion of the electronic device 100. Thus, a system according to an embodiment of the present invention can be embodied.

[0240] Fig. 20 FIG. 1 is a diagram showing an embodiment of a head-up display (HUD) according to an embodiment of the present invention. Fig. 20 , the head-up display can display an augmented reality guidance screen on the head-up display through wired and wireless communication with other devices.

[0241] As an example, augmented reality can be provided by using a head-up display on the front glass of the vehicle or by using an image overlap of an additional image output device, and as described above, the augmented reality processing unit 160 can generate an interface image overlapped on the augmented reality image or glass, etc. Thus, augmented reality navigation or vehicle infotainment system, etc. can be embodied.

[0242] According to the various embodiments of the present invention described above, a route guidance object is displayed based on the result of identifying lane markings on both sides of a vehicle's driving lane (Lane), and the route guidance object is displayed using route data stored in a storage unit at long distances when the lane markings are erased or unclear, thereby allowing the route guidance object to be more accurately displayed in augmented reality.

[0243] Also, according to various embodiments of the present invention, a dynamic route guidance object that the driver can intuitively recognize may be represented on the augmented display.

[0244] Moreover, according to various embodiments of the present invention, when a lane line deviation occurs while a vehicle is traveling, a route guidance object and a lane line deviation guidance object may be intuitively displayed on an enhanced display.

[0245] In addition, at least one of the modules constituting the augmented reality processing device 10 and the electronic device 100 according to the present invention may be implemented as a module of a system for automatic driving to perform a navigation function. Figure 21 to Figure 22 This is explained in more detail.

[0246] Reference Fig.21 According to this embodiment, the autonomous driving car 2000 may include a control device 2100, sensing modules 2004a, 2004b, 2004c, 2004d, an engine 2006 and a user interface 2008.

[0247] The self-driving car 2000 may have an automatic driving mode or a manual mode. As an example, according to a user input received through the user interface 2008, the manual mode may be switched to the automatic driving mode, or the automatic driving mode may be switched to the manual mode.

[0248] When the vehicle 2000 is operating in the automatic driving mode, the self-driving car 2000 can operate under the control of the control device 2100.

[0249] In this embodiment, the control device 2100 may include a controller 2120 having a memory 2122 and a processor 2124 , a sensor 2110 , a communication device 2130 , and an object detection device 2140 .

[0250] In the present embodiment, the object detection device 2140 is a device for detecting an object located outside the vehicle 2000. The object detection device 2140 detects the object located outside the vehicle 2000 and can generate object information according to the detection result.

[0251] The object information may include information on the presence or absence of the object, position information of the object, distance information between the vehicle and the object, and relative speed information between the vehicle and the object.

[0252] The object may include various objects located outside the vehicle 2000, such as lane lines, other vehicles, pedestrians, traffic signals, lights, roads, structures, speed bumps, terrain objects, and animals. Here, the traffic signal may be a concept including a traffic light, a traffic sign, and a pattern or text drawn on the road surface. In addition, the light may be light generated from lights of other vehicles, light generated from street lamps, or sunlight.

[0253] In addition, structures can be objects located around the road and fixed on the ground. For example, structures can include street lights, street trees, buildings, telephone poles, traffic lights, and bridges. Terrain objects can include mountains, hillsides, etc.

[0254] The object detecting device 2140 may include a camera module. The controller 2120 may extract object information from an external image captured by the camera module and cause the controller 2120 to process the information.

[0255] In addition, the object detection device 2140 may also include an imaging device for identifying the external environment. In addition to LIDAR, RADAR, GPS equipment, odometer (Odometry) and other computer vision equipment, ultrasonic sensors, infrared sensors may also be used, and these devices may be selected or started simultaneously as needed for more precise detection.

[0256] In addition, the sensor 2110 can be connected to the sensing modules 2004a, 2004b, 2004c, and 2004d to obtain various sensing information of the internal / external environment of the vehicle. Among them, the sensor 2110 may include a posture sensor (for example, a yaw sensor, a roll sensor, a pitch sensor, a collision sensor, a wheel sensor, a speed sensor, a tilt sensor, a weight sensor, a heading sensor, a gyro sensor, a positioning module, a vehicle forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor rotated by a steering wheel, a vehicle internal temperature sensor, a vehicle internal humidity sensor, an ultrasonic sensor, an illumination sensor, an accelerator pedal position sensor, a brake pedal position sensor, etc.

[0257] Thus, sensor 2110 can obtain sensing signals of vehicle posture information, vehicle collision information, vehicle direction information, vehicle position information (GPS information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle tilt information, vehicle forward / backward information, battery information, fuel information, tire information, headlight information, in-vehicle temperature information, in-vehicle humidity information, steering wheel rotation angle, outside vehicle illumination, pressure applied to the accelerator pedal, pressure applied to the brake pedal, and the like.

[0258] In addition, sensor 2110 may also include an accelerator pedal sensor, a pressure sensor, an engine speed sensor, an air flow sensor (AFS), an intake air temperature sensor (ATS), a water temperature sensor (WTS), a throttle position sensor (TPS), a TDC sensor, a crank angle sensor (CAS), etc.

[0259] In this way, the sensor 2110 can generate vehicle status information based on the sensing data.

[0260] The wireless communication device 2130 is used to implement wireless communication between the autonomous driving vehicles 2000. For example, the user's mobile phone, other wireless communication devices 2130, other vehicles, central devices (traffic control devices), servers, etc. can communicate with the autonomous driving vehicle 2000. The wireless communication device 2130 can send and receive wireless signals according to the connection wireless protocol. The wireless communication protocol can be Wireless Fidelity (Wi-Fi), Bluetooth (Bluetooth), Long-Term Evolution (Long-Term Evolution, LTE), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (WCDMA), Global Systems for Mobile Communications (Global Systems for Mobile Communications, GSM), but the communication protocol is not limited to this.

[0261] Furthermore, in this embodiment, the autonomous driving car 2000 can realize communication between vehicles through the wireless communication device 2130. That is, the wireless communication device 2130 can communicate with other vehicles on the road and multiple other vehicles through vehicle-to-vehicle (V2V) communication. The autonomous driving car 2000 can send and receive information such as driving warnings and traffic information through communication between vehicles, and can also request information from other vehicles or receive information from other vehicles. For example, the wireless communication device 2130 can perform vehicle-to-vehicle communication through a dedicated short-range communication (DSRC) device or a cellular vehicle-to-vehicle communication (C-V2V) device. In addition, in addition to communication between vehicles, communication (V2X, Vehicle to Everything communication) between vehicles and other things (for example, electronic devices carried by pedestrians, etc.) can also be performed through the wireless communication device 2130.

[0262] In this embodiment, the controller 2120 is a unit that controls the overall operation of each unit in the vehicle 2000, and is manufactured by the vehicle manufacturing company or is additionally configured after manufacturing in order to perform the function of automatic driving. Alternatively, it may include a structure that continuously performs additional functions by updating the controller 2120 configured during manufacturing. The controller 2120 may also be named ECU (Electronic Control Unit).

[0263] The controller 2120 collects various data from the connected sensors 2110, the object detection device 2140, the communication device 2130, etc., and can transmit control signals based on the collected data to the sensors 2110, the engine 2006, the user interface 2008, the communication device 2130, and the object detection device 2140 included as other structures in the vehicle. In addition, although not shown in the figure, control signals can also be transmitted to the acceleration device, the braking system, the steering device, or the navigation device related to the driving of the vehicle.

[0264] In this embodiment, the controller 2120 can control the engine 2006. For example, the autonomous driving car 2000 senses the speed limit of the road on which it is traveling and controls the engine 2006 to prevent the driving speed from exceeding the speed limit, and can control the engine 2006 to accelerate the driving speed of the autonomous driving car 2000 within a range that does not exceed the speed limit.

[0265] In addition, when the autonomous driving car 2000 approaches or deviates from the lane line while the autonomous driving car 2000 is traveling, the controller 2120 can determine whether the approach and deviation of the lane line are caused by normal driving conditions or other driving conditions, and can control the engine 2006 to control the driving of the vehicle according to the judgment result. Specifically, the autonomous driving car 2000 can be equipped with a lane line detection device 10 according to an embodiment of the present invention, and the lane line detection device 10 can detect the lane lines formed on both sides of the lane in which the vehicle is traveling. In this case, the controller 2120 determines whether the autonomous driving car 2000 approaches or deviates from the lane line. If it is determined that the autonomous driving car 2000 approaches or deviates from the lane line, it can be determined whether it is caused by normal driving conditions or other driving conditions. Among them, as an example of a normal driving condition, there may be a situation where the vehicle needs to change lanes. In addition, as an example of other driving conditions, there may be a situation where the vehicle does not need to change lanes. In the case where the vehicle does not need to change lanes, if the controller 2120 determines that it is approaching or deviating from the lane line, it can control the driving of the autonomous driving car 2000 to drive normally without deviating from the lane line.

[0266] The lane line detection device 10 may be constituted by a module in the control device 2100 of the autonomous driving vehicle 2000. That is, the memory 2122 and the processor 2124 of the control device 2100 may implement the lane line detection method according to the present invention through software.

[0267] In the case where there are other vehicles or obstacles in front of the vehicle, the controller 2120 can control the engine 2106 or the brake system to slow down the vehicle. In addition to the speed, the track, the route, and the steering angle can also be controlled. The controller 2120 can generate the required control signal and control the vehicle's travel based on the recognition information of the vehicle's driving lane, driving signals, and other external environments.

[0268] In addition to generating the self-control signal, the controller 2120 communicates with surrounding vehicles or a central server and transmits instructions for controlling surrounding devices through the received information, thereby controlling the driving of the vehicle.

[0269] Furthermore, when the position of the camera module 2150 is changed or the shooting angle is changed, it is difficult for the controller 2120 to accurately identify the vehicle or the lane line according to the present embodiment, and therefore, in order to prevent the above phenomenon, a control signal is generated to perform calibration of the camera module 2150. Therefore, in the present embodiment, the controller 2120 generates a calibration control signal through the camera module 2150, thereby, even if the installation position of the camera module 2150 is changed due to vibration or impact, etc., which occurs with the movement of the autonomous driving car 2000, the normal installation position, direction, shooting angle, etc. of the camera module 2150 can be continuously maintained. When the initial installation position, direction, shooting angle information of the camera module 2120 stored in advance and the initial installation position, direction, shooting angle information of the camera module 2120 measured during the driving of the autonomous driving car 2000 become above a critical value, the controller 2120 can generate a control signal to perform calibration of the camera module 2120.

[0270] In this embodiment, the controller 2120 may include a memory 2122 and a processor 2124. The processor 2124 may execute software stored in the memory 2122 according to a control signal of the controller 2120. Specifically, the controller 2120 stores data and instructions for executing the lane line detection method according to the present invention in the memory 2122. In order to implement one or more methods described herein, the instructions may be executed by the processor 2124.

[0271] Here, the memory 2122 may be stored in a recording medium that can be executed in the non-volatile processor 2124. The memory 2122 may store software and data through appropriate internal and external devices. The memory 2122 may be composed of a random access memory (RAM), a read only memory (ROM), a hard disk, and a memory 2122 device connected to a software protector.

[0272] The memory 2122 can at least store an operating system (OS), user applications, and executable instructions. The memory 2122 can also store application data and arrange data structures.

[0273] Processor 2124 is a microprocessor or a suitable electronic processor, and may also be a controller, a microcontroller or a state machine.

[0274] The processor 2124 may be embodied by a combination of computing devices, which may be constituted by a digital signal processor, a microprocessor, or a suitable combination of these.

[0275] In addition, the autonomous driving vehicle 2000 may further include a user interface 2008 for the user to input information to the control device 2100. The user interface 2008 may allow the user to input information through appropriate interactions. For example, it may be embodied by a touch screen, a keyboard, an operation button, etc. The user interface 2008 transmits the input or instruction to the controller 2120, and the controller 2120 may execute a control action of the vehicle in response to the input or instruction.

[0276] Furthermore, the user interface 2008, as an external device of the autonomous driving vehicle 2000, can communicate with the autonomous driving vehicle 2000 via the wireless communication device 2130. For example, the user interface 2008 can establish linkage with a mobile phone, a tablet computer, or other computer devices.

[0277] Furthermore, in the present embodiment, the case where the autonomous driving vehicle 2000 includes the engine 2006 is described, but other types of propulsion systems may also be included. For example, the vehicle may be operated by electric energy, hydrogen energy, or a hybrid system combining these. Therefore, the controller 2120 includes a propulsion mechanism based on the propulsion system of the autonomous driving vehicle 2000, and may provide the control signal to the structure of each propulsion mechanism.

[0278] Below, refer to Fig. 22 The detailed structure of the control device 2100 that executes the lane line detection method according to this embodiment is described in more detail.

[0279] The control device 2100 includes a processor 2124. The processor 2124 may be a common single or multi-core microprocessor, a dedicated microprocessor, a microcontroller, a programmable gate array, etc. The processor may be referred to as a central processing unit (CPU). Moreover, in this embodiment, the processor 2124 may also be used in combination with multiple processors.

[0280] The control device 2100 further includes a memory 2122. The memory 2122 may be any electronic component that stores electronic information. In addition to a single memory, the memory 2122 may also include a combination of multiple memories 2122.

[0281] Data and instruction language 2122a for executing the lane line detection method according to the present invention may also be stored in the memory 2122. When the processor 2124 executes a plurality of instruction language 2122a, all or part of the instruction language 2122a and the data 2122b required for executing the instruction may be loaded into 2124a, 2124b.

[0282] The control device 2100 may further include a transmitter 2130a, a receiver 2130b or a transceiver 2130c for transmitting and receiving signals. One or more antennas 2132a, 2132b may be electrically connected to the transmitter 2130a, the receiver 2130b or each transceiver 2130c, and may further include multiple antennas.

[0283] The control device 2100 may further include a digital signal processor (DSP) 2170 , so that the vehicle may process digital signals more quickly through the digital signal processor 2170 .

[0284] The control device 2100 may further include a communication interface 2180. The communication interface 2180 may further include one or more ports and / or communication modules for connecting other devices to the control device 2100. The communication interface 2180 may enable a user to interact with the control device 2100.

[0285] The various structures of the control device 2100 may be connected via one or more buses 2190 , and the bus 2190 may also include a power bus, a control signal bus, a status signal bus, a data bus, etc. Under the control of the processor 2124 , the multiple structures may transmit information to each other via the bus 2190 and perform the required functions.

[0286] On the one hand, the terms "first", "second", "third" and "fourth" in the specification and claims are used, if recorded, to distinguish between similar components, but are not used to describe a specific order or order of occurrence. The terms used together with them should be understood to be interchangeable under appropriate circumstances so that the embodiments of the present invention recorded herein can be performed in other orders not illustrated or described herein. Similarly, when it is recorded herein that the method includes a series of steps, the order of the steps disclosed herein is not necessarily the order in which the steps are performed, and any recorded steps can be omitted and / or any other steps not recorded herein can be added to the method. For example, without departing from the scope of the present invention, the first component can be named as the second component, and similarly, the second component can be named as the first component.

[0287] In addition, the terms "left side", "right side", "front", "rear", "top", "bottom", "above", "below", etc. in the specification and claims are used for explanation purposes and do not necessarily describe unchanging relative positions. The terms used therewith should be understood to be interchangeable under appropriate circumstances so that the embodiments of the present invention described herein can, for example, move in other directions not illustrated or described herein. The term "connected" used herein is defined as being directly or indirectly connected by electrical or non-electrical means. Objects described herein as being "adjacent" to each other may appropriately be objects that are in physical contact with each other or adjacent to each other or in the same general range or area as appropriate, depending on the article in which the sentence is used. The sentences "in one embodiment" present herein refer to the same embodiment, but are not certain.

[0288] In addition, various variations of expressions such as "connected", "connected", "coupled", "coupled", "combined", and "combined" in the specification and claims are used to mean direct connection with other components or indirect connection through other components.

[0289] In contrast, when it is mentioned that a certain structural element is “directly connected to” or “directly coupled to” another structural element, it should be understood that there are no other structural elements intervening.

[0290] In addition, the suffixes "module" and "unit" of the constituent elements used in this specification are given or used interchangeably only for the convenience of writing the specification, and they themselves do not have mutually distinguishing meanings or functions.

[0291] In addition, the terms used in this specification are used to illustrate the embodiments, rather than to limit the present invention. In this specification, when not specifically mentioned in the text, the singular includes the plural. The present invention shall not be interpreted as "constituting" or "comprising" and other terms must include all the constituent elements or steps mentioned in the specification, and may not include some of the constituent elements or steps, or may also include other additional constituent elements or steps.

[0292] The present invention has been described above with its preferred embodiments as the center. All embodiments and conditional examples disclosed in this specification are recorded by those skilled in the art to facilitate readers to understand the principles and concepts of the present invention. Those skilled in the art should understand that the present invention can be embodied in a deformed form within the scope of the essential characteristics of the present invention.

[0293] Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. Furthermore, the scope of the present invention should be interpreted as being described in the claims rather than in the description, and all differences within the same scope are included in the present invention.

[0294] On the one hand, the above methods according to various embodiments of the present invention can be embodied by a program and provided to a server or device. Thus, each device is connected to a server or device storing the program, so that the program can be downloaded.

[0295] In addition, the control methods according to various embodiments of the present invention described above can be embodied by programs and stored in various non-transitory computer readable mediums. Non-transitory computer readable mediums refer to media that store data semi-permanently and can be read by a device rather than media that store data in a short period of time such as registers, caches, and memories. Specifically, the various applications or programs described above can be stored in non-transitory readable media such as CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, ROMs, etc.

[0296] The above diagrams illustrate the preferred embodiments of the present invention. However, the present invention is not limited to the above-mentioned specific embodiments. Those skilled in the art can make various modified implementations without departing from the gist of the present invention as claimed in the claims. Moreover, these modified implementations should not be understood separately from the technical ideas or prospects of the present invention.

Claims

1. A route guidance method based on augmented reality, which is a route guidance method based on augmented reality using a processor, characterized in that: include: The step of acquiring a driving image captured by a camera of the vehicle while driving; The step of obtaining route data to the destination of the vehicle; A step of identifying lane lines on both sides of the lane in which the vehicle is traveling from the acquired driving image; For an area where lane lines on both sides are identified in the driving image, a step of generating first route guidance linear data based on the identified lane lines on both sides; For an area where lane lines on both sides are not identified in the driving image, a step of using link linear data of the route data to generate second route guidance linear data; a step of combining the first route guidance linear data and the second route guidance linear data to generate combined route guidance linear data; as well as The step of displaying a route guidance object on augmented reality using the generated combined route guidance linear data.

2. The route guidance method based on augmented reality according to claim 1, characterized in that: The step of generating the second route guidance linear data comprises: A step of calculating movement values ​​of a plurality of points constituting the link linear data based on the position of a first point constituting the first route guidance linear data; and A step of moving a plurality of points constituting the link linear data based on the calculated movement value.

3. The route guidance method based on augmented reality according to claim 2, characterized in that: The step of generating the second route guiding linear data further includes: A step of calculating a rotation angle of the link linear data based on an angle difference between a driving direction vector of the vehicle and a route direction vector of the vehicle; and The step of rotating the link linear data generated by the movement based on the calculated rotation angle.

4. The route guidance method based on augmented reality according to claim 1, characterized in that: The method further includes generating a virtual lane line for the lane line on the other side based on the lane line width data when only the lane line on one side is identified in the driving image.

5. The route guidance method based on augmented reality according to claim 1, characterized in that: The route guidance object includes: A first route guidance object in the shape of an arrow, having a center in the combined route guidance linear data and pointing to a route along which the vehicle should travel; a second route guidance object displayed between the left side of the first route guidance object and the left lane line; and The third route guidance object is displayed on the right side of the first route guidance object and between the right lane line.

6. The route guidance method based on augmented reality according to claim 1, characterized in that: When the vehicle is moving forward, the route guidance object is displayed by moving from front to rear, The moving speed of the route guidance object is changed according to the speed of the vehicle.

7. The route guidance method based on augmented reality according to claim 1, characterized in that: The method further includes displaying a curve location guidance object instead of displaying the route guidance object when the vehicle approaches a curve location.

8. The route guidance method based on augmented reality according to claim 5, characterized in that: The method further includes changing a display state of at least one of the second route guidance object and the third route guidance object to display the object when the vehicle deviates from a lane line.

9. An electronic device for performing route guidance based on augmented reality, characterized in that: include: A display unit, used for displaying a screen; A driving image acquisition unit, used to acquire a driving image captured by a camera of a moving vehicle; a route data acquisition unit, configured to acquire route data to a destination of the vehicle; A lane line recognition unit, used to recognize lane lines on both sides of the lane in which the vehicle is traveling from the acquired driving image; a route guidance linear data generating unit, for an area where lane lines on both sides are identified in the driving image, generating first route guidance linear data based on the identified lane lines on both sides, and for an area where lane lines on both sides are not identified in the driving image, generating second route guidance linear data using link linear data of the route data, and generating combined route guidance linear data by combining the first route guidance linear data and the second route guidance linear data; and The control unit is used to control the display unit so that the route guidance object is displayed in the augmented reality image by using the generated combined route guidance linear data.

10. The electronic device according to claim 9, characterized in that: The route guidance object includes: A first route guidance object in the shape of an arrow, having a center in the combined route guidance linear data and pointing to a route along which the vehicle should travel; a second route guidance object displayed between the left side of the first route guidance object and the left lane line; and The third route guidance object is displayed on the right side of the first route guidance object and between the right lane line.

11. The electronic device according to claim 9, characterized in that: When the vehicle moves forward, the route guidance object moves from front to back and is displayed in the augmented reality image. The moving speed of the route guidance object is changed according to the speed of the vehicle.

12. The electronic device according to claim 9, characterized in that: When the vehicle approaches a curve point, the control portion controls the display portion so that a curve point guidance object is displayed instead of the route guidance object.

13. The electronic device according to claim 10, characterized in that: When the vehicle deviates from a lane line, the control unit controls the display unit so as to change a display state of at least one of the second route guidance object and the third route guidance object and display the object.

14. An augmented reality processing device, characterized in that: include: A driving image acquisition unit, used to acquire a driving image captured by a camera of the vehicle while driving; a route data acquisition unit, configured to acquire route data to a destination of the vehicle; A lane line recognition unit, which recognizes lane lines on both sides of the lane in which the vehicle is traveling from the acquired driving image; a route guidance linear data generating unit, for an area where lane lines on both sides are identified in the driving image, generating first route guidance linear data based on the identified lane lines on both sides, for an area where lane lines on both sides are not identified in the driving image, generating second route guidance linear data using link linear data of the route data, and generating combined route guidance linear data by combining the first route guidance linear data and the second route guidance linear data; as well as The augmented reality image generating unit generates an augmented reality image that maps the route guidance object by using the generated combined route guidance linear data.

15. The augmented reality processing device according to claim 14, characterized in that: The route guidance linear data generating unit calculates movement values ​​of a plurality of points constituting the link linear data based on the position of a first point constituting the first route guidance linear data, and moves the plurality of points constituting the link linear data based on the calculated movement values.

16. The augmented reality processing device according to claim 15, characterized in that: The route guidance linear data generating unit calculates a rotation angle of the link linear data based on an angle difference between a traveling direction vector of the vehicle and a route direction vector of the vehicle, and rotates the link linear data generated by the movement based on the calculated rotation angle.

17. The augmented reality processing device according to claim 14, characterized in that: The route guidance linear data generating unit further includes a step of generating a virtual lane line for the lane line on the other side based on the lane line width data when only the lane line on one side is identified in the driving image. 18 . A program stored in a computer-readable recording medium including a program code for executing the route guidance method based on augmented reality according to claim 1 .

19. A computer-readable recording medium having recorded therein a program for executing the route guidance method based on augmented reality according to claims 1 to 8.

Images