Dynamic adjustment of augmented reality images

Abstract

A system and method for adjusting an augmented reality (AR) image, involving a forward-facing camera in a vehicle. The camera acquires an image including a road surface in front of the vehicle. The system also includes a processor for detecting lane lines of the road surface in three dimensions, performing interpolation using the lane lines to determine a ground plane of the road surface, and adjusting the AR image to obtain an adjusted AR image so that all points of the adjusted AR image correspond to points on the ground plane.

Description

Technical Field

[0001] The present invention relates to dynamic adjustment of augmented reality images. Background Art

[0002] Vehicles (such as cars, trucks, construction equipment, farm equipment) are increasingly used to obtain information about the vehicle and its surroundings, and to provide information to the occupants of the vehicle. Exemplary interfaces include infotainment systems and head-up displays (HUDs, also known as head-up displays). Infotainment systems may include displays to display maps or images obtained by a camera of the vehicle. For example, the HUD projects an image onto the windshield of the vehicle. Augmented reality (AR) images refer to images that enhance real-world images. For example, an AR image of an arrow can be used to enhance the image of the road ahead obtained by a camera at the front of the vehicle to indicate an upcoming turn. As another example, instead of displaying speed or other data, the HUD can project an AR image to enhance what the driver sees through the windshield. Road dynamics or other sources of change in the relative direction of the vehicle to the outside world may cause the AR image to be displayed in the wrong location (e.g., an arrow indicating the direction is projected on the side of the road in the camera image, not on the road). Therefore, it is desirable to provide dynamic adjustment of the AR image. Summary of the invention

[0003] In one exemplary embodiment, a system for adjusting an augmented reality (AR) image includes a front camera in a vehicle to obtain an image including a road surface in front of the vehicle. The system also includes a processor for detecting lane lines of the road surface in three dimensions, performing interpolation using the lane lines to determine a ground plane of the road surface, and adjusting the AR image to obtain an adjusted AR image so that all points of the adjusted AR image correspond to points on the ground plane.

[0004] In addition to one or more features described herein, the processor obtains a base image showing a road surface.

[0005] In addition to one or more features described herein, the processor also adds the adjusted AR image to the base image for display to the vehicle driver.

[0006] In addition to one or more features described herein, the processor obtains a base image from the camera or a second forward-facing camera in the vehicle.

[0007] In addition to one or more of the features described herein, the processor monitors the eye position of the vehicle driver.

[0008] In addition to one or more features described herein, the processor determines a head-up display area on the vehicle windshield to correspond to the driver's eye position.

[0009] In addition to one or more features described herein, the processor projects the adjusted AR image in the heads-up display area.

[0010] In addition to one or more features described herein, the processor adjusts the AR image based on a rotation and a translation of a point on a ground plane.

[0011] In addition to one or more features described herein, the processor detects from the image an object on the road directly in front of the vehicle.

[0012] In addition to one or more features described herein, the processor selects or creates an adjusted AR image size based on a road surface between the vehicle and the object locations.

[0013] In another exemplary embodiment, a method of adjusting an augmented reality (AR) image includes acquiring an image from a front-facing camera in a vehicle, the image including a road surface in front of the vehicle. The method also includes detecting lane lines of the road surface in three dimensions and performing interpolation using the lane lines to determine a ground plane of the road surface. The AR image is adjusted to obtain an adjusted AR image so that all points of the adjusted AR image correspond to points on the ground plane.

[0014] In addition to one or more features described herein, the method also includes obtaining a base image showing a road surface.

[0015] In addition to one or more features described herein, the method also includes the processor adding the adjusted AR image to the base image for display to the vehicle driver.

[0016] In addition to one or more features described herein, a method of acquiring a base image is to acquire it from a camera in the vehicle or a second front camera.

[0017] In addition to one or more features described herein, the method also includes the processor monitoring the eye position of the vehicle driver.

[0018] In addition to one or more features described herein, the method also includes the processor determining a head-up display area on the vehicle windshield to correspond to the driver's eye position.

[0019] In addition to one or more features described herein, the method also includes the processor projecting the adjusted AR image in the heads-up display area.

[0020] In addition to one or more features described herein, the adjustment of the AR image is based on a rotation and translation of a point on a ground plane.

[0021] In addition to one or more features described herein, the method also includes a processor detecting, from the image, an object on the road surface directly in front of the vehicle.

[0022] In addition to one or more features described herein, the method also includes the processor selecting or creating an adjusted AR image size based on a road surface between the vehicle and the object location.

[0023] The above features and advantages as well as other features and advantages of the present invention will become apparent from the following detailed description when taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Other features, advantages and details appear by way of example only in the following detailed description, which refers to the accompanying drawings, in which:

[0025] Figure 1 is a block diagram of a vehicle that performs dynamic adjustment of augmented reality (AR) images according to one or more embodiments;

[0026] Figure 2 illustrates adjustment of an AR image according to one or more embodiments;

[0027] Figure 3 A vehicle is shown performing dynamic adjustment of an AR image according to one or more embodiments;

[0028] Figure 4 is a process flow of a method for performing dynamic adjustment of an AR image according to various embodiments;

[0029] Figure 5 illustrates aspects of extracting road geometry to dynamically adjust AR images according to one or more embodiments;

[0030] Figure 6 The operations involved in adjusting an AR image displayed as a heads-up display are shown;

[0031] Figure 7 illustrates the operations involved in adjusting an AR image displayed on a base image; and

[0032] Figure 8 It is shown that the ground plane is divided into sub-planes to improve the adjustment efficiency of the AR image according to one or more embodiments. DETAILED DESCRIPTION

[0033] The following description is for exemplary purposes only and is not intended to limit the present invention and its application or use. It should be understood that in the accompanying drawings, corresponding reference numerals indicate similar or corresponding components and features.

[0034] As previously described, AR images can be added to real-world images (e.g., obtained by a vehicle camera and displayed by an infotainment system) or can be projected into a real-world view (e.g., in the form of a HUD). Embodiments of the systems and methods detailed herein relate to dynamic adjustment of AR images. While driving, AR images may be displayed on the road surface or other road features. As previously described, AR images may appear in the wrong position in the real-world image or real-world view due to changes in the relative direction of the vehicle with respect to the road surface. This can happen if the road ahead is uneven. Even if the road is flat, the relative direction can change when the vehicle has an inclined angle (e.g., due to a heavy load in the trunk). Changes in the vehicle's position relative to the road may also cause this problem. Finally, other vehicles or objects in the field of view may require changes in the position or other aspects of the AR symbol.

[0035] Previous approaches to solving the relative orientation problem involve creating an accurate map of the terrain in advance. Then, with precise vehicle positioning and inclination, the orientation of the vehicle relative to the road surface at a given location can be known. However, this requires driving on all possible terrains to obtain a map of the terrain. In addition, sensors are required to obtain positioning and inclination information. In addition, this approach cannot handle the presence of other vehicles or objects in real time. Another previous approach involves the use of three-dimensional sensors, such as light detection and ranging (lidar) systems. However, light detection and ranging (lidar) systems are generally not applicable to vehicles, and the processing of data obtained by light detection and ranging (lidar) systems is generally more processor-intensive than the processing of data obtained by other sensors (e.g., cameras, radar systems).

[0036] According to one or more embodiments detailed herein, a single image from a single camera is used to obtain information about the road geometry or structure of a road segment. The camera may be a front-facing camera, which is commonly provided in vehicles. Techniques for reducing lag and improving accuracy are also described. According to an exemplary embodiment, a road geometry is determined for an upcoming road segment captured by the camera. According to another exemplary embodiment, a large stretch of road may be divided into multiple segments so that the road geometry of each segment can be more accurately determined.

[0037] According to an exemplary embodiment, Figure 1 is to execute the AR image 210 ( Figure 2 ) is a block diagram of a dynamically adjusted vehicle 100. Figure 1The exemplary vehicle 100 shown is a car 101. Lane lines 150 are shown on a road 160 showing the vehicle 100. The vehicle 100 includes a controller 110 that performs dynamic adjustment of the AR image 210 as described in detail herein. The controller 110 may also perform the functions of an electronic control unit (ECU) of the vehicle 100 or communicate with the ECU. Exemplary functions of the ECU may include semi-autonomous actions such as automatic braking and adaptive cruise control. The controller 110 includes a processing circuit that may include an application-specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated or group) and a memory that executes one or more software or firmware programs, a combinational logic circuit, and / or other suitable components that provide the functions. Three non-limiting exemplary forward cameras 120a, 120b, 120c (generally referred to as 120) are shown.

[0038] For example, the controller 110 may obtain an image 220 (eg, a road 160 in front of the vehicle 100) from the camera 120b. Figure 2 ) to perform dynamic adjustment of the AR image 210. One of the other cameras 120a, 120c or the camera 120b itself may provide an image 220 (referred to as a base image 225 for illustration purposes) Figure 2 )), the AR image 210 is added to the image 220 to create a complete image 230 ( Figure 2 ). That is, the camera 120 that obtains the image 220 used to determine the geometry of the road 160 to facilitate the adjustment of the AR image 210 can be, but need not be, the same camera 120 that obtains the base image 225 that is enhanced by the AR image 210 to create the complete image 230. When different cameras 120 are used, the relative positions of the two cameras 120 in the vehicle 100 must be considered to ensure that the AR image 210 adjusted based on one image 220 from one camera 120 is added to the correct position in the base image 225 obtained by the other camera 120. Figure 1 The number and location of the cameras 120 shown is only one example. In alternative embodiments, dynamic adjustment of the AR image 210 may be performed using another camera 120 located elsewhere that obtains the image 220 of the road 160 in front of the vehicle 100 .

[0039] The HUD region 130 is displayed on the windshield 135 of the vehicle 100. The HUD region 130 corresponds to the driver 330 ( Figure 3 )'s eyes 310( Figure 3 ) so that the image projected in the HUD area 130 will be visible to the driver 330. Because the position of the eye 310 moves, the position of the HUD area 130 on the windshield 135 moves based on the monitoring of the eye 310. Figure 1 In the view in FIG. 1 , the infotainment system 140 is displayed in the dashboard of the vehicle 100, below the windshield 135. The infotainment system 140 includes a display 145, which can display a base image 225 obtained by one of the cameras 120 and an AR image 210 adjusted according to one or more embodiments. According to an alternative embodiment, the base image 225 and the adjusted AR image 210 (i.e., the complete image 230) can be displayed on a tablet or phone or any device 115 connected to the controller 110 in a wired or wireless manner. As shown in the figure, the device 115 can be located inside the vehicle 100 or outside the vehicle 100. In FIG. Figure 2 An exemplary adjustment of the AR image 210 is shown in FIG.

[0040] Figure 2 FIG. 2 shows an adjustment of the AR image 210 according to one or more embodiments. Figure 1 Two complete images 230a, 230b (generally referred to as 230) are shown. The complete image 230 may refer to a real-world view of the road 160 in the HUD area 130 with the AR image 210 projected thereon. Figure 2 In the exemplary case shown, the complete image 230 may alternatively reference a base image 225 of the road 160, wherein the AR image 210 is added and displayed on the display 145 of the infotainment system 140 or on the screen of another device (e.g., a tablet, a phone). As previously described, the image 220 obtained by any of the cameras 120 may be used to determine the road geometry. As also noted, the image 220 obtained using any of the cameras 120 may be used as the base image 225 to which the AR image 210 is added to create the complete image 230. The complete image 230a shows that the road 160 rises (i.e., an increase in the height of the road 160 in front of the vehicle 100) and then flattens, and without any adjustment to account for this elevation change, the AR image 210a is misaligned. As shown in the complete image 230a, the AR image 210a of the arrow appears to be higher than the road 160, rather than being fully projected on the surface of the road 160.

[0041] The complete image 230b shows the same road 160 in front of the vehicle 100. However, Figure 2 As shown, the AR image 210a is adjusted to the AR image 210b so that it appears entirely on the surface of the road 160. For explanation purposes, the surface of the road 160 is specifically discussed as the target of the projection of the AR image 210. For example, a change in the direction of the vehicle 100 toward the road 160 can also affect the placement of the AR image 210 on a road sign or another location near the road 160. Figure 4As further discussed, determination of relative direction involves determining the geometry of road 160 .

[0042] Figure 3 2 shows a vehicle 100 performing dynamic adjustment of an AR image 210 according to one or more embodiments. A side view of the vehicle 100 is shown between lane lines 150. Figure 3 In the exemplary case shown, the AR image 210 is displayed in the HUD area 130 of the windshield 135. Thus, in this case, there is no base image 225 to which the AR image 210 is added. Instead, the AR image 210 in the HUD area 130 of the windshield 135 appears directly on the real view of the road 160. Figure 3 An object 320 (eg, another vehicle 100) is shown on the road 160 ahead. In addition to the road geometry, the other object 320 must also be considered when adjusting the AR image 210, as shown in FIG. Figure 4 Further discussed. In order for the AR image 210 to appear in the correct position, the HUD area 130 of the windshield 135 corresponds to the position of the driver's eyes 310, which is determined by tracking the position of one or both eyes 310 of the driver 330 using known head and eye tracking processes to locate the driver's eye position. Unlike the camera 120 that obtains the image 220, the eyes 310 cannot remain in one position relative to the vehicle 100. Therefore, adjusting the AR image 210 based on the geometry of the road 160 also includes real-time tracking of the movement of the eyes 310, as described in reference Figure 4 and 6 As discussed.

[0043] Figure 4 is a process flow of a method 400 for performing dynamic adjustment of an AR image 210 according to various embodiments. For example, the process flow may be performed by the controller 110. At block 410, obtaining the image 220 may include, for example, using Figure 1 A front-facing camera 120 is shown capturing an image of the road 160 in front of the vehicle 100. At block 420, reference is made to Figure 5 Further discussion on extracting road geometry. Determine the ground plane P G Coordinate system At block 420 , in addition to the road geometry, information about objects 320 (eg, other vehicles 100 ) on the road 160 is obtained. This information is used to avoid projecting the AR image 210 onto the object 320 when the AR image 210 should be projected on the surface of the road 160 .

[0044] At block 430, selecting an AR image 210 refers to selecting an image from predefined images or creating an image. The road geometry information extracted at block 420 and the object 320 information obtained may affect the creation or selection of the AR image 210. For example, if there is an object 320 (e.g., another vehicle 100) directly in front of the vehicle 100, the size of the surface of the road 160 onto which the AR image 210 is projected may be reduced, thereby limiting the size of the AR image 210 that is created or selected.

[0045] At block 440, homography is used to project the AR image 210 selected at block 430 to the correct location of the base image 225 or HUD area 130. Homography relates pixel coordinates in two images. Therefore, any two images on the same plane are related by homography. Specifically, at block 440, the rotation and translation between two viewpoints are determined, one viewpoint being perpendicular to the ground plane P G (i.e., a bird's eye view), and the other viewpoint is the viewpoint of the camera that created the base image 225 or the driver's viewpoint, as further described. Processing at block 440 requires obtaining information from block 450 generated by the monitoring eye 310 or obtaining the base image 225. Then, at a point in the real world Projected onto the HUD area 130 or a point in the real world The processing at block 440 is an adjustment of the AR image 210 that takes into account the relative orientation of the vehicle 100 with respect to the road 160 and references the coordinate system of the camera 120 that obtained the base image 225. Figure 6 and Figure 7 At block 460 , displaying the full image 230 includes projecting the adjusted AR image 210 in the HUD area 130 or providing the adjusted AR image 210 with the base image 225 in the display 145 or on a screen of the device 115 .

[0046] Figure 5 In block 420 ( Figure 4 ) extracts road geometry to dynamically adjust aspects of the AR image at box 440. The single image 220 from one of the front cameras 120 obtained at box 410 is used to detect lane lines 150 in the image 220 in three dimensions. Obtaining three-dimensional lane detection in the coordinate system of the camera 120 that obtained the image 220 is known and will not be described in detail here. According to one example, this process is described in U.S. application Ser. No. 16 / 807,735. Based on obtaining a three-dimensional point corresponding to the lane line 150, interpolation is used to determine a three-dimensional point on the road 160 defined by the lane line 150. (See, for example, Figure 1Specifically, in the coordinate system of the camera 120, from the two dimensions of the point point The third dimension of is calculated by interpolation to calculate the elevation of road 160. Given a point The two dimensions of (e.g., the point between lane lines 150 on the surface of road 160), find the nearest point in lane line 150, and calculate the interpolated value of weighted elevation Qz. Where {P} is the distance from lane line 150 to the X, Y plane (e.g., Figure 5 shown) The set of closest points to The mean elevation is given by the following formula:

[0047]

[0048] In formula 1, as shown, is a point in the set {P} and the function d gives the point and Point The two-dimensional Euclidean distance between . The function d is given by:

[0049]

[0050] Figure 6 and Figure 7 1 shows factors used to adjust the AR image 210 based on the relative orientation of the vehicle 100 with respect to the road 160 (assuming that the AR image 210 is projected onto the road 160 or the base image 225 of the road 160). Specifically, the adjustment of the AR image 210 involves adjusting the points in the real world (i.e., on the road 160) to the same value as the vehicle 100. Projected to a point in the HUD area 130 like Figure 6 As shown, or to a point in the coordinate system of the camera 120 that obtained the basic image 225 like Figure 7 As shown. Figure 6 and Figure 7 , lane line 150 and a coordinate system of the ground plane PG and the ground with arrows projected as an exemplary adjusted AR image 210 are shown. (ie, road 160). It is shown that at ground level P G Demonstration point on

[0051] exist Figure 6 , the position of the eyes 310 of the driver 330 in the vehicle 100 is shown Such as HUD area 130. HUD area plane P H and coordinate system Example points with HUD area 130 The position of the driver's 330 eye 310 is shown together. To point The mean distance is also expressed as In the real world The rotation R and translation (ie, on the road 160) provides a corresponding point in the HUD area 130 In particular,

[0052]

[0053] When Formula 3 is used to adjust the AR image 210 displayed in the HUD area 130, f is given by H and the position of the driver's 330 eyes 310 f x and f y The plane distance or two-dimensional distance given by Figure 6 In addition, c is the distance between the eye 310 and the center of the HUD area 130 with a component c x and c y The two-dimensional distance of Figure 6 shown.

[0054] exist Figure 7 , the position of the focus 120-fp of the camera 120 for obtaining the base image 225 is shown. Also shown is the camera plane P C and coordinate system And in the coordinate system of the camera 120 Demonstration points in It is also shown that the x and f y The focal length f and has component c x and c y Then, Equation 3 can be used to determine the coordinate system for the camera The points of the adjusted AR image 210 to add the AR image 210 to the correct portion of the base image 225 .

[0055] According to an exemplary embodiment, the process of adjusting the AR image 210 can be more efficient. As an example of efficiency, if the image 220 is obtained farther in front of the vehicle 100, the processing of block 420 is performed before the processing of block 440. Specifically, pre-calculation can be performed:

[0056]

[0057] For example, in the exemplary case of display 145, A relates to the estimated rotation and translation of road 160 relative to the viewpoint From this viewpoint, a base image 225 of the road 160 will be captured (when the vehicle 100 gets closer to the area captured by the image 220) and the AR image 210 will be displayed at this viewpoint. Then, upon display of the AR image 210, it can be confirmed whether the estimated pose A matches the current pose in front of the vehicle 100 within a specified threshold. If so, the second part of the calculation can be performed as follows:

[0058]

[0059] In this way, latency associated with the processing of method 400 may be reduced. Figure 8 Another efficiency that can be introduced using Equation 4 and Equation 5, either alone or in combination, is shown. Figure 4 The illustrated method 400 is implemented in Equation 4 and Equation 5.

[0060] Figure 8 8 shows that the ground plane PG is divided into sub-planes 810 according to one or more embodiments to improve the adjustment efficiency of the AR image 210. Figure 8 As shown, the exemplary ground plane PG includes three elevation changes resulting in four sub-planes 810-1, 810-2, 810-3, 810-4 (generally referred to as 810). Figure 8 , three regions 820 are indicated as transition points of the sub-plane 810. Each of these regions 820 exhibits a change in curvature or elevation gradient that exceeds a threshold in the three-dimensional representation of the lane line 150 determined at box 420. By using the sub-planes 810 instead of the full ground plane PG at box 440, a small set of homographs (one for each sub-plane 810) is calculated and used to adjust the AR image 210. This reduces projection errors and the amount of computational work that is feasible compared to a single homography for the full ground plane PG. Although calculating a single projection for each point would be accurate when using the full ground plane PG, it is computationally infeasible.

[0061] Although the above disclosure has been described with reference to exemplary embodiments, it will be appreciated by those skilled in the art that various changes may be made and equivalents may be substituted for its elements without departing from the scope thereof. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the present invention without departing from the basic scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed, but includes all embodiments within its scope.

Claims

1. A system for adjusting an augmented reality image, the system comprising: a front-facing camera in the vehicle, the camera being configured to obtain an image including a road surface in front of the vehicle; and A processor is configured to detect lane lines of a road surface in three dimensions, perform interpolation using the lane lines to determine a ground plane of the road surface, and adjust the augmented reality image to obtain an adjusted augmented reality image so that all points of the adjusted augmented reality image correspond to points on the ground plane.

2. The system according to claim 1, wherein: The processor is also configured to obtain a base image showing the road surface, wherein the processor is also configured to add the adjusted augmented reality image to the base image for display to a driver of the vehicle, and the processor obtains the base image from the camera or a second front camera in the vehicle.

3. The system according to claim 1, wherein: The processor is further configured to monitor the eye position of a driver of the vehicle, the processor is further configured to determine a head-up display area on the vehicle windshield to correspond to the eye position of the driver, and the processor is further configured to project the adjusted augmented reality image in the head-up display area. 4 . The system of claim 1 , wherein the processor is configured to adjust the augmented reality image based on a rotation and a translation of a point on the ground plane.

5. The system according to claim 1, wherein: The processor is further configured to detect an object on a road surface directly in front of the vehicle from the image, and the processor is configured to select or create an adjusted augmented reality image size based on the road surface between the vehicle and the location of the object.

6. A method for adjusting an augmented reality image, the method comprising: obtaining, by a processor, an image from a forward-facing camera in the vehicle, the image including a road surface in front of the vehicle; Detecting, by a processor, lane lines on a road surface in three dimensions; performing, by the processor, interpolation using the lane lines to determine a ground plane of the road surface; and The augmented reality image is adjusted by the processor to obtain an adjusted augmented reality image such that all points of the adjusted augmented reality image correspond to points on the ground plane.

7. The method according to claim 6 also includes obtaining, by the processor, a base image showing the road surface, and the processor adding the adjusted augmented reality image to the base image for display to the driver of the vehicle, wherein the base image is obtained from the camera or a second front camera in the vehicle.

8. The method of claim 6 further includes the processor monitoring an eye position of a driver of the vehicle, determining a head-up display area on a windshield of the vehicle to correspond to the eye position of the driver, and projecting the adjusted augmented reality image in the head-up display area.

9. The method of claim 6, wherein the adjusting the augmented reality image is based on a rotation and a translation of a point on the ground plane.

10. The method of claim 6 further comprising the processor detecting an object on a road surface directly in front of the vehicle from the image and selecting or creating the adjusted augmented reality image size based on the road surface between the vehicle and the location of the object.

Images