A method, vehicle and device for displaying AR information in real-time mobile scenes

Abstract

The present application relates to the field of augmented reality technology, and discloses a method, vehicle, and device for displaying AR information in a real-time mobile scene. The method includes: obtaining a first road real-scene image of an electronic device at a first position at a first moment during movement; determining a predicted display position of AR information corresponding to a first target object based on the first road real-scene image; determining calibration information of the predicted display position based on the moving speed of the electronic device during the process of the electronic device moving from the first position to the second position; obtaining a calibrated display position based on the calibration information; and displaying AR information in the front field of view based on the calibrated display position when the electronic device is at the second position. In this way, the electronic device can display AR information in the front field of view based on the calibrated display position. To a certain extent, the accuracy of the display position of the AR information can be improved, the sense of reality of the virtual and real fit between the AR information and the front field of view can be improved, and the user experience can be enhanced.

Description

Technical Field

[0001] The present application relates to the field of augmented reality technology, and in particular to a method, vehicle, and device for displaying AR information in a real-time mobile scene. Background Art

[0002] An augmented reality head-up display (AR HUD) is a device that displays images containing AR information in the forward field of view. This AR information includes driving assistance and navigation guidance. This allows drivers to view AR information directly in the forward field of view without having to look down at their phone or vehicle display. This eliminates the need to switch sight lines, reduces crisis response time, and improves driving safety.

[0003] Among them, the display method of AR information is generally that the vehicle first collects the real-scene image of the road, determines the position where the AR information needs to be displayed in the front field of view based on the real-scene image of the road, and then displays the AR information at the corresponding position in the front field of view.

[0004] However, when the vehicle is in motion, it takes a certain amount of time for the vehicle to determine the position of the AR information displayed in the front field of view based on the collected real-life road images. During this period, the vehicle will move. As a result, when the vehicle displays the aforementioned AR information, the front field of view has changed, making the display position of the AR information inaccurate. The sense of reality of the AR information and the front field of view is low, and the user experience is poor. Summary of the Invention

[0005] The embodiments of the present application provide a method, vehicle, and device for displaying AR information in a real-time mobile scene.

[0006] In the first aspect, an embodiment of the present application provides a method for displaying AR information in a real-time mobile real scene, which is applied to an electronic device, and the method includes: obtaining a first real-scene image of a road when the electronic device is at a first position at a first moment during movement; determining a predicted display position of AR information corresponding to a first target object based on the first real-scene image of the road; determining calibration information of the predicted display position based on the moving speed of the electronic device during the process of moving the electronic device from the first position to the second position; obtaining a calibrated display position based on the calibration information; and displaying the AR information in the front field of view based on the calibrated display position when the electronic device is at the second position.

[0007] It is understood that based on the embodiments of the present application, the electronic device can display the AR information in the front field of view based on the calibrated display position. This can improve the accuracy of the display position of the AR information to a certain extent, improve the realism of the AR information and the front field of view, and enhance the user experience.

[0008] In a possible implementation of the first aspect above, when the electronic device is at the second position, before displaying the AR information in the front field of view based on the calibrated display position, the method further includes: converting the calibrated display information from the image coordinate system to the human eye coordinate system, and then converting it into the augmented reality head-up display coordinate system.

[0009] In a possible implementation of the first aspect above, determining the predicted display position of the AR information corresponding to the first target object based on the first real-scene image of the road includes: determining spatial feature information based on the first real-scene image of the road; and determining the predicted display information of the AR information corresponding to the first target object based on the spatial feature information.

[0010] In a possible implementation of the first aspect, determining the spatial feature information based on the first real-scene image of the road includes determining the spatial feature information based on a simultaneous positioning and mapping technology and the first real-scene image of the road.

[0011] In a possible implementation of the first aspect above, determining the spatial feature information based on the first real-scene image of the road includes: if there is a curve or uphill or downhill road in the first real-scene image of the road, converting the first real-scene image of the road captured by the camera angle into an image in the world coordinate system.

[0012] In a possible implementation of the first aspect above, the AR information includes driving assistance information and / or navigation guidance information.

[0013] In a second aspect, an embodiment of the present application provides a vehicle, comprising a head-up display and an AR information display system; the AR information display system is used to obtain a first real-scene image of a road at a first position at a first moment during the movement of the vehicle; the AR information display system is used to determine a predicted display position of AR information corresponding to a first target object based on the first real-scene image of the road; the AR information display system is used to determine calibration information of the predicted display position based on the moving speed of the vehicle during the movement of the vehicle from the first position to the second position; the AR information display system is used to obtain a calibrated display position based on the calibration information; and the AR information display system is used to control the head-up display to display the AR information based on the calibrated display position when the vehicle is at the second position.

[0014] It is understandable that the AR information display system can be a controller in a smart car system, a head-up display system, etc., but is not limited to this.

[0015] In a possible implementation of the first aspect above, before displaying the AR information in the front field of view based on the calibrated display position, the AR information display system is used to convert the calibrated display information from the image coordinate system to the human eye coordinate system, and then into the augmented reality head-up display coordinate system.

[0016] In a possible implementation of the first aspect above, the AR information display system is used to determine the predicted display position of the AR information corresponding to the first target object based on the first real-scene image of the road, including: the AR information display system is used to determine spatial feature information based on the first real-scene image of the road; the AR information display system is used to determine the predicted display information of the AR information corresponding to the first target object based on the spatial feature information.

[0017] In a possible implementation of the first aspect above, the AR information display system determines the spatial feature information based on the first road real scene image, including: the AR display system is used to determine the spatial feature information based on synchronous positioning and mapping technology and the first road real scene image.

[0018] In a possible implementation of the first aspect above, the AR information display system is used to determine spatial feature information based on the first road real-scene image, including: the AR information display system is used to convert the first road real-scene image captured by the camera angle into an image in the world coordinate system when there is a curve or uphill or downhill road in the first road real-scene image.

[0019] In a possible implementation of the first aspect above, the AR information includes driving assistance information and / or navigation guidance information.

[0020] In a third aspect, an embodiment of the present application provides a device comprising: one or more memories storing instructions; a processor coupled to the one or more memories, wherein when the instructions are executed by the processor, the device executes the real-time mobile real-scene AR information display method described in any one of the first aspects.

[0021] In a possible implementation of the first aspect above, the device is a vehicle, a mobile phone, a watch, or AR glasses.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes the real-time mobile real-scene AR information display method described in any one of the first aspects.

[0023] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when run on a computer, enables the computer to execute the real-time mobile AR information display method in a real scene as described in any one of the first aspects.

[0024] In a sixth aspect, an embodiment of the present application provides a chip, which is coupled to a memory and is used to read and execute program instructions stored in the memory to implement the real-time mobile AR information display method in a real scene as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram of an imaging principle of a virtual image provided by an embodiment of the present application;

[0026] Figure 2A A schematic diagram of an application scenario of a method for displaying AR information in a real-time mobile scene is shown;

[0027] Figure 2B A schematic diagram showing a vehicle 1 displaying a turn icon A001 in the field of view in front of a windshield using head-up display technology while the vehicle 1 is driving is shown;

[0028] Figure 3 A flow chart showing a method for displaying AR information in a real-time mobile scene is shown;

[0029] Figure 4A shows the angles between different road slopes and the horizontal line where the vehicle 1 is located;

[0030] Figure 4B The figure shows possible vanishing lines P', G' or Q' in the road real scene image A01;

[0031] Figures 5A to 5C A schematic diagram showing the principle of determining the distance between the target object and the camera in the previous frame of road real scene image captured by vehicle 1;

[0032] Figure 6A shows a positional relationship of a camera relative to a photographed object;

[0033] Figure 6B It shows the conversion of an image L2 in an image coordinate system captured by a camera angle into a top view L1 in a world coordinate system;

[0034] Figure 7A schematic diagram showing the distribution of element position information in a real road scene image is shown;

[0035] Figure 8 A schematic diagram showing a process in which a vehicle 1 determines a positional relationship between the vehicle 1 and the lane line in which it is located;

[0036] Figure 9 A schematic diagram of the overall principle of converting an image captured by a camera into an image from the perspective of the human eye is shown;

[0037] Figure 10 A possible functional framework schematic diagram of a vehicle 1 is shown. DETAILED DESCRIPTION

[0038] The embodiments of the present application include but are not limited to a method, medium, and electronic device for displaying AR information in real-time mobile scenes.

[0039] The following introduces some related concepts in the embodiments of this application.

[0040] (1) Augmented Reality Head-up Display (AR HUD)

[0041] AR HUD is generally installed in a vehicle. It is a display device that projects images into the front field of view of the user (such as the driver) in the vehicle based on the principle of virtual imaging.

[0042] In an embodiment of the present application, an AR HUD is a device that displays an image containing AR information on a real road surface and / or a target object on the real road surface in the field of view ahead. The AR information includes driving assistance information and / or navigation guidance information. The driving assistance information may include steering instructions, lane departure, collision warning, following distance, pedestrian prompts, obstacles, navigation routes and other information. The navigation guidance information may include vehicle speed, navigation information, traffic signs, driving information, points of interest (POIs) and other information. The POIs may be parking lots, restaurants, shopping malls, theaters, gas stations, etc.

[0043] For example, AR HUD can provide driving assistance information within a preset distance (e.g., 10 meters) from the vehicle, and AR HUD can provide navigation guidance information within a preset distance (e.g., 2.5 meters) from the vehicle.

[0044] (2)Virtual image principle

[0045] A virtual image is an optical phenomenon that can be viewed directly by the eye but cannot be received by a light screen.

[0046] In the embodiment of the present application, the imaging principle of the AR HUD in the vehicle can be found in Figure 1 , Figure 1 This is a schematic diagram of an imaging principle of a virtual image provided by an embodiment of the present application. Figure 1 As shown, an augmented reality head-up display E generates a source image and, based on the source image, emits a light beam L with a certain divergence angle α1. Light beam L is represented by light rays L1 and L2. After being reflected by mirrors M1 and M2 and the car's windshield, light beam L enters the human eye at a divergence angle α2. Based on the common understanding that light travels in straight lines, the brain traces the light in the opposite direction and interprets the intersection of the light beam L with the reverse extension as the object point, i.e., the virtual image point. The content of this virtual image point can be the aforementioned AR information.

[0047] As mentioned above, when the vehicle is in motion, it takes a certain amount of time for the vehicle to determine the position of the AR information displayed in the front field of view based on the collected real-life road images. During this period of time, the vehicle will move. As a result, when the vehicle displays the aforementioned AR information, the front field of view has changed, making the display position of the AR information inaccurate. The sense of reality of the AR information and the front field of view is low, and the user experience is low.

[0048] For example, Figure 2A The following is a schematic diagram showing an application scenario of a method for displaying AR information in a real-time mobile scene. Figure 2A As shown, vehicle 1 is traveling in the current environment. At time t0, vehicle 1 is at position P1, and at time t1, vehicle 1 is at position P2. From time t0 to time t1, vehicle 1 has moved from position P1 to current position P2, and vehicle 1 has moved a distance d1.

[0049] Figure 2B A schematic diagram is shown of a vehicle 1 displaying a turn icon A001 in the field of view in front of a windshield using head-up display technology while the vehicle 1 is driving.

[0050] like Figure 2B As shown, when the vehicle 1 is in a driving state, and the navigation application scenario is to display a turn icon A001 at the turning point where the traffic light is located, wherein the positioning of the turning point is determined by the vehicle 1 based on the position of at least the target object of the traffic light, and the turn icon A001 is used to prompt the vehicle 1 to change the driving direction at the turning point where the traffic light is located.

[0051] At time t0, vehicle 1 is at position P1. Vehicle 1 can collect real-world images of the road, and then obtain the position P1' where the turn icon A001 is displayed based on the collected real-world images of the road. When the turn icon A001 is displayed at time t1, vehicle 1 has already moved from position P1 to the current position P2, having moved a distance d1. Therefore, the field of view ahead has changed, and the distance between vehicle 1 and the turning point where the traffic light is located has shortened. However, the turn icon A001 is still displayed based on the distance between the turning point where the traffic light was previously located and the position of vehicle 1. Therefore, the turn icon A001 will be displayed at position P1', causing the display position of the turn icon A001 to not correspond to the position where it should actually be displayed. The sense of reality of the turn icon A001 fitting the field of view ahead is low, which reduces the user experience.

[0052] In order to solve the technical problems raised in the background technology, an embodiment of the present application provides a method for displaying AR information in a real-time mobile real scene. The method includes: when a vehicle is driving, a real-scene image of a road is collected at a first position at a first moment, and a predicted display position of AR information corresponding to a target object is determined based on the real-scene image of the road. In the process of determining the predicted display position of AR information corresponding to a target object (or target point) based on the real-scene image of the road, vehicle 1 has traveled from the first position to a second position. At the second position at a second moment, vehicle 1 calibrates the predicted display position of AR information based on the first position and the second position, and displays AR information in the field of view in front of vehicle 1 based on the calibrated predicted display position.

[0053] In this way, the accuracy of the display position of AR information is improved, the realism of the AR information and the front field of view is improved, and the user experience is enhanced.

[0054] For example, Figure 2B As shown, in this navigation application scenario, if vehicle 1 is in a driving state, a real-scene image of the road is collected at position P1 at time t0, and the predicted display position of the AR information (i.e., the turn icon A001) corresponding to the turning point where the target object, the traffic light, is located, is determined based on the real-scene image of the road as position P1'. In the process of determining the predicted display position of the turn icon A001 corresponding to the target object based on the real-scene image of the road, vehicle 1 has traveled from position P1 to position P2. At position P2 at time t1, vehicle 1 calibrates the predicted display position of the AR information (i.e., the turn icon A001) based on positions P1 and P2, and obtains the calibrated position as position P2', and displays the turn icon A001 in the field of view in front of vehicle 1 based on the calibrated predicted display position P2'.

[0055] Based on the above solution, vehicle 1 will obtain a more accurate display position (position P2') for turn icon A001, rather than position P1' that partially covers the traffic light support column. This improves the accuracy of the display position of turn icon A001, thereby enhancing the realism of turn icon A001 and the real-world fit between the forward field of view, and enhancing the user experience.

[0056] The method for displaying AR information in a real-time mobile scene provided by the embodiment of the present application can be applied to the AR real-scene navigation process, such as turning, changing lanes, etc., but is not limited thereto.

[0057] The method for displaying AR information in a real-time mobile real scene provided by the embodiment of the present application can be applied not only to AR real scene navigation of vehicles, but also to AR real scene navigation of mobile phones, AR real scene navigation of smart watches, AR real scene navigation of AR glasses, etc., but is not limited thereto.

[0058] It is understood that AR information can be displayed in the form of text, images, three-dimensional (3D) models, etc. The turn icon A001 can be displayed in 3D style, thereby presenting a better sense of three-dimensionality and realism, improving the visual effect of the real-scene AR information display, and providing users with better navigation guidance.

[0059] Figure 3 A flow chart of a method for displaying AR information in a real-time mobile scene according to an embodiment of the present application is provided. Figure 3 As shown, the execution subject of the process may be vehicle 1, and the process includes the following steps:

[0060] 301: Acquire spatial feature information of a driving environment of a vehicle 1 at a first position at a first moment.

[0061] It is understood that spatial feature information may include road feature information and / or road space information. Road feature information may include various objects in the driving environment of vehicle 1. Each object may include buildings, the vehicle 1 in front, traffic lights, lane lines, telephone poles, roadside, intersections, etc. Road space information may include the position information of vehicle 1 in the actual environment. For example, the position information may include the vehicle posture of vehicle 1. It is understood that the vehicle posture refers to the up and down pitch angles, or left and right pitch angles, etc. of the vehicle relative to the road. Among them, the up and down pitch angles refer to the angles between the vehicle and the parallel line of the road ahead. Road space information may also include the position information of various objects in the actual environment of vehicle 1.

[0062] In some embodiments, vehicle 1 can obtain a real-world image of the road environment in which vehicle 1 is traveling and extract spatial feature information from the obtained real-world image. Optionally, the road spatial information can be obtained using a simultaneous localization and mapping (SLAM) method (or technology). The purpose of SLAM is to construct a real-time map of the surrounding environment based on the real-world image of the road obtained by vehicle 1. It is understood that based on this, vehicle 1 can further infer the position information of various objects in the driving environment of vehicle 1 based on geometric perspective and the surrounding environment map. In other embodiments, if vehicle 1 has a high-precision map and a lidar, vehicle 1 can use the high-precision map and lidar to obtain the absolute position (or absolute geographic location) of vehicle 1 and various objects in the driving environment. Absolute geographic location refers to longitude and latitude, which is based on the entire Earth as a reference system and is measured in latitude and longitude. Every location on Earth has its own unique longitude and latitude values. For example, vehicle 1 can use a high-precision map and lidar to obtain the absolute position of vehicle 1 and the location annotated by AR information (e.g., the corner where a traffic light is located).

[0063] In addition to obtaining the aforementioned road space information through SLAM, in some other embodiments, vehicle 1 may also obtain road space information through the following methods to correct the road space information obtained through SLAM. For example, vehicle 1 may determine the angle between the horizontal line where vehicle 1 is located and the parallel line of the road ahead based on the angle between the user's eyes in vehicle 1 and the straight line where the vanishing line in the real-world image of the road is located and the horizontal line of the road. The "vanishing line" refers to the visual intersection of the parallel lines.

[0064] For example, Figure 4A Indicates the angle between different road slopes and the horizontal line where vehicle 1 is located. Figure 4A As shown, O c represents the position of the user's eyes in the vehicle 1, and the user's eye position O c The straight line connecting the vanishing lines P', G', Q', and H' is the horizontal line O c The angles of H are 0, α, φ, and β respectively. Figure 4B Indicates the possible vanishing lines P', G', Q', and H' in the real road scene image A01.

[0065] The position of the vanishing line in the road real scene image A01 is different, and the angle between the horizontal line where the vehicle 1 is located and the road is different. If the vanishing line in the road real scene image A01 is P', then according to the eye O cThe distance from the plane of the real road image A01 displayed on the windshield of the vehicle 1, the distance between the vanishing line P' and the edge of the real road image A01, and the tangent and inverse tangent trigonometric formulas are used to obtain the position O of the user's eye in the real road image A01 in the vehicle 1. c The straight line O with the vanishing line P' c P' and horizontal line O c The angle α of H.

[0066] Since the position of the user's eyes in the road real scene image A01 is O c The straight line O with the vanishing line P' c P' and horizontal line O c The angle of H is angle α, and the position of the user's eyes is O c The straight line where the vanishing line P is located and the horizontal line O where the vehicle 1 is located c The angle between the planes parallel to H is also the angle α. The angle between the vehicle 1 and the horizontal line O is c The angle between the plane parallel to H and the road in front of the vehicle 1 is also the same as the angle α. c The angle between the plane parallel to H and the road 3 is angle α.

[0067] Similarly, if the vanishing line in the road scene image A01 is G', then according to the eye O c The distance from the plane of the road real scene image A01, the distance between the vanishing line G' and the edge of the road real scene image A01, and the tangent formula and the inverse tangent trigonometric formula are used to obtain the distance between the vehicle 1 and the horizontal line O c The angle between the plane parallel to H and the road 2 is angle φ.

[0068] Similarly, if the vanishing line in the road scene image A01 is Q', then according to the eye O c The distance from the plane of the road real scene image A01, the distance between the vanishing line Q' and the edge of the road real scene image A01, and the tangent formula and the inverse tangent trigonometric formula are used to obtain the distance between the vehicle 1 and the horizontal line O c The angle between the plane parallel to H and the road 3 is angle β.

[0069] During the driving process of the vehicle 1, the vehicle 1 may capture a front-frame real-scene image of the road (e.g., the real-scene image of the road at the first position) and a rear-frame real-scene image of the road (e.g., the real-scene image of the road at the second position) through the camera. During the process from the camera capturing the front-frame real-scene image to the camera capturing the rear-frame real-scene image, the vehicle 1 travels a preset distance. Then, based on the preset distance traveled by the vehicle 1 and the movement change and imaging changes (viewing angle, geometric perspective, distortion, etc.) of the target object in the rear-frame real-scene image captured by the vehicle 1, the vehicle 1 may determine the distance between the target object and the camera in the rear-frame real-scene image.

[0070] For example, Figures 5A to 5C The schematic diagram shows the principle of determining the distance between the target object and the camera in the previous frame of the road real scene image collected by the vehicle 1. Figure 5A As shown, vehicle 1 captures the current frame of road real scene image and the previous frame of road real scene image through the camera. In the process of capturing the previous frame of road real scene image to the current frame of road real scene image, vehicle 1 travels a preset distance d. Since the imaging change is proportional reduction and no distortion occurs, vehicle 1 can be based on the preset distance d traveled by vehicle 1 and the movement change of the target object in the current frame of road real scene image and the previous frame of road real scene image (for example, Figure 5B As shown in FIG1 , the target object area accounts for a proportion K1 of the area of the shooting imaging screen and the target object area accounts for a proportion K2 of the area of the shooting imaging screen), and the preset distance d traveled by the vehicle 1 is determined by the following formula:

[0071] d=D2*(K2-K1) / K1

[0072] Among them, Figure 5C As shown, D2 represents the distance between vehicle 1 and the target object when the camera captures the road scene image of the previous frame before the current frame.

[0073] It is understandable that because the camera is tilted downward, but not completely pointed directly downward, the obtained real-world image of the road may exhibit trapezoidal distortion, meaning that objects appearing near the real-world image of the road appear wide, while objects appearing far away appear narrow. This trapezoidal distortion can cause large errors in subsequent navigation line recognition and calculated deflection angles, making it impossible to achieve accurate positioning and route travel. Therefore, vehicle 1 can directly derive the mapping relationship between the image coordinate system and the real-world coordinate system based on the imaging principle of the camera's optical lens and other geometric relationships, thereby converting the real-world image of the road captured in the image coordinate system from the camera's perspective into a top-down real-world image of the road captured in the real-world coordinate system.

[0074] In some embodiments, if vehicle 1 determines that the road in the real-life image of the road is curved or has ups and downs, it is necessary to calculate according to the scale of the curved line in the geometric perspective space, and fit and restore the curvature of the foreground curve or ups and downs from a bird's-eye view in the actual world coordinate system.

[0075] Specifically, based on the imaging principle of the camera's optical lens, vehicle 1 derives the correspondence between the image coordinate system and the real-world coordinate system. Specifically, as long as the pixel coordinates of objects such as curved roads and uphill and downhill roads in the image coordinate system are known, the actual horizontal and vertical distances between the objects and the bottom of the camera can be derived.

[0076] For example, Figure 6A It shows the position relationship of a camera relative to the object being photographed. Figure 6B It shows the conversion of an image L2 in an image coordinate system shot from a camera angle into a top view L1 in a world coordinate system.

[0077] Vehicle 1 can calculate the position of each pixel in the image L2 ("square" image B) in the image coordinate system captured at the camera angle corresponding to the top view L1 ("trapezoidal" image A) in the world coordinate system using the following formula, that is, converting the "perspective view" into the "top view".

[0078]

[0079]

[0080]

[0081] Y1=H*tan(α+Δθ);

[0082]

[0083]

[0084] Where (X0, Y0) represents the coordinates of the pixel in image B (image coordinate system) captured from the camera angle, and (X1, Y1) represents the coordinates of the pixel in top view A converted from image B to the world coordinate system. α represents the pitch angle between the camera and the line perpendicular to the bottom edge of the image in the side view, θ represents the vertical field of view angle between the camera and the line perpendicular to the bottom edge of the image in the side view, H represents the height of the camera above the ground, Dmin represents the actual distance from the bottom edge of the image to the camera, and Dmax represents the actual distance from the top edge of the image to the camera.

[0085] β represents the horizontal field of view of the camera in the top view, width represents the image width, height represents the image height, D1 represents the focal difference between the camera coordinates and the horizontal plane line, X1 represents the horizontal distance from the camera, and Y1 represents the vertical distance from the camera.

[0086] Furthermore, after obtaining a real-world image of the road in the world coordinate system via SLAM, vehicle 1 can also calibrate the road spatial information in the real-world image obtained via SLAM using other methods. For example, vehicle 1 can calculate the position information of each element in the real-world image of the road, and use the more accurate position information of vehicle 1 obtained by the autonomous driving perception system to calibrate the position information of other elements in the real-world image of the road, excluding vehicle 1.

[0087] For example, Figure 7 A schematic diagram showing the distribution of element position information in a real road image is shown. Figure 7 As shown in the figure, assume that vehicle 1, vehicle 2, object 3, object 4, and object 5 are distributed in the real road scene image. The position information of vehicle 1, vehicle 2, object 3, object 4, and object 5 obtained by SLAM are (x1, y1, z1), (x2, y2, z2), (a, b, c), (d, e, f), and (h, i, j), respectively. Since the road spatial information of the aforementioned objects in the real road scene image obtained by SLAM is relatively inaccurate, in order to improve the accuracy. Vehicle 1 first accurately perceives the positions of vehicle 1 and vehicle 2 as (x1', y1', z1') and (x2', y2', z2') based on the autonomous driving perception system. Based on the difference Δ(x1, y1, z1) between (x1', y1', z1') and (x1, y1, z1), or the difference Δ(x2, y2, z2) between (x2', y2', z2') and (x2, y2, z2), and the known intrinsic parameter transformation matrix information of vehicle 1, the offset matrix δT can be calculated. The recalibrated spatial position information is X'=X*δT. The calibrated position information of vehicle 1, vehicle 2, object 3, object 4, and object 5 are (x1', y1', z1'), (x2', y2', z2'), (a, b, c)*δT, (d, e, f)*δT, and (h, i, j)*δT, respectively.

[0088] It is understood that in addition to obtaining road space information in the above manner, as previously mentioned, when the vehicle 1 can also use a high-precision map and a laser radar, it can obtain its own positioning based on the high-precision map and the laser radar.

[0089] When vehicle 1 can only access standard definition (SD) maps and conventional global positioning system (GPS) / Beidou positioning, the lane information is unclear due to interference such as occlusion and multipath. Vehicle 1 can combine the vision, inertial navigation, real time kinematic (RTK), vehicle speed, map, lidar and other information of the digital video recorder (DVR) driving recorder to identify the lane position and correct the horizontal lane. For example, vehicle 1 can also obtain data through the camera, radar, lidar, global navigation satellite system, inertial sensor, and standard definition map on vehicle 1, and process these data to obtain the road structure and the positional relationship between vehicle 1 and the road. For example, which lane line vehicle 1 is in.

[0090] In order to identify the position (lateral and / or longitudinal position) of the vehicle 1 , for example, sub-meter high-precision positioning is required. Figure 8 FIG. 1 shows a schematic diagram of a process in which a vehicle 1 determines the position relationship between the vehicle 1 and the lane line in which it is located. Figure 8 As shown, the process of vehicle 1 determining the position relationship between vehicle 1 and the lane line is as follows:

[0091] A. Obtain the target distance of the multi-frame fusion information image. In this step, vehicle 1 can obtain sensor data from the sensor, identify the markers on the road based on the sensor data, and further determine the distance between vehicle 1 and the identified markers on the road. The markers can be telephone poles, traffic lights, etc. The sensor can be a camera, radar, lidar, etc. Exemplarily, the sensor data can include a real-life image of the road captured by the camera, and vehicle 1 can extract the markers on the road from the real-life image of the road captured by the camera. The sensor data can also include the distance to the markers on the road measured by the lidar.

[0092] B. Obtaining the drivable area (FreeSpace). In this step, the vehicle 1 can obtain sensor data from cameras, radars, lidars, etc., and determine the location of the road ahead that the vehicle 1 can drivable based on the sensor data.

[0093] C. Obtain traffic flow. Traffic flow refers to vehicles 1 surrounding vehicle 1. In this step, vehicle 1 can obtain sensor data from cameras, radars, lidars, etc., and use this sensor data to identify the number of traffic flows ahead of vehicle 1. It can then determine the lane in which vehicle 1 is located based on the number of traffic flows.

[0094] D. Static Feature Perception. For example, static features may include lane markings, road edges, stop signs, road markings, cones, traffic lights, and the like. In this step, vehicle 1 can acquire sensor data from cameras, radar, lidar, and other sensors and identify static features based on this sensor data. These static feature recognition results can be used to assist in locating vehicle 1.

[0095] E. Vehicle Information Perception. For example, vehicle information may include vehicle speed, acceleration, road pitch angle, vehicle head orientation, and vehicle body position. In this step, vehicle 1 may acquire sensor data from a global navigation satellite system (GNSS) or inertial sensors, and identify vehicle information based on this sensor data.

[0096] F. Laser-Assisted Positioning. In this step, vehicle 1 can infer the lane where vehicle 1 is located based on the information obtained in steps A, B, C, and D. It is understood that if vehicle 1 is currently in a dark environment, the road image captured by vehicle 1's camera cannot accurately locate the lane where vehicle 1 is located. In this case, because the lidar can still accurately locate the lane where vehicle 1 is located in a dark environment, vehicle 1 can use laser-assisted positioning of the lane where vehicle 1 is located.

[0097] G. Road Structure Inference: In this step, vehicle 1 infers the road structure of the road on which vehicle 1 is located based on the information obtained in steps A, B, C, and D.

[0098] H. Calculation of the vehicle's posture. In this step, the vehicle 1 calculates its posture based on the data obtained in step B.

[0099] I. Surrounding Road Network Query: In this step, the vehicle 1 can query the surrounding road network information based on the standard definition map.

[0100] J. Extracting Key Intersection Information. In this step, vehicle 1 can extract key intersection information based on the surrounding road network results obtained in step 1. For example, the key intersection information can include the turn where the traffic light is located. This key intersection information can be used to assist in locating the position of vehicle 1.

[0101] K. Complex Road Condition Reasoning: In this step, vehicle 1 can obtain the positional relationship between vehicle 1 and the road structure based on steps G and H.

[0102] L. Extreme Weather Road Perception: In this step, vehicle 1 can obtain the positional relationship between vehicle 1 and the road structure according to step F.

[0103] In the complex road condition reasoning in the above solution, the lane line where vehicle 1 is located can be determined in the following two ways:

[0104] Fusion Positioning Technology 1: Vehicle 1 uses a visual algorithm to identify lane lines and curbs, and can deduce whether the current lane is on the left or right. If there are few lanes and there is not much obstruction from the preceding vehicle, the lane index can be directly identified. If there is obstruction, the lane position can be deduced by identifying the traffic flow through a visual algorithm. Finally, the total number of lanes in the current section of the map is integrated to determine the current lane index.

[0105] Fusion positioning technology 2: While vehicle 1 is driving, a historical information list of the lane in which vehicle 1 is located (including the current lane index, the total number of lanes on the road, etc.) can be maintained. The lane change behavior of vehicle 1 can be judged by the angle between the lane line and the vehicle direction angle, and the lane in which vehicle 1 is located can be updated based on the number of lanes in the map.

[0106] 302: Determine a predicted display position of the AR information corresponding to the target object based on the spatial feature information.

[0107] For example, Figure 2B As shown, in this navigation application scenario, if vehicle 1 is in a driving state, vehicle 1 collects a real-scene image of the road at position P1 at time t0, and determines based on the real-scene image of the road that the predicted display position of the turn icon A001 at the turning point where the traffic light is located is position P1', wherein the positioning of this position at the turning point is determined by vehicle 1 based on the position of at least the target object, the traffic light.

[0108] It is understandable that AR information may include display content in addition to the display location, wherein the display content of AR information may include route navigation, obstacles, areas of interest, etc., AR images that need to be superimposed on the real scene, but is not limited thereto.

[0109] For example, the navigation information for a square obstacle is a trapezoid in view A and an oblique trapezoid in view B. The display of the obstacle navigation information in view B is composed of the position transformations of multiple locations. It can be understood that the final display positions of the navigation, obstacle, and area of interest are all changed by key locations among the multiple locations.

[0110] 303 : Based on the relationship between the second position to which the vehicle 1 travels at the second moment and the first position, calibrate the predicted display position of the AR information to obtain calibrated AR information.

[0111] It can be understood that when vehicle 1 is in a driving state, it takes a certain amount of time for vehicle 1 to determine the position where the AR information is displayed in the front field of view based on the collected real-life image of the road. During this period of time, vehicle 1 will move. In this way, when vehicle 1 displays the aforementioned AR information, the front field of view has changed, and there is a deviation between the predicted display position of the AR information and the position where the AR information should actually be displayed.

[0112] To compensate for this deviation, vehicle 1 can perform location correction and compensation based on the navigation route and time delay before AR imaging. Specifically, vehicle 1 can read the AR information and identification trajectory information of the navigation map, and its current location. Based on the vehicle's speed and time delay, it can estimate the vehicle's upcoming new location according to the route.

[0113] As you can understand, an inertial navigation system (INS) is a navigation parameter calculation system that uses gyroscopes and accelerometers as sensitive components. This system establishes a navigation coordinate system based on gyroscope output and calculates vehicle 1's speed and position within that coordinate system based on accelerometer output. If vehicle 1 determines that the navigation signal is poor, it can use inertial navigation and time delay to perform position corrections and compensation before AR imaging.

[0114] Specifically, assume that the predicted time for vehicle 1 to calculate the position displayed by the AR information based on the real-world image of the road is p seconds, and the current speed of vehicle 1 is calculated as v (m / s) based on the tire speed. Then, based on the AR information, it is determined that the current road needs to be rotated counterclockwise by a degrees relative to the east direction. A coordinate system is established, with the east direction of vehicle 1 as the x-axis, the north direction as the y-axis, and the current vehicle 1 as the origin (0,0). The calibrated AR display information is the position displayed by the AR information calculated based on the real-world image of the road minus the predicted deviation: (v*p*sin(a), v*p*cos(a)).

[0115] It can be understood that the prediction compensation of the time delay is not only based on vision and inertial navigation trajectory, but the vehicle 1 can also read the information in the navigation map to compensate for the deviation.

[0116] In some embodiments, when vehicle 1 selects road feature information in a real-scene image of a road based on SLAM, it mainly selects road feature information at a preset distance from the location of the AR information, and estimates the three-dimensional coordinates of the aforementioned road feature information in stereoscopic space based on geometric perspective.

[0117] 304: Convert the calibrated display information from the image coordinate system to the AR HUD coordinate system.

[0118] It is understood that the camera is generally set on the bumper of the vehicle 1, below the human eye. Since the angle at which the human eye sees the image is different from the angle at which the camera forms the image, in order to improve the user experience, it is necessary to convert the image captured by the camera from the image coordinate system to the image from the human eye's perspective. Among them, the image captured by the camera can be a real-scene image of the road. The vehicle 1 can convert the real-scene image of the road in the image coordinate system into the real-scene image of the road from the human eye's perspective, and based on the image from the human eye's perspective, convert the calibrated display information from the image coordinate system to the human eye coordinate system, and then from the human eye coordinate system to the AR HUD coordinate system.

[0119] For example, Figure 9 According to some embodiments of the present application, a schematic diagram of the overall principle of converting an image captured by a camera into an image from the perspective of the human eye is shown.

[0120] E represents the AR HUD, A represents the camera, B represents the origin of the rear wheel axis of vehicle 1, C represents the world coordinate system, and D represents the human eye.

[0121] like Figure 9 As shown, the vehicle 1 needs to convert the road scene image captured by the camera from the camera imaging angle to the human eye angle. The matrix transformation parameters required for the above conversion can be the default parameters carried by the vehicle 1 itself.

[0122] The aforementioned angle change can be transformed into a variety of coordinate systems, for example, 5. The following uses the 5 examples to illustrate the transformation of traffic feature information between the camera imaging angle and the human eye angle.

[0123] Specifically, the real-scene image of the road captured by the camera on vehicle 1 can be converted from the image coordinate system to the camera coordinate system, from the camera coordinate system to the vehicle 1 coordinate system, from the vehicle 1 coordinate system to the world coordinate system, from the world coordinate system to the human eye coordinate system, and then from the human eye coordinate system to the AR HUD coordinate system.

[0124] Based on the above five coordinate system transformation steps, the following introduces the formula for each coordinate system transformation step:

[0125] (A) Image coordinate system is converted to camera coordinate system:

[0126]

[0127]

[0128] In the above formula, f represents the focal length of the camera, (x, y) is the point in the road scene image coordinate system, and (Xc, Yc, Zc) is the point in the camera coordinate system.

[0129] (B) The camera coordinate system is converted to the vehicle 1 coordinate system:

[0130]

[0131] In the above formula, R is a 3*3 rotation matrix, representing the angular difference of the camera relative to the forward direction of vehicle 1, and t is a 3*1 translation matrix, representing the translation of the camera relative to the rear axle coordinate system of vehicle 1.

[0132] (C) Convert vehicle 1 coordinate system to world coordinate system

[0133]

[0134] In the above formula, R is a 3*3 rotation matrix, representing the angular difference between the forward direction of vehicle 1 and the world coordinate system, and t is a 3*1 translation matrix, representing the translation of the rear axle of vehicle 1 relative to the origin of the world coordinate system.

[0135] (D) Convert the world coordinate system to the human eye coordinate system

[0136]

[0137] In the above formula, R is a 3*3 rotation matrix, representing the angular difference between the forward direction of vehicle 1 and the human eye, and t is a 3*1 translation matrix, representing the translation of the rear axle of vehicle 1 relative to the human eye.

[0138] (E) Conversion of human eye coordinate system to AR HUD coordinate system

[0139]

[0140]

[0141] In the above formula, (x, y) represents the coordinate position on the AR HUD, (Xc, Yc, Zc) represents the position of the object in the human eye coordinate system, and f represents the focal length of the AR HUD (the distance from the focal plane to the eyebox).

[0142] The conversion method and conversion parameters of the entire process can be calibrated when each vehicle leaves the factory, and a five-step end-to-end transformation matrix X from the "real-world road image coordinate system to the AR HUD coordinate system" and the human eye position correction matrix E are generated. If the coordinates of the real-world road image containing AR information to be rendered are matrix V, and the real-time spatial position of the human eye is P, then the coordinates of the AR information displayed by the AR HUD are: O = V*X*P*E.

[0143] 305: Displaying AR information in the front field of view based on the display information in the AR HUD coordinate system.

[0144] For example, as shown in FIG2 , the position where the turn icon A001 is displayed on the road real scene image A01 is the pedestrian passage P2 ′ at the calibrated red light turn.

[0145] Figure 10 This is a possible functional framework diagram of a vehicle 1 provided in an embodiment of the present application. Figure 10As shown, the functional framework of the vehicle 1 may include various subsystems, such as the sensor system 10, the control system 20, one or more peripheral devices 16 (one is shown as an example), the power supply 40, the computer system 50 and the head-up display system 60. Optionally, the vehicle 1 may also include other functional systems, such as an engine system that provides power to the vehicle 1, etc., which are not limited in this application.

[0146] The sensor system 10 may include several detection devices that can sense the measured information and convert the sensed information into electrical signals or other required forms of information output according to certain rules. As shown in the figure, these detection devices may include a global positioning system 11 (GPS), a vehicle speed sensor 12, an inertial measurement unit 13 (IMU), a radar unit 14, a laser rangefinder 15, a camera unit 16, a wheel speed sensor 17, a steering sensor 18, a gear position sensor 19, or other components used for automatic detection, etc., and this application does not limit them.

[0147] The Global Positioning System (GPS) 11 utilizes GPS positioning satellites for real-time global positioning and navigation. In this application, the GPS 11 can be used to locate the vehicle 1 in real time, providing information about the vehicle's geographic location. The vehicle speed sensor 12 is used to detect the vehicle's speed. The inertial measurement unit 13, which may include a combination of an accelerometer and a gyroscope, measures the angular rate and acceleration of the vehicle 1. For example, while the vehicle 1 is traveling, the inertial measurement unit can measure changes in the vehicle's position and angle based on the vehicle's inertial acceleration.

[0148] Radar unit 14 can also be referred to as a radar system. The radar unit uses wireless signals to sense objects in the current environment of vehicle 1. Optionally, the radar unit can also sense information such as the object's speed and direction of travel. In practical applications, the radar unit can be configured as one or more antennas for receiving or transmitting wireless signals. The laser rangefinder 15 is an instrument that uses modulated laser light to measure the distance to a target object. In practical applications, the laser rangefinder may include, but is not limited to, any combination of one or more of the following elements: a laser source, a laser scanner, and a laser detector.

[0149] The camera unit 16 is used to capture images, such as pictures and videos. In this application, while the vehicle 1 is in motion or after the camera device is activated, the camera device can capture images of the vehicle 1's environment in real time. For example, while the vehicle 1 is entering or exiting a tunnel, the camera device can continuously capture corresponding images in real time. In practical applications, the camera device includes, but is not limited to, a driving recorder, a camera, a still camera, or other components for taking photos / videos. This application does not limit the number of such cameras.

[0150] The wheel speed sensor 17 is a sensor for detecting the rotational speed of the wheels of the vehicle 1. Commonly used wheel speed sensors 17 may include, but are not limited to, magnetoelectric wheel speed sensors and Hall-effect wheel speed sensors. The steering sensor 18, also known as a steering angle sensor, may represent a system for detecting the steering angle of the vehicle 1. In practical applications, the steering sensor 18 may be used to measure the steering angle of the steering wheel of the vehicle 1, or to measure an electrical signal representing the steering angle of the steering wheel of the vehicle 1. Optionally, the steering sensor 18 may also be used to measure the steering angle of the tires of the vehicle 1, or to measure an electrical signal representing the steering angle of the tires of the vehicle 1, etc., and this application does not limit this.

[0151] That is, the steering sensor 18 can be used to measure any one or more combinations of the following: the steering angle of the steering wheel, the electrical signal representing the steering angle of the steering wheel, the steering angle of the wheel (the tire of the vehicle 1) and the electrical signal representing the steering angle of the wheel, etc.

[0152] Gear sensor 19 is used to detect the current gear position of vehicle 1. The gear positions of vehicle 1 may vary depending on the manufacturer. For example, autonomous vehicle 1 supports six gear positions: P, R, N, D, 2nd, and L. P (parking) is used for parking, utilizing the vehicle's mechanical system to lock the brakes, preventing movement. R (reverse), also known as reverse gear, is used for reversing. D (drive), also known as forward gear, is used for driving. 2nd (second gear), also known as forward gear, is used to adjust the vehicle's speed. 2nd gear is typically used for driving up and down slopes. L (low), also known as low speed, limits the vehicle's speed. For example, on a downhill road, vehicle 1 in L gear uses engine power for braking, eliminating the driver's need to apply the brakes for extended periods, potentially overheating the brake pads and causing danger.

[0153] The control system 20 may include several components, such as a steering unit 21, a braking unit 22, a lighting system 23, an automatic driving system 24, a map navigation system 25, a network timing system 26, and an obstacle avoidance system 27. Optionally, the control system 20 may also include components such as a throttle controller and an engine controller for controlling the speed of the vehicle 1, which are not limited in this application.

[0154] The steering unit 21 may represent a system for adjusting the direction of travel of the vehicle 1, which may include but is not limited to a steering wheel, or any other structural device for adjusting or controlling the direction of travel of the vehicle 1. The braking unit 22 may represent a system for slowing down the travel speed of the vehicle 1, and may also be referred to as the braking system of the vehicle 1. It may include but is not limited to a brake controller, a decelerator, or any other structural device for slowing down the vehicle 1. In actual applications, the braking unit 22 may use friction to slow down the tires of the vehicle 1, thereby slowing down the travel speed of the vehicle 1. The lighting system 23 is used to provide lighting or warning functions for the vehicle 1. For example, when the vehicle 1 is driving at night, the lighting system 23 may enable the headlights and taillights of the vehicle 1 to provide light brightness for the vehicle 1 to travel and ensure the safe driving of the vehicle 1. In actual applications, the lighting system includes but is not limited to headlights, taillights, width lights, and warning lights.

[0155] The autonomous driving system 24 may include hardware and software systems for processing and analyzing data input to the autonomous driving system 14 to obtain actual control parameters for each component in the control system 20, such as the desired brake pressure of the brake controller in the brake unit and the desired torque of the engine. This facilitates the control system 20 to implement corresponding controls and ensure the safe driving of the vehicle 1. Optionally, the autonomous driving system 14 can also determine obstacles facing the vehicle 1 and characteristics of the vehicle 1's environment (such as the lane the vehicle 1 is currently traveling in, road boundaries, and upcoming traffic lights) through data analysis. The data input to the autonomous driving system 14 can be image data captured by a camera device or data collected by various components in the sensor system 10, such as the steering wheel angle provided by a steering angle sensor and the wheel speed provided by a wheel speed sensor, etc., although this application does not limit this.

[0156] The map navigation system 25 is used to provide map information and navigation services for the vehicle 1. In actual applications, the map navigation system 25 can plan an optimal driving route, such as a route with the shortest distance or less traffic, based on the positioning information of the vehicle 1 provided by the GPS (specifically, the current location of the vehicle 1) and the destination address entered by the user. This facilitates the vehicle 1 to navigate along the optimal driving route to reach the destination address. Optionally, in addition to providing navigation functions, the map navigation system can also provide or display corresponding map information to the user based on the user's actual needs, such as displaying the road section currently traveled by the vehicle 1 on a map in real time, etc. This is not limited in this application.

[0157] The network time system (NTS) 26 provides a time synchronization service to ensure that the vehicle 1's current system time is synchronized with the network standard time, thereby providing more accurate time information for the vehicle 1. Specifically, the NTS 26 obtains a standard time signal from a GPS satellite and uses this time signal to synchronize the vehicle 1's current system time, ensuring that the vehicle 1's current system time is consistent with the obtained standard time signal.

[0158] The obstacle avoidance system 27 is used to predict obstacles that the vehicle 1 may encounter during its travel and to control the vehicle 1 to circumvent or bypass the obstacles to ensure normal travel. For example, the obstacle avoidance system 27 can analyze and determine possible obstacles in the vehicle 1's path using sensor data collected by the various components of the sensor system 10. If the obstacle is large, such as a fixed structure (building) on the roadside, the obstacle avoidance system 27 can control the vehicle 10 to circumvent the obstacle for safe travel. Conversely, if the obstacle is small, such as a small rock on the road, the obstacle avoidance system 27 can control the vehicle 1 to bypass the obstacle and continue forward.

[0159] Peripheral devices 16 may include several components, such as the illustrated communication system 31, touch screen 32, user interface 33, microphone 34, and speaker 35. Communication system 31 is used to enable network communication between vehicle 1 and other devices. In practice, communication system 31 may utilize wireless or wired communication technologies to enable network communication between vehicle 1 and other devices. Wired communication technologies may involve communication between vehicle 1 and other devices via network cables or optical fibers. The wireless communication technology includes but is not limited to global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC) and infrared technology (IR), etc.

[0160] The touch screen 32 can be used to detect operating instructions on the touch screen 32. For example, the user performs touch operations on the content data displayed on the touch screen 32 according to actual needs to implement the function corresponding to the touch operation, such as playing multimedia files such as music and videos. The user interface 33 can specifically be a touch panel for detecting operating instructions on the touch panel. The user interface 33 can also be a physical button or a mouse. The user interface 34 can also be a display screen for outputting data, displaying images or data. Optionally, the user interface 34 can also be at least one device belonging to the category of peripheral devices, such as a touch screen, a microphone, and a speaker.

[0161] Microphone 34, also known as a mouthpiece or speaker, converts audio signals into electrical signals. When making a call or sending a voice message, the user approaches the microphone and speaks, which inputs the audio signal into the microphone. Speaker 35, also known as a horn, converts audio electrical signals into audio signals. Vehicle 1 uses speaker 35 to listen to music, make hands-free calls, and more.

[0162] Power supply 40 represents a system that provides power or energy to vehicle 1 and may include, but is not limited to, rechargeable lithium batteries or lead-acid batteries. In practical applications, one or more battery assemblies in the power supply are used to provide the power or energy required to start vehicle 1. This application does not limit the type and material of the power supply. Alternatively, power supply 40 may be an energy source, such as gasoline, diesel, ethanol, solar cells or panels, for providing energy to vehicle 1. This application does not limit this.

[0163] Several functions of the vehicle 1 are controlled and implemented by the computer system 50. The computer system 50 may include one or more processors 51 (one processor is shown as an example) and a memory 52 (also referred to as a storage device). In actual applications, the memory 52 is also inside the computer system 50, or it can be outside the computer system 50, for example, as a cache in the vehicle 1, etc., which is not limited in this application.

[0164] The processor 51 may include one or more general-purpose processors, such as a graphics processing unit (GPU). The processor 51 may be configured to execute relevant programs or instructions corresponding to the programs stored in the memory 52 to implement corresponding functions of the vehicle 1 .

[0165] The memory 52 may include volatile memory, such as RAM; the memory may also include non-volatile memory, such as ROM, flash memory, HDD or solid-state drive (SSD); the memory 52 may also include a combination of the above types of memory. The memory 52 may be used to store a set of program codes or instructions corresponding to the program codes, so that the processor 51 can call the program codes or instructions stored in the memory 52 to implement the corresponding functions of the vehicle 1. This function includes but is not limited to Figure 10 Some or all of the functions shown in the functional framework diagram of the vehicle 1 are shown. In this application, the memory 52 may store a set of program codes for controlling the vehicle 1, and the processor 51 may call the program codes to control the safe driving of the vehicle 1. How to achieve safe driving of the vehicle 1 is described in detail below in this application.

[0166] Optionally, in addition to storing program code or instructions, memory 52 may also store information such as road maps, driving routes, and sensor data. Computer system 50 may be combined with other components shown in the functional framework diagram of vehicle 1, such as sensors and GPS in the sensor system, to implement relevant functions of vehicle 1. For example, computer system 50 may control the driving direction or speed of vehicle 1 based on data input from sensor system 10, although this application does not limit this.

[0167] The head-up display system 60 may include several components, such as the illustrated windshield 61, a controller 62, and a head-up display 63. The controller 222 is used to generate images based on user instructions and transmit these images to the head-up display 200. The head-up display 200 may include an image generation unit, a pluggable lens assembly, and a reflector assembly. The windshield cooperates with the head-up display to implement the optical path of the head-up display system, thereby presenting the target image in front of the driver. It should be noted that the functions of some components of the head-up display system may also be performed by other subsystems of the vehicle 1. For example, the controller 62 may also be a component of the control system.

[0168] Among them, this application Figure 10 The four subsystems shown in the figure—the sensor system 10, the control system 20, the computer system 50, and the head-up display system 60—are merely examples and are not intended to be limiting. In actual applications, vehicle 1 may combine several components within vehicle 1 according to different functions to create subsystems with corresponding functions. For example, vehicle 1 may also include an electronic stability program (ESP) and an electric power steering (EPS) system, not shown. The ESP system may consist of some sensors from the sensor system 10 and some components from the control system 20. Specifically, the ESP system may include wheel speed sensors 17, steering sensors 18, lateral acceleration sensors, and control units associated with the control system 20. The EPS system may consist of some sensors from the sensor system 10, some components from the control system 20, and a power supply 40. Specifically, the EPS system may include some sensors from the sensor system 10, some components from the control system 20, and some components such as a power supply 40. Specifically, the EPS system may include the steering sensor 18, the generator and reducer associated with the control system 20, and a battery power supply. For another example, the head-up display system may also include a user interface 33 and a touch screen 32 in peripheral devices to implement the function of receiving user instructions. The head-up display system may also include a camera unit in the sensor system to cooperate with the controller 1203 to generate images. For example, the camera unit sends the image to the controller 1203.

[0169] It should be noted that the above Figure 10This is only a schematic diagram of a possible functional framework of the vehicle 1. In actual applications, the vehicle 1 may include more or fewer systems or components, which is not limited in this application.

[0170] The above-mentioned vehicle 1 can be a car, truck, motorcycle, bus, ship, airplane, helicopter, lawn mower, entertainment vehicle, amusement park vehicle 1, construction equipment, tram, golf cart, train, and cart, etc., and the embodiments of the present application are not particularly limited.

[0171] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0172] Program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0173] Program code can be implemented with a high-level programming language or an object-oriented programming language to communicate with the processing system. Where necessary, program code can also be implemented in assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0174] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including but not limited to floppy disks, optical disks, optical discs, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, a machine-readable medium includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0175] In the accompanying drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order may not be required. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of a structural or method feature in a particular figure does not imply that such feature is required in all embodiments, and in some embodiments, such features may not be included or may be combined with other features.

[0176] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems raised by this application. In addition, in order to highlight the innovative part of this application, the above-mentioned device embodiments of this application do not introduce units / modules that are not closely related to solving the technical problems raised by this application. This does not mean that other units / modules do not exist in the above-mentioned device embodiments.

[0177] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "including a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0178] Although the present application has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the application.

Claims

1. A method for displaying AR information in real-time mobile scenes, applied to electronic devices, characterized in that: The method comprises: Acquire a first real-scene image of a road at a first position at a first moment during movement of the electronic device; If the first real road image contains a bend or an uphill or downhill road, convert the first real road image captured by the camera angle into an image in a world coordinate system, determine spatial feature information based on the first real road image in the world coordinate system, and determine predicted display information of the AR information corresponding to the first target object in the first real road image based on the spatial feature information; Determining a movement distance of the electronic device during the process of moving the electronic device from the first position to the second position based on the distance between the electronic device and the target object at the moment when the camera captures the real road scene image at the second position, and a movement change of the target object between the first real road scene image and the real road scene image captured by the camera at the second position; determining calibration information of the predicted display information based on a movement distance of the electronic device during a process of the electronic device moving from the first position to the second position; acquiring a calibrated display position based on the calibration information; When the electronic device is at the second position, displaying the AR information in a front field of view based on the calibrated display position; The determining of spatial feature information based on the first road real scene image in the world coordinate system includes: Determining spatial feature information of the first road real scene image in the world coordinate system based on simultaneous positioning and mapping technology; The spatial feature information of the electronic device obtained by the perception system based on autonomous driving and the offset matrix calibration are based on the spatial feature information determined by the synchronous positioning and mapping technology.

2. The method according to claim 1, characterized in that When the electronic device is at the second position, before displaying the AR information in the front field of view based on the calibrated display position, the method further includes: converting the calibrated display information from the image coordinate system to the human eye coordinate system, and then converting it into the augmented reality head-up display coordinate system.

3. The method according to claim 1, characterized in that The AR information includes driving assistance information and / or navigation guidance information.

4. A vehicle, characterized in that: The vehicle includes a heads-up display and an AR information display system; The AR information display system is used to obtain a first road real scene image at a first position at a first moment during the movement of the vehicle; The AR information display system is configured to, if the first real road scene image contains a bend or an uphill or downhill road, convert the first real road scene image captured by the camera into an image in a world coordinate system, determine spatial feature information based on the first real road scene image in the world coordinate system, and determine predicted display information of the AR information corresponding to the first target object based on the spatial feature information; The AR information display system is configured to determine a movement distance of the electronic device during the process of moving the electronic device from the first position to the second position based on a distance between the electronic device and a target object at the moment when the camera captures a real road scene image at the second position, and a movement change amount of the target object between the first real road scene image and the real road scene image captured by the camera at the second position; and determining calibration information of the predicted display information based on a movement distance of the electronic device during the process of moving the electronic device from the first position to the second position; The AR information display system is configured to obtain a calibrated display position based on the calibration information; The AR information display system is configured to control the head-up display to display the AR information based on the calibrated display position when the vehicle is at the second position; The AR display system is configured to determine spatial feature information of the first road real scene image in the world coordinate system based on a synchronous positioning and mapping technology; The spatial feature information of the electronic device obtained by the perception system based on autonomous driving and the offset matrix calibration are based on the spatial feature information determined by the synchronous positioning and mapping technology.

5. The vehicle according to claim 4, characterized in that Before displaying the AR information in the front field of view based on the calibrated display position, the AR information display system is used to convert the calibrated display information from the image coordinate system to the human eye coordinate system, and then to the augmented reality head-up display coordinate system.

6. The vehicle according to claim 4, characterized in that The AR information includes driving assistance information and / or navigation guidance information.

7. A real-time mobile AR information display device, characterized in that: The device comprises: one or more memories storing instructions; A processor, wherein the processor is coupled to the one or more memories, and when the instructions are executed by the processor, the device executes the real-time mobile AR information display method in a real scene according to any one of claims 1 to 3.

8. The device according to claim 7, characterized in that The device is a vehicle, a mobile phone, a watch or AR glasses.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on the electronic device, the electronic device executes the real-time mobile AR information display method in a real scene according to any one of claims 1 to 3.

10. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the real-time mobile AR information display method according to any one of claims 1 to 3.

Images