Augmented Reality-Based Information Display Method, System, Device and Projection Device

Abstract

Embodiments of the present application disclose an information display method, system, device, projection device, and storage medium based on augmented reality. The method includes: obtaining real-scene information collected by a scene perception device; obtaining a target display area determined based on a first spatial pose of a user, the first spatial pose being determined based on an eye tracking device; obtaining a coordinate transformation rule corresponding to mapping the real-scene information to the target display area; generating driving guidance information based on the real-scene information; and displaying the driving guidance information at a corresponding position in the target display area based on the coordinate transformation rule. It realizes that the driving guidance information is displayed at the corresponding position in the target display area determined by the first spatial pose of the user determined based on the eye tracking device through the coordinate transformation rule, and realizes real-time tracking of the first spatial pose of the user by the eye tracking device, so that it can adapt to the change of the eye pose and dynamically adjust the display area and display position of the driving guidance information.

Description

Technical Field

[0001] The present application relates to the technical field of coordinate transformation, and more specifically, to an information display method, system, device, projection device, and storage medium based on augmented reality. Background Art

[0002] HUD (head up display) is a head-up display, which can display important information on a transparent glass in front of the line of sight. It was first applied to fighter planes, and its main purpose is to enable pilots to avoid having to frequently focus on looking down at the data in the instrument panel, so as to prevent pilots from being unable to observe the environmental information in the front area of flight when viewing the data in the instrument panel. In order to reduce accidents caused by users looking down at the instrument panel or the center console, HUD has been introduced from airplanes to the automotive field.

[0003] However, the existing way of displaying information by HUD lacks intelligence. Taking car driving as an example, with the addition of more assisted driving information such as road conditions, navigation, and danger warnings, the user's body and head will swing with changes in road conditions or personal activities, etc., causing changes in the user's spatial pose. In this case, it may lead to problems such as offset and other mismatches between the image displayed by the user on the HUD and the real scene image. Summary of the Invention

[0004] In view of the above problems, the present application proposes an information display method, system, device, projection device, and storage medium based on augmented reality to improve the above problems.

[0005] In a first aspect, an embodiment of the present application provides an information display method based on augmented reality, the method including: acquiring real-scene information collected by a scene perception device; acquiring a target display area determined based on a first spatial pose of a user, the first spatial pose being determined based on an eye tracking device; acquiring a coordinate transformation rule corresponding to mapping the real-scene information to the target display area; generating driving guidance information based on the real-scene information; and displaying the driving guidance information at a corresponding position in the target display area based on the coordinate transformation rule.

[0006] Second aspect, an embodiment of the present application provides an augmented reality information display device, and the information display device includes an image perception module, a coordinate transformation module, and a display module: The image perception module is configured to obtain real-scene information collected by an image perception device; The coordinate transformation module is configured to obtain a target display area determined based on a first spatial pose of a user, and the first spatial pose is determined based on an eye tracking device; The coordinate transformation module is further configured to obtain a coordinate transformation rule corresponding to mapping the real-scene information to the target display area; The display module is configured to generate driving guidance information based on the real-scene information; The display module is further configured to display the driving guidance information at a corresponding position in the target display area based on the coordinate transformation rule.

[0007] Third aspect, an embodiment of the present application provides an augmented reality vehicle-mounted information display system, and the system includes: A scene perception device, configured to collect real-scene information of the external environment of the vehicle; An eye tracking device, configured to obtain the line-of-sight movement range of a user; An image processing device, configured to obtain a first spatial pose of the user based on the line-of-sight movement range, obtain a target display area determined based on the first spatial pose, obtain a coordinate transformation rule corresponding to mapping the real-scene information to the target display area, generate driving guidance information based on the real-scene information, and generate target position coordinates for displaying the driving guidance information in the target display area based on the coordinate transformation rule; A HUD display device, configured to display the driving guidance information at the target position coordinates in the target display area.

[0008] Fourth aspect, an embodiment of the present application provides a projection device, including one or more processors and a memory; One or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs are configured to execute the method described in the first aspect above.

[0009] Fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which program code is stored, and when the program code runs, the method described in the first aspect above is executed.

[0010] An information display method, system, device, projection device, and storage medium based on augmented reality provided by the present application obtain real-scene information collected by an image sensing device, then obtain a target display area determined based on the first spatial pose of the user, where the first spatial pose is determined based on an eye tracking device, then obtain a coordinate transformation rule for mapping the real-scene information to the coordinates corresponding to the target display area, and generate driving guidance information based on the real-scene information. Then, based on the coordinate transformation rule, the driving guidance information is displayed at the corresponding position in the target display area. Thus, through the above method, the driving guidance information generated based on the real-scene information is displayed at the corresponding position in the target display area determined by the first spatial pose of the user determined based on the eye tracking device through the coordinate transformation rule. It realizes real-time tracking of the first spatial pose of the user by the eye tracking device, enables adaptation to changes in the eye pose, dynamically adjusts the display area and display position of the driving guidance information, and enables the user to accurately and conveniently view virtual driving guidance information corresponding to the driving scene during driving, improving driving safety and comfort, and thus enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those skilled in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0012] Figure 1 FIG. shows a flowchart of a method for an information display method based on augmented reality proposed in an embodiment of the present application.

[0013] Figure 2 FIG. shows a structural example diagram of an in-vehicle information display system based on augmented reality for the information display method based on augmented reality provided in this embodiment.

[0014] Figure 3 FIG. shows an example diagram of the HUD virtual image plane of the HUD display device in this embodiment.

[0015] Figure 4 FIG. shows a schematic diagram of the relationship between the eyes of the driver and the HUD virtual image plane in this embodiment.

[0016] Figure 5 FIG. shows an example diagram of displaying driving guidance information through the information display system based on augmented reality proposed in the present application in a dangerous scenario provided in this embodiment.

[0017] Figure 6 FIG. shows a flowchart of a method for an information display method based on augmented reality proposed in another embodiment of the present application.

[0018] Figure 7 Shows an example diagram of the display effect of the augmented reality-based information display system provided in this embodiment.

[0019] Figure 8 Shows another example diagram of the display effect of the augmented reality-based information display system provided in this embodiment.

[0020] Figure 9 Shows the flowchart of a method for an augmented reality-based information display method proposed in another embodiment of this application.

[0021] Figure 10 Shows an example diagram of adjusting a target display area based on a change vector provided in this embodiment.

[0022] Figure 11 Shows an example diagram of the processing process of the augmented reality-based information display method proposed in this embodiment.

[0023] Figure 12 Shows the structural block diagram of an augmented reality-based information display device proposed in an embodiment of this application.

[0024] Figure 13 Shows the structural block diagram of a projection device for executing an augmented reality-based information display method according to an embodiment of this application.

[0025] Figure 14 Shows the storage unit for storing or carrying the program code for implementing an augmented reality-based information display method according to an embodiment of this application. Detailed implementation manners

[0026] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0027] HUD (head up display) is a head-up display (or heads-up display), which can display important information on a transparent glass in front of the line of sight. It was first applied to fighter jets, and its main purpose is to enable pilots not to need to frequently concentrate on looking down at the data on the instrument panel, thereby avoiding the situation where pilots cannot observe the environmental information in the front flight area when viewing the data on the instrument panel. In order to reduce accidents caused by users looking down at the instrument panel or the center console, HUD has been introduced from airplanes to the automotive field.

[0028] HUD is mainly divided into two types: rear-mounted (also known as Combine HUD, C-type HUD) and front-mounted (also known as WindshieldHUD, W-type HUD). Among them, the front-mounted HUD uses the windshield as a combiner, and projects the content required by the driver to the front windshield through the optical system. The human eye can observe the HUD virtual image and the external scene at the same time through the windshield within the field of vision, thereby improving driving safety and driving comfort. However, some existing HUD devices only display virtual information in front of the driver's line of sight, and are not integrated with the real environment. With the addition of more auxiliary driving information such as road conditions, navigation, and hazard warnings, this mismatch between virtual content and real scenes will instead distract the driver's attention.

[0029] Augmented Reality (AR) is a technology that cleverly integrates virtual information with the real world.

[0030] As a way, with the development of autonomous driving and augmented reality and mixed reality technology, AR technology can be introduced into the HUD field. AR-HUD can solve the problem of separation and mismatch between traditional HUD virtual information and actual scenes through the combination of AR technology and front-mounted HUD, and improve driving safety and comfort while enriching the HUD display content. However, the existing HUD information display method lacks intelligence. Taking car driving as an example, with the addition of more auxiliary driving information such as road conditions, navigation and hazard warnings, the user's body and head will swing with changes in road conditions or personal activities, causing the user's spatial posture to change. In this case, the user may see that the image displayed by the HUD is offset from the real scene image and other mismatches.

[0031] Therefore, in order to improve the above-mentioned problems, the inventors proposed the method provided by the present application, which can obtain real-scene information collected by an image perception device, and then obtain a target display area determined based on the user's first spatial posture, wherein the first spatial posture is determined based on a human eye tracking device, and then obtain the coordinate transformation rule corresponding to the real-scene information mapped to the target display area, and then generate driving guidance information based on the real-scene information, and then display the driving guidance information at the corresponding position of the target display area based on the coordinate transformation rule, so that the driving guidance information generated based on the real-scene information is displayed at the corresponding position of the target display area determined based on the first spatial posture of the user determined by the human eye tracking device through the coordinate transformation rule, and the first spatial posture of the user is tracked in real time by the human eye tracking device, so that it can adapt to changes in the human eye posture, and dynamically adjust the display area and display position of the driving guidance information, so that the user can accurately and conveniently view the virtual driving guidance information corresponding to the driving scene during driving, thereby improving driving safety and comfort, and thus improving user experience.

[0032] The embodiments of the present application will be specifically described below in conjunction with the accompanying drawings.

[0033] Please refer to Figure 1 , which is a flowchart of a method for an information display method based on augmented reality provided by an embodiment of the present application. The method of this embodiment can be executed by a device for processing real-scene information based on augmented reality, and the device can be implemented in a hardware and / or software manner. The method includes:

[0034] Step S110: Obtain real-scene information collected by a scene perception device.

[0035] Among them, the real-scene information in the embodiments of the present application can be real-scene information corresponding to multiple scenarios. Optionally, the multiple scenarios can include, but are not limited to, a driving scenario, a tourism scenario, and an outdoor activity scenario, etc. For example, if it is a driving scenario, the real-scene information can include lanes, signs, dangerous pedestrians (such as vulnerable groups such as blind people, elderly people walking alone, pregnant women or children), and vehicles, etc.; if it is a tourism scenario, the real-scene information can include tourist destination signs, tourist routes, tourist attraction information, and tourist attraction weather information, etc.; if it is an outdoor activity scenario, the real-scene information can include the current location information and the information of nearby convenience stores, etc.

[0036] Optionally, the scene perception device can include sensing devices such as lasers and lidars, and can also include image acquisition devices such as cameras (including monocular cameras, binocular cameras, and RGB-D cameras, etc.). As a way, real-scene information corresponding to the current scene can be obtained through the scene perception device. For example, assume that the current scene is a driving scenario and the scene perception device is a camera, and the camera can be installed on the vehicle (optionally, the installation position can be adjusted according to the vehicle model structure or actual needs), so that the camera can obtain real-scene information related to driving in real time. Among them, for the acquisition principle and implementation of the scene perception device (including lasers, lidars or cameras) to collect real-scene information, reference can be made to related technologies and will not be elaborated here.

[0037] Step S120: Obtain a target display area determined based on the first spatial pose of the user, where the first spatial pose is determined based on an eye tracking device.

[0038] In this embodiment, the first spatial pose may be the human eye pose determined based on a human eye tracking device. It can be understood that if the line of sight range of the user changes, the corresponding human eye pose may also change. However, if the user's body rotates while the line of sight range of the eyes remains unchanged, the human eye pose can remain unchanged. In this way, since the user's eyes can see the real scene information that needs attention in the driving scenario (for example, road conditions, dangerous pedestrians and vehicles, etc.), safe driving can still be carried out. Then, as a way, the first spatial pose may be determined based on the user's current human eye pose. In this way, the target display area determined based on the user's current human eye pose can be obtained. Optionally, the human eye tracking device may be a device with a camera function such as a camera, and the specific type may not be limited.

[0039] Optionally, if the human eye pose of the user changes, but the change range is very small (for example, less than a pre-set threshold), the first spatial pose of the user may be the sitting pose of the user in the driving state, or the sitting pose after adjusting the seat (here it can be the seat adjusted by the current user for the first time). It can be understood that different sitting poses of the user correspond to different spatial poses. As a way, the sitting pose state of the user after adjusting the seat can be used as the first spatial pose of the user. Optionally, if the change range of the human eye pose of the user is not less than the pre-set threshold, the first spatial pose of the user can be re-determined according to the user's current human eye pose.

[0040] In this embodiment, the target display area is an area for displaying virtual image information corresponding to the real scene information. Taking the driving scenario as an example, the target display area may be an area on the windshield of the vehicle for displaying the projected virtual image information corresponding to the real scene information. Optionally, the target display areas corresponding to different spatial poses of the same user may be different, and the target display areas corresponding to the spatial poses of different users may be different.

[0041] In order to eliminate the display difference between the position of the virtual image information displayed corresponding to the real scene information and the actual position of the real scene information, as a way, the target display area determined based on the user's first spatial pose can be obtained, so that the virtual image information corresponding to the real scene information can be displayed in the target display area, reducing the aforementioned display difference, and further improving the accuracy of the display position of the virtual image information corresponding to the real scene information.

[0042] Step S130: Obtain the coordinate transformation rule for mapping the real scene information to the corresponding target display area.

[0043] Among them, the coordinate transformation rule can be used to map the coordinates of the real scene information to the corresponding coordinates of the target display area. As a way, when the real scene information and the target display area are obtained, the coordinate transformation rule for mapping the real scene information to the target display area can be obtained, so that subsequently, the driving guidance information corresponding to the real scene information can be accurately displayed at the corresponding position of the target display area based on the coordinate transformation rule.

[0044] Optionally, the coordinate transformation rule may include a first transformation matrix and a second transformation matrix. Among them, the first transformation matrix can be used to determine the reference world coordinates corresponding to the coordinates of the real scene information collected by the scene perception device, and the second transformation matrix can be used to convert the reference world coordinates into view coordinates that match the offset of the user's perspective within the target display area. Among them, the reference world coordinates can be understood as the relative position coordinates of the real scene information in the coordinate system established corresponding to the scene perception device. Optionally, the reference world coordinates in this embodiment can be understood as the world coordinates relatively stationary with respect to the vehicle. The view coordinates can be understood as the relative position coordinates of the reference world coordinates in the coordinate system corresponding to the target display area.

[0045] As a way of implementation, the first transformation matrix and the second transformation matrix can be obtained, and then the product of the parameters represented by the first transformation matrix and the parameters represented by the second transformation matrix is used as the coordinate transformation rule for mapping the real scene information to the target display area.

[0046] Optionally, the first transformation matrix may include a first rotation matrix and a first translation vector. Among them, the first rotation matrix can be used to rotate the coordinates of the real scene information collected by the scene perception device, and the first translation vector can be used to translate the coordinates. As a way, the reference world coordinates corresponding to the coordinates of the real scene information collected by the scene perception device can be determined based on the first rotation matrix and the first translation vector.

[0047] Optionally, the second transformation matrix may include a perspective offset matrix, a transpose matrix, and a projection matrix. Among them, the projection matrix can be used to determine the mapping range for mapping the real scene information to the target display area, the perspective offset matrix can be used to determine the degree of offset of the user's perspective detected by the eye tracking device, and the transpose matrix can be used to determine the relative position for displaying the driving guidance information within the mapping range. As a way, the reference world coordinates can be converted into view coordinates that match the offset of the user's perspective within the target display area based on the perspective offset matrix, the transpose matrix, and the projection matrix. Among them, the perspective offset matrix may include a first view coordinate, the transpose matrix represents the transpose facing the first transformation matrix, the transpose matrix may include a spatial unit vector located in the target display area, and the projection matrix may include a field of view angle parameter. Optionally, the field of view angle parameter may include a distance parameter and a scale parameter associated with the user's perspective.

[0048] Taking the driving scenario as an example, the present embodiment will be exemplarily described as follows:

[0049] Please refer to Figure 2 , which is a structural example diagram of an in-vehicle information display system based on augmented reality applicable to the augmented reality-based information display method provided in this embodiment. As Figure 2 shown, the in-vehicle information display system based on augmented reality may include a scene perception device, an image processing device, a HUD display device, and an eye tracking device. Among them, the scene perception device may be used to collect real-scene information of the vehicle's external environment. The eye tracking device may be used to obtain the range of the user's eye movement. The image processing device may be used to obtain the user's first spatial pose based on the range of eye movement, obtain the target display area determined based on the first spatial pose, obtain the coordinate transformation rule corresponding to mapping the real-scene information to the target display area, generate driving guidance information based on the real-scene information, and generate the target position coordinates for displaying the driving guidance information in the target display area based on the coordinate transformation rule. The HUD display device may be used to display the driving guidance information at the target position coordinates of the target display area.

[0050] Among them, the image processing device may be the processor chip of the in-vehicle system, or the processing chip of an independent in-vehicle computer system, or the processor chip integrated in the scene perception device (such as a lidar), etc., which is not limited herein.

[0051] In one implementation, the in-vehicle information display system may include a vehicle, a driver, an eye tracking device, a scene perception device, an image processing device, and a HUD display device with AR-HUD function. As an implementation, the scene perception device may be installed on the vehicle and can obtain driving-related scene information (which can also be understood as the aforementioned real-scene information). The driver sits in the driver's seat of the vehicle. The eye tracking device is installed inside the vehicle. The eye tracking device can track the reasonable positions of the basic movement range of the driver's eyes. Optionally, the reasonable positions here can be understood as the positions where the driver's line of sight may turn in cooperation with driving needs during driving, such as turning left, turning right, or turning backward, etc. The specific turning direction and angle may not be limited. The HUD display device is installed on the vehicle's front windshield, and the position of the HUD display device can be adjusted so that the driver's eyes can see the entire virtual image corresponding to the driving scene information. Optionally, the image processing device can adapt to the changes in the spatial pose of the driver's eyes. In this way, the image processing device can real-time convert the real-scene information collected by the scene perception device into an image fused with the real-scene information of the real scene and send it to the HUD display device for display.

[0052] As a way, after the scene perception device acquires the real scene information, it can obtain the position coordinates of the real scene information in the world coordinate system (such as Figure 2 the O-xyz shown) through position acquisition methods such as GPS positioning. Subsequently, it can select the world coordinate origin and the coordinate axis directions based on the traveling direction of the vehicle, and determine the reference world coordinate system that is relatively stationary with the vehicle according to the world coordinate origin and the coordinate axis directions. By determining the reference world coordinate system that is relatively stationary with the vehicle, the reference world coordinates corresponding to the coordinates of the real scene information in the reference world coordinate system can be obtained. Among them, the selection method of the world coordinate origin and the coordinate axis directions can refer to the related technology, which will not be elaborated here. It should be noted that the reference world coordinate system can be understood as the coordinate system obtained after rotating and / or translating the world coordinate system.

[0053] For example, as an implementation method, in the case of determining the reference world coordinate system that is relatively stationary with the vehicle, the spatial pose of the scene perception device in the reference world coordinate system can be obtained. In this way, the perception module transformation matrix M (i.e., the aforementioned first transformation matrix) can be calculated according to the spatial pose of the scene perception device in the reference world coordinate system. Exemplarily, it can be assumed that the transformation process of changing the world coordinate system to the reference world coordinate system includes the first rotation matrix (which can also be understood as the total rotation matrix of the scene perception device) R M and the first translation vector T M . Optionally, the relationship between the perception module transformation matrix M, the first rotation matrix R M and the first translation vector T M can be expressed as:

[0054]

[0055] Among them,

[0056]

[0057]

[0058] Among them, R Mx , R My , R Mz are respectively the rotation matrices of the perception module transformation matrix M around the x-axis, y-axis, and z-axis of the world coordinate system, and the Euler angles of rotation are α M , β M , γ M in sequence. (T Mx , T My , T Mz ) are the coordinates of the real scene information in the reference world coordinate system.

[0059] Optionally, in the reference world coordinate system, the spatial pose of the virtual image of the plane where the HUD display device is located and the spatial pose of the driver's eyes can be measured. In this way, the aforementioned second transformation matrix can be obtained based on the spatial pose of the virtual image and the spatial pose of the driver's eyes (i.e., the human eye pose of the driver).

[0060] The second transformation matrix (which can also be understood here as the perspective matrix of the virtual image) C can include a view offset matrix T, a transpose matrix N T and a projection matrix P. The view offset matrix T can be determined by the human eye pose of the driver, and the transpose matrix N T can be determined by the spatial pose of the virtual image plane of the HUD display device. The projection matrix P can be jointly determined by the human eye pose of the driver and the spatial pose of the virtual image plane of the HUD display device. As a way, the relationship among the view offset matrix T, the transpose matrix N T and the projection matrix P can be expressed as:

[0061] C = PN T T.

[0062] Among them, the view offset matrix T can include the first view coordinates, and the first view coordinates are the position coordinates of the human eye pose of the driver in the reference world coordinate system. As a way, in this embodiment, the first view coordinates can be represented by (P ex , P ey , P ez ). In this way, the view offset matrix T can be expressed as:

[0063]

[0064] Optionally, the transpose matrix N T can include the spatial unit vectors located in the target display area. As a way, the transpose matrix N in this embodiment T can be expressed as:

[0065]

[0066] Among them, Vr, Vu, and Vn are the spatial unit vectors represented by the virtual image plane corresponding to the HUD display module. Exemplarily, please refer to Figure 3 , which shows an example diagram of the HUD virtual image plane of the HUD display device in this embodiment. As Figure 3 shown, V r is the right vector, V u is the up vector, and V n is the normal vector of the HUD virtual image plane.

[0067] Optionally, the projection matrix includes a field of view angle parameter, and the field of view angle may include a distance parameter and a scale parameter associated with the user's perspective. The relational expression satisfied by the projection matrix P may be:

[0068]

[0069] Among them, the parameters n and f are the far and near distances of the human eye's field of view (i.e., the above-mentioned distance parameter), and the parameters l, r, b, and t can represent the left, right, lower, and upper scales determined by the size and pose relationship between the eye and the HUD virtual image plane (i.e., the above-mentioned scale parameter). Optionally, please refer to Figure 4 , which shows a schematic diagram of the relationship between the driver's eyes and the HUD virtual image plane in this embodiment. As shown in Figure 4 , if d = -(v n ·v a ), then the calculation formulas for the parameters l, r, b, and t can be respectively expressed as:

[0070] l = (v r ·v a )n / d r = (v r ·v b )n / d

[0071] b = (v u ·v a )n / d t = (v u ·v c )n / d

[0072] As a way, when the first transformation matrix and the second transformation matrix are obtained, the product of the parameters represented by the first transformation matrix and the parameters represented by the second transformation matrix can be obtained as the coordinate transformation rule corresponding to mapping the real scene information to the target display area. That is, in this way, the coordinate transformation rule can be expressed as:

[0073] F = MC = MP N T T.

[0074] It should be noted that in this embodiment, the eye tracking device can capture the spatial pose of the driver's eyes in real time and calculate the pose difference between the current moment and the previous moment of the driver's eyes. Optionally, when the pose difference exceeds a pre-set specified threshold (the specific value may not be limited), the above-mentioned second transformation matrix C can be recalculated and updated. Among them, the calculation method of the pose difference of the driver's eyes can be various. For example, it can be the mean square error MSE or the mean absolute error MAE, etc. It can be understood that the calculation method of the pose difference of the driver's eyes can also be customized according to actual needs. Optionally, the specified thresholds corresponding to different calculation methods can be different.

[0075] Step S140: Generate driving guidance information based on the real scene information.

[0076] The driving guidance information in this embodiment may include navigation indication information corresponding to the road conditions, pedestrian warning information, tourist attraction prompt information, etc. The types and specific contents of the driving guidance information may not be limited. For example, as Figure 5 shown, it shows an example diagram of displaying driving guidance information through the information display system based on augmented reality proposed in this application in a dangerous scenario. As Figure 5 shown, the image processing device can convert the real scene information collected by the scene perception device into a HUD virtual image for display on the HUD display device. The specific content of the display is as Figure 5 shown in the right image. In this case, the scene seen by the driver's eyes may include lane guidance information (i.e., Figure 5 the "navigation indication in the virtual image" shown in Figure 5 ) and pedestrian warning information (i.e., the "pedestrian prompt box in the virtual image" shown in

[0077] As a way, when the real scene information is obtained, driving guidance information can be generated based on the real scene information. Optionally, the prompting method of the driving guidance information in this embodiment may not be limited. For example, it can be prompted in the form of icons (such as arrows), pictures, animations, voices or videos, etc. Then, for the driving guidance information with different prompting methods, it can be generated in the corresponding way. Optionally, the generation principle of the driving guidance information for each prompting method based on the real scene information can refer to the related technology and will not be elaborated here. Optionally, the driving guidance information in this embodiment may include at least one prompting method. For example, on the basis of displaying the navigation indication icon corresponding to the road, voice can be combined to prompt the user, so as to more accurately prompt the user for driving guidance, ensure driving safety and improve the user experience.

[0078] Step S150: Display the driving guidance information at the corresponding position in the target display area based on the coordinate transformation rule.

[0079] Optionally, by displaying the driving guidance information at the corresponding position in the target display area based on the coordinate transformation rule, it is possible to avoid the difference between the position where the HUD displays the real scene information and the actual position of the real scene information, and improve the accuracy and reliability of the display.

[0080] An information display method based on augmented reality provided by the present application obtains the real-world information collected by an image sensing device, then obtains a target display area determined based on the first spatial pose of the user, where the first spatial pose is determined based on an eye tracking device. Then, it obtains the coordinate transformation rule corresponding to mapping the real-world information to the target display area, and generates driving guidance information based on the real-world information. Then, based on the coordinate transformation rule, the driving guidance information is displayed at the corresponding position in the target display area. Thus, through the above method, the driving guidance information generated based on the real-world information is displayed at the corresponding position in the target display area determined by the first spatial pose of the user determined based on the eye tracking device through the coordinate transformation rule. It realizes real-time tracking of the first spatial pose of the user by the eye tracking device, enables adaptation to changes in the eye pose, dynamically adjusts the display area and display position of the driving guidance information, and enables the user to accurately and conveniently view the virtual driving guidance information corresponding to the driving scene during driving, improving driving safety and comfort, and thus enhancing the user experience.

[0081] Please refer to Figure 6 , which is a flowchart of an information display method based on augmented reality provided by another embodiment of the present application. The method of this embodiment can be executed by a device for processing real-world information based on augmented reality, and the device can be implemented in a hardware and / or software manner. The method includes:

[0082] Step S210: Obtain the real-world information collected by the scene sensing device.

[0083] Step S220: Obtain the target display area determined based on the first spatial pose of the user.

[0084] Among them, the first spatial pose is determined based on the eye tracking device, and the specific description can refer to the description in the foregoing embodiment and will not be elaborated here.

[0085] Step S230: Obtain the coordinate transformation rule corresponding to mapping the real-world information to the target display area.

[0086] Step S240: Generate driving guidance information based on the real-world information.

[0087] Step S250: Input the position coordinates of the real-world information in the coordinate system corresponding to the scene sensing device into the first transformation matrix to obtain a to-be-processed coordinate transformation matrix.

[0088] As a way, the position coordinates of the real-world information in the coordinate system corresponding to the scene sensing device can be input into the first transformation matrix, and the output result can be used as the to-be-processed coordinate transformation matrix. For example, in a specific application scenario, assume that the position coordinates of the real-world information in the coordinate system corresponding to the scene sensing device are Ow (x, y, z), optionally, after inputting the position coordinates O w into the aforementioned first transformation matrix, the following can be obtained:

[0089]

[0090]

[0091] wherein, O' can be used as the coordinate transformation matrix to be processed, and O W uses homogeneous coordinates.

[0092] Step S260: Perform coordinate transformation on the coordinate transformation matrix to be processed according to the second transformation matrix to obtain the relative position coordinates of the real scene information in the target display area.

[0093] As a way, the coordinate transformation matrix to be processed can be subjected to coordinate transformation according to the aforementioned second transformation matrix to obtain the relative position coordinates of the real scene information in the target display area. Optionally, the specific implementation process of the coordinate transformation can refer to related technologies and will not be elaborated here.

[0094] For example, taking the above example as an example, assuming that in the HUD image of the target display area of the HUD display device, the position coordinates corresponding to the position coordinates O w (x, y, z) are represented as O h (u, v), then after performing coordinate transformation on the coordinate transformation matrix O' to be processed according to the second transformation matrix, the following can be obtained:

[0095]

[0096] wherein, width and height are the width and height of the HUD image, and the unit can be pixels. In this way, O h (u, v) can be used as the relative position coordinates of the real scene information in the target display area.

[0097] Step S270: Display the driving guidance information at the position characterized by the relative position coordinates.

[0098] Optionally, in this way, the driving guidance information corresponding to the real scene information can be displayed at the position characterized by the relative position coordinates.

[0099] The following uses a specific example to illustrate this embodiment:,...

[0100] Please refer to Figure 7 , which shows an example diagram of the display effect of the information display system based on augmented reality provided by this embodiment. As Figure 7As shown, a virtual-real scene fusion system (which can be understood as the augmented reality-based information display system in this application) can be built in relevant modeling software (such as Unity3D, etc.). The virtual-real scene fusion system can include a vehicle, Camera 1 (for simulating the driver's eyes), Script Program 1 for simulating the eye tracking device, the HUD imaging module simulated by Camera 2 and a plane (i.e., the aforementioned HUD display device), the spatial scene information (which can be the spatial scene information under different scenarios) obtained by the image perception module simulated by a checkerboard (implemented by Script Program 2), and the information transformation and image drawing and rendering of the image processing device completed by Script Program 3.

[0101] Optionally, in the simulated virtual-real scene fusion system, the central position at the bottom of the vehicle can be selected as the coordinate origin, the forward direction of the vehicle can be the positive direction of the Z axis, and the right-hand coordinate system can be adopted. Assuming that the driver is sitting in the driver's seat, the eyes of the driver simulated by Camera 1 face forward, and the pose of the HUD virtual camera simulated by Camera 2 is the same as the pose of the driver's eyes. In this way, the scene perception device can obtain the spatial corner point information of the checkerboard on the vehicle, and the image processing device can draw the corner points into the HUD image space (as shown in the lower left corner of the figure below, which is the corner point image drawn by the image processing device into the HUD image space), and then send the drawn image to the HUD display device for display (as shown in the figure below, which conveys the corner point image to the HUD for virtual image display). In this way, the driver's perspective scene as shown in the figure below can be obtained. From the enlarged view of the virtual-real fusion result as shown in the figure below, it can be seen that the virtual and real scenes can be accurately fused. Figure 7 As shown, the lower left corner is the corner point image drawn by the image processing device into the HUD image space Figure 7 As shown, which conveys the corner point image to the HUD for virtual image display. Figure 7 In this way, the driver's perspective scene as shown in the figure below can be obtained. Figure 7 From the enlarged view of the virtual-real fusion result as shown in the figure below, it can be seen that the virtual and real scenes can be accurately fused.

[0102] Among them, the eye tracking device in this embodiment can be configured with a 3D sensor for perceiving the spatial pose of the human eye and an eye tracking algorithm adapted to the output data of the sensor. Optionally, the 3D sensor for perceiving the spatial pose of the human eye can be a binocular camera, or an RGB-D camera, etc., and the eye tracking algorithm can be a computer vision algorithm, or a deep learning algorithm, etc., and specific details can be not limited. As a way, the scene perception device can obtain the real scene information in the driving scene such as lanes, navigation instructions, (hazardous) pedestrians, etc. through a camera combined with GPS, IMU sensors and perception processing algorithms. Optionally, if the user's eye pose changes, the image processing device can adjust the aforementioned first transformation matrix and second transformation matrix according to the changed eye pose, and then draw an image adapted to the changed eye pose, and display the virtual image information through the HUD display device.

[0103] As a way, if the spatial pose of the driver's eyes changes during driving, for example, the driver's line of sight moves as shown in Figure 5 In this case, the eye tracking device can obtain the spatial pose of the driver's eyes in real time and determine whether the change in the eye pose at the current moment exceeds a specified threshold compared to the eye pose at the previous moment. The value of the specified threshold can be set according to the actual situation. Optionally, if it is determined that the change in the driver's eye pose exceeds the specified threshold, in this case, the spatial pose of the driver can be re-obtained based on the changed eye pose of the driver, and then the target display area can be re-determined according to the spatial pose, and then the virtual-real fusion scene graph can be displayed at the corresponding position of the re-determined target display area.

[0104] For example, as an implementation, in Figure 7 Under the shown display effect, if the change in the driver's eye pose has changed compared to the previous moment and the range of change exceeds the specified threshold, then an example diagram of the display effect as shown in Figure 8 can be obtained. The specific value of the moment can be not limited. In some possible implementation manners, the judgment interval can also be changed to a time period or a cycle, etc. As shown in Figure 8 When the change range of the driver's eye pose exceeds the specified threshold, the position where the virtual-real scene fusion graph is displayed in the driver's perspective scene is shifted, so that when the driver's eye pose changes, the driving guidance information corresponding to the real scene information in the driving scene can be presented to the user from the best perspective, avoiding driving safety accidents caused by the user's line of sight deviation, improving the safety and flexibility of driving, and thus improving the user experience.

[0105] It should be noted that in this embodiment, if the driver can still see the HUD virtual image plane after the line of sight moves, the driver can accurately see the navigation instructions marked on the driving lane or the warning frame surrounding the pedestrian and other driving guidance information through the HUD virtual image plane to ensure safe driving. Optionally, if the driver cannot see the HUD virtual image plane in front of the vehicle head after the line of sight moves, in some possible implementation manners, multiple HUD virtual image planes can be configured as needed, so that no matter in which direction the driver's line of sight deviates (or rotates), the driver can still see the driving guidance information that needs attention in the current driving scene, improving the flexibility and diversity of information display and the user experience.

[0106] An information display method based on augmented reality provided by the present application. After coordinate transformation of the driving guidance information generated based on real-scene information through a first transformation matrix and a second transformation matrix respectively, it is displayed at the corresponding position of the target display area determined based on the first spatial pose of the user determined by the eye tracking device. It realizes real-time tracking of the first spatial pose of the user through the eye tracking device, enables adaptation to changes in the eye pose, dynamically adjusts the display area and display position of the driving guidance information, so that the user can accurately and conveniently view the virtual driving guidance information corresponding to the driving scene during driving without repeatedly confirming the accuracy of the driving guidance information, reduces the fatigue caused by frequent line-of-sight conversion due to viewing road conditions and navigation and other driving guidance information, and improves the safety and comfort of driving.

[0107] Please refer to Figure 9 , which is a flowchart of an information display method based on augmented reality provided by another embodiment of the present application. The method of this embodiment can be executed by a device for processing real-scene information based on augmented reality, and the device can be implemented in a hardware and / or software manner. The method includes:

[0108] Step S310: Obtain real-scene information collected by a scene perception device.

[0109] Step S320: Obtain a target display area determined based on the first spatial pose of the user, where the first spatial pose is determined based on an eye tracking device.

[0110] Step S330: Detect a change in the first spatial pose by obtaining the eye pose change parameter of the eye tracking device.

[0111] It can be understood that during the user's driving, as the driving scene information such as the road conditions of the driving lane changes, the user's pose may change. For example, the user's head may turn in directions such as forward, backward, left, and right. In this way, the user's line-of-sight range will change (for example, line-of-sight deviation, etc.). In this case, if the original HUD display method is still used to display the driving guidance information corresponding to the real-scene information, there may be a safety hazard due to display errors in the position.

[0112] As a way to improve this problem, the eye tracking device in this embodiment can detect the user's eye posture in real time. If it is detected that the user's current eye posture has changed compared to the eye pose at the previous moment (or previous time period), then parameters corresponding to the change in the eye pose can be obtained. Optionally, in this way, the change in the user's first spatial pose can be detected through the eye posture change (i.e., eye pose change) parameters, so that if it is detected that the first spatial pose has changed, the target display area can be re-determined based on the changed spatial pose, thereby ensuring the accuracy of the display position of the driving guidance information corresponding to the real scene information, without the user having to repeatedly confirm the accuracy of the driving guidance information, improving the flexibility of displaying the driving guidance information, and further enhancing the user experience.

[0113] Optionally, the eye posture change parameters can include the direction, angle, or range of the user's eye line of sight, etc. As a way, relevant face recognition algorithms can be used to determine whether the eye pose in the eye pose image collected by the eye tracking device has changed. If it has changed, the corresponding eye posture change parameters can be obtained.

[0114] As a way, a change vector corresponding to the first spatial pose can be obtained according to the eye posture change parameters. Optionally, the specific calculation process can refer to related technologies and will not be elaborated here.

[0115] Step S340: Obtain the coordinate transformation rule corresponding to mapping the real scene information to the target display area.

[0116] Step S350: Generate driving guidance information based on the real scene information.

[0117] Step S361: If the change amount of the eye posture change parameters is greater than a preset threshold, update the coordinate transformation rule according to the eye posture change parameters.

[0118] Optionally, a preset threshold corresponding to the change vector can be pre-configured. This preset threshold can be used to distinguish whether subsequent adjustment of the target display area is required. As an implementation method, if the change value of the change vector is greater than the preset threshold, the target display area can be adjusted based on the change vector to obtain the re-determined target display area.

[0119] For example, in a specific application scenario, please refer to Figure 10 , which shows an example diagram of adjusting the target display area based on the change vector provided in this embodiment. As Figure 10As shown, the user's first spatial pose changes from 22 to 23', where 23' is the current first spatial pose, which is determined based on the user's current eye pose. As a way, if the change vector of the eye pose corresponding to the user's current first spatial pose 23' is greater than a preset threshold, in this way, the position of the target display area on the screen 21 of the front windshield of the vehicle can change from 23 to 23', where 23' is the re-determined target display area.

[0120] Optionally, if the user's first spatial pose changes, in the above way, the coordinate transformation rule can be updated according to the eye pose change parameter, and the second coordinate transformation rule corresponding to the real scene information mapped to the re-determined target display area can be obtained. The specific determination process of the second coordinate transformation rule can refer to the determination principle and process of the foregoing coordinate transformation rule, which will not be elaborated here.

[0121] Step S362: Display the driving guidance information at the corresponding position of the target display area based on the updated coordinate transformation rule.

[0122] Optionally, on the basis of obtaining the re-determined target display area, the driving guidance information can be displayed at the corresponding position of the re-determined target display area based on the second coordinate change rule.

[0123] As an implementation method, the target display area in this embodiment can be adjusted according to the change of the user's first spatial pose. For example, if it is detected that the user has a posture such as lowering the head, the target display area can be displayed at the corresponding position on the central control display screen; if it is detected that the user looks at the mobile phone frequently during driving, the target display area can be displayed on the display screen of the mobile phone; or other screens that can be used as the target display area in the driving scenario, such as the windows on the left and right sides of the driver's seat.

[0124] Step S371: If the change amount of the eye pose change parameter is not greater than the preset threshold, display the driving guidance information at the corresponding position of the target display area based on the coordinate transformation rule.

[0125] As another implementation method, if the change value of the change vector is not greater than the preset threshold, the target display area determined based on the user's first spatial pose can be obtained. In this way, the user's first spatial pose can be the user's sitting posture, etc., and the specific can refer to the description in the foregoing embodiments.

[0126] Optionally, in this embodiment, the sequence order between each step can be not limited. For example, step S330 can be implemented after step S340, etc.

[0127] Exemplarily, a specific implementation process is shown below:

[0128] As Figure 11 shown, a process example diagram of the information display method based on enhanced display proposed in this embodiment is shown. In Figure 11 it, the process pointed by the hollow arrow can be the initial process, and the process pointed by the solid arrow can be the real-time continuous process. As an implementation manner, a coordinate system can be established first, and then the spatial poses of the scene perception module (which can be understood as the aforementioned scene perception device) and the HUD virtual image plane are measured, and the spatial position coordinates of the driver's eyes are initialized. Then, the scene perception module matrix M and the HUD imaging matrix C are calculated respectively, and then the total transformation matrix (i.e., the aforementioned coordinate transformation rule) F = CM is obtained. Optionally, the scene perception module can obtain real-time scene information, use the real-time scene information as the information to be displayed and send it to the image processing module (which can be understood as the aforementioned image processing device). The image processing module performs coordinate transformation processing on the coordinates corresponding to the real-time scene information and draws the finally obtained image, and projects the image onto the HUD display screen (i.e., the aforementioned target display area) for display, so as to improve the accuracy of the display position of the driving guidance information, reduce user operations, and further improve the user experience.

[0129] Step S364: Display the driving guidance information at the corresponding position in the target display area based on the coordinate transformation rule.

[0130] An information display method based on augmented reality provided by the present application detects the change of the first spatial pose by obtaining the change parameters of the eye pose of the eye tracking device. Then, when the change value of the change vector corresponding to the change parameters of the eye pose is greater than a preset threshold, the target display area is readjusted based on the change vector, realizing that the driving guidance information is displayed at the corresponding position in the target display area determined based on the first spatial pose of the user determined by the eye tracking device, realizing real-time tracking of the first spatial pose of the user by the eye tracking device, enabling adaptation to the change of the eye pose, dynamically adjusting the display area and display position of the driving guidance information, so that the user can accurately and conveniently view the virtual driving guidance information corresponding to the driving scene during driving without repeatedly confirming the accuracy of the driving guidance information, reducing the fatigue caused by frequent line-of-sight conversion due to viewing driving guidance information such as road conditions and navigation, and improving the safety and comfort of driving.

[0131] Please refer to Figure 12 , an information display device 400 based on augmented reality provided by an embodiment of the present application can run on a projection device. The device 400 includes:

[0132] An image perception module 410, configured to obtain real-time scene information collected by an image perception device.

[0133] A coordinate transformation module 420, configured to obtain a target display area determined based on a first spatial pose of a user, where the first spatial pose is determined based on an eye tracking device.

[0134] Optionally, the device 400 may further include a parameter change detection module, configured to detect a change in the first spatial pose by obtaining an eye pose change parameter of the eye tracking device. In this way, the coordinate transformation module 420 may specifically be configured to obtain an eye pose change parameter collected by the eye tracking device; obtain a change vector corresponding to the first spatial pose based on the eye pose change parameter; if a change amount of the change vector is greater than a preset threshold, adjust the target display area based on the change vector to obtain a re-determined target display area. Optionally, if the change amount of the change vector is not greater than the preset threshold, execute the step of obtaining the target display area determined based on the first spatial pose of the user.

[0135] As a way, the coordinate transformation module 420 may further be configured to obtain a coordinate transformation rule corresponding to mapping the real scene information to the target display area.

[0136] Optionally, the coordinate transformation rule in this embodiment may include a first transformation matrix and a second transformation matrix. The first transformation matrix is configured to determine a reference world coordinate corresponding to the coordinate of the real scene information collected by the scene perception device, and the second transformation matrix is configured to convert the reference world coordinate into a view coordinate that matches an offset of the user's perspective in the target display area.

[0137] Optionally, the first transformation matrix may include a first rotation matrix and a first translation vector. The first rotation matrix is configured to rotate the coordinate of the real scene information collected by the scene perception device, and the first translation vector is configured to translate the coordinate. The first transformation matrix determines a reference world coordinate corresponding to the coordinate of the real scene information collected by the scene perception device based on the first rotation matrix and the first translation vector.

[0138] Optionally, the second transformation matrix may include a perspective offset matrix, a transpose matrix, and a projection matrix. The projection matrix is configured to determine a mapping range for mapping the real scene information to the target display area. The perspective offset matrix is configured to determine an offset degree of the user's perspective detected by the eye tracking device. The transpose matrix is configured to determine a relative position for displaying the driving guidance information within the mapping range. The second transformation matrix converts the reference world coordinate into a view coordinate that matches an offset of the user's perspective in the target display area based on the perspective offset matrix, the transpose matrix, and the projection matrix.

[0139] Optionally, the perspective shift matrix may include first view coordinates, the transpose matrix represents the transpose of the first transformation matrix, the transpose matrix includes a spatial unit vector located in the target display area, the projection matrix includes a field of view angle parameter, and the field of view angle parameter includes a distance parameter and a scale parameter associated with the user's perspective.

[0140] The display module 430 is configured to generate driving guidance information based on the real scene information.

[0141] As a way, the display module 430 may also be configured to display the driving guidance information at a corresponding position in the target display area based on the coordinate transformation rule.

[0142] Optionally, the display module 430 may specifically be configured to input the position coordinates of the real scene information in the coordinate system corresponding to the scene perception device into the first transformation matrix to obtain a to-be-processed coordinate transformation matrix; perform coordinate transformation on the to-be-processed coordinate transformation matrix according to the second transformation matrix to obtain the relative position coordinates of the real scene information in the target display area; and display the driving guidance information at the position represented by the relative position coordinates.

[0143] It should be noted that the device embodiments in this application correspond to the foregoing method embodiments. The specific principles in the device embodiments can be referred to the content in the foregoing method embodiments, and will not be elaborated here.

[0144] Next, Figure 13 A projection device provided by the present application will be described.

[0145] Please refer to Figure 13 , based on the above augmented reality-based information display method, system, and device, another projection device 100 that can execute the foregoing augmented reality-based information display method is further provided in an embodiment of the present application. The projection device 100 includes one or more (only one is shown in the figure) processors 102, a memory 104, an image perception module 11, a coordinate transformation module 12, an eye tracking module 14, and a display module 13 that are coupled to each other. Among them, a program that can execute the content in the foregoing embodiments is stored in the memory 104, and the processor 102 can execute the program stored in the memory 104. The memory 104 includes the device 400 described in the foregoing embodiments.

[0146] Among them, the processor 102 may include one or more processing cores. The processor 102 connects various parts within the entire projection device 100 through various interfaces and lines, and executes various functions of the projection device 100 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 104, and by calling the data stored in the memory 104. Optionally, the processor 102 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 102 may integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing the display content; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 102 and may be implemented separately through a communication chip.

[0147] The memory 104 may include random access memory (RAM), and may also include read-only memory. The memory 104 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 104 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for implementing at least one function (such as touch function, sound playback function, video image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created during the use of the projection device 100 (such as audio and video data), etc.

[0148] The image perception module 11 is used to obtain the real-scene information collected by the image perception device; the coordinate transformation module 12 is used to obtain the target display area determined based on the user's first spatial pose, where the first spatial pose is determined based on the eye tracking device 14; the eye tracking device 14 is used to detect the user's eye pose in real time; the coordinate transformation module 12 is also used to obtain the coordinate transformation rule corresponding to mapping the real-scene information to the target display area; the display module 13 is used to generate driving guidance information based on the real-scene information; the display module 13 is also used to display the driving guidance information at the corresponding position of the target display area based on the coordinate transformation rule.

[0149] Please refer to Figure 14 , which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. Program code is stored in the computer-readable medium 500, and the program code can be called by a processor to execute the method described in the above method embodiment.

[0150] The computer-readable storage medium 500 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium 500 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 500 has a storage space for the program code 510 that executes any method step in the above method. These program codes can be read from or written into one or more computer program products. The program code 510 can be compressed in an appropriate form, for example.

[0151] In summary, an information display method, system, device, projection device, and storage medium based on augmented reality provided by the present application obtain real-scene information collected by an image sensing device, then obtain a target display area determined based on a first spatial pose of a user, the first spatial pose being determined based on an eye tracking device, then obtain a coordinate transformation rule corresponding to mapping the real-scene information to the target display area, and generate driving guidance information based on the real-scene information, and then display the driving guidance information at a corresponding position in the target display area based on the coordinate transformation rule. Thus, through the above method, the driving guidance information generated based on the real-scene information is displayed at a corresponding position in the target display area determined based on the first spatial pose of the user determined by the eye tracking device through the coordinate transformation rule, and the first spatial pose of the user is tracked in real time by the eye tracking device, so that the display area and display position of the driving guidance information can be dynamically adjusted to adapt to the change of the eye pose, and the user can accurately and conveniently view the virtual driving guidance information corresponding to the driving scene during driving, improving the safety and comfort of driving, and thus enhancing the user experience.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An information display method based on augmented reality, characterized in that, the method includes: Obtaining real - scene information collected by a scene perception device; Obtaining a target display area determined based on the first spatial pose of the user, where the first spatial pose is determined based on an eye - tracking device; Obtaining a coordinate transformation rule corresponding to mapping the real - scene information to the target display area, where the coordinate transformation rule includes a first transformation matrix and a second transformation matrix. The first transformation matrix is used to determine a reference world coordinate corresponding to the coordinate of the real - scene information collected by the scene perception device, and the second transformation matrix is used to convert the reference world coordinate into a view coordinate that matches the offset of the user's perspective within the target display area; the second transformation matrix includes a perspective offset matrix, a transpose matrix, and a projection matrix. The perspective offset matrix includes a first view coordinate, the transpose matrix represents the transpose facing the first transformation matrix, the transpose matrix includes a spatial unit vector located in the target display area, and the projection matrix includes a field - of - view angle parameter, and the field - of - view angle parameter includes a distance parameter and a scale parameter associated with the user's perspective; Generating driving guidance information based on the real - scene information; Displaying the driving guidance information at a corresponding position in the target display area based on the coordinate transformation rule.

2. The method according to claim 1, characterized in that, the first transformation matrix includes a first rotation matrix and a first translation vector. The first rotation matrix is used to rotate the coordinate of the real - scene information collected by the scene perception device, and the first translation vector is used to translate the coordinate. The first transformation matrix determines a reference world coordinate corresponding to the coordinate of the real - scene information collected by the scene perception device based on the first rotation matrix and the first translation vector.

3. The method according to claim 1, characterized in that, the second transformation matrix converts the reference world coordinate into a view coordinate that matches the offset of the user's perspective within the target display area based on the perspective offset matrix, the transpose matrix, and the projection matrix.

4. The method according to any one of claims 1 - 3, characterized in that, displaying the driving guidance information at a corresponding position in the target display area based on the coordinate transformation rule includes: Inputting the position coordinate of the real - scene information in the coordinate system corresponding to the scene perception device into the first transformation matrix to obtain a to - be - processed coordinate transformation matrix; Performing coordinate transformation on the to - be - processed coordinate transformation matrix according to the second transformation matrix to obtain the relative position coordinate of the real - scene information within the target display area; Displaying the driving guidance information at the position characterized by the relative position coordinate.

5. The method according to claim 1, characterized in that, before obtaining the coordinate transformation rule corresponding to mapping the real - scene information to the target display area, it further includes: Detecting the change of the first spatial pose by obtaining the eye - pose change parameter of the eye - tracking device; the method further includes: If the change amount of the human eye pose change parameter is greater than a preset threshold, update the coordinate transformation rule according to the human eye pose change parameter.

6. The method according to claim 5, wherein, the method further includes: If the change amount of the human eye pose change parameter is not greater than a preset threshold, execute the step of obtaining the coordinate transformation rule corresponding to the real scene information mapped to the target display area.

7. An information display device based on augmented reality, wherein, the information display device includes an image perception module, a coordinate transformation module, and a display module: The image perception module is configured to obtain real scene information collected by an image perception device; The coordinate transformation module is configured to obtain a target display area determined based on a first spatial pose of a user, where the first spatial pose is determined based on an eye tracking device; The coordinate transformation module is further configured to obtain a coordinate transformation rule corresponding to the real scene information mapped to the target display area; the coordinate transformation rule includes a first transformation matrix and a second transformation matrix, the first transformation matrix is used to determine a reference world coordinate corresponding to the coordinate of the real scene information collected by the image perception device, and the second transformation matrix is used to convert the reference world coordinate into a view coordinate that matches the offset of the user's perspective within the target display area; the second transformation matrix includes a perspective offset matrix, a transpose matrix, and a projection matrix, the perspective offset matrix includes a first view coordinate, the transpose matrix represents the transpose facing the first transformation matrix, the transpose matrix includes a spatial unit vector located in the target display area, and the projection matrix includes a field of view angle parameter, and the field of view angle parameter includes a distance parameter and a scale parameter associated with the user's perspective; The display module is configured to generate driving guidance information based on the real scene information; The display module is further configured to display the driving guidance information at a corresponding position in the target display area based on the coordinate transformation rule.

8. An in-vehicle information display system based on augmented reality, wherein, the system includes: A scene perception device for collecting real scene information of the vehicle external environment; An eye tracking device for obtaining the line-of-sight movement range of the user; An image processing device is configured to obtain a first spatial pose of the user based on the line-of-sight movement range, obtain a target display area determined based on the first spatial pose, obtain a coordinate transformation rule corresponding to the mapping of the real scene information to the target display area, generate driving guidance information based on the real scene information, and generate target position coordinates of the driving guidance information displayed in the target display area based on the coordinate transformation rule; the coordinate transformation rule includes a first transformation matrix and a second transformation matrix, the first transformation matrix is used to determine a reference world coordinate corresponding to the coordinate of the real scene information collected by the scene perception device, and the second transformation matrix is used to convert the reference world coordinate into a view coordinate that matches the offset of the user's perspective within the target display area; the second transformation matrix includes a perspective offset matrix, a transpose matrix, and a projection matrix, the perspective offset matrix includes a first view coordinate, the transpose matrix represents the transpose facing the first transformation matrix, the transpose matrix includes a spatial unit vector located in the target display area, and the projection matrix includes a field-of-view angle parameter, and the field-of-view angle parameter includes a distance parameter and a scale parameter associated with the user's perspective; An HUD display device is configured to display the driving guidance information at the target position coordinates in the target display area.

9. A projection device, characterized in that, it includes one or more processors and a memory; one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, program code is stored in the computer-readable storage medium, and when the program code is run by a processor, the method according to any one of claims 1-6 is executed.

Images