Information display method, information display device, electronic device, and computer program

Abstract

An information display method and related device applicable to the fields of transportation and Internet of Vehicles. The method includes the steps of: generating an information display area projected from an AR HUD device (110) on the target vehicle (100) via the windshield of the target vehicle (100), the information display area including at least a virtual prompt element corresponding to a key object, the key object being an object in a driving environment in which the target vehicle (100) is present, the display position of the virtual prompt element corresponding to the position of the key object in the driving environment; and, when a change occurs in the key object satisfying a predetermined condition, changing the position of the information display area on the windshield so that the information display area includes a virtual prompt element corresponding to the changed key object. This method enhances the prompt information provided by the AR HUD and improves the driver's user experience.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of priority to Chinese Patent Application No. 2022114475443, entitled "Information Display Device and Related Apparatus," filed with the State Intellectual Property Office of the People's Republic of China on November 18, 2022, the entire contents of which are incorporated herein by reference.

[0002] [Technical field] This application relates to the field of transportation technology, and more particularly to head-up display technology. [Background technology]

[0003] AR HUD is a combination of Augmented Reality (AR) technology and a Head-Up Display (HUD) system. AR technology is used to "augment" the real world by cleverly blending virtual information with it. HUD systems project driving assistance information, such as vehicle speed and road guidance, onto the vehicle's windshield, allowing the driver to view the information without having to lower or turn their head. AR HUDs work by detecting the real driving environment and presenting relevant information in the real driving environment to the driver by projecting virtual elements corresponding to the information onto the vehicle's windshield.

[0004] Currently, the field of view (FOV) of AR HUDs is generally narrow, typically only 13 x 5 degrees, and the FOV position is generally fixed and unchanging. Therefore, when using augmented reality displays based on this FOV, only information within a specific field of view in the actual driving environment can be displayed. For example, only traffic signs on the road (lane guide signs at intersections, lane markings, etc.) can often be displayed. The prompt information provided by AR HUDs is not rich enough, resulting in a poor user experience for drivers. Summary of the Invention [Means for solving the problem]

[0005] According to an embodiment of the present application, an information display method and related device are provided that can enrich the prompt information provided by the AR HUD and improve the user experience of the driver.

[0006] In view of this, according to a first aspect of the present application, there is provided a method for displaying information performed by an electronic device, the method comprising: generating an information display area projected from an augmented reality head-up display (AR HUD) in the target vehicle by a windshield of the target vehicle, the information display area including at least a virtual prompt element corresponding to a key object, the key object being an object in a driving environment in which the target vehicle exists, and a display position of the virtual prompt element corresponding to a position of the key object in the driving environment; and when a change occurs in the key object that satisfies a predetermined condition, changing the position of the information display area on the windshield so that the information display area includes a virtual prompt element corresponding to the changed key object.

[0007] According to a second aspect of the present application, there is provided an information display device, the device comprising: a projection display module configured to perform the steps of: generating, via a windshield of the target vehicle, an information display area projected from the AR HUD in the target vehicle, the information display area including at least a virtual prompt element corresponding to a key object, the key object being an object in a driving environment in which the target vehicle resides, and a display position of the virtual prompt element corresponding to a position of the key object in the driving environment; and a position change module configured to perform a step of changing the position of the information display area on the windshield when a change that satisfies a predetermined condition occurs in the key object so that the information display area includes a virtual prompt element corresponding to the changed key object.

[0008] According to a third aspect of the present application, there is provided an electronic device, said device comprising: a memory in which a computer program is stored; and a processor for executing the steps of the information display method according to the first aspect in accordance with the computer program.

[0009] According to a fourth aspect of the present application, there is provided a computer-readable storage medium having stored thereon a computer program for use in performing the steps of the information display method according to the first aspect.

[0010] According to a fifth aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored on a computer-readable storage medium, the computer instructions being read by a processor of a computing device from the computer-readable storage medium and executed by the processor to cause the computing device to perform the steps of the information display method set out in the first aspect above. [Effects of the Invention]

[0011] As can be seen from the above technical solutions, the embodiments of the present application have the following advantages: An embodiment of the present application provides an information display method. In this method, the position of the FOV of the AR HUD device is dynamically adjusted in response to changes in key objects in a driving environment in which a target vehicle is present. That is, the position of the information display area formed by the AR HUD device and projected onto the windshield of the target vehicle is dynamically adjusted on the windshield. This allows a virtual prompt element corresponding to the key object in the driving environment to be continuously displayed in the information display area. This ensures that the display position of the virtual prompt element corresponds to the position of the key object in the driving environment, thereby providing the driver with more realistic prompt information. Furthermore, in the information display method according to the embodiment of the present application, when the key object changes from a currently focused field of view of the AR HUD device to a different field of view, the AR HUD device can adjust the position of the information display area formed by the projection on the windshield so that the information display area corresponds to the changed field of view. In this way, the information displayed by the AR HUD device is no longer limited to a specific field of view but can dynamically adapt to a wider field of view. Accordingly, the driver can learn more prompt information through the dynamic information display area, thereby improving the user experience. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a schematic diagram of an application scene of an information display method according to an embodiment of the present application; [Figure 2] 1 is a flowchart of an information display method according to an embodiment of the present application. [Figure 3] FIG. 1 is a schematic diagram of an image plane of an AR HUD device according to an embodiment of the present application. [Figure 4] 1 is a schematic diagram of an exemplary information display area according to an embodiment of the present application. [Figure 5] 10A and 10B are schematic diagrams illustrating a change in the display position of an information display area according to an embodiment of the present application; [Figure 6]10A and 10B are schematic diagrams illustrating a change in the display position of an information display area according to an embodiment of the present application; [Figure 7] 1 is a schematic diagram of an exemplary information display area according to an embodiment of the present application. [Figure 8] 3 is a flowchart of a method for determining a display position of an information display area according to an embodiment of the present application; [Figure 9] 2 is a schematic diagram of a vehicle coordinate system of a target vehicle according to an embodiment of the present application; [Figure 10] 1 is a schematic diagram of a movable range of a target imaging area according to an embodiment of the present application; [Figure 11] 1 is a schematic configuration diagram of an information display device according to an embodiment of the present application. [Figure 12] FIG. 1 is a schematic configuration diagram of a terminal device according to an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0013] The information display method according to the embodiment of the present application can be performed by an electronic device. Specifically, the electronic device can be a controller for controlling an AR HUD device mounted on a vehicle. Specifically, the controller can be integrated into the AR HUD device or can be independent from the AR HUD device. The controller can control an image projected from the AR HUD device and can also control a display position of the image projected from the AR HUD device.

[0014] To facilitate understanding of the information display method according to the embodiment of the present application, an application scenario of the information display method will be described below by taking a controller independent from an AR HUD device as an example.

[0015]

[0023] Referring to Fig. 1, there is shown a schematic diagram of an application scenario of the information display method according to an embodiment of the present application. As shown in Fig. 1, the application scenario includes a target vehicle 100. The target vehicle 100 is equipped with an AR HUD device 110 and a controller 120 that can communicate with each other via a wired or wireless network. The controller 120 is used to control the information display area generated by the AR HUD device 110 and projected onto the windshield of the target vehicle 100.

[0016] In practical applications, the controller 120 can send control data to the optical mechanism 111 in the AR HUD device 110 and control the optical mechanism 111 to generate an information display area based on the control data. For example, the controller 120 can send image rendering data to the optical mechanism 111 and use the optical mechanism 111 to render an information display area based on the image rendering data. The optical mechanism 111 displays an image generated by the rendering. Light from the image is reflected by reflective members 112 and 113 in the AR HUD device 110 and finally projected onto the windshield of the target vehicle 100. An information display area corresponding to the image generated by the optical mechanism 111 is displayed on the windshield. The information display area corresponds to a target imaging area in a virtual image forming plane of the AR HUD device 110.

[0017] The information display area projected by the AR HUD device 110 includes a virtual prompt element corresponding to a key object. The key object here refers to an object in the driving environment in which the target vehicle 100 exists, specifically an object that requires the driver's attention, such as a lane, a vehicle, or a pedestrian on the road surface. The virtual prompt element (also referred to as a virtual presentation element) here refers to a virtual element created based on AR technology to indicate the presence of the key object, such as a virtual arrow indicating the lane direction or guidance direction, a virtual vehicle element indicating a vehicle, or a virtual pedestrian element indicating a pedestrian. In an embodiment of the present application, the display position of the virtual prompt element corresponding to the key object in the information display area corresponds to the position of the key object in the driving environment. For example, from the driver's viewpoint, a virtual arrow indicating the lane direction overlaps with a lane direction sign on the road surface, a virtual vehicle element indicating a vehicle overlaps with the position of the vehicle, and a virtual pedestrian element indicating a pedestrian overlaps with the position of the pedestrian.

[0018] When a change that satisfies a predetermined condition occurs in a key object in the driving environment, the controller 120 needs to change the position of the information display area on the windshield of the target vehicle 100 by controlling the AR HUD device 110 to change the position of a target imaging area corresponding to the information display area in the virtual image forming surface. As an example, when a key object exceeds the field of view currently corresponding to the information display area, the controller 120 determines the imaging position of a virtual prompt element corresponding to the key object on the virtual image forming surface of the AR HUD device 110 according to the position of the key object in the driving environment, and then adjusts the position of the target imaging area corresponding to the information display area on the virtual image forming surface so that the target imaging area covers the imaging position of the virtual prompt element corresponding to the key object. It should be understood that adjusting the position of the target imaging area on the virtual image forming surface can adjust the position of the information display area on the windshield, and ensuring that the target imaging area covers the imaging position of the virtual prompt element corresponding to the key object can ensure that the virtual prompt element corresponding to the key object is included in the information display area.

[0019] It should be understood that the application scenario shown in FIG. 1 is merely an example, and no limitations are placed here on the application scenario of the information display method according to the embodiment of the present application.

[0020] The information display method according to the present application will be described in detail below with reference to the method embodiment.

[0021] 2, which is a flowchart of an information display method according to an embodiment of the present application. As shown in FIG. 2, the information display method includes the following steps:

[0022] Step 201: Generate an information display area projected from the AR-HUD device of the target vehicle by the windshield of the target vehicle, the information display area including at least a virtual prompt element corresponding to a key object, the key object being an object in a driving environment where the target vehicle exists, and the display position of the virtual prompt element corresponds to the position of the key object in the driving environment.

[0023] In an embodiment of the present application, the AR HUD device in the target vehicle can generate an information display area displayed on the windshield of the target vehicle through projection. Specifically, an optical device in the AR HUD device can receive image rendering data transmitted from a controller and render and display a target image based on the image rendering data, where the target image corresponds to the information display area. A light-reflecting member in the AR HUD device reflects light of the target image generated by the optical device and projects and displays the information display area corresponding to the target image on the windshield of the target vehicle, thereby realizing the generation of the information display area on the windshield of the target vehicle.

[0024] The information display area generated by the windshield of the target vehicle corresponds to the target imaging area in the image plane of the AR HUD device. FIG. 3 is a schematic diagram of the image plane of an AR HUD device according to an embodiment of the present application. As shown in FIG. 3, the AR HUD device uses the vehicle's windshield as a projection medium to reflect and form an image. The image formed on the windshield corresponds to a virtual image located in a corresponding imaging area in the virtual image formation plane of the AR HUD device. The size of the imaging area corresponds to the FOV of the AR HUD device. Typically, the virtual image formation plane of the AR HUD device is perpendicular to the ground and located approximately 7 meters in front of the main driver's seat of the vehicle. The FOV of the AR HUD is 13 × 5 degrees. The FOV indicates the angle formed by the driver's eyes and the visible edge of the virtual image. The FOV of 13 × 5 degrees means that the driver's horizontal viewing angle to the virtual image is 13 degrees and the vertical viewing angle is 5 degrees.

[0025] In the embodiment of the present application, the position of the information display area on the windshield can be adjusted by adjusting the position of the target imaging area corresponding to the information display area in the image plane. Specific implementation methods for determining the position of the information display area on the windshield and for adjusting the position of the information display area on the windshield will be described in detail in another method embodiment below.

[0026] It should be noted that the information display area projected from the AR HUD device includes at least a virtual prompt element corresponding to a key object. The key object here refers to an object in the driving environment where the target vehicle is located, specifically, an object that requires the driver's attention in the driving environment. The key object can be determined based on a reference image collected by a front camera mounted on the target vehicle. For example, image detection can be performed on the reference image collected by the front camera mounted on the target vehicle to determine specific types of objects included in the reference image as candidate key objects. Then, according to a predetermined key object determination rule, an object that requires the driver's particular attention can be selected as the key object from among the candidate key objects included in the reference image. The key object can be, for example, a traffic sign on the road surface (e.g., a lane change sign at an intersection, a lane marking, etc.), a vehicle, a pedestrian, an obstacle, etc. on the road surface. However, the present application does not impose any restrictions on the type of the key object.

[0027] It should be noted that key objects in the driving environment are different from general objects in the driving environment. Key objects in the driving environment include objects that affect driving safety, but specifically, they may be objects in the driving environment that are likely to cause traffic accidents, such as vehicles, pedestrians, and obstacles whose distance from a target vehicle is less than a predetermined distance threshold (e.g., 20 meters), or objects in the driving environment that indicate traffic rules, such as intersection signs, lane signs, and speed limit signs. On the other hand, general objects in the driving environment may be any objects that do not affect driving safety, such as vehicles, pedestrians, and obstacles whose distance from a target vehicle is equal to or greater than a predetermined distance threshold.

[0028] It should be understood that in practical applications, a judgment condition corresponding to the key object can be set in advance, and when an object that meets the judgment condition is detected in the driving environment, the object is judged as a key object. For example, the judgment condition can specify the type of key object (vehicle, person, animal, obstacle, traffic sign, etc.) and can also specify the distance requirement from the target vehicle, but in the embodiment of the present application, no restriction is placed on the judgment condition of the key object.

[0029] A virtual prompt element corresponding to a key object is a virtual element generated based on AR technology to indicate the presence of the key object. The virtual prompt element can guide the driver to pay closer attention to the key object. The virtual prompt element corresponding to a key object may be a pre-designed template virtual element. For example, a virtual prompt element corresponding to a lane guide sign may be a virtual arrow, a virtual prompt element corresponding to a lane sign may be a virtual lane, a virtual prompt element corresponding to a vehicle may be a pre-designed virtual vehicle or a virtual element indicating the distance between a target vehicle and the vehicle, and a virtual prompt element corresponding to a pedestrian may be a pre-designed virtual pedestrian or a virtual element indicating the distance between a target vehicle and the pedestrian. The virtual prompt element corresponding to a key object may be a virtual element generated in real time according to an actual key object existing in the driving environment, and the virtual prompt element can reflect the characteristics of the key object to some extent. This application does not impose any restrictions on the expression format of the virtual prompt element corresponding to the key object.

[0030] In an embodiment of the present application, in order to improve the matching between the virtual prompt element and the actual driving environment and provide the driver with a more realistic information display effect, the display position of the virtual prompt element in the information display area corresponds to the position of the key object in the driving environment. That is, from the driver's perspective, the virtual prompt element in the information display area overlaps with the corresponding key object, or the distance between the virtual prompt element and the corresponding key object is less than a predetermined distance threshold (e.g., 5 cm). For example, if the key object is a lane guide sign at an intersection, from the driver's perspective, the virtual prompt element corresponding to the lane guide sign is displayed superimposed on the actual lane guide sign. For example, if the key object is a vehicle on the road surface, from the driver's perspective, the virtual prompt element corresponding to the vehicle is displayed superimposed on the actual vehicle. A specific implementation method for determining the display position of the virtual prompt element matching the key object will also be described in detail in another method embodiment below.

[0031] Optionally, the information display area may include driving assistance information on the central control screen of the target vehicle, such as the target vehicle's speed, remaining fuel, remaining battery, engine RPM, and guidance information, in addition to virtual prompt elements generated based on AR technology, but this application does not impose any restrictions on the content displayed in the information display area.

[0032] Optionally, if the driving environment in which the target vehicle resides does not include key objects requiring the driver's attention, the information display area may also not include virtual prompt elements corresponding to the key objects.

[0033] FIG. 4 is a schematic diagram of an exemplary information display area according to an embodiment of the present application. As shown in FIG. 4, an information display area 420 is displayed on the windshield 410 of a target vehicle. The information display area 420 includes a virtual prompt element 421 corresponding to a lane guide sign. From the driver's perspective, the virtual prompt element 421 is displayed superimposed on the actual lane guide sign on the road surface. The information display area 420 also includes the target vehicle's current speed of 47 km / h. It should be understood that content not numbered in FIG. 4 is not content within the information display area formed by projection of the AR HUD, but background content within the driving environment in which the target vehicle exists.

[0034] Step 202: If a change occurs in the key object that satisfies a predetermined condition, the position of the information display area on the windshield is changed so that the information display area includes a virtual prompt element corresponding to the changed key object.

[0035] When a change that satisfies a predetermined condition occurs in a key object in the driving environment, the AD HUD device changes the position of the image formed by its projection, i.e., changes the position of the target imaging area on its image plane, thereby correspondingly changing the position of the information display area displayed on the windshield of the target vehicle, so that the target imaging area after the change in position can completely carry the virtual prompt element corresponding to the changed key object, i.e., the information display area whose display position has been changed can completely carry the virtual prompt element corresponding to the changed key object.

[0036] The predetermined condition is a condition set in advance to determine whether the display position of the information display area needs to be changed. For example, the predetermined condition may be that the changed key object exceeds or does not completely fall within the field of view currently focused on in the AR HUD device. The field of view currently focused on in the AR HUD device can be understood as the field of view corresponding to the information display area. If the changed key object exceeds or does not completely fall within the field of view currently focused on in the AR HUD device, the virtual prompt element corresponding to the changed key object cannot be displayed or cannot be completely displayed in the information display area. Therefore, it is necessary to adjust the display position of the information display area so that the virtual prompt element corresponding to the changed key object can be completely displayed in the information display area.

[0037] The change in the key object may specifically refer to a change in the key object requiring the driver's attention, i.e., a change from one key object to another. For example, when a target vehicle is traveling in a normal straight line, the key object that the driver should pay attention to is a traffic sign on the road. However, if another vehicle or pedestrian suddenly appears ahead of the target vehicle, the key object that the driver should pay attention to becomes the vehicle or pedestrian that has appeared. Furthermore, the change in the key object may specifically refer to a change in the position of the key object itself, i.e., the movement of the key object from one position to another. For example, when a target vehicle is tracking a specific object, the position of the specific object relative to the target vehicle may change.

[0038] In one possible embodiment, when the key object changes from a first key object to a second key object and the information display area does not satisfy the first target display condition for the virtual prompt element corresponding to the second key object, the position of the information display area on the windshield is changed so that the information display area satisfies the first target display condition for the virtual prompt element corresponding to the second key object.

[0039] For example, while the target vehicle is traveling, the key object requiring the driver's attention may change, i.e., the key object that originally received attention may change from a first key object to a second key object. If the target imaging area corresponding to the information display area on the image plane of the AR HUD device does not satisfy the first target bearing condition for the virtual prompt element corresponding to the second key object, i.e., if the information display area does not satisfy the first target display condition for the virtual prompt element corresponding to the second key object, the AR HUD device needs to adjust the position of the target imaging area on its image plane to correspondingly adjust the position of the information display area on the windshield. This allows the adjusted target imaging area to satisfy the first target bearing condition for the virtual prompt element corresponding to the second key object, and the adjusted display position of the information display area to satisfy the first target display condition for the virtual prompt element corresponding to the second key object.

[0040] The first target display condition is a condition for determining whether the display status of the virtual prompt element corresponding to the key object in the information display area satisfies a predetermined requirement, and the first target support condition is a condition for determining whether the support status of the virtual prompt element corresponding to the key object in the target imaging area satisfies a predetermined requirement. For example, the first target display condition may be set to completely display the virtual prompt element corresponding to the key object in the information display area, and the first target support condition may be set to completely support the virtual prompt element corresponding to the key object in the target imaging area. Of course, in actual applications, the first target display condition and the first target support condition may be set to other conditions according to actual needs, such as the virtual prompt element corresponding to the key object being located in a specific display area in the information display area or the virtual prompt element corresponding to the key object being located in a specific support area in the target imaging area. However, the present application does not particularly limit the first target display condition and the first target support condition.

[0041] It should be understood that in practical applications, even if the key object requiring the driver's attention changes from a first key object to a second key object, there is no need to change the display position of the information display area if the information display area meets the first target display condition for the virtual prompt element corresponding to the second key object.

[0042] As an example, when the first key object is a target ground object and the second key object is a target ground object, the position of the information display area on the windshield may be moved upward when changing the position of the information display area on the windshield. Also, when the first key object is a target ground object and the second key object is a target ground object, the position of the information display area on the windshield may be moved downward when changing the position of the information display area on the windshield. The position of the target ground object is lower than the position of the target ground object.

[0043] The target ground object may be an object on the road surface that requires the driver's attention, such as a traffic sign painted on the road surface (lane guide signs at intersections, lane markings, etc.) or a guide sign superimposed on the road surface by an AR HUD device.The target ground object may also be an object on the ground that requires the driver's attention, such as a vehicle, pedestrian, or obstacle warned about by an Advanced Driving Assistance System (ADAS).

[0044] FIG. 5 is a schematic diagram illustrating a change in the display position of the information display area according to an embodiment of the present application. As shown in FIG. 5(a), when the target vehicle is traveling in a normal straight-ahead state, the first key object that the vehicle originally focused on is a lane guide sign on the road surface. In this case, the information display area is area 510 shown in FIG. 5(a), which includes a virtual prompt element (i.e., virtual element 511) corresponding to the lane guide sign. If another vehicle suddenly cuts in front of the target vehicle and issues an ADAS warning for the target vehicle, the information display area accordingly moves upward to area 520 shown in FIG. 5(b), which includes a virtual prompt element (i.e., virtual element 521 indicating the distance between the target vehicle and the other vehicle that caused the ADAS warning) corresponding to the other vehicle that caused the ADAS warning. Furthermore, if there is another vehicle that caused the ADAS warning ahead of the target vehicle while the target vehicle is traveling, a virtual prompt element corresponding to the other vehicle that caused the ADAS warning is displayed in the information display area. When the guidance information indicates that the target vehicle will soon change lanes, the information display area moves downward accordingly, and a virtual prompt element corresponding to the guidance information is displayed in the information display area. From the driver's perspective, the virtual prompt element is typically displayed superimposed on the road surface. It should be understood that the content not numbered in Figure 5 is not the content in the information display area formed by the projection of the AR HUD, but the background content in the driving environment of the target vehicle.

[0045] In this way, by using the above method, when the key object requiring the driver's attention changes from a target ground object to a target ground object, or when the key object requiring the driver's attention changes from a target ground object to a target ground object, the position of the information display area on the windshield can be flexibly adjusted vertically so that the information display area after the position adjustment can fully carry the virtual prompt element corresponding to the changed key object, thereby effectively notifying the driver to pay timely attention to the changed key object.

[0046] As another example, when the first key object is a target object on the left side of the line of sight and the second key object is a target object on the right side of the line of sight, the position of the information display area on the windshield may be moved to the right when changing the position of the information display area on the windshield.Also, when the first key object is a target object on the right side of the line of sight and the second key object is a target object on the left side of the line of sight, the position of the information display area on the windshield may be moved to the left when changing the position of the information display area on the windshield.

[0047] It should be noted that the target object on the left side of the line of sight may be an object located on the left side of the driver's line of sight and requiring the driver's attention, and the target object on the right side of the line of sight may be an object located on the right side of the driver's line of sight and requiring the driver's attention. The target objects on the left side of the line of sight and the target objects on the right side of the line of sight include, but are not limited to, traffic signs painted on the road surface, guide signs superimposed on the road surface, vehicles on the road, pedestrians, obstacles, etc.

[0048] FIG. 6 is a schematic diagram illustrating a change in the display position of an information display area according to an embodiment of the present application. As shown in FIG. 6(a), assume that a first vehicle that has issued an ADAS warning for the target vehicle is present to the left front of the target vehicle while the target vehicle is traveling. In this case, the information display area is the area 610 shown in FIG. 6(a), which includes a virtual prompt element corresponding to the first vehicle (i.e., a virtual element 611 indicating the distance between the target vehicle and the first vehicle). If a second vehicle that has issued an ADAS warning for the target vehicle suddenly appears to the right front of the target vehicle, the information display area accordingly shifts to the right, to the area 620 shown in FIG. 6(b). This area includes a virtual prompt element corresponding to the second vehicle (i.e., a virtual element 621 indicating the distance between the target vehicle and the second vehicle). Similarly, if the key object requiring the driver's attention changes from a third vehicle located to the right front to a fourth vehicle located to the left front, the information display area accordingly shifts to the left, and a virtual prompt element corresponding to the fourth vehicle is displayed. It should be understood that the content not numbered in FIG. 6 is not content within the information display area formed by the projection of the AR HUD, but background content within the driving environment in which the target vehicle exists.

[0049] In this way, by using the above method, when the key object requiring the driver's attention changes from a target object on the left side of the line of sight to a target object on the right side of the line of sight, or when the key object requiring the driver's attention changes from a target object on the right side of the line of sight to a target object on the left side of the line of sight, the position of the information display area on the windshield can be flexibly adjusted horizontally so that the information display area after the position adjustment can fully carry the virtual prompt element corresponding to the changed key object, thereby effectively notifying the driver to pay timely attention to the changed key object.

[0050] In actual applications, the direction in which the display position of the information display area moves is not limited to the up, down, left, and right directions introduced above, but may also be moved in directions such as the upper left, lower left, upper right, and lower right depending on actual needs.However, it should be understood that this application does not impose any restrictions on the direction in which the display position of the information display area moves.

[0051] It should be noted that in the above example, the moving direction and moving distance of the information display area can be specifically determined by the imaging positions of the virtual prompt elements corresponding to the first key object and the second key object, respectively, on the image plane of the AR HUD device. That is, a first imaging position of the virtual prompt element corresponding to the first key object on the image plane is determined, and a second imaging position of the virtual prompt element corresponding to the second key object on the image plane is determined, where the image plane is a virtual image formation plane formed by projection of the AR HUD device. Then, the display position of the information display area may be changed according to the positional relationship between the first imaging position and the second imaging position.

[0052] Specifically, when the key object requiring the driver's attention changes from a first key object to a second key object, a first imaging position of a virtual prompt element corresponding to the first key object on the image plane of the AR HUD device can be determined according to the position of the first key object in the actual driving environment, and a second imaging position of a virtual prompt element corresponding to the second key object on the image plane of the AR HUD device can be determined according to the position of the second key object in the actual driving environment. A specific manner of determining the corresponding imaging position of the virtual prompt element corresponding to the key object on the image plane will be described in detail in another method embodiment below. Next, a movement manner of a target imaging area corresponding to the information display area on the image plane is determined according to the positional relationship between the first imaging position and the second imaging position, and the target imaging area is moved according to the movement manner so that the target imaging area satisfies the first target carrying condition with respect to the virtual prompt element corresponding to the second key object. Furthermore, based on the determined movement manner of the target imaging area, movement of the AR HUD device is controlled to move the target imaging area according to the determined movement manner. When the target imaging area is moved, the display position of the information display area changes, so that the information display area whose display position has been changed can satisfy the first target display condition for the virtual prompt element corresponding to the second key object.

[0053] In this way, by using the above method, when the key object requiring the driver's attention changes from one key object to another, the display position of the information display area formed by the projection of the AD HUD device can be flexibly and accurately adjusted so that the information display area can fully carry the virtual prompt elements corresponding to the changed key object, thereby guiding the driver to pay full attention to the changed key object.

[0054] In another possible embodiment, when the position of a key object in the driving environment changes from a first position to a second position and the information display area does not satisfy a second target display condition for the virtual prompt element corresponding to the key object at the second position, the position of the information display area on the windshield is changed so that the information display area satisfies the second target display condition for the virtual prompt element corresponding to the key object at the second position.

[0055] As an example, while the target vehicle is traveling, the position of a key object requiring the driver's attention relative to the target vehicle may change, i.e., the position relative to the target vehicle may change from a first position to a second position. For example, assuming that the key object that the driver is paying attention to is a reference vehicle traveling ahead, if the reference vehicle suddenly accelerates, the distance to the target vehicle may increase rapidly, if the reference vehicle suddenly decelerates, the distance to the target vehicle may decrease rapidly, and if the reference vehicle suddenly moves to the left or right, the relative direction between the reference vehicle and the target vehicle may change; alternatively, if the target vehicle suddenly accelerates, the distance to the reference vehicle may decrease rapidly, if the target vehicle suddenly decelerates, the distance to the reference vehicle may increase rapidly, and if the target vehicle suddenly moves to the left or right, the relative direction between the reference vehicle and the target vehicle may change.

[0056] If the target imaging area corresponding to the information display area on the image plane of the AR HUD device does not satisfy the second target bearing condition for the virtual prompt element corresponding to the key object at the second position, i.e., if the information display area does not satisfy the second target display condition for the virtual prompt element corresponding to the key object at the second position, the AR HUD device needs to adjust the position of the target imaging area on the image plane and correspondingly adjust the position of the information display area on the windshield, so that the target imaging area after the position adjustment can satisfy the second target bearing condition for the virtual prompt element corresponding to the key object at the second position, and the information display area after the position adjustment can satisfy the second target display condition for the virtual prompt element corresponding to the key object at the second position.

[0057] It should be noted that the second target display condition is a condition for determining whether the display status of the virtual prompt element corresponding to the key object in the information display area satisfies a predetermined requirement, and the second target support condition is a condition for determining whether the support status of the virtual prompt element corresponding to the key object in the target imaging area satisfies a predetermined requirement. For example, the second target display condition may be to completely display the virtual prompt element corresponding to the key object in the information display area, and the second target support condition may be to completely support the virtual prompt element corresponding to the key object in the target imaging area. Of course, in actual applications, the second target display condition and the second target support condition may be set to other conditions according to actual needs, and the present application does not particularly limit the second target display condition and the second target support condition.

[0058] It should be understood that in practical applications, even if the position of a key object requiring the driver's attention in the driving environment changes from a first position to a second position, there is no need to change the display position of the information display area if the information display area meets the second target display condition for the virtual prompt element corresponding to the key object.

[0059] In this embodiment, the display position of the information display area can be changed by determining the image positions of the virtual prompt elements corresponding to the key object at the first position and the key object at the second position on the AR HUD device image plane. That is, a third image position of the virtual prompt element corresponding to the key object at the first position on the image plane is determined, and a fourth image position of the virtual prompt element corresponding to the key object at the second position on the image plane is determined, where the image plane is a virtual image formation plane formed by projection of the AR HUD. Then, the position of the information display area on the windshield may be changed according to the positional relationship between the third image position and the fourth image position.

[0060] Specifically, when the position of a key object requiring the driver's attention in the driving environment changes from a first position to a second position, a third imaging position in the image plane of the AR HUD of the virtual prompt element corresponding to the key object at the first position can be determined according to the first position, and a fourth imaging position in the image plane of the AR HUD of the virtual prompt element corresponding to the key object at the second position can be determined according to the second position. A specific manner of determining the imaging position in the image plane of the virtual prompt element corresponding to the key object will be described in detail in another method embodiment below. Next, a movement manner of a target imaging area corresponding to the information display area in the image plane is determined based on the positional relationship between the third imaging position and the fourth imaging position, and the target imaging area is moved according to the movement manner so that the target imaging area satisfies the second target carrying condition with respect to the virtual prompt element corresponding to the key object at the second position. Furthermore, based on the determined movement manner of the target imaging area, movement of the AR HUD device is controlled to move the target imaging area according to the determined movement manner. When the target imaging area is moved, the display position of the information display area changes, so that the information display area whose display position has been changed can satisfy a second target display condition for the virtual prompt element corresponding to the key object at the second position.

[0061] In this way, when the key object requiring the driver's attention changes from a first position to a second position, the display position of the information display area formed by the projection of the AD HUD device can be flexibly and accurately adjusted to fully display the virtual prompt element corresponding to the key object after the position change in the information display area, thereby making it easier to keep the driver's attention on the key object.

[0062] In some possible embodiments, a driving environment may include multiple basic key objects that require the driver's attention. However, due to the limited FOV corresponding to the information display area, the virtual prompt elements corresponding to each of these multiple basic key objects cannot all satisfy the third target display condition. In this case, among these multiple basic key objects, a main key object and a sub-key object can be distinguished. In the information display area, the virtual prompt element corresponding to the main key object is displayed, and the prompt content corresponding to the sub-key object is displayed according to a predetermined display method (prompt method).

[0063] The basic key object is an object in the driving environment that requires the driver's attention, and is also called an object of interest. Specifically, the basic key object can be obtained by performing image detection on images collected by the front camera mounted on the target vehicle. That is, an image detection model can be used to detect images collected by the front camera mounted on the target vehicle, and a predetermined type of object in the images can be obtained as the basic key object. Here, the predetermined type of object is a predetermined object in the driving environment that requires the driver's attention.

[0064] The main key object is an object with a high importance among the basic key objects, and the subkey object is an object with a low importance among the basic key objects, where the importance of the basic key object may be determined according to the type of the basic key object and / or the location between the basic key object and the target vehicle.

[0065] As an example, the importance corresponding to each of the determined basic key objects may be determined in combination with the state of the target vehicle. For example, the importance corresponding to the basic key object may be determined according to the type of the basic key object and / or the distance between the basic key object and the target vehicle. Next, the basic key objects are sorted in order of decreasing importance, and the first k basic key objects (k is an integer equal to or greater than 1) are used as main key objects, and the remaining basic key objects are used as subkey objects.

[0066] More specifically, importance classification rules corresponding to various driving states (including, but not limited to, a straight driving state, a lane change state, an intersection arrival state, etc.) can be pre-set, and the importance classification rules are used to define the correspondence relationship between the type of basic key object and the importance in the corresponding driving state. For example, assume that the importance classification rule corresponding to the straight driving state indicates that, for basic key objects whose distance to the target vehicle is less than a predetermined distance threshold, the pedestrian basic key object has the highest importance, the vehicle basic key object has the second highest importance, and the road sign basic key object has the lowest importance. Accordingly, if the current driving environment simultaneously includes three basic key objects, namely, a pedestrian, a vehicle, and a road sign, it is determined that the pedestrian has the highest importance, the vehicle has the second highest importance, and the road sign has the lowest importance. Accordingly, the pedestrian can be classified as the main key object, and the vehicle and the road sign can be classified as sub-key objects.

[0067] As an example, assume that the driving environment in which the target vehicle exists contains two basic key objects that require the driver's attention: a lane guide sign and a reference vehicle that caused an ADAS warning. Furthermore, the information display area cannot satisfy the third target display condition for both the virtual prompt element corresponding to the lane guide sign and the virtual prompt element corresponding to the reference vehicle. In this case, based on a predetermined key object importance classification rule and in combination with the current state of the target vehicle, a main key object and a sub-key object must be determined from the two basic key objects. For example, if the target vehicle is in an intersection guidance state, the lane guide sign can be determined as the main key object and the reference vehicle can be determined as the sub-key object. Alternatively, if the target vehicle is in a normal straight-ahead state, the reference vehicle can be determined as the main key object and the lane guide sign can be determined as the sub-key object. Of course, in a practical application, the driving environment in which the target vehicle exists may contain more basic key objects, and the number of main key objects and sub-key objects may not be limited to one.

[0068] The third target display condition is a condition for determining whether the display status of the virtual prompt element corresponding to the basic key object in the information display area satisfies a predetermined requirement. For example, the third target display condition may be to completely display the virtual prompt element corresponding to the basic key object in the information display area. Of course, in actual applications, the third target display condition may be set to other conditions according to actual needs, and the present application does not particularly limit the third target display condition.

[0069] Furthermore, the display position of the information display area is determined according to the imaging position of the virtual prompt element corresponding to the main key object on the image plane, and the virtual prompt element corresponding to the main key object is displayed in the information display area.Since the sub-key objects also require the driver's attention, the corresponding prompt contents are displayed in the information display area according to a predetermined display method, for example, a prompt sign or prompt text corresponding to the sub-key object may be displayed in the information display area.

[0070] For example, the prompt content corresponding to the sub-key object may be displayed based on a target edge region of the information display area, and the relative direction between the target edge region and the center of the information display area matches the relative direction between the sub-key object and the target vehicle in the driving environment.

[0071] 7 is a schematic diagram of an information display area according to an embodiment of the present application. As shown in FIG. 7, assuming that the sub-key object is a reference vehicle located to the left front of the target vehicle and the distance between the reference vehicle and the target vehicle is relatively short, the presence of the reference vehicle can be indicated in the lower left edge area of ​​the information display area. For example, the presence of the reference vehicle can be indicated by displaying the lower left edge area in red, or a prompt sign or prompt text corresponding to the reference vehicle can be displayed in the lower left edge area of ​​the information display area.

[0072] The target edge area may be determined depending on the relative direction between the subkey object and the target vehicle, and may specifically be at least one of the left edge area, right edge area, upper edge area, lower edge area, lower left edge area, upper left edge area, lower right edge area, and upper right edge area, but it should be understood that the present application does not impose any restrictions on the position of the target edge area.

[0073] In this way, when it is necessary to draw the driver's attention to multiple key objects simultaneously but it is not possible to simultaneously display the virtual prompt elements corresponding to each of the multiple key objects in the information display area, the simultaneous display of the multiple key objects can be realized by using the above method. Furthermore, by distinguishing between the main key object and the sub-key objects and displaying them in different ways, the driver can pay different degrees of attention to the multiple key objects. For example, by displaying the main key object in the form of a virtual prompt element and displaying the sub-key objects in the target edge area of ​​the information display area, the driver will first pay attention to the main key object and then to the sub-key objects, which contributes to improving driving safety.

[0074] In an information display method according to an embodiment of the present application, the position of the FOV of the AR HUD device is dynamically adjusted in response to changes in key objects in the driving environment where the target vehicle is located. That is, the position of the information display area formed by the AR HUD device and projected onto the windshield of the target vehicle is dynamically adjusted on the windshield. This allows virtual prompt elements corresponding to key objects in the driving environment to be continuously displayed in the information display area, and ensures that the display position of the virtual prompt element corresponds to the position of the key object in the driving environment, thereby providing the driver with more realistic prompt information. Furthermore, in an information display method according to an embodiment of the present application, when the key object changes from the currently focused field of view of the AR HUD device to another field of view, the AR HUD device can adjust the position of the information display area formed by the projection on the windshield so that the information display area corresponds to the changed field of view. In this way, the information displayed by the AR HUD device is no longer limited to a specific field of view but can dynamically adapt to a wider field of view. Accordingly, the driver can learn more prompt information through the dynamic information display area, thereby improving the user experience.

[0075] The above-described method for determining the display position of the information display area will be described in detail below with reference to a method embodiment. In this embodiment, the above-described method for determining the key object, the method for determining the position of the key object in the driving environment, and the method for determining the imaging position of the key object on the image plane of the AR HUD device will also be described.

[0076] 8 is a flowchart of a method for determining a display position of an information display area according to an embodiment of the present application. The method is performed by an electronic device in a target vehicle that controls an AR HUD device. As shown in FIG. 8, the method includes the following steps:

[0077] Step 801: Obtain a reference image collected by the front camera mounted on the target vehicle.

[0078] In an embodiment of the present application, the on-board front camera of the target vehicle can collect reference images corresponding to the driver's field of view in the driving environment in which the target vehicle exists, i.e., reference images for showing the driving environment in front of the target vehicle, at a specific frame rate (e.g., 30 frames / second). It should be understood that the on-board front camera here refers to a camera mounted on the target vehicle for collecting environmental images in front of the target vehicle, and specifically may be a dashcam.

[0079] Furthermore, the vehicle front camera can transmit the collected reference image to an electronic device for controlling the AR HUD device, so that the electronic device can determine key objects in the driving environment that require the driver's attention and the locations of the key objects based on the received reference image.

[0080] Step 802: Detect candidate key objects contained in the reference image using an image detection model.

[0081] After receiving the reference image transmitted from the vehicle front camera, the electronic device can input the reference image into a pre-trained image detection model, which can analyze and process the input reference image to output corresponding detection results, including the positions of candidate key objects in the reference image and the types of the candidate key objects.

[0082] The image detection model may be any deep neural network model that can detect a specific target in an input image and the position of the specific target in the input image, and examples of the image detection model include an R-CNN (Region-Convolutional Neural Networks) model and an R-FCN (Region-based Fully Convolutional Network) model, but the present application does not impose any restrictions on the model structure of the image detection model. The candidate key object may be understood as a certain type of object in a driving environment, such as a traffic sign, a vehicle, a pedestrian, etc.

[0083] As an example, in an embodiment of the present application, an image detection model can be used to detect traffic signs (including, but not limited to, lane guide signs on the road, lanes, etc.), vehicles, and pedestrians in a reference image. That is, when an electronic device inputs a reference image transmitted from a vehicle's front camera into an image detection model, the image detection model can detect whether traffic signs, vehicles, or pedestrians exist in the reference image. If present, the image detection model displays the positions of the traffic signs, vehicles, and pedestrians in the reference image. Other information in the driving environment that does not require special attention by the driver does not need to be detected by the image detection model.

[0084] In this way, the above method uses an image detection model to detect reference images collected by the front camera mounted on the target vehicle, and determines certain types of objects as candidate key objects, thereby accurately recognizing objects that require the driver's attention in the driving environment where the target vehicle currently exists, and then further determining key objects that actually require the driver's attention from these candidate key objects, thereby ensuring that the determined key objects are of high importance and are real objects in the driving environment where the target vehicle actually exists. Accordingly, the presence of the key objects can be displayed using virtual prompt elements, thereby further improving driving safety.

[0085] Step 803: Determine the base position of the candidate key object in the vehicle coordinate system according to the position of the vehicle-mounted front camera in the vehicle coordinate system of the target vehicle and the position of the candidate key object in the reference image.

[0086] After using the image detection model to determine each candidate key object included in the reference image, the electronic device can then determine the base position of each candidate key object in the vehicle coordinate system of the target vehicle according to the position of the candidate key object in the image coordinate system of the reference image, that is, switch the position information of the candidate key object between the image coordinate system and the vehicle coordinate system.

[0087] Specifically, the position of the onboard front camera in the vehicle coordinate system of the target vehicle can be calibrated in advance. FIG. 9 is a schematic diagram of the vehicle coordinate system of the target vehicle. The vehicle coordinate system has an origin at the intersection of the diagonal lines of the four wheels of the chassis of the target vehicle, with the x-axis pointing from the origin toward the front of the target vehicle, the y-axis pointing from the origin toward the right side of the target vehicle, and the z-axis pointing from the origin toward the top of the target vehicle. Next, based on the vehicle coordinate system, the position of the onboard front camera in the vehicle coordinate system can be calibrated offline, i.e., the x-, y-, and z-coordinates (unit: meters) of the onboard front camera in the vehicle coordinate system, as well as the pitch, yaw, and roll angles of the onboard front camera can be calibrated. In addition, the intrinsic parameters of the onboard front camera also need to be calibrated offline. It should be understood that the above calibration process is usually completed before the target vehicle leaves the factory.

[0088] If the position of the candidate key object in the image coordinate system of the reference image, the position of the on-board front camera used to collect the reference image in the vehicle coordinate system, and the intrinsic parameters of the on-board front camera are known, the electronic device can determine the basic position of the candidate key object in the vehicle coordinate system of the target vehicle by conventional spatial position transformation calculations.

[0089] Step 804: Determine a predicted position of the candidate key object at a future reference time according to a kinematic model corresponding to the target vehicle and the basic position, and use the predicted position as the position of the candidate key object in the driving environment, where the kinematic model corresponding to the target vehicle is determined according to the motion state parameters of the target vehicle.

[0090] One of the main functions of forming an information display area by projection from an AR HUD device is to convey early warning information to the driver. To ensure the timeliness of the early warning, the prompt content displayed in the information display area is usually predicted based on historical information. For example, the early warning prompt content displayed in the information display area at time t is usually predicted based on historical information collected at or before time t-1, but is not determined based on information collected at time t. Based on this, for each candidate key object in the reference image, the electronic device needs to predict the predicted position of the candidate key object at a future reference time according to the motion state corresponding to the target vehicle and the basic position of the candidate key object, use the predicted position as the position of the candidate key object in the driving environment described above, and accordingly determine the display position of the corresponding virtual prompt element in the information display area.

[0091] Specifically, the electronic device can acquire motion state parameters of the target vehicle collected by sensors of the target vehicle, such as vehicle speed, steering angle, inertial measurement unit (IMU) data, images collected by a camera, radar parameters, etc. The IMU data may include the acceleration and rotational speed of the target vehicle. Next, the electronic device can use the motion state parameters to establish a kinematic model corresponding to the target vehicle using an algorithm for establishing a vehicle kinematic model (e.g., a visual inertial odometry (VIO) algorithm). The kinematic model reflects the current motion state of the target vehicle and can estimate the future motion state of the target vehicle, such as its position and orientation, at a future time. According to the kinematic model of the target vehicle and the base position of the candidate key object in the vehicle coordinate system, the electronic device can predict the predicted position of the candidate key object at a future reference time (i.e., a specific future time, such as n seconds (n is greater than 0) from the current time). It should be understood that the predicted position is also a position in the vehicle coordinate system. Next, the predicted position is used as the position of the candidate key object in the driving environment in which the target vehicle exists. The position can be used to determine whether the candidate key object is a key object that requires the driver's attention, and to determine the imaging position of the corresponding virtual prompt element.

[0092] It should be noted that the operation of determining the predicted locations of the candidate key objects can be performed by a tracker in the electronic device, i.e., each candidate key object in the reference image is input to the tracker, which can then be used in combination with a kinematic model corresponding to the target vehicle to automatically determine the predicted location of each candidate key object at a future time.

[0093] In this way, the above method determines the predicted position of the key object at a future time based on the kinematic model corresponding to the target vehicle, thereby ensuring the accuracy of the determined predicted position of the key object, and accordingly uses the predicted position as the position of the key object in the driving environment, and determines the position of the virtual prompt element corresponding to the key object in the information prompt area based on the predicted position, thereby ensuring that the virtual prompt element accurately corresponds to the position of the key object in the actual driving environment.

[0094] Step 805: Determine the key object according to the candidate key objects and a key object determination rule.

[0095] Furthermore, the electronic device can select a key object that requires the driver's attention from each candidate key object based on the position of each candidate key object in the driving environment according to a predetermined key object determination rule, and in a subsequent step, project and display a virtual prompt element corresponding to the key object onto the information display area formed by projection from the AR HUD device, thereby alerting the driver to pay attention to the key object.

[0096] In one possible embodiment, the electronic device can determine the key object in the following manner in combination with the current driving state of the target vehicle: When the target vehicle is in a normal straight-ahead state, if there is an obstacle object among the candidate key objects whose distance to the target vehicle is less than a predetermined distance threshold, the obstacle object is determined as the key object; When the target vehicle is in a pre-lane-change state, lane-change warning information superimposed on the road surface is determined as the key object; When the target vehicle is in a pre-intersection state, if there is a lane guide sign at the intersection among the candidate key objects, the lane guide sign is determined as the key object.

[0097] As an example, early warning prompt information provided by an AR HUD device can be divided into two categories: guidance prompt information (used to guide the direction of the target vehicle) and warning prompt information (used to warn of obstacles present in the driving environment). Guidance prompt information typically corresponds to three states: straight ahead state (i.e., the vehicle is normally driving straight), pre-lane change state (i.e., the vehicle is about to change lanes), and pre-intersection arrival state (i.e., the vehicle is about to arrive at an intersection). Warning prompt information typically corresponds to two situations: leading vehicle warning and pedestrian warning.

[0098] In an embodiment of the present application, an exemplary key object determination rule is provided for determining a key object requiring the driver's attention based on the classification of the above-mentioned early warning prompt information. That is, when a target vehicle is in a normal straight-ahead state, if a candidate key object that may trigger an ADAS alarm is detected based on the position of each candidate key object in the driving environment, the candidate key object that may trigger an ADAS alarm is used as a key object requiring the driver's attention, such as a vehicle or pedestrian that may trigger an ADAS alarm. When the target vehicle is in a pre-lane change state, the lane change warning information indicated in the guidance information can be used as a key object. When the target vehicle is in a pre-intersection arrival state, the lane guidance sign at the intersection can be used as a key object.

[0099] The above key object determination rule is merely an example, and in actual applications, key objects that require the driver's attention may be determined based on other key object determination rules, but it should be understood that this application does not place any restrictions on the key object determination rules.

[0100] In this way, the above method determines the key object that requires the driver's attention from each candidate key object according to factors such as the type of each candidate key object and the distance from the target vehicle, in combination with the current driving state of the target vehicle, thereby ensuring that the determined key object is the key object that currently requires the driver's attention the most. In response to this, a virtual prompt element corresponding to the key object is displayed in the information display area, thereby providing a more effective warning effect and contributing to improved driving safety.

[0101] Step 806: Determine a target imaging position of a virtual prompt element corresponding to the key object on an image plane according to the position of an eyebox of the AR HUD device in the vehicle coordinate system and the position of the key object in the driving environment, where the image plane is a virtual image formation surface projected from the AR HUD device.

[0102] After determining the key object requiring the driver's attention and the position of the key object in the driving environment, the electronic device can determine a target imaging position of a virtual prompt element corresponding to the key object in the image plane (virtual image forming surface) of the AR HUD device according to the position of the eyebox of the AR HUD device in the vehicle coordinate system of the target vehicle and the position of the key object in the driving environment.

[0103] It should be noted that the eyebox of an AR HUD device refers to the moving area of ​​the driver's eyes. Factors such as the driver's height, sitting posture, and head position typically affect the eye position and gaze direction, which can change the driver's field of view. When the driver's eyes are within the eyebox of the AR HUD device, the driver can clearly and completely see the information display area projected from the AR HUD device. In an embodiment of the present application, the position of the eyebox of the AR HUD device in the vehicle coordinate system of the target vehicle can be calibrated before the target vehicle leaves the factory. The vehicle coordinate system of the target vehicle is shown in FIG. 9. Based on the vehicle coordinate system, the x-, y-, and z-coordinates (unit: meters) of the eyebox in the vehicle coordinate system, as well as the pitch, yaw, and roll angles of the eyebox, can be calibrated offline. Furthermore, the projection-specific parameters of the AR HUD device also need to be calibrated offline.

[0104] If the position of a key object in the vehicle coordinate system is known (the position of the key object in the driving environment is essentially the predicted position of the key object in the vehicle coordinate system), if the position of the eyebox of the AR HUD device in the vehicle coordinate system and the projection-specific parameters of the AR HUD device are known, the electronic device can determine the target imaging position of the virtual prompt element corresponding to the key object on the virtual image forming surface of the AR HUD device by spatial position transformation calculation.

[0105] The position of the on-board front camera, the position of the eyebox of the AR HUD device, and the position of the key object are all determined based on the vehicle coordinate system of the target vehicle, so that the accuracy of the target imaging position in the image plane of the virtual prompt element corresponding to the determined key object can be effectively ensured, that is, the display position corresponding to the target imaging position in the information display area can be accurately matched with the key object in the actual driving environment.

[0106] Step 807: Determine a target imaging area on the image plane corresponding to the information display area according to the target imaging position, where the target imaging area covers the target imaging position, and the position of the target imaging area on the image plane determines the position of the information display area on the windshield.

[0107] After determining the target imaging position of the virtual prompt element corresponding to the key object on the image plane of the AR HUD device, a corresponding target imaging area in the image plane of the information display area formed by the projection of the AR HUD device can be determined according to the target imaging position. Specifically, the position of the target imaging area may be determined based on the principle that the target imaging area completely covers the target imaging position.

[0108] It should be understood that determining a target imaging area on an image plane means determining the display position of an information display area corresponding to the target imaging area, and determining a target imaging position on an image plane means determining the display position of a virtual prompt element corresponding to the target imaging position. That is, according to the principle of light reflection, the target imaging area and the target imaging position on the image plane can be projected as an information display area and a virtual prompt element, respectively.

[0109] As described in the embodiment shown in FIG. 2, the key object may change while the target vehicle is traveling. If the key object changes from one key object to another, the electronic device can detect this through steps 801 to 805. In this case, the electronic device executes step 806 to determine the target imaging position of the virtual prompt element corresponding to the new key object on the image plane, and then executes step 807 to determine whether the position of the target imaging area needs to be adjusted according to the positional relationship between the target imaging position and the original target imaging area. If the target imaging position exceeds the target imaging area, step 807 must be executed to re-determine the position of the target imaging area so that the target imaging area completely covers the target imaging position. By adjusting the position of the target imaging area, the display position of the information display area can be adjusted. If the change in the key object is a change in position relative to the target vehicle, the electronic device determines the target imaging position of the virtual prompt element corresponding to the key object on the image plane from steps 801 to 806, and then executes step 807 to determine whether the position of the target imaging area needs to be adjusted according to the positional relationship between the target imaging position and the original target imaging area. If the target imaging position exceeds the target imaging area, step 807 needs to be executed to re-determine the position of the target imaging area so that the target imaging area completely covers the target imaging position. By adjusting the position of the target imaging area, the display position of the information display area can be adjusted.

[0110] In practical applications, to prevent the information display area from exceeding the driver's field of vision or from becoming completely invisible to the driver due to adjustment of the display position of the information display area, a movable range corresponding to the target imaging area can be set and the target imaging area can be controlled to move only within that movable range.

[0111] FIG. 10 is a schematic diagram of a movable range of a target imaging area according to an embodiment of the present application. As shown in FIG. 10, in the image plane, there is a boundary between all parallel lines (such as lanes) on the ground and the sky, and this line is usually called the horizon. In an embodiment of the present application, the upper and lower boundary lines of the movable range can be determined based on the imaging position corresponding to the horizon line in the image plane. For example, an imaging position corresponding to N meters ahead of the target vehicle in the image plane can be defined as the lower boundary line, and an imaging position corresponding to M meters ahead of the target vehicle in the image plane can be defined as the upper boundary line. Here, N is smaller than the distance between the horizon line and the target vehicle, and M is larger than the distance between the horizon line and the target vehicle. Furthermore, in an embodiment of the present application, the left and right boundaries of the movable range can be defined according to actual needs, and the present application does not particularly limit the area of ​​the movable range.

[0112] In the method for determining the display position of an information display area according to an embodiment of the present application, the positions of the on-board front camera of the target vehicle and the eyebox of the AR HUD device are determined offline in the vehicle coordinate system of the target vehicle, and then the position of a key object in a reference image collected by the on-board front camera is mapped onto the vehicle coordinate system, and the position of the key object in the vehicle coordinate system is mapped onto the projection plane of the AR HUD device, thereby ensuring that the virtual prompt element corresponding to the key object generated from the projection matches the key object in the actual driving environment, thereby ensuring augmented reality.In addition, in the embodiment of the present application, the target imaging area corresponding to the information display area on the windshield is dynamically adjusted according to the target imaging position of the virtual prompt element corresponding to the key object on the image plane, thereby ensuring that the virtual prompt element corresponding to the key object is always completely displayed in the information display area, thereby improving the richness of information provided by the information display area.

[0113] To enable practical application and implementation of the above-described information display method, the present application also provides a corresponding information display device.

[0114] 11 is a schematic diagram showing the configuration of an information display device 1100 corresponding to the information display method shown in FIG. 2. As shown in FIG. 11, the information display device 1100 includes: a projection display module 1101 configured to perform the steps of: generating, via a windshield of a target vehicle, an information display area projected from the AR HUD in the target vehicle, the information display area including at least a virtual prompt element corresponding to a key object, the key object being an object in a driving environment in which the target vehicle is located, and a display position of the virtual prompt element corresponding to a position of the key object in the driving environment; and a position change module 1102 configured to perform a step of changing the position of the information display area on the windshield when a change that satisfies a predetermined condition occurs in the key object so that the information display area includes a virtual prompt element that corresponds to the changed key object.

[0115] Optionally, the position change module 1102 specifically: the step of changing the position of the information display area on the windshield when a change that satisfies a predetermined condition occurs in the key object, When the key object changes from a first key object to a second key object and the information display area does not satisfy a first target display condition for a virtual prompt element corresponding to the second key object, the method is configured to perform a step of changing the position of the information display area on the windshield so that the information display area satisfies the first target display condition for a virtual prompt element corresponding to the second key object.

[0116] Optionally, the position change module 1102 specifically: a step of moving a position of the information display area on the windshield upward when the first key object is a target ground object and the second key object is a target ground object, wherein the position of the target ground object is lower than the position of the target ground object; If the first key object is a target ground object and the second key object is a target ground object, moving the position of the information display area on the windshield downward.

[0117] Optionally, the position change module 1102 specifically: When the first key object is a target object on the left side of the line of sight and the second key object is a target object on the right side of the line of sight, moving a position of the information display area on the windshield to the right; When the first key object is a target object on the right side of the line of sight and the second key object is a target object on the left side of the line of sight, the method is configured to execute a step of moving the position of the information display area on the windshield to the left.

[0118] Optionally, the position change module 1102 specifically: determining a first imaging position in an image plane of a virtual prompt element corresponding to the first key object and a second imaging position in the image plane of a virtual prompt element corresponding to the second key object, the image plane being a virtual image plane projected from the AR HUD; and changing the position of the information display area on the windshield according to a positional relationship between the first imaging position and the second imaging position.

[0119] Optionally, the position change module 1102 specifically: When the position of the key object in the driving environment changes from a first position to a second position and the information display area does not satisfy a second target display condition for a virtual prompt element corresponding to the key object at the second position, the system is configured to perform a step of changing the position of the information display area on the windshield so that the information display area satisfies a second target display condition for a virtual prompt element corresponding to the key object at the second position.

[0120] Optionally, the position change module 1102 specifically: determining a third imaging position in an image plane of a virtual prompt element corresponding to the key object at the first position and a fourth imaging position in the image plane of a virtual prompt element corresponding to the key object at the second position, the image plane being a virtual image plane projected from the AR HUD; and changing a position of the information display area on the windshield according to a positional relationship between the third imaging position and the fourth imaging position.

[0121] Optionally, the projection display module 1101 further comprises: When the driving environment includes a plurality of basic key objects and the information display area does not satisfy a third target display condition for the virtual prompt element corresponding to each of the plurality of basic key objects, the method is configured to execute a step of displaying a virtual prompt element corresponding to a main key object among the plurality of basic key objects in the information display area and displaying prompt content corresponding to a subkey object among the plurality of basic key objects according to a predetermined display method.

[0122] Optionally, the projection display module 1101 specifically includes: The device is configured to perform a step of displaying prompt content corresponding to the subkey object based on a target edge area of ​​the information display area, wherein the relative direction between the target edge area and the center of the information display area matches the relative direction between the subkey object and the target vehicle in the driving environment.

[0123] Optionally, the apparatus further comprises a key object determination module, the key object determination module comprising: acquiring a reference image collected by a front camera mounted on the target vehicle; detecting candidate key objects in the reference image using an image detection model; determining the key object according to the candidate key objects and a key object determination rule.

[0124] Optionally, the key object determination module specifically: When the target vehicle is traveling straight ahead, if there is an obstacle object among the candidate key objects whose distance from the target vehicle is less than a predetermined distance threshold, determining the obstacle object as the key object; When the target vehicle is in a pre-lane change state, determining lane change warning information superimposed on a road surface as the key object; When the target vehicle is in a pre-intersection arrival state, if there is a lane guide sign at an intersection among the candidate key objects, determining the lane guide sign as the key object.

[0125] Optionally, the apparatus further comprises a target location determination module, the target location determination module comprising: determining a base position of the key object in the vehicle coordinate system according to a position of the on-board front camera in the vehicle coordinate system of the target vehicle and a position of the key object in the reference image; and determining a predicted position of the key object at a future reference time according to a kinematic model corresponding to the target vehicle and the base position, and using the predicted position as the position of the key object within the driving environment, wherein the kinematic model corresponding to the target vehicle is determined according to motion state parameters of the target vehicle.

[0126] Optionally, the apparatus further comprises a projection position determination module, the projection position determination module comprising: determining a target imaging position on an image plane of a virtual prompt element corresponding to the key object according to a position of an eyebox of the AR HUD in a vehicle coordinate system of the target vehicle and a position of the key object in the driving environment, wherein the image plane is a virtual image formation surface projected from the AR HUD; The system is configured to perform a step of determining a target imaging area in the image plane corresponding to the information display area according to the target imaging position, wherein the target imaging area covers the target imaging position, and the position of the target imaging area in the image plane determines the position of the information display area on the windshield.

[0127] In an information display device according to an embodiment of the present application, the position of the FOV of the AR HUD device is dynamically adjusted in response to changes in key objects in the driving environment where the target vehicle is located. That is, the position of the information display area formed by the AR HUD device and projected onto the windshield of the target vehicle is dynamically adjusted on the windshield. This allows virtual prompt elements corresponding to key objects in the driving environment to be continuously displayed in the information display area, and ensures that the display position of the virtual prompt element corresponds to the position of the key object in the driving environment, thereby providing the driver with more realistic prompt information. Furthermore, in an information display device according to an embodiment of the present application, when the key object changes from the currently focused field of view of the AR HUD device to another field of view, the AR HUD device can adjust the position of the information display area formed by the projection on the windshield so that the information display area corresponds to the changed field of view. In this way, the information displayed by the AR HUD device is no longer limited to a specific field of view but can dynamically adapt to a wider field of view. Accordingly, the driver can learn more prompt information through the dynamic information display area, thereby improving the user experience.

[0128] According to an embodiment of the present application, there is further provided an electronic device for implementing the above information display method. The electronic device may specifically be a terminal device corresponding to the controller described above. The terminal device according to the embodiment of the present application will be described below in terms of hardware instantiation.

[0129] 12 is a schematic configuration diagram of a terminal device according to an embodiment of the present application. As shown in FIG. 12, for convenience of explanation, only parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method embodiment section of the present application. The terminal can be any terminal device, such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sale (POS), an in-vehicle terminal, etc., and the in-vehicle terminal is used here as an example.

[0130] 12 is a block diagram showing a partial configuration of an in-vehicle terminal related to a terminal according to an embodiment of the present application. As shown in FIG. 12, the in-vehicle terminal includes components such as a radio frequency (RF) circuit 1210, a memory 1220, an input unit 1230 (including a touch panel 1231 and other input devices 1232), a display unit 1240 (including a display panel 1241), a sensor 1250, an audio circuit 1260 (connectable to a speaker 1261 and a microphone 1262), a wireless fidelity (WiFi) module 1270, a processor 1280, and a power supply 1290. Those skilled in the art will understand that the in-vehicle terminal structure shown in FIG. 12 is not limiting and may include more or fewer components than those shown, or a combination of specific components, or a different arrangement of components.

[0131] The memory 1220 can be used to store software programs and modules. The processor 1280 executes the software programs and modules stored in the memory 1220 to perform various functional applications and data processing of the vehicle terminal. The memory 1220 mainly includes a program storage area and a data storage area. The program storage area stores an operating system and applications required for at least one function (such as audio playback function or image playback function), and the data storage area stores data created in accordance with the use of the vehicle terminal (such as audio data and a phone book). The memory 1220 further includes a high-speed random access memory and may also include non-volatile memory such as at least one disk storage device, flash memory, or other volatile solid-state storage device.

[0132] Processor 1280 is the control center of the in-vehicle terminal. It connects various components of the in-vehicle terminal using various interfaces and lines, executes software programs and / or modules stored in memory 1220, and accesses data stored in memory 1220 to perform various functions of the in-vehicle terminal and process data. Optionally, processor 1280 may include one or more processing units. Preferably, processor 1280 may integrate an application processor and a modem processor. The application processor mainly processes the operating system, user interface, and application programs, while the modem processor mainly processes wireless communications. It should be understood that the modem processor does not have to be integrated into processor 1280.

[0133] In the embodiment of the present application, the processor 1280 included in the terminal is further used to perform steps of any one of the implementations of the information display method according to the embodiment of the present application.

[0134] According to an embodiment of the present application, there is further provided a computer-readable storage medium having a computer program stored thereon, which executes any one of the information display methods according to the above embodiments.

[0135] According to an embodiment of the present application, there is further provided a computer program product or a computer program including computer instructions stored in a computer-readable storage medium, wherein a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, thereby causing the computer device to perform any one of the information display methods described in the above embodiments.

[0136] Those skilled in the art will clearly understand that for convenience and conciseness of description, the specific operation processes of the systems, devices, and units described above can refer to the corresponding processes described in the above method embodiments, but will not be further described.

[0137] It should be understood that in some embodiments according to the present application, the disclosed systems, devices, and methods may be implemented in other ways. For example, the device embodiment described above is merely an example. For example, the division of the units is merely a logical division of function, and other division methods may exist in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some functions may be ignored or not performed. Furthermore, the illustrated or described mutual couplings or direct couplings or communication connections may be indirect couplings or communication connections via electrical, mechanical, or other types of interfaces, devices, or units.

[0138] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. To achieve the objectives of the technical solution of this embodiment, some or all of the units therein may be selected according to actual needs.

[0139] Furthermore, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist physically independently, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or a software functional unit.

[0140] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the portion contributing to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (such as a personal computer, a server, or a network device) to execute all or part of the steps of the method according to each embodiment of the present application. The aforementioned storage medium includes other media capable of storing a computer program, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0141] In this application, "at least one" means one or more, and "plurality" means two or more. "And / or" is used to describe a relationship between related objects and indicates that three relationships can exist. For example, "A and / or B" means that only A exists, only B exists, or both A and B exist simultaneously. Here, A and B can be singular or plural. The character " / " typically indicates that the related objects before and after it are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of a single item or multiple items. For example, "at least one of a, b, and c" can refer to "a and b," "a and c," "b and c," or "a, b, and c" among a, b, and c. Here, a, b, and c can be singular or plural.

[0142] Therefore, the above embodiments do not limit the technical solutions of the present application, but are used only to explain them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may be amended or some of the technical features may be replaced with equivalents. However, such amendments or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application. [Explanation of symbols]

[0143] 100 Target Vehicles 110 HUD Device 111 Optical machinery 111 Optical equipment 112, 113 Reflective member 120 Controller 410 Windshield 420 Information display area 421 Virtual Prompt Element 510 areas 511 Virtual Elements 520 areas 521 Virtual Elements 610 areas 611 Virtual Elements 620 areas 621 Virtual Elements 1100 Information display device 1101 Projection Display Module 1102 Position Change Module 1210 Radio Frequency (RF) Circuits 1220 memory 1230 input units 1231 Touch Panel 1232 Input Devices 1240 display unit 1241 Display Panel 1250 Sensor 1260 Audio Circuit 1261 Speaker 1262 Mike 1280 processor 1290 Power supply

Claims

1. 1. A method of displaying information performed by an electronic device, comprising: generating an information display area projected from an Augmented Reality Head Up Display (AR HUD) of the target vehicle by a windshield of the target vehicle, the information display area including at least a virtual prompt element corresponding to a key object, the key object being an object in a driving environment in which the target vehicle exists, and a display position of the virtual prompt element corresponding to a position of the key object in the driving environment; and when a change occurs in the key object that satisfies a predetermined condition, changing the position of the information display area on the windshield so that the information display area includes a virtual prompt element corresponding to the changed key object.

2. the step of changing the position of the information display area on the windshield when a change that satisfies a predetermined condition occurs in the key object, 2. The information display method of claim 1, further comprising: when the key object changes from a first key object to a second key object and the information display area does not satisfy a first target display condition for a virtual prompt element corresponding to the second key object, changing the position of the information display area on the windshield so that the information display area satisfies the first target display condition for a virtual prompt element corresponding to the second key object.

3. The step of changing the position of the information display area on the windshield includes: a step of moving a position of the information display area on the windshield upward when the first key object is a target ground object and the second key object is a target ground object, wherein the position of the target ground object is lower than the position of the target ground object; 3. The information display method according to claim 2, further comprising the step of: moving a position of the information display area on the windshield downward when the first key object is a target ground object and the second key object is a target ground object.

4. The step of changing the position of the information display area on the windshield includes: a step of moving a position of the information display area on the windshield to the right when the first key object is a target object on the left side of the line of sight and the second key object is a target object on the right side of the line of sight; 4. The information display method according to claim 2, further comprising: a step of moving a position of the information display area on the windshield to the left when the first key object is a target object on the right side of the line of sight and the second key object is a target object on the left side of the line of sight.

5. The step of changing the position of the information display area on the windshield includes: determining a first imaging position in an image plane of a virtual prompt element corresponding to the first key object, and determining a second imaging position in the image plane of a virtual prompt element corresponding to the second key object, the image plane being a virtual image formation surface projected from the AR HUD; and changing the position of the information display area on the windshield in accordance with the positional relationship between the first imaging position and the second imaging position.

6. the step of changing the position of the information display area on the windshield when a change that satisfies a predetermined condition occurs in the key object, An information display method as described in any one of claims 1 to 5, comprising a step of changing the position of the information display area on the windshield so that, when the position of the key object in the driving environment changes from a first position to a second position and the information display area does not satisfy a second target display condition for a virtual prompt element corresponding to the key object at the second position, the information display area satisfies a second target display condition for a virtual prompt element corresponding to the key object at the second position.

7. The step of changing the position of the information display area on the windshield includes: determining a third imaging position in an image plane of a virtual prompt element corresponding to the key object at the first position, and determining a fourth imaging position in the image plane of a virtual prompt element corresponding to the key object at the second position, the image plane being a virtual image plane projected from the AR HUD; The information display method according to claim 6, further comprising: changing a position of the information display area on the windshield in accordance with a positional relationship between the third imaging position and the fourth imaging position.

8. The information display method according to any one of claims 1 to 7, further comprising the step of: when the driving environment includes a plurality of basic key objects and the information display area does not satisfy a third target display condition for the virtual prompt element corresponding to each of the plurality of basic key objects, displaying in the information display area a virtual prompt element corresponding to a main key object among the plurality of basic key objects, and displaying prompt content corresponding to a subkey object among the plurality of basic key objects according to a predetermined display method.

9. The step of displaying prompt content corresponding to a subkey object among the plurality of basic key objects according to a predetermined display method includes:

9. The information display method of claim 8, further comprising a step of displaying prompt content corresponding to the sub-key object based on a target edge area of ​​the information display area, wherein the relative direction between the target edge area and the center of the information display area matches the relative direction between the sub-key object and the target vehicle in the driving environment.

10. The key object acquires a reference image collected by a front camera mounted on the target vehicle; detecting candidate key objects contained in the reference image by an image detection model; and 10. The information display method according to claim 1, wherein the key object is determined by the step of determining the key object according to the candidate key objects and a key object determination rule.

11. The step of determining the key object according to the candidate key objects and a key object determination rule includes: When the target vehicle is traveling straight ahead, if there is an obstacle object among the candidate key objects whose distance from the target vehicle is less than a predetermined distance threshold, determining the obstacle object as the key object; When the target vehicle is in a pre-lane change state, determining lane change warning information superimposed on a road surface as the key object; and when the target vehicle is in a pre-intersection arrival state, if there is a lane guide sign at the intersection among the candidate key objects, determining the lane guide sign as the key object.

12. determining a base position of the key object in the driving environment in the vehicle coordinate system according to a position of the on-board front camera in the vehicle coordinate system of the target vehicle and a position of the key object in the reference image; and 12. The information display method according to any one of claims 1 to 11, wherein the key object is determined by a step of determining a predicted position of the key object at a future reference time according to a kinematic model corresponding to the target vehicle and the base position, and setting the predicted position as the position of the key object in the driving environment, wherein the kinematic model corresponding to the target vehicle is determined according to motion state parameters of the target vehicle.

13. The position of the information display area on the windshield is determining a target imaging position in an image plane of a virtual prompt element corresponding to the key object according to a position of an eyebox of the AR HUD in a vehicle coordinate system of the target vehicle and a position of the key object in the driving environment, wherein the image plane is a virtual image formation surface projected from the AR HUD; and An information display method as claimed in any one of claims 1 to 12, wherein the method comprises the steps of: determining a target imaging area in the image plane corresponding to the information display area according to the target imaging position, the target imaging area covering the target imaging position; and determining the position of the target imaging area in the image plane to determine the position of the information display area on the windshield.

14. a projection display module configured to perform the steps of: generating, by a windshield of a target vehicle, an information display area projected from an Augmented Reality-Head Up Display (AR HUD) in the target vehicle, the information display area including at least a virtual prompt element corresponding to a key object, the key object being an object in a driving environment in which the target vehicle is located, and a display position of the virtual prompt element corresponding to a position of the key object in the driving environment; and a position change module configured to perform a step of changing a position of the information display area on the windshield when a change that satisfies a predetermined condition occurs in the key object so that the information display area includes a virtual prompt element that corresponds to the changed key object.

15. An electronic device including a processor and a memory in which a computer program is stored, An electronic device that causes the processor to execute the information display method according to any one of claims 1 to 13 in accordance with the computer program.

16. A computer-readable storage medium on which a computer program is stored, A computer-readable storage medium that causes the information display method according to any one of claims 1 to 13 to be executed in accordance with the computer program.

17. A computer program product comprising a computer program or instructions, A computer program product which, when the computer program or the instructions are executed by a processor, causes the computer program product to implement the information display method according to any one of claims 1 to 13.

Images