POI (Point of Interest) enhanced display method and related equipment

Abstract

The embodiment of the invention discloses a POI (Point of Interest) enhanced display method and related equipment. The vehicle machine can run a navigation application program and open a navigation route in the navigation application program. The vehicle machine can turn on a camera to collect an image in front of a windshield of a vehicle. Then, the vehicle machine can judge that the target scenery appears in the driving field of view and is located on the navigation route based on the image, the vehicle machine projects an identifier of the target scenery to the front of a windshield of the vehicle, and the identifier is superposed on the target scenery in the driving field of view; the display position of the identifier in front of the windshield of the vehicle is determined based on a first three-dimensional coordinate, the first three-dimensional coordinate can be measured and determined by a vehicle machine through a first sensor, and the first sensor comprises one or more of the following items: a radar, a depth-of-field sensor and a laser sensor. Therefore, according to the method, accurate positioning and enhanced display of the target scene can be realized, and the navigation experience of a user is further optimized.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a POI enhanced display method and related equipment. Background Technology

[0002] Augmented reality head-up display (AR HUD) is a technology that projects vehicle information, navigation prompts, and warnings from driver assistance systems directly into the driver's line of sight in the form of graphics and / or text. The combination of AR HUD and in-vehicle navigation systems provides users with a new navigation experience. It offers drivers an intuitive and easy-to-understand interface that enhances their perception of their surroundings, enabling them to react more quickly. It also reduces the risk of driver distraction from looking at the instrument panel or center console screen, thereby improving driving safety and convenience.

[0003] However, due to the limited accuracy of points of interest (POIs) and deviations in map drawing processes in traditional navigation systems, the location of the destination is often not accurate enough. In other words, the destination located by the navigation system often falls near the actual destination. As a result, the AR HUD's labeling of the navigation destination is also inaccurate when approaching it. That is, because the AR HUD does not actually match the displayed content of the HUD with the actual destination, users still need to make further independent judgments to find the destination entrance when they are close to it. Summary of the Invention

[0004] This application provides a POI (Point of Interest) enhancement display method and related equipment. The vehicle's infotainment system can run a navigation application and activate a navigation route within that application. The system can activate its camera to capture an image of the area in front of the vehicle's windshield. Subsequently, based on this image, the system can determine that a target object appears in the driver's field of view and is located on the navigation route. The system then projects an icon of the target object onto the windshield. In the driver's field of view, this icon is superimposed on the target object. The display position of the icon on the windshield is determined based on a first three-dimensional coordinate system, which can be measured by the system using a first sensor. This first sensor includes one or more of the following: radar, a depth sensor, and a laser sensor. Therefore, this method can achieve accurate positioning and enhanced display of the target object, thereby optimizing the user's navigation experience.

[0005] In a first aspect, this application provides a point of interest (POI) enhancement display method applied to an in-vehicle device. The method includes: running a navigation application and activating a first route in the navigation application; activating a camera to capture an image in front of the vehicle's windshield; determining, based on the image, that a target object appears in the driver's field of view and is located on the first route; the in-vehicle device projects a first virtual image onto the windshield of the vehicle; in the driver's field of view, the first virtual image is superimposed on the target object; the display position of the first virtual image in front of the vehicle's windshield is determined based on a first three-dimensional coordinate, which is measured and determined by the in-vehicle device through a first sensor, the first sensor including one or more of the following: radar, depth sensor, and laser sensor.

[0006] By implementing the method provided in the first aspect, during navigation, after determining that the target scene appears in the driver's field of vision based on the image captured by the camera, the on-board equipment (i.e., the vehicle system) can use first sensors such as radar, depth sensor, and laser sensor to measure the accurate three-dimensional coordinates (i.e., the first three-dimensional coordinates) of the target scene. Subsequently, this accurate three-dimensional coordinates can be used to determine the accurate position of the target scene's marker (i.e., the first virtual image) projected onto the front of the vehicle's windshield, so that the target scene's marker can be accurately superimposed on the target scene, thereby achieving precise positioning and enhanced display of the target scene, thus optimizing the user's navigation experience.

[0007] In conjunction with the first aspect, in some embodiments, the target scene includes a first entrance of a first building, and before the in-vehicle device projects the first virtual image onto the windshield of the vehicle, the method further includes: determining, based on images captured by a camera, that multiple entrances of the first building appear in the driver's field of view; and selecting the first entrance from the multiple entrances.

[0008] Implementing the method provided in the above embodiments, the target scene can be an entrance (i.e., the first entrance) of a large building. If the vehicle-mounted device determines, through the image captured by the camera, that multiple entrances of the large building (i.e., the first building) appear in the driver's field of vision, that is, multiple entrances can be seen by the driver, the vehicle-mounted device can select the first entrance from the multiple entrances. Optionally, the vehicle-mounted device can select the first entrance (i.e., the optimal entrance point of interest) from the multiple entrances based on the driving distance from the vehicle-mounted device to the multiple entrances.

[0009] In conjunction with the first aspect, in some embodiments, the in-vehicle device projects a first virtual image onto the front of the vehicle's windshield, specifically including: the in-vehicle device projects the first virtual image onto a virtual image plane in front of the windshield, the virtual image plane being perpendicular to the ground, and the first virtual image, the target scene, and the driver's viewpoint on the virtual image plane being located on the same straight line.

[0010] When implementing the method provided in the above embodiments, the in-vehicle device projects the first virtual image, that is, the image of the target scene's identifier, by using the optical engine system of the AR HUD to project the identifier onto a virtual image plane perpendicular to the ground in front of the vehicle, so that the identifier, the target scene, and the driver's viewpoint are on the same straight line, so that the user can perceive that the identifier is perfectly aligned with the real scene, and it actually serves as a navigation indicator.

[0011] In conjunction with the first aspect, in some embodiments, the first three-dimensional coordinates are the three-dimensional coordinates of the target scene in a coordinate system with the driver's viewpoint as the origin, and the two-dimensional coordinates of the first virtual image on the virtual image plane are obtained by transforming the first three-dimensional coordinates.

[0012] Implementing the method provided in the above embodiments, the in-vehicle device needs to project the target object's identifier onto the virtual image plane by determining the projection position using two-dimensional coordinates. These two-dimensional coordinates are obtained by converting the accurate three-dimensional coordinates (i.e., the first three-dimensional coordinates) measured by the first sensor. These accurate three-dimensional coordinates are three-dimensional coordinates in a coordinate system with the driver's viewpoint as the origin. In other words, these accurate three-dimensional coordinates are three-dimensional coordinates with the driver's eye as the origin, thus ensuring that the target object's identifier can be accurately projected onto the corresponding position in the driver's line of sight, improving the accuracy of the in-vehicle device's recognition and positioning of the target object.

[0013] In conjunction with the first aspect, in some embodiments, the method further includes: determining the vehicle's pose using vehicle control sensors, including accelerometers, gyroscopes, global navigation satellite systems, lidar, and cameras; and determining first three-dimensional coordinates based on the vehicle's pose and the distance between the target scene and the vehicle measured by the first sensor.

[0014] By implementing the method provided in the above embodiments, the on-board device can determine the vehicle's position and pose using vehicle control sensors, and jointly determine the accurate three-dimensional coordinates of the target object in space based on the vehicle's position and pose and the data measured by the first sensor. This is because the accurate three-dimensional coordinates are based on the driver's viewpoint, that is, the driver's eye. The axes of this relative coordinate system can be determined by the vehicle's orientation. Therefore, the accurate three-dimensional coordinates of the target object can be determined based on the vehicle's position and pose and the data measured by the first sensor, thereby obtaining accurate two-dimensional coordinates in the subsequent transformation process and accurately projecting them.

[0015] In conjunction with the first aspect, in some embodiments, determining that a target object appears in the driver's field of view based on an image specifically includes:

[0016] Based on the image captured by the camera at time t1, it is determined that the target scene appears in the driver's field of vision; the vehicle-mounted device projects the first virtual image onto the windshield of the vehicle, specifically including: the vehicle-mounted device projects the first virtual image at time t2 onto the windshield of the vehicle, the first virtual image at time t2 is determined by the vehicle-mounted device based on the first three-dimensional coordinates determined by the first sensor at time t2, and time t2 is the same as time t1 or the time difference between time t2 and time t1 is less than the first time threshold.

[0017] Implementing the method provided in the above embodiments, the first three-dimensional coordinates calculated at time t2 are actually the first three-dimensional coordinates of the target scene captured by the camera at time t1. This t2 can be equal to t1, or the time difference between t2 and t1 can be within the first time threshold range, thereby ensuring the real-time performance of this solution. This allows the first three-dimensional coordinates generated at time t2 to be accurately identified and superimposed on the target scene when used for ARHUD projection rendering, thereby ensuring the realization of the AR HUD augmented reality function.

[0018] In conjunction with the first aspect, in some embodiments, the closer the vehicle-mounted device is to the target scene, the larger the first virtual image superimposed on the target scene; or, the farther the vehicle-mounted device is from the target scene, the smaller the first virtual image superimposed on the target scene.

[0019] By implementing the method provided in the above embodiments, the in-vehicle device can overlay the target scene's identifier onto the target scene in real time, regardless of how the in-vehicle device moves with the vehicle, thereby ensuring the realization of the AR HUD augmented reality function and providing users with accurate navigation prompts.

[0020] In conjunction with the first aspect, in some embodiments, the method further includes: determining, based on the image, that the target scene has disappeared from the driver's field of vision, and canceling the projection of the first virtual image onto the windshield of the vehicle by the in-vehicle device.

[0021] In conjunction with the first aspect, in some embodiments, before activating the camera, the method further includes: detecting that the distance between the vehicle-mounted device and the target scene is less than or equal to a first distance threshold.

[0022] By implementing the method provided in the above embodiments, the in-vehicle device can activate the camera to detect whether the target object appears in the driver's field of view only after detecting that the distance between itself and the target object is less than a first distance threshold. This ensures that in the initial stage of navigation, when the distance between the vehicle and the target object is too large, the in-vehicle device does not need to trigger the method provided in this application embodiment, thereby reducing the power consumption of the in-vehicle system during navigation.

[0023] In conjunction with the first aspect, in some embodiments, the first virtual image includes one or more of the following: a location icon, a name of the target scene, and an outline of the target scene.

[0024] In conjunction with the first aspect, in some embodiments, the target scenery includes the entrance of the first building, the road in the first building, and the first building itself.

[0025] In a second aspect, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the method described in the first aspect and any possible implementation thereof.

[0026] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first aspect and any possible implementation thereof.

[0027] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect and any possible implementation thereof.

[0028] Understandably, the electronic device provided in the second aspect, the computer storage medium provided in the third aspect, and the computer program product provided in the fourth aspect are all used to execute the method provided in this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the imaging principle of an AR HUD provided in an embodiment of this application;

[0030] Figure 2 This is a schematic diagram of the POI range when a large building is the destination, provided in an embodiment of this application;

[0031] Figure 3 This is a simplified flowchart illustrating a POI enhanced display method provided in an embodiment of this application;

[0032] Figure 4 This is a schematic diagram illustrating a use case of a POI enhanced display method provided in an embodiment of this application;

[0033] Figure 5 This is a schematic diagram illustrating the effect of a POI enhancement display method provided in an embodiment of this application;

[0034] Figure 6 This is a schematic diagram of a smart cockpit vehicle infotainment software architecture provided in an embodiment of this application;

[0035] Figure 7 This is a flowchart illustrating a POI enhanced display method provided in an embodiment of this application;

[0036] Figure 8 This is a schematic diagram of the interface display of an AR HUD provided in an embodiment of this application;

[0037] Figure 9 yes Figure 8 A magnified view of a portion of the enhanced POI display;

[0038] Figure 10 This is a flowchart illustrating another POI enhanced display method provided in an embodiment of this application;

[0039] Figure 11 This is a schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of this application. Detailed Implementation

[0040] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be a limitation of this application.

[0041] Augmented reality head-up display (AR HUD) is a technology that projects vehicle information, navigation prompts, and warnings from driver assistance systems directly into the driver's line of sight in the form of graphics and / or text. In other words, it overlays digital images onto the real world as seen by the driver, making the projected information blend seamlessly with the real driving environment.

[0042] Figure 1 This is a schematic diagram of the imaging principle of an AR HUD provided in an embodiment of this application.

[0043] The design principle of AR HUD is to utilize an off-axis three-mirror optical system to achieve the effect of overlaying digital images in the real world. For example... Figure 1 As shown, the image generation unit (PGU), also known as a graphics display, uses digital light processing (DLP) or other display technologies to create high-resolution images. These images are then guided and magnified by a series of mirrors before being projected onto the vehicle's windshield. For the driver, the image projected onto the windshield can be viewed... Figure 1 A virtual image is formed on the virtual image plane, and this virtual image is aligned with the actual objects in the driving environment and the driver's eye. This allows the driver to perceive the aforementioned image (i.e., the various information mentioned earlier) as if it were floating above the real road conditions ahead. This design achieves a seamless integration of the HUD's display content and the actual driving environment, which is the technical effect of AR HUD.

[0044] This technology, which projects images directly into the driver's line of sight, is the key to the widespread application of AR HUDs in in-vehicle navigation. With AR HUDs, navigation information and various driving data are no longer statically displayed on the dashboard, but dynamically integrated into the actual field of vision, providing users with more intuitive navigation prompts. For example, navigation arrows can be aligned with the actual road, providing more precise lane-level navigation, allowing drivers to better understand when to turn or change lanes. Furthermore, AR HUDs can display other relevant information, such as traffic signs, pedestrian detection, and obstacle warnings, which enhance the driver's perception of the surrounding environment, improving driving safety and convenience. In short, AR HUDs, through technologies such as... Figure 1 The optical design and augmented reality algorithms shown provide users with a new navigation experience. They not only improve navigation accuracy but also offer drivers an intuitive and easy-to-understand interface to enhance their perception of the surrounding environment, enabling them to react more quickly. At the same time, AR HUD technology reduces the risk of driver distraction from the instrument panel or central control screen, thereby improving driving safety and convenience.

[0045] However, due to the limited accuracy of points of interest (POIs) and deviations in map drawing processes in traditional navigation systems, the location of the destination is often not accurate enough. In other words, the destination located by the navigation system often falls near the actual destination. As a result, the AR HUD's labeling of the navigation destination is also inaccurate when approaching it. That is, because the AR HUD does not actually match the displayed content of the HUD with the actual destination, users still need to make further independent judgments to find the destination entrance when they are close to it.

[0046] Figure 2 This is a schematic diagram of the POI range when a large building is the destination, provided in an embodiment of this application, which can be used to illustrate how the above-mentioned technical problems occur.

[0047] For example, the destination of this navigation is a large building shopping mall A, and the scope of shopping mall A is as follows: Figure 2The area shown is filled with a diagonal line. Understandably, assuming mall A has only one entrance, denoted as entrance 1, parking near entrance 1 is the most convenient way for a user to enter mall A. Therefore, for a user, the ideal navigation result to mall A would be to navigate to the vicinity of entrance 1. However, since destinations are usually identified as Points of Interest (POIs) in navigation systems, for a large area or large building (such as mall A), this POI can be a representative point within mall A. For example, if... Figure 2 As shown, the current POI coordinates of store A in the navigation system are located at the southwest corner of store A's actual location, and the current driving direction is from... Figure 2 If the vehicle is traveling from the southwest corner to the southeast corner of shopping mall A, then when the vehicle reaches the southwest corner of mall A, since the vehicle is already near the POI coordinates of mall A in the navigation system, the navigation system will determine that it has reached its destination and end the navigation. However, for the user, since the navigation ends at the southwest corner of mall A, which is still a distance from entrance 1 (e.g., 100m as mentioned above), the user may still not know how to enter mall A, thus the navigation function is not truly realized.

[0048] In other instances, the POI coordinates of the aforementioned shopping mall A in the navigation system may not even fall within the actual location range of shopping mall A. For example, the POI coordinates of shopping mall A in the navigation system may be offset to the location range of building B next to shopping mall A. In this case, when the AR HUD projects, it will project the logo of shopping mall A onto building B, which will mislead the driver.

[0049] Therefore, this application provides a method for enhancing POI display, applied to an electronic device 100 with AR and navigation functions, such as an in-vehicle infotainment system equipped with an AR HUD. The aforementioned in-vehicle infotainment system is also called an in-vehicle terminal or in-vehicle device, i.e., an electronic computer device integrated inside a vehicle. The following description of the POI enhancement display method will use the aforementioned electronic device 100 as an example of an in-vehicle infotainment system.

[0050] In the aforementioned POI enhanced display method, the vehicle's infotainment system can run a navigation application and activate a navigation route within that application, also referred to as the first route. The system can activate its camera to capture an image of the area in front of the vehicle's windshield. Subsequently, based on this image, the system can determine that a target object appears in the driver's field of view and is located on the navigation route. The system then projects an image of the target object onto the windshield. The driver's field of view refers to the viewing angle visible to the driver from the driver's seat. The image of the target object projected onto the windshield is also called the first virtual image. In the driver's field of view, this image is superimposed on the target object. The display position of the image on the windshield is determined based on first three-dimensional coordinates, which can be measured and determined by the system using a first sensor. This first sensor includes one or more of the following: radar, a depth sensor, and a laser sensor.

[0051] Taking the aforementioned target scenery as the entrance to the destination in the navigation route as an example, such as Figure 3 As shown, through the aforementioned POI enhancement display method, the vehicle-mounted system can associate the POI of the destination with the area of ​​interest (AOI) and obtain POI data for one or more entrances under the destination AOI. Then, the vehicle-mounted system can combine navigation information with the vehicle's position and attitude information to determine or select the target entrance from the aforementioned one or more entrances. During this process, the vehicle-mounted system can obtain the accurate three-dimensional coordinates (i.e., the aforementioned first three-dimensional coordinates) of the target entrance based on sensors such as radar and depth cameras (i.e., the aforementioned first sensor). Based on the accurate three-dimensional coordinates of the target entrance, the system can precisely locate and enhance the display of the target entrance on the AR HUD, thereby optimizing the user's navigation experience. In the above description, the destination of the navigation route can also be referred to as the first building, and the aforementioned target entrance can also be referred to as the first entrance; that is, the aforementioned target scenery can include the first entrance of the first building.

[0052] For example, such as Figure 4 As shown, based on the POI enhancement display method provided in this application embodiment, if the user's navigation destination is Mall A, in addition to Figure 2 As shown in Entrance 1, Mall A also has Entrance 2 and Entrance 3. Therefore, according to... Figure 4 Given the vehicle's location, the in-vehicle system can use entrance 1 as the entrance to shopping mall A, that is, entrance 1 as the target entrance and the destination of this navigation. Subsequently, if the system determines that entrance 1 is within the driver's field of view based on the image captured by the camera, it can project the sign of entrance 1 onto a virtual image plane in front of the windshield, for example... Figure 5The diagram shows the selection and labeling of entrance 1 of shopping mall A. Specifically, the four vertices of the selected area drawn by the vehicle's AR HUD correspond to the four vertices of the main entrance 1, thus clearly identifying the target area in the driver's field of vision. (Not limited to...) Figure 5 The display is enhanced by selecting and marking. The vehicle system can also enhance the display of entrance 1 in the AR HUD through other display methods such as positioning icons, jumping icons, the name of the target scene (entrance 1 in this case), and the outline of the target scene (entrance 1 in this case), so that entrance 1 can be distinguished from other buildings in the user's field of vision in the AR HUD.

[0053] In the above process, enhancing the display of POI 1 (Entrance 1) means that the sign projected by the vehicle-mounted system is accurately superimposed on Entrance 1, thereby achieving the effect of the sign and the real scene being aligned. It can be understood that to achieve the effect of the sign accurately superimposed on the target scene, the sign on the virtual image plane, the target scene, and the driver's viewpoint must be on the same straight line. The virtual image plane is perpendicular to the ground, the sign projected on the virtual image plane is the first virtual image, and the driver's viewpoint refers to the preset position of the driver's eyes during vehicle movement. Furthermore, when executing the POI enhancement display method provided in this application embodiment, the sign is updated in real time. That is, as the vehicle moves, the closer the vehicle-mounted system is to the target scene (such as Entrance 1), the larger the sign superimposed on the target scene can be; conversely, the farther the vehicle-mounted system is from the target scene, the smaller the sign superimposed on the target scene can be.

[0054] The above display effect can be achieved through, for example Figure 6 The intelligent cockpit in-vehicle infotainment software architecture shown is implemented.

[0055] like Figure 6 As shown, the above-mentioned intelligent cockpit vehicle software architecture includes, but is not limited to, a vehicle control status detection module, a physics engine module, and a POI enhancement module.

[0056] The aforementioned vehicle control status detection module can utilize sensor data from the cockpit perception service to acquire vehicle control status data, thereby providing necessary input parameters for the physics engine module. This vehicle control status data reflects the vehicle's current state, including but not limited to vehicle driving data, vehicle control data, vehicle location information, and environmental information surrounding the vehicle. Specifically, the vehicle driving data includes, but is not limited to, driving speed and distance traveled; the vehicle control data includes, but is not limited to, steering wheel angle and four-wheel angles; the vehicle location information includes, but is not limited to, the vehicle's current latitude and longitude coordinates; and the environmental information surrounding the vehicle includes, but is not limited to, road conditions, surrounding buildings, obstacles, and real-time traffic conditions (such as road congestion).

[0057] The aforementioned cabin perception service refers to the function of detecting the vehicle's interior environment, passenger status, and dynamic vehicle behavior through the integration of multiple sensors. For example... Figure 6 As shown, these sensors include, but are not limited to, accelerometers, gyroscopes, sensors for detecting the speed of four wheels (such as wheel speed sensors), sensors for detecting the angle of four wheels (such as tilt sensors), sensors for detecting the steering wheel angle (such as steering wheel angle sensors), global positioning system (GPS) receivers, lidar, cameras, millimeter-wave radar, etc.

[0058] The image data collected by the aforementioned sensors can be output as video signals, forming part of the vehicle control status data. For example, high-definition image data captured by a camera can be output as video signals to provide environmental information about the vehicle's surroundings within the vehicle control status data. Optionally, such as... Figure 6 As shown, the aforementioned video signal can be transmitted using low-voltage differential signaling (LVDS) technology. In addition to image data, data detected by other types of sensors (such as accelerometers, gyroscopes, wheel speed sensors, etc.) can be output as real-time signals via an onboard gateway (such as Ethernet), thus forming part of the vehicle control status data. The aforementioned onboard gateway refers to a device that aggregates and transmits various sensor data from the vehicle's internal network to the vehicle's central processing unit or external systems based on Ethernet or other communication protocols. Subsequently, the aforementioned vehicle control status data is input from the vehicle control status detection module to the physical engine module.

[0059] The aforementioned physics engine module can utilize vehicle control status data obtained from the vehicle control status detection module, combined with vehicle models and physical laws, to perform calculations and simulations in order to predict and control vehicle behavior.

[0060] like Figure 6As shown, the physics engine module includes, but is not limited to, the following four sub-modules: a 3D modeling module, a coordinate transformation module, an occlusion detection module, and an image comparison module. The 3D modeling module can be used to perform 3D modeling based on latitude and longitude coordinates, altitude, and other data of multiple points. In this embodiment, the 3D modeling module can be used to construct a 3D spatial model with the vehicle as the origin of the coordinate system within the vehicle's infotainment system. The coordinate transformation module is used to calculate the three-dimensional coordinates of points in the space (e.g., multiple entrance points (POIs) under the destination-associated AOI) in the 3D space; that is, the coordinate transformation module is used to convert the coordinates of points in the space from an absolute coordinate system based on Earth's latitude and longitude to values ​​in a relative coordinate system with the vehicle as the origin. Furthermore, the coordinate transformation module can further convert the coordinates of points in the space from a relative coordinate system with the vehicle as the origin to a relative coordinate system with the driver's viewpoint as the origin. The occlusion detection module can identify whether each entrance to the destination is occluded using images captured by the camera, thereby determining whether the entrance point (POI) enters the driver's field of view. The aforementioned image comparison module is used to compare the image feature points in the image data of the vehicle control status data with the road condition features stored in the navigation system, thereby correcting the coordinates of each entrance POI in the 3D space with the vehicle as the origin, so as to further improve the accuracy of subsequent calculations and the accuracy of the final ARHUD display of POI enhancement.

[0061] The function of the aforementioned POI enhancement module is to convert the POI from a three-dimensional coordinate system to a two-dimensional coordinate system, so that the AR HUD system can draw the target scene's marker on the virtual image plane. This process allows the projected marker to be accurately displayed on the virtual image plane, thereby aligning the marker, the target scene, and the driver's viewpoint on the same straight line, thus achieving an enhanced display effect for the POI.

[0062] Understandably, in this embodiment, the function of the aforementioned physics engine is to construct a spatial model of the vehicle and target scenery (such as the aforementioned entrance POI) using vehicle control state data, and then convert the coordinates of each point in this 3D spatial model into three-dimensional coordinates in a specific coordinate system (i.e., a coordinate system with the driver's viewpoint as the origin) and output them to the subsequent POI enhancement module for drawing the logo on the virtual image plane. In other words, the coordinate transformation step ultimately yields the three-dimensional coordinates of the target scenery (such as the aforementioned entrance POI) in a coordinate system with the driver's viewpoint as the origin (i.e., the aforementioned first three-dimensional coordinates), and the two-dimensional coordinates of the projected logo on the virtual image plane are obtained by transforming these first three-dimensional coordinates. Therefore, only when the three-dimensional coordinates output by the physics engine are accurate can the POI enhancement module accurately draw the logo during subsequent calculations, and only then can the drawn logo truly match the real driving environment. Therefore, the image comparison module in the aforementioned physics engine module is a further verification of the vehicle's pose, thereby ensuring the correct construction of the relative coordinate system with the vehicle as the origin, and thus ensuring the correct coordinates of the destination entrance POI in the aforementioned relative coordinate system. The occlusion judgment module in the physics engine module is to ensure that the current entrance POI is visible within the driver's field of view, that is, to meet the prerequisite for POI enhancement, and to avoid incorrect rendering by the POI enhancement module. It can be seen that the aforementioned occlusion judgment module and image comparison module are both used to further improve the accuracy of the 3D spatial model in the embodiments of this application, thereby improving the degree of fit between the content displayed on the final AR HUD and the actual road conditions.

[0063] Understandable. Figure 6 The illustrated smart cockpit in-vehicle infotainment system software architecture diagram places multiple sensors under the cockpit perception service only to illustrate that the cockpit perception service can be implemented by these sensors, and does not imply that the aforementioned sensors are integrated inside the in-vehicle infotainment system. In other words, optionally, the in-vehicle infotainment system can directly integrate the aforementioned sensors. Alternatively, the in-vehicle infotainment system may not integrate all or some of the aforementioned sensors, but may connect to and control the sensors via wired or wireless means. This application embodiment does not impose special limitations on the specific configuration of the in-vehicle infotainment system and sensors.

[0064] Based on the aforementioned intelligent cockpit software architecture, the vehicle system can utilize, for example... Figure 7 The process shown executes the POI enhanced display method provided in the embodiments of this application.

[0065] S101, The vehicle's infotainment system obtains the coordinates of the destination POI.

[0066] Understandably, in navigation and map services, larger locations or areas, such as the aforementioned shopping mall A, are defined as Areas of Interest (AOIs). To facilitate positioning and navigation, the system selects specific points within the AOI as Points of Interest (POIs) and provides their coordinates. Users navigate to these POI coordinates to approach or enter the AOI, thus achieving navigation functionality. Therefore, when a user begins navigation, the vehicle's infotainment system can obtain the name of the user's destination from the navigation system, and then obtain the coordinates of the corresponding POI. The destination POI obtained here is the representative POI defined for this AOI area for ease of positioning and navigation. Understandably, the specific representative POI within the defined AOI area for the destination is determined by the navigation system. At this time, the POI coordinates obtained by the vehicle's infotainment system from the navigation system can be two-dimensional coordinates, indicating the POI's position on the ground plane. Optionally, the POI coordinates can be its latitude and longitude coordinates.

[0067] S102. The vehicle-mounted system associates the destination POI coordinates with the AOI area and obtains the coordinates of each entrance POI under the AOI area.

[0068] Since the destination POI provided by the navigation system is actually a specific point within the AOI area, the navigation system should internally store the mapping relationship between the AOI area and the coordinates of the destination POI. Therefore, after obtaining the destination POI, the vehicle's system can determine the corresponding AOI area based on the above mapping relationship.

[0069] Understandably, the destination POI coordinates mentioned above are a sub-POI within the AOI range. In a navigation system, there can be one or more sub-POIs within the AOI range. Furthermore, for a large building (such as shopping mall A mentioned above), the navigation system typically stores the POIs of multiple entrances to that AOI area. That is, the one or more sub-POIs within the AOI range usually include the POIs of the entrances to that AOI area. Therefore, after the vehicle's infotainment system associates the destination POI coordinates with the AOI area, it can also obtain the two-dimensional coordinates of all entrance POIs within that AOI area.

[0070] S103, the vehicle-mounted system performs 3D modeling and coordinate transformation based on the coordinates of each entrance POI.

[0071] The vehicle-mounted infotainment system's 3D modeling based on the coordinates of each entrance POI refers to constructing a three-dimensional spatial model using the coordinates and other data of each POI stored in the navigation system. This model reflects the spatial relationship between the vehicle and the various POIs. Understandably, the latitude and longitude of the POIs alone are insufficient to construct this 3D spatial model. Therefore, the vehicle-mounted infotainment system can also obtain the altitude data of each entrance POI from the navigation system to determine its position in 3D space. In addition to the entrance POIs, the system also needs to acquire the vehicle's absolute coordinates (latitude and longitude) and altitude data for 3D modeling. Optionally, the system can use GPS technology to obtain the vehicle's absolute coordinates and altitude data. Subsequently, the system can perform 3D modeling based on the coordinates of each entrance POI and the vehicle's own absolute coordinates.

[0072] Afterwards, the vehicle-mounted system can call the aforementioned coordinate transformation module to perform coordinate transformation operations. In these operations, the vehicle-mounted system first converts the 3D modeling results into a three-dimensional coordinate system with the vehicle as the origin. This three-dimensional coordinate system is also called the vehicle coordinate system. Optionally, the vehicle-mounted system can define the vehicle coordinate system using a right-handed coordinate system. For example, the system can establish the vehicle coordinate system with the vehicle center as the origin, the front of the vehicle as the positive half-axis of the X-axis, the left side of the vehicle (e.g., perpendicular to the door) as the positive half-axis of the Y-axis, and the direction pointing upwards as the positive half-axis of the Z-axis. Not limited to the above method, the vehicle-mounted system can also establish the vehicle coordinate system using a left-handed coordinate system. This application does not impose any special limitations on the method of establishing the vehicle coordinate system.

[0073] Understandably, a vehicle's absolute coordinates are only discrete points. After obtaining the vehicle's absolute coordinates, the vehicle's infotainment system can only calculate the distance between the vehicle and the aforementioned AOI area and one or more entrance POIs. This is insufficient to determine how the vehicle's coordinate system should be established based on the vehicle's absolute coordinates, and consequently, it cannot determine the coordinates of the AOI and each entrance POI within the vehicle's coordinate system. For example, for the same entrance POI, its coordinates in the vehicle's coordinate system differ when the vehicle is facing south and when it is facing east. Therefore, in addition to the vehicle's absolute coordinates, such as... Figure 7 As shown, the vehicle's infotainment system can use vehicle control sensors to obtain the vehicle's pose to further determine the vehicle's position and attitude, thereby determining the orientation of each axis in the vehicle coordinate system and improving the accuracy of subsequent 3D modeling. The aforementioned vehicle control sensors used for detecting the vehicle's pose include, but are not limited to, accelerometers, gyroscopes, global navigation satellite systems, lidar, and cameras.

[0074] Inside the vehicle's infotainment system, converting the absolute coordinates of a point to coordinates in a relative coordinate system with a point in space as the origin can be achieved using a transformation matrix. For example, the coordinates of point (1, 2, 1) in a new coordinate system with (1, 1, 1) as the origin are (0, 1, 0). This transformation can be achieved using the following transformation matrix C:

[0075]

[0076] Specifically, if the axes of the absolute coordinate system and the vehicle coordinate system have the same direction, the vehicle-mounted system can determine the translation vector as (-1, -1, -1) based on the transformation of the origin coordinates from (0, 0, 0) to (1, 1, 1), and then use this translation vector to construct the transformation matrix C. Subsequently, the vehicle-mounted system can multiply the homogeneous coordinates (i.e., adding 1 to the end of the 3D coordinates) of the point requiring coordinate transformation (such as the entrance POI) with the transformation matrix. For example, the homogeneous coordinates of point (1, 2, 1) are (1, 2, 1, 1). Multiplying this point with the transformation matrix yields its coordinates in the new coordinate system. Understandably, the result of the matrix multiplication is (0, 1, 0, 1), where the first three elements (0, 1, 0) represent the coordinates of point (1, 2, 1) in the new coordinate system (i.e., the vehicle coordinate system), and the last element, 1, represents the "depth" of the point in the new coordinate system, which can be ignored in the coordinate transformation step.

[0077] Understandably, in the example above, the transformation from the absolute coordinate system to the vehicle coordinate system only involves translation; that is, the vehicle's current driving direction (i.e., the X-axis of the vehicle coordinate system) matches the X-axis in the absolute coordinate system. However, if the vehicle's driving direction differs from the X-axis in the absolute coordinate system, the coordinate system transformation should also include rotation. In other words, the vehicle's infotainment system should consider the rotation angles of each axis when constructing the transformation matrix C. The subsequent coordinate transformation process is the same as for a coordinate transformation involving only translation, and will not be elaborated here.

[0078] While the vehicle coordinate system facilitates handling the space around the vehicle, the AR HUD system needs to accurately present information within the driver's field of view. Therefore, to achieve this, the vehicle's infotainment system needs to further transform the coordinate data from the vehicle coordinate system to the human eye coordinate system, with the driver's viewpoint as the origin. This is understandable because only the human eye coordinate system can accurately simulate the perspective effect seen from the human eye's position, more accurately simulate the depth of objects, and ensure that the information displayed on the ARHUD is aligned with the driver. Therefore, only by converting the vehicle coordinate system to the human eye coordinate system can the ARHUD system ensure that the image projected onto the windshield seamlessly blends with the actual driving environment, achieving a straight line between the data projection point (i.e., the point where the image is projected onto the windshield), the driver's eye, the virtual image, and the real scene, thus providing the driver with an intuitive, accurate, and safe enhanced display view.

[0079] Optionally, the vehicle's infotainment system can set the directions of each axis of the human eye coordinate system to be consistent with those of each axis of the vehicle coordinate system. In this case, the transformation from the vehicle coordinate system to the human eye coordinate system only involves translation. Subsequently, the vehicle's infotainment system can determine the transformation matrix C' from the vehicle coordinate system to the human eye coordinate system based on the position of the driver's eyes relative to the origin of the vehicle coordinate system, thereby obtaining the three-dimensional coordinates of each entrance POI in the human eye coordinate system. The position of the driver's eyes relative to the origin of the vehicle coordinate system can be determined in advance based on data such as seat position, vehicle size, and driver's height; that is, the translation vector for the transformation from the vehicle coordinate system to the human eye coordinate system can be determined in advance. The specific transformation process is similar to that for converting from an absolute coordinate system to a vehicle coordinate system, and will not be elaborated here.

[0080] In some embodiments, before the vehicle system performs 3D modeling, the vehicle system can also capture images of the front of the vehicle using a camera. The vehicle system can use a large model A to perform image recognition on the captured images. By comparing the features of the destination buildings in the navigation system with the features of the buildings in the captured images, it can determine whether the destination buildings have appeared in the user's field of vision, thereby further verifying whether the vehicle has driven to the vicinity of the destination and determining the current driving direction of the vehicle. On the one hand, this can reduce the computational load and power consumption of the vehicle system, and on the other hand, it can verify the vehicle's position and pose detected by the vehicle control sensors to improve the accuracy of 3D modeling. Figure 7 This step is highlighted with a dashed line to indicate that it is an optional processing step within step S103. The image recognition steps described above can be executed by the image comparison module within the aforementioned physical engine module.

[0081] S104, The vehicle's infotainment system performs an obstruction detection.

[0082] Understandably, the occlusion determination here does not refer to whether the destination building is visible within the driver's field of view, but rather whether the destination's entrance POI is visible within the driver's field of view; that is, whether the image captured by the camera contains the aforementioned destination's entrance POI. Understandably, these two situations are not equivalent. For example, in... Figure 3 In the scenario shown, if shopping mall A does not have entrance 1, then according to the vehicle's direction of travel, although the driver can see the building of shopping mall A, entrances 2 and 3 are not visible in the driver's field of vision. In other words, the destination's entrance POI is not visible within the driver's field of vision. The key to occlusion judgment here is whether the destination's entrance POI is visible, i.e., whether it is within the driver's field of vision. This is because, as explained above, the target scene is the destination's entrance POI. In other words, the occlusion judgment performed by the vehicle's infotainment system in step S104 is actually the system determining whether the target scene is obstructed.

[0083] The vehicle's infotainment system can use a large model (e.g., large model A) to implement the aforementioned occlusion determination. For example, the input to large model A can be an image captured by a camera and an image of the entrance point of interest (POI) obtained by the vehicle's infotainment system from the navigation system; the output can be a determination of whether the entrance POI is occluded. The details of the occlusion determination process described above will not be elaborated here.

[0084] Based on the aforementioned occlusion detection, the vehicle-mounted system can verify the accuracy of the entrance POI coordinates calculated in the previous steps. Understandably, for a large building, for example... Figure 3 The shopping mall A shown may have multiple entrances. The navigation system can prioritize the entrance visible within the driver's field of view as the destination, thus quickly entering the building. If the entrance POI of the destination remains obscured in the first instance, meaning that the image captured by the camera does not contain any entrance POI of the destination, it may indicate an error in the previous calculation process. Optionally, the vehicle system can repeat step S103 above.

[0085] S105, The vehicle's infotainment system determines the distance.

[0086] Understandably, for some destinations with complex road layouts, such as scenarios where multiple similar entrances exist within a short distance, or where buildings and their entrances in certain neighborhoods may have similar designs due to urban development requirements, step S104 might incorrectly identify all of these similar entrances as the destination's POI. Therefore, the vehicle-mounted system can perform a distance assessment again before projection rendering to better distinguish the destination's POI from these similar entrances, thereby enabling effective cross-validation and improving the accuracy of POI display during projection rendering.

[0087] Specifically, the vehicle's infotainment system can use sensors such as radar and depth cameras to obtain the distance between the entrance POI and the vehicle. Understandably, the entrance POI here should be one visible within the driver's field of vision. The distances obtained using radar and depth cameras are the measured distances between the vehicle and each visible entrance POI. The infotainment system can then compare these measured distances with the distances estimated from the destination on the navigation system, thereby distinguishing the destination's entrance POI from multiple similar entrances.

[0088] For example, there are two similar entrances, entrance a and entrance b, where entrance b is the POI for the navigation destination. After executing step S104, the vehicle's infotainment system might identify that neither entrance a nor entrance b is obstructed. This means that the system incorrectly determines in this step that there are two entrances within the user's field of vision that lead to a large building. If the vehicle passes entrance a first based on its direction of travel, and only step S104 is executed, the system might treat entrance a as the destination's POI in subsequent intelligent planning and enhance its display, leading to navigation errors. Therefore, step S105 is executed after step S104. At this point, the system can use sensors such as radar and depth cameras to measure distances and determine that entrance a is near the vehicle. If the navigation system estimates the distance between the vehicle and the destination to be 100m, by comparing the measured distance with the estimated distance, the system can rule out entrance a as the destination's POI.

[0089] Since the vehicle system uses radar, depth cameras, and other primary sensors to measure multiple entrance POIs in the above distance determination, the data obtained from the above measurements can be used to further improve the accuracy of the three-dimensional coordinates of the entrance POIs, in addition to cross-validation.

[0090] Understandably, the entrance is not a discrete point, but a spatial region occupying a certain area. Therefore, in the aforementioned distance measurement process, optionally, sensors such as radar or depth cameras can measure the actual distance between the center point of the entrance and the vehicle. Alternatively, sensors such as radar or depth cameras can also perform more precise distance measurement, including measuring the width, height, and position of the entrance relative to the vehicle. For example, measuring the relative positions and actual distances between the four vertices of the entrance gate and the vehicle. This data helps the vehicle's infotainment system more accurately construct a spatial model of the current entrance in the human eye coordinate system, thereby enabling more precise projection rendering in subsequent projection rendering.

[0091] S106, The vehicle's infotainment system performs intelligent route planning based on the navigation system.

[0092] In steps S104 and S105 above, there can be multiple entrances suitable as the final destination of the navigation system. Therefore, in step S106, the vehicle system determines that multiple entrances to the destination appear in the driver's field of view based on the images captured by the camera. Subsequently, the vehicle system can select the best entrance POI as the entrance to the final destination of the navigation system. This best entrance POI is also called the first entrance. It can be understood that the entrance suitable as the final destination of the navigation system refers to the entrance that can be captured by the camera in step S104 (i.e., visible in the driver's field of view) and whose actual distance can be measured in step S105.

[0093] Optionally, the vehicle system can select the entrance with the shortest driving distance from the above multiple entrances as the entrance to the final destination of the navigation system.

[0094] Understandably, for these multiple entrances, the entrance with the shortest straight-line distance is not necessarily the entrance with the shortest driving distance. For example, in a large building area, two entrances (entrance c and entrance d) are both located on the opposite side of the road in the direction of vehicle travel. Although entrance c is closer to the vehicle in a straight line and entrance d is farther away, vehicles actually need to drive to the next intersection to make a U-turn to reach these two entrances. Therefore, in terms of driving distance, entrance d is the shorter one.

[0095] Therefore, the vehicle's infotainment system can call the navigation system to determine the driving distance from the vehicle to the aforementioned visible entrances, and then determine the entrance with the shortest driving distance as the best POI for navigation to the destination.

[0096] Understandably, after determining the optimal entry point POI, the vehicle's navigation system can replace the destination POI in the navigation system with the aforementioned optimal entry point POI.

[0097] S107, The vehicle-mounted system performs projection drawing.

[0098] The vehicle's infotainment system can obtain the 3D coordinates of the Point of Interest (POI) at the optimal entry point in the human eye coordinate system, convert these 3D coordinates into 2D coordinates on the virtual image plane, and then send the calculated 2D coordinates to the AR HUD system for projection rendering. This projection rendering process is also the process of using the AR HUD to enhance the display of the POI. This is understandable. Figure 5Regarding the enhanced POI display effect shown, for the vehicle's infotainment system, the most important thing to determine when performing the above projection drawing is the two-dimensional coordinates of the four vertices of the selected POI on the virtual image plane. The subsequent steps of the vehicle's infotainment system performing projection drawing will be explained using a single point as an example. The determination of the two-dimensional coordinates of multiple points (such as the four vertices that need to be determined by selecting the POI) on the virtual image plane and the projection drawing follow similar steps, and will not be repeated here.

[0099] Optionally, the vehicle's infotainment system can use a perspective projection matrix to convert the aforementioned three-dimensional coordinates into two-dimensional coordinates.

[0100] Specifically, the vehicle-mounted system can obtain the homogeneous coordinates of the optimal entrance POI based on the perspective projection matrix and the three-dimensional coordinates of the optimal entrance POI. Then, it can normalize the homogeneous coordinates of the optimal entrance POI and map the normalized coordinates onto the virtual image plane to finally obtain the coordinates of the optimal entrance POI in the virtual image plane coordinate system.

[0101] For example, assuming that the three-dimensional coordinates of the upper left vertex (also called vertex P) of the optimal entrance POI gate are (x, y, z) based on the above human eye coordinate system, the vehicle system can calculate the homogeneous coordinates of vertex P on the virtual image plane using the following formula:

[0102]

[0103] In the above formula (1) Used to represent the three-dimensional coordinates of vertex P in the human eye coordinate system. The homogeneous coordinates of vertex P are used to represent the perspective projection matrix constructed by the vehicle-mounted system. When the coordinates in formula (1) are already three-dimensional coordinates in the human eye coordinate system, as shown in formula (2), the above perspective projection matrix is ​​the intrinsic parameter matrix, used to reflect the inherent optical characteristics in the AR HUD imaging process. It can be understood that the above... Although there are three parameters, the last parameter w is a homogeneous coordinate, used to remove it in the final normalization step to obtain the standard two-dimensional coordinates of vertex P. Therefore, the above formula (1) is used to represent the two-dimensional coordinates of a three-dimensional point (such as vertex P) in space after transformation by the perspective projection matrix. The above formula (2) is the specific expansion formula of the above intrinsic parameter matrix, where f x f is the focal length of the camera on the x-axis of the human eye coordinate system. y c is the focal length of the camera on the y-axis of the human eye coordinate system; x c is the coordinate of the principal point of the image on the x-axis of the image coordinate system. u This refers to the coordinates of the principal point of the image on the y-axis of the image coordinate system. This principal point can be the center point during imaging. Understandably, throughout the imaging process, the intrinsic parameter matrix is ​​adjusted by setting a suitable focal length (i.e., f).x and f y The size of the image is adjusted by setting a suitable principal point so that the image center, data projection point, and real scene are aligned on a straight line, thereby ensuring that the virtual image is projected to the correct position and fits the real scene. The intrinsic parameter matrix in the above formula can be obtained in advance through a calibration program after the vehicle design is completed. That is, during real-time calculation, the above intrinsic parameter matrix K is fixed.

[0104] Subsequently, the vehicle-mounted system can normalize the homogeneous coordinates of vertex P obtained using formulas (1) and (2) above. Specifically, the normalization is achieved by dividing u and v by w, i.e., using perspective division to obtain (u', v'). For example, Then (u', v') = (0.4, -0.6).

[0105] The physical meaning of the normalized coordinates mentioned above is the position of a point within the normalized device coordinates (NDC). The NDC is a virtual coordinate system that provides a unified method for handling coordinates from different viewpoints and projection transformations.

[0106] Understandably, for the user, the virtual image plane in front of the car can be considered the imaging screen. However, this screen is not infinitely large; its size defines the projection range and position of the AR HUD. Therefore, the actual size of the aforementioned virtual image plane is determined by the AR HUD's screen resolution and field of view (FOV). Based on the normalization process described above, the vehicle's infotainment system ensures that the processed coordinate system perfectly adapts to the resolution and layout requirements of the AR HUD display device. This guarantees that any point within the entire projection range of the AR HUD can be accurately projected and displayed. In other words, regardless of which point within the dashed plane is selected for POI enhancement display, precise image projection can be achieved.

[0107] Optionally, the vehicle-mounted system can translate and scale the normalized coordinates (u', v') to coordinates (m, n) in the virtual image plane, thereby further aligning the image projected by the AR HUD with the real scene.

[0108] For example, if the HUD screen resolution is 1280 pixels × 720 pixels, then for the virtual image plane "screen," the vehicle's infotainment system can define the screen center as the origin (0,0), and the upper right corner as (640, 360). The specific conversion process can be based on the following formula:

[0109] m = u ′ ×0.5W(3)

[0110] n = v ′ ×0.5H(4)

[0111] In formula (3), W represents the width of the HUD screen resolution, and in formula (4), H represents the height of the screen resolution. It is understandable that since the center of the virtual image plane defined by the vehicle-mounted system is the origin, consistent with the origin of the coordinate system in the NDC, in formulas (3) and (4), the vehicle-mounted system can obtain the two-dimensional coordinates of the image on the virtual image screen simply by magnifying the normalized coordinates. These two-dimensional coordinates can then be given to the AR HUD system for projection rendering. However, if the origin of the coordinate system defining the virtual image plane by the vehicle-mounted system is inconsistent with the origin of the coordinate system in the NDC, then in this step, the vehicle-mounted system not only needs to magnify the normalized coordinates but also needs to translate them to make the origins of the two coordinate systems consistent.

[0112] Not limited to the perspective projection matrix mentioned above, the embodiments of this application do not impose any special restrictions on the specific method by which the vehicle-mounted system determines the two-dimensional coordinates of a point in space (such as the vertex P of the optimal entrance POI) on the virtual image plane based on the three-dimensional coordinates of the point.

[0113] Understandably, the curvature of the windshield causes distortion in the projected image. Therefore, in the above calculation process, the vehicle-mounted system can also correct the projected image using a reverse distortion processing method. This is not limited to the aforementioned reverse distortion processing method; other methods capable of correcting coordinate data can also be applied. This application does not impose any special restrictions on the specific method used to correct the data.

[0114] In the above method, from step S103 to step S107, the vehicle system can combine spatial computing and various sensing technologies to construct a more accurate spatial model of the POI in the human eye coordinate system, and then use AR HUD to make more accurate annotations, thereby demonstrating the enhanced display effect of the POI on AR HUD.

[0115] Understandably, in the POI enhancement display method provided in this application embodiment, the marker drawing, i.e., the POI enhancement display, is updated in real time. That is, in step S104, based on the image captured by the camera at time t1, it is determined that the target scene appears in the driver's field of view. Then, in step S106, the vehicle system can project the marker at time t2 onto the front of the vehicle's windshield. The marker at time t2 is determined by the vehicle system based on the accurate three-dimensional coordinates determined by the first sensor at time t2. Here, time t2 is the same as time t1, or the time difference between time t2 and time t1 is less than a first time threshold. For example, the first time threshold can be 50 milliseconds.

[0116] In some embodiments, step S103 is optional. That is, the POI coordinates of each entrance to the destination obtained by the vehicle-mounted system in step S102 are only used to confirm multiple entrances under the destination building of the navigation route, thereby performing an occlusion judgment on these multiple entrances in step S104. Subsequently, the vehicle-mounted system can determine the vehicle's pose based on the vehicle control sensors in step S105, and directly determine the accurate three-dimensional coordinates of the entrance POI based on the vehicle's pose and the distance between the entrance POI and the vehicle-mounted system measured by the first sensor (such as radar, depth sensor, etc.). It is understandable that in the method flow including step S103, the vehicle-mounted system actually determines the accurate three-dimensional coordinates of the entrance POI based on the vehicle's pose and the distance between the entrance POI and the vehicle-mounted system measured by the first sensor. However, since the absolute coordinates of the entrance POI itself are also introduced as a reference in this process, the global consistency and accuracy of the data obtained by different sensors can be effectively ensured, and the accuracy of the final calculated three-dimensional coordinate result can be effectively improved.

[0117] In some embodiments, before executing step S104, the vehicle-mounted system can first calculate the distance d between the vehicle itself and the destination POI (i.e., the target scene). If the calculated d is less than a first distance threshold, the vehicle-mounted system will be triggered to execute step S104. If the calculated d is greater than or equal to the first distance threshold, the vehicle-mounted system may not trigger step S104. If the execution step includes step S103, optionally, the calculation of the distance d can also be performed before executing step S103. Similarly, step S103 will only be triggered if the calculated d is less than the first distance threshold. It is understood that the POI enhancement display method provided in this application embodiment is to solve the problem of insufficient destination POI calibration accuracy. Therefore, during navigation, when the distance between the vehicle and the destination POI is greater than or equal to the first distance threshold, the vehicle-mounted system does not need to execute subsequent steps of this method in real time, thereby reducing the computational load and power consumption of the vehicle-mounted system.

[0118] Understandably, in the above method, since the premise for projecting the marker or enhancing the display is that the target scene appears in the driver's field of vision, if it is determined from the image captured by the camera that the target scene has disappeared from the driver's field of vision, the vehicle system can cancel the projection of the marker of the target scene (i.e., the first virtual image) onto the front of the vehicle's windshield.

[0119] Understandably, the above solution uses the destination as the entrance to the navigation route as an example. In reality, it's not limited to the entrance; the target scenery can also include roads within the destination and the destination itself (i.e., the AOI area). These target scenery can be pre-set in the vehicle's infotainment system. For example, the system can set target scenery in the order of destination and entrance. If the system fails to capture the destination via camera, it can use the destination as the target scenery. If the system captures an image of the destination, it can change the target scenery to the entrance. If the system detects that the vehicle has entered the destination, it can change the target scenery to a road within the destination. Correspondingly, the above method can be adaptively adjusted when the target scenery is modified.

[0120] In other embodiments, in addition to enhancing the display of the entrance POI by means of box selection when finally arriving near the destination, the vehicle system can also further enhance the display of the POI based on various data directly provided by the navigation system within a first distance threshold from the destination.

[0121] Figure 8 This is a schematic diagram of the interface display of an AR HUD provided in an embodiment of this application. Figure 9 yes Figure 8 A magnified view of a portion of the enhanced POI display.

[0122] In one possible implementation, the vehicle's infotainment system can also differentiate and display the associated AOI area on the AR HUD. Optionally, such as... Figure 9 As shown, the vehicle-mounted system can use left and right parentheses to select the associated AOI area. Optionally, the vehicle-mounted system can also display outlines of the buildings within the AOI area, meaning the displayed identifiers are the outlines of the buildings within the AOI area.

[0123] In one possible implementation, the vehicle-mounted system may not be limited to marking entrance POIs visible within the driver's field of vision; it may also mark other POIs below the destination AOI that are obscured within the field of vision. These other POIs below the destination AOI include entrance POIs not visible within the driver's field of vision, and may also include some POIs within the destination AOI. If the destination (such as building A mentioned above) has multiple entrances, such as... Figure 9 As shown, when a vehicle is traveling on XX Road, only entrance 1 is visible, while entrances 2 and 3 are located behind buildings. At this time, the vehicle's infotainment system can mark the currently invisible entrance points (POIs) such as entrances 2 and 3 on the AR HUD, thus displaying complete entrance information for the destination to the user.

[0124] Optionally, the vehicle's infotainment system can use different location icons to distinguish whether an entrance is visible. For example... Figure 9 As shown, the vehicle-mounted system can use black location icons to represent currently visible entrances and white location icons to represent currently invisible entrances. Not limited to using different location icons to distinguish between visible and invisible entrances, the vehicle-mounted system can also use animations to differentiate between visible and obscured entrances. For example, the vehicle-mounted system can use a jumping location icon on the AR HUD to represent a visible entrance and a stationary location icon to represent an obscured entrance. Not limited to the above methods, other ways to distinguish between visible and obscured entrances can be used on the vehicle-mounted system's AR HUD, and this application embodiment does not impose any special limitations on this.

[0125] Optionally, the vehicle-mounted system can also display the distance between the current vehicle and each POI included in the destination AOI. For example... Figure 9 As shown, the "50m" display below entrance 1 of building A indicates that the current driving distance between the vehicle and entrance 1 is 50m. Optionally, this distance can be obtained from a navigation system or calculated based on three-dimensional coordinates.

[0126] In one possible implementation, the vehicle-mounted HUD can also display the internal roads of Building A, that is, the roads between multiple Points of Interest (POIs) beneath Building A. Understandably, these internal roads are provided by the navigation system; only when the navigation system stores these internal roads can AR HUD technology be used for such operations. Figure 9 The internal roads shown are displayed. If the navigation system does not store the aforementioned internal roads, they will not be displayed.

[0127] Understandable. Figure 8 and Figure 9 The above-described possible implementations demonstrate multiple effects on POI enhancement, but this does not mean that these effects need to be implemented simultaneously on the AR HUD. That is, each of the above-described effects can be implemented independently, or multiple effects can be implemented simultaneously, depending on the algorithm settings within the navigation system. This application does not impose any special limitation on whether the POI enhancement effect is one or multiple.

[0128] In the above embodiments, further POI enhancement display effects can be based on, for example... Figure 10 The flowchart shown is implemented. Compared to Figure 7 The flowchart shown is for the method of enhancing the display of POI. Figure 10 The flowchart shown adds steps S108 to S110 after step S103.

[0129] S108, The vehicle-mounted system calibrates the outer frame of the destination AOI.

[0130] The aforementioned vehicle-mounted system's calibration of the destination AOI outline refers to the system determining the precise position and extent of the destination AOI within the AR HUD projection, ensuring that the enhanced image can be accurately overlaid on the destination AOI. For example, for... Figure 9 The enhanced display achieved by selecting an AOI area, as shown in the diagram, hinges on the vehicle-mounted system calibrating the destination AOI outline. The key aspect is determining the two-dimensional coordinates of the four vertices of the destination AOI outline on the aforementioned virtual image plane. Understandably, these two-dimensional coordinates of the destination AOI outline on the virtual image plane are obtained through a transformation of the destination AOI's three-dimensional coordinates. The three-dimensional coordinates of the destination AOI refer to a three-dimensional spatial region formed by the multiple points constituting the AOI.

[0131] Therefore, for the vehicle's navigation system to calibrate the destination AOI outline, it must first obtain the AOI's regional coordinates and the height information of buildings within the AOI area (such as the altitude data of the highest point of each building). The AOI's regional coordinates can be defined within the navigation system using one or more closed polygon boundaries; that is, the AOI's regional coordinates can be composed of a series of vertex coordinates on the polygon boundaries. Similarly, the AOI's regional coordinates can be stored in two-dimensional coordinates (latitude and longitude) or in three-dimensional coordinates (including altitude data).

[0132] Subsequently, the vehicle's infotainment system can construct a spatial model of the AOI region in an absolute coordinate system using the region coordinates of the aforementioned AOI and the height information of buildings within the AOI region. Then, it transforms this spatial model of the AOI region from the absolute coordinate system to the human eye coordinate system. Afterward, the vehicle's infotainment system can convert the spatial model of the AOI region from three dimensions into two-dimensional points on a virtual image plane adapted to the resolution and layout of the AR HUD display device. The specific conversion process can be referred to in the previous section on converting vertex P from three-dimensional coordinates in the human eye coordinate system to two-dimensional coordinates on the virtual image plane, and will not be repeated here.

[0133] To achieve such Figure 9 As shown in the diagram, the vehicle-mounted system can determine four vertices based on the projection of the aforementioned three-dimensional coordinates onto a two-dimensional virtual image plane. These four vertices define the boundary of the AOI (Area of ​​Interest) region. Optionally, the vehicle-mounted system can determine these four vertices using the minimum and maximum values ​​of the X and Y axes of the AOI region on the virtual image plane. Optionally, the vehicle-mounted system can also fine-tune the values ​​of the four vertices determined above, so that the planar area formed by the finely adjusted four vertices can completely cover the projection of the AOI region onto the virtual image plane.

[0134] Compared to other solutions, the method of calculating the transformation from 3D coordinates to 2D coordinates on the virtual image plane through perspective relationships allows the vehicle's navigation system to calibrate the AOI outline even when the destination building is not visible during the current driving process (such as when the destination building is obscured or when the weather is bad and visibility is low), thereby providing navigation prompts to the user and improving the user's navigation experience.

[0135] S109, Vehicle-to-everything (V2X) system calculates internal road projection.

[0136] The aforementioned vehicle-to-everything (V2X) internal road projection refers to the location representation of the internal road transformation of the destination AOI on the virtual image plane, which can be represented by two-dimensional coordinates on the virtual image plane.

[0137] Understandably, a navigation system's map database defines a path by recording the latitude and longitude coordinates (i.e., absolute coordinates) of each point along the path. Therefore, when a large building is the destination, if the navigation system stores internal roads between multiple Points of Interest (POIs) within the large building (including the entrance POI and other POIs under that AOI), these internal roads should also be recorded in the navigation system as two-dimensional coordinates of multiple points.

[0138] Therefore, after step S103, if the vehicle system can obtain the two-dimensional coordinates and elevation data of each point in the internal road between multiple sub-POIs from the navigation system, the vehicle system can also construct a spatial model of the internal road and then obtain the two-dimensional coordinates of the internal road projected onto the virtual image plane through a calculation method similar to that in step S107. This will not be elaborated here.

[0139] S110, Vehicle Infotainment System Calibration Input Point (POI).

[0140] Following step S103, the aforementioned vehicle-mounted system calibration of entrance POIs refers to the vehicle-mounted system calculating the two-dimensional coordinates of each entrance POI projected onto the virtual image plane based on the results of 3D modeling. The calculation method for these two-dimensional coordinates can be referenced from the relevant operations in step S107 above, and will not be repeated here.

[0141] In some embodiments, if in step S103 the vehicle-mounted system also calculates the three-dimensional coordinates of the internal POIs under the AOI area in the human eye coordinate system, then the vehicle-mounted system can also calibrate these internal POIs, that is, calculate the two-dimensional coordinates of these internal POIs projected onto the virtual image plane. It is understandable that when storing POIs associated with the AOI (i.e., sub-POIs of the AOI), the navigation system will assign them different tags or attributes to distinguish their functions and locations. For example, the navigation system can store these sub-POIs in the form of a structure. Therefore, the vehicle-mounted system can distinguish these multiple POIs through attributes or tags.

[0142] Understandable, Figure 10 In the POI enhancement method flow shown, which includes steps S108 to S110, steps S108 to S110 are all optional. If none of these three steps are executed, then... Figure 7 The special cases shown are as follows. Furthermore, when two or more of the above three steps are executed by the vehicle-mounted system, the order of execution can be interchanged; they do not need to follow the above-mentioned order. This application embodiment does not impose any special restrictions on this. In this case, in step S107, the vehicle-mounted system performs projection drawing, which includes not only accurate projection drawing of the optimal entrance POI, but also drawing of the AOI area, the internal projection road, and one or more of all entrance POIs under that AOI based on the results of steps S108 to S110.

[0143] Understandably, the vehicle-mounted system's calibration in steps S108-S110 is a coarse calibration. This is because the calibration utilizes the latitude, longitude, and altitude coordinates of the destination AOI or POI, or the midpoint of the internal road, provided by the navigation system, to construct a spatial model. After coordinate transformation, perspective is used to calculate the corresponding two-dimensional coordinates of the points on the virtual image plane. However, the latitude, longitude, and altitude data provided by the navigation system have certain errors. Therefore, the image drawn in step S107 based on the two-dimensional coordinates calculated in steps S108-S110 cannot accurately match the actual driving environment. Therefore, in subsequent processing, the vehicle-mounted system needs to obtain the precise three-dimensional coordinates of the POI relative to the vehicle-mounted system using sensors such as radar and depth cameras (as described in step S105). Only then can the image drawn in step S107 accurately match the POI in the actual driving environment, achieving the POI enhancement display effect.

[0144] The POI enhancement display method provided in this application embodiment is not limited to in-vehicle infotainment systems. Other head-mounted display devices with AR and navigation functions can also be applied. This application embodiment does not further limit the specific type of electronic device 100. It is understood that in the above method, other electronic devices 100 should also have corresponding sensors capable of position positioning and attitude detection, and should also have corresponding radar, depth sensors, and other sensors capable of accurate distance measurement of the POI. It is understood that for the head-mounted display device in the above electronic device 100, the windshield of the in-vehicle infotainment system in the above solution is equivalent to the glass panel on the head-mounted display device that provides the user with a field of vision. The aforementioned driving field of vision is the range of vision that the user can see through this glass panel, and the aforementioned driving viewpoint is the user's eye. The remaining content can be referred to the relevant description above and will not be repeated here.

[0145] Figure 11 This is a schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of this application.

[0146] like Figure 11 As shown, the electronic device 100 may include components such as a processor 311, a memory 312, a wireless communication processing module 313, a power switch 314, an optomechanical system 315, and a sensor module 316. The components in the electronic device are connected to each other via a bus and communicate based on the bus.

[0147] Processor 311 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0148] Memory 312 is coupled to processor 311 and is used to store various software programs and / or multiple sets of instructions. Memory 312 can be used to store computer executable program code, which includes instructions. Processor 311 executes various functional applications and data processing of the electronic device by running the instructions stored in memory 312. Memory may also be provided in processor 311 for storing instructions and data.

[0149] The memory 312 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM). The RAM can be directly read and written by the processor 311. The RAM can be used to store executable programs (e.g., machine instructions) of the operating system or other running programs, as well as user and application data. The NVM can also store executable programs and user and application data. Executable programs, i.e., user data, stored in the NVM can be pre-loaded into the RAM for direct read and write operations by the processor 311.

[0150] The executable program code and user data used to implement the POI enhanced display method provided in this application embodiment can be stored in non-volatile memory. During the implementation of the above-described POI enhanced display method, the electronic device can load the non-volatile memory executable program code and user data into random access memory, and finally accurately locate and enhance the target entrance on the AR HUD based on its accurate three-dimensional coordinates, thereby optimizing the user's navigation experience.

[0151] The wireless communication processing module 313 can provide wireless communication solutions including WLAN, such as Wi-Fi, Bluetooth communication, ZigBee communication, NFC communication, infrared communication, and UWB communication. In this embodiment, the GPS function used for positioning by the electronic device 100 can be provided by the aforementioned wireless communication processing module 313. In addition, the electronic device 100 can also use the wireless communication processing module 313 to request various data from the navigation system's server, including but not limited to various latitude and longitude coordinates and altitude data.

[0152] The power switch 314 can be used to control the power supply to electronic devices, thereby supplying power to the processor 311, memory 312, wireless communication processing module 313, optomechanical system 315, sensor module 316, etc.

[0153] The optomechanical system 315 may include components such as an image generation unit, a light source, and optical elements, for generating and projecting images, thereby accurately overlaying digital information onto the real environment in the form of images. In this embodiment, the electronic device 100 can perform projection drawing functions in the aforementioned AR HUD based on the optomechanical system 315. Specifically, after determining the two-dimensional coordinates of the image on the virtual image plane, the electronic device 100 can send the two-dimensional coordinates in the virtual image plane to the optomechanical system 315 for projection drawing.

[0154] The sensor module may include various sensors such as depth cameras, accelerometers, gyroscopes, and wheel speed sensors. In this embodiment, the sensor module can provide cockpit perception services and accurate detection of Points of Interest (POIs). Specifically, sensors such as radar and depth cameras can accurately detect the location of the POI in space, thereby obtaining an accurate spatial model of the POI. This accurate spatial model is then used for projection rendering to achieve enhanced display of the POI. The aforementioned accelerometers, gyroscopes, and wheel speed sensors provide vehicle control status data, which is then used by the electronic device 100 to assist in determining the current spatial position and orientation of the electronic device 100, thereby helping the electronic device 100 obtain more accurate results in 3D modeling and coordinate transformation.

[0155] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. Components may be implemented in hardware, software, or a combination of software and hardware.

[0156] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. Embodiments of this application also provide a computer program product, including a computer program that, when run on a processor, implements the steps in the various method embodiments described above.

[0157] The above detailed embodiments further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above are merely specific embodiments of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A method for enhancing the display of Points of Interest (POIs), characterized in that, Applied to in-vehicle equipment, the method includes: Run the navigation application and start the first route within the navigation application; Turn on the camera to capture an image of the area in front of the vehicle's windshield; Based on the image, it is determined that the target scene appears in the driver's field of vision. The target scene is located on the first route. The vehicle-mounted device projects a first virtual image onto the windshield of the vehicle. In the driver's field of vision, the first virtual image is superimposed on the target scene. The display position of the first virtual image in front of the windshield of the vehicle is determined based on a first three-dimensional coordinate. The first three-dimensional coordinate is determined by the vehicle-mounted device through a first sensor. The first sensor includes one or more of the following: radar, depth sensor, and laser sensor.

2. The method according to claim 1, characterized in that, The target scene includes a first entrance to a first building. Before the in-vehicle device projects the first virtual image onto the windshield of the vehicle, the method further includes: Based on the images captured by the camera, it was determined that multiple entrances to the first building appeared in the driver's field of view; The first entry is selected from the plurality of entry points.

3. The method according to claim 1 or 2, characterized in that, The in-vehicle device projects a first virtual image onto the windshield of the vehicle, specifically including: The in-vehicle device projects the first virtual image onto a virtual image plane in front of the windshield. The virtual image plane is perpendicular to the ground, and the first virtual image, the target scene, and the driver's viewpoint on the virtual image plane are located on the same straight line.

4. The method according to claim 3, characterized in that, The first three-dimensional coordinates are the three-dimensional coordinates of the target scene in a coordinate system with the driving viewpoint as the origin, and the two-dimensional coordinates of the first virtual image on the virtual image plane are obtained by transforming the first three-dimensional coordinates.

5. The method according to claim 4, characterized in that, The method further includes: The vehicle's position and orientation are determined by vehicle control sensors, including accelerometers, gyroscopes, global navigation satellite systems, lidar, and cameras. The first three-dimensional coordinates are determined based on the vehicle's position and the distance between the target object and the vehicle measured by the first sensor.

6. The method according to any one of claims 1-5, characterized in that, The step of determining that the target object appears in the driver's field of vision based on the image specifically includes: Based on the image captured by the camera at time t1, it is determined that the target scene appears in the driver's field of vision; The in-vehicle device projects a first virtual image onto the windshield of the vehicle, specifically including: The vehicle-mounted device projects a first virtual image at time t2 onto the windshield of the vehicle. The first virtual image at time t2 is determined by the vehicle-mounted device based on the first three-dimensional coordinates determined by the first sensor at time t2. Time t2 is the same as time t1 or the time difference between time t2 and time t1 is less than a first time threshold.

7. The method according to any one of claims 1-6, characterized in that, The closer the vehicle-mounted device is to the target scene, the larger the first virtual image superimposed on the target scene; conversely, the farther the vehicle-mounted device is from the target scene, the smaller the first virtual image superimposed on the target scene.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: Based on the image, if the target object is determined to have disappeared from the driver's field of vision, the in-vehicle device cancels the projection of the first virtual image onto the windshield of the vehicle.

9. The method according to any one of claims 1-8, characterized in that, Before turning on the camera, the method further includes: The distance between the vehicle-mounted device and the target scene is detected to be less than or equal to a first distance threshold.

10. The method according to any one of claims 1-9, characterized in that, The first virtual image includes one or more of the following: a location icon, the name of the target scene, and the outline of the target scene.

11. The method according to any one of claims 1-10, characterized in that, The target scenery includes the entrance of the first building, the road in the first building, and the first building itself.

12. An electronic device, characterized in that, The method includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when executed by the one or more processors, cause the method as described in any one of claims 1-11 to be performed.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run on an electronic device, it causes the method described in any one of claims 1-11 to be performed.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-11.

Images