A head-up display method and device for multiple users

Abstract

The application relates to a multi-user head-up display method and device, which comprises the following steps: establishing a line-of-sight vector corresponding to each user according to the head posture and eyeball gaze direction of at least two users in a vehicle; constructing a three-dimensional space model of an external environment of the vehicle and identifying and labeling a target object; intersecting each line-of-sight vector with the three-dimensional space model to determine a target object currently gazed at by each user; locking the target object gazed at by at least two users as a unified target when it is determined that the target object is gazed at by the at least two users; and mapping the world coordinates of the unified target to a corresponding HUD display plane of each user through coordinate transformation and driving each HUD to synchronously lock and display. Through cross-view angle consistency determination and unified coordinate mapping, the application realizes multi-HUD cooperative display of multi-user view angles, locks the same target, and takes into account the safety of a main driver and the information richness of a copilot.

Description

Technical Field

[0001] This application relates to the field of head-up display technology, and more specifically to a head-up display method and apparatus for multiple users. Background Technology

[0002] With the rapid development of smart cockpits and in-vehicle augmented reality (AR) technology, head-up displays (HUDs) have become an important feature of mid-to-high-end smart cars. Existing HUD systems are typically deployed in the driver's seat to overlay two-dimensional information such as vehicle speed, navigation, and advanced driver assistance system (ADAS) prompts onto the windshield, expanding the driver's perception of the driving environment. To further enhance the interactive experience, some augmented reality HUD solutions construct a static three-dimensional spatial model of the vehicle's external environment and pre-label various components in the model (such as buildings, road signs, and vehicles) with attribute information. This information is then matched with the driver's line-of-sight information and the preset target object attributes. A coordinate fusion algorithm is used to calculate the projection position of the target object on the windshield, thereby displaying target-related information in an augmented reality manner. Existing technical solutions have the following main shortcomings: The limitations of single-viewpoint and single-user systems. Existing HUDs and corresponding target recognition and information display solutions mainly revolve around a single driver. Passengers in the front passenger seat and rear seats cannot share the HUD display information and find it difficult to participate in natural interactions based on the real-world scene in front, resulting in low user engagement. Furthermore, they lack cross-viewpoint target consistency capabilities. In actual driving scenarios, occupants in different seats have significantly different observation positions of the same external target (such as a tree or a building) due to differences in their viewing angles. Existing solutions cannot perform convergence calculations and consistency determinations for the gazes of multiple occupants, thus failing to confirm whether multiple people are looking at the same target object, and therefore cannot simultaneously lock onto and present the same target on multiple display terminals.

[0003] Therefore, there is an urgent need in this field for a method that can support unified target recognition and collaborative display under conditions of multiple users, multiple HUDs, and multiple viewing angles. Summary of the Invention

[0004] This application provides a head-up display method and apparatus for multiple users to solve the problems of inconsistent user perspectives, inconsistent target recognition, and lack of collaborative display and interaction in existing HUD solutions.

[0005] Firstly, this application provides a head-up display method for multiple users, including: Based on the head posture and eye gaze direction of at least two users in the vehicle, establish a gaze vector for each user. A three-dimensional spatial model of the vehicle's external environment is constructed, target objects in the three-dimensional spatial model are detected, and each target object is assigned annotation information; wherein, the annotation information includes at least one of the following: object identifier, object category, center coordinates, object size, and object orientation; The intersection of each user's gaze vector with the three-dimensional spatial model is calculated, and the target object currently being gazed at by each user is determined based on the target object at the intersection point. Based on the spatial proximity and semantic consistency of the target objects currently being gazed at by the at least two users, when it is determined that the at least two users are gazing at the same target object, the target object is locked as a unified target being gazed at by the at least two users. The position of the unified target in the world coordinate system is mapped to the display plane of the HUD corresponding to the user who is looking at the unified target through coordinate transformation, thereby driving the HUD to synchronously lock and display the unified target.

[0006] Optionally, the step of establishing a gaze vector for each user based on the head posture and eye gaze direction of at least two users inside the vehicle includes: Acquire the user's facial image and extract facial key points from the facial image; The PnP algorithm is used to determine the user's head pose based on the facial key points; Based on the head posture and the preset facial anatomy model, the coordinates of the user's eye center are determined; The user's gaze vector is calculated based on the eye center coordinates and the head rotation matrix.

[0007] Optionally, the construction of a three-dimensional spatial model of the vehicle's external environment, target detection of target objects in the three-dimensional spatial model, and assignment of annotation information to each target object include: Acquire multimodal perception data of the external environment collected by multimodal sensors; Unify the multimodal perception data to a vehicle coordinate system with a preset origin; Based on the multimodal perception data in the vehicle coordinate system, a dense point cloud map of the external environment is generated, and the dense point cloud map is used as the three-dimensional spatial model. The target objects in the dense point cloud map are identified and segmented into instances. Each target object is assigned a target identifier and fitted with a 3D bounding box to obtain a structured target representation containing the annotation information. The structured target representation is used as the target object in the three-dimensional space model.

[0008] Optionally, the step of intersecting the gaze vector of each user with the three-dimensional spatial model and determining the target object currently being gazed at by each user based on the target object at the intersection point includes: Extend the gaze vector of each user to the three-dimensional space model, and obtain the intersection point between the gaze vector and the target object in the three-dimensional space model by ray intersection; The target object corresponding to the intersection point closest to the user along the line of sight is determined as the target object that the user is currently looking at.

[0009] Optionally, when determining that the at least two users are looking at the same target object based on the spatial proximity and semantic consistency of the target objects currently being looked at by each of the at least two users, locking the target object as a unified target being looked at by the at least two users includes: Spatial clustering is performed on the intersection points of at least two users' gaze vectors and the three-dimensional spatial model to obtain intersection point clusters; semantic matching is performed on the target objects corresponding to each intersection point within the intersection point clusters; Based on the relationship between the distance between each intersection point and a preset distance threshold, the degree of proximity of the target objects currently being viewed by the at least two users in the spatial dimension is obtained; based on the semantic category of the target objects corresponding to each intersection point, the degree of consistency of the target objects currently being viewed by the at least two users in the semantic dimension is obtained. If the distance between each intersection point is less than or equal to a preset distance threshold, and the semantic category similarity of the target objects corresponding to each intersection point is greater than or equal to a preset similarity value, then it is determined that the at least two users are looking at the same target object. Optionally, when determining that the at least two users are looking at the same target object based on the spatial proximity and semantic consistency of the target objects currently being looked at by each of the at least two users, locking the target object as a unified target being looked at by the at least two users includes: The semantic category similarity of the target objects viewed by the at least two users is calculated as the degree of consistency of the target objects in the semantic dimension, and the spatial distance proximity of the center points of the target objects viewed by the at least two users is calculated as the degree of proximity of the target objects in the spatial dimension. A weighted fusion method is used to calculate the consistency confidence based on the semantic category similarity and the normalized spatial proximity; wherein, the consistency confidence is equal to the product of the semantic category similarity weight and the semantic category similarity, plus the product of the spatial proximity weight and the spatial proximity, and the sum of the semantic category similarity weight and the spatial proximity weight is 1; The consistency confidence level is used to determine whether the at least two users are looking at the same target object.

[0010] Optionally, calculating the spatial proximity of the center points of the target objects viewed by the at least two users includes: Calculate the Euclidean distance between the world coordinates of the center points of the target objects viewed by the at least two users, and use it as the spatial distance between the center points of the target objects viewed by the at least two users; Based on the spatial distance and the preset spatial distance threshold, the spatial distance proximity of the center points of the target objects being viewed by the at least two users is calculated.

[0011] Optionally, mapping the position of the unified target in the world coordinate system to the display plane of the HUD corresponding to the user gazing at the unified target through coordinate transformation, and driving the HUD to synchronously lock and display the unified target, includes: A rigid body transformation is established from the world coordinate system to the HUD coordinate system, and the homogeneous coordinates of the unified target in the world coordinate system are projected onto the HUD camera intrinsic parameter matrix through the rigid body transformation to obtain the projected coordinates of the unified target on the HUD display plane. Based on the projection coordinates, the unified target is synchronously locked and displayed on each of the HUDs.

[0012] Optionally, driving the HUD to synchronously lock and display the unified target includes: Different HUD rendering strategies are adopted for different user roles, and the unified target is synchronously locked and displayed on each HUD.

[0013] Optionally, driving the HUD to synchronously lock and display the unified target includes: When at least two users include the driver's seat user, the projection position of the unified target on the HUD display plane corresponding to the driver's seat user is recalculated based on the head posture and eye gaze direction of the driver's seat user, so as to obtain the projection position after parallax compensation for the driver's seat. Based on the projection position after parallax compensation for the driver's seat, the unified target is rendered on the HUD corresponding to the driver's seat user.

[0014] Optionally, driving the HUD to synchronously lock and display the unified target includes: Assign a unified target identifier corresponding to the unified target to each of the HUDs, and render the unified target independently on each of the HUDs; When a change in the user's head posture and eye gaze direction is detected, the display position of the unified target on each of the HUDs is dynamically corrected according to the change in the user's head posture and eye gaze direction.

[0015] Optionally, the method further includes: Receive the user's interaction request initiated by voice or gesture; Based on the interaction request, data corresponding to the unified goal is provided to the user.

[0016] Secondly, this application provides a head-up display device for multiple users, comprising: The gaze vector establishment module is used to establish a gaze vector for each user based on the head posture and eye gaze direction of at least two users in the vehicle. The target object determination module is used to construct a three-dimensional spatial model of the vehicle's external environment, perform target detection on target objects in the three-dimensional spatial model, and assign annotation information to each target object; wherein, the annotation information includes at least one of the following: object identifier, object category, center coordinates, object size, and object orientation; The target object determination module is used to find the intersection of each user's gaze vector with the three-dimensional space model, and determine the target object currently being gazed at by each user based on the target object at the intersection point; The target consistency judgment module is used to determine, based on the spatial proximity and semantic consistency of the target objects currently being gazed at by the at least two users, that when the at least two users are gazing at the same target object, the target object is locked as a unified target being gazed at by the at least two users. The display driver module is used to map the position of the unified target in the world coordinate system to the display plane of the HUD corresponding to the user who is looking at the unified target through coordinate transformation, and drive the HUD to synchronously lock and display the unified target.

[0017] Thirdly, this application also provides a vehicle, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to execute the head-up display method for multiple users as described in any of the preceding claims of this application.

[0018] Fourthly, this application also provides a computer storage medium storing computer-executable instructions for executing the head-up display method for multiple users described above.

[0019] The head-up display method for multiple users provided in this application achieves the following technical effects through a complete technical chain: "establishing gaze vectors for each user—3D spatial modeling and identification annotation—gauge intersection to determine the target object for each user—determining whether multiple users are looking at the same target object and locking onto a unified target—mapping the coordinates of the unified target to each HUD for synchronous locking and display." (1) This application establishes a gaze vector for each user's head posture and eye gaze direction, and intersects each gaze vector with a three-dimensional spatial model to determine the target object being gazed at. It then further determines whether at least two users are gazing at the same target object. Therefore, it can dynamically confirm whether multiple people are focusing on the same external target even when there are natural differences in the perspectives of occupants in different seats, without having to pre-set the target object attributes as required by existing technologies. Thus, this application solves the problem of "lack of cross-view target consistency capability" in existing solutions, enabling the system to dynamically lock onto external targets being gazed at by multiple users without pre-setting target attributes.

[0020] (2) This application maps the position of the locked unified target in the world coordinate system to the display plane of the HUD corresponding to the user looking at the unified target through coordinate transformation, and drives each HUD to lock and display the unified target synchronously. Therefore, multiple HUDs can present the same target under a unified spatial reference, avoiding target misalignment or unclear reference caused by independent projection of each HUD. This mechanism directly responds to the deficiency of the existing solution "lacking a multi-HUD collaborative mechanism", enabling the front passenger and rear passengers to share the recognition and display results of the same target with the driver, improving the overall user participation and naturalness of interaction.

[0021] (3) This application adopts differentiated HUD rendering strategies for different users (for example, focusing on safety and noise reduction in the driver's seat, presenting only a minimalist outline or arrow indicator, while providing 3D wireframes, labels, and encyclopedic information in the passenger seat), thus ensuring the driver's safety and avoiding obstruction of the real road view while providing rich target interactive information for the passenger. Therefore, this application resolves the contradiction between "driving safety" and "information richness" in the prior art.

[0022] (4) Based on locking a unified target, this application can also receive interactive requests initiated by users through voice or gestures, and provide users with data corresponding to the unified target (such as encyclopedia, navigation, etc.) accordingly. Thus, a complete closed loop of "multi-source perception - line-of-sight intersection - consistency matching - collaborative display - active interaction" is formed. Compared with the existing solution that only performs one-way information superposition, it significantly enhances the encyclopedic interactive experience that integrates the inside and outside of the vehicle. Attached Figure Description

[0023] Figure 1A flowchart illustrating a head-up display method for multiple users provided in an embodiment of this application; Figure 2 A flowchart illustrating a multi-user head-up display method provided in another embodiment of this application; Figure 3 A flowchart illustrating a multi-user head-up display method provided in another embodiment of this application; Figure 4 A flowchart illustrating a multi-user head-up display method provided in another embodiment of this application; Figure 5 A flowchart illustrating a multi-user head-up display method provided in another embodiment of this application; Figure 6 A flowchart illustrating a multi-user head-up display method provided in another embodiment of this application; Figure 7 This application provides a schematic diagram of the structure of a head-up display device for multiple users. Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 9 This is a schematic diagram of HUD coordinate mapping provided in an embodiment of this application. Detailed Implementation

[0024] The following description, with reference to the accompanying drawings and optional embodiments, illustrates the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the optional embodiments are merely illustrative of this application and not intended to limit the scope of protection of this application.

[0025] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0026] In the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance. For ease of understanding, some English abbreviations in this embodiment are explained below: Head-Up Display (HUD), Augmented Reality (AR), Advanced Driver Assistance System (ADAS), Perspective-n-Point (PnP), Simultaneous Localization and Mapping (SLAM), Light Detection and Ranging (LiDAR), Controller Area Network with Flexible Data-Rate (CAN FD), Convolutional Neural Network (CNN), and Point of Interest (POI).

[0027] Existing HUD solutions are generally designed around a single driver, making it impossible for the front passenger and rear passengers to share display information. Furthermore, it's difficult to confirm whether multiple occupants are looking at the same external target, and maintaining consistency in the recognition, display, and interaction with the same target across different HUDs is challenging. This application aims to address these problems, and the technical solution of this application will be described in detail below through a series of embodiments.

[0028] The HUD display system of this application mainly includes: a multi-HUD display module (driver's HUD, passenger's HUD), an in-vehicle visual perception module (face recognition camera, eye tracking), an external environment perception module (camera, LiDAR), a three-dimensional space modeling module, a gaze calculation module, a multi-view fusion module, an AR rendering module, and an interactive control module (voice / gesture).

[0029] Among them, the vehicle external environment perception module and the 3D space modeling module work together to complete the construction of the 3D space model; the line of sight calculation module completes the establishment of the line of sight vector; the multi-view fusion module completes the consistency judgment of the same target object; the AR rendering module and the display driving module complete the mapping and rendering of the same target object; and the interaction control module completes the interaction closed loop.

[0030] Example 1 The fundamental reason why existing technologies struggle to allow multiple occupants to share the display of the same target is that each occupant has different eye positions (i.e., head posture and eye gaze direction), resulting in different focal points for their gaze in real 3D space. If projections are directly made according to each occupant's eye position, the same real object will appear misaligned and unclear on different HUDs. Therefore, this embodiment introduces a process of "intersecting gaze vectors to determine individual target objects, identifying whether they are the same target object, locking onto a unified target, and synchronously displaying it using unified coordinate mapping." This achieves target unification across viewing angles from a spatial calculation perspective, rather than simply projecting based on eye position.

[0031] Figure 1 This is a flowchart illustrating a multi-user head-up display method provided in an embodiment of this application, applied to a vehicle cabin equipped with at least a driver's seat HUD and a passenger's seat HUD. Figure 1 As shown, the method includes: Step S11: Based on the head posture and eye gaze direction of at least two users in the vehicle, establish the gaze vector corresponding to each user.

[0032] For example, the vehicle cabin is equipped with a high-precision in-vehicle sensing camera array covering the driver's and passenger's areas (with infrared illumination to ensure clear facial images can be captured even at night or in backlight conditions); the system uses in-vehicle cameras or infrared sensors to obtain the head posture and eye gaze direction of the driver and passenger, and establishes a gaze vector pointing out of the vehicle for each user.

[0033] Step S12: Construct a three-dimensional spatial model of the vehicle's external environment, perform target detection on the target objects in the three-dimensional spatial model, and assign annotation information to each target object; wherein, the annotation information includes at least one of the following: object identifier, object category, center coordinates, object size, and object orientation.

[0034] The system can acquire external environmental data using vehicle-mounted front / side-view cameras, LiDAR and other multimodal sensors, and use SLAM technology to construct a dense point cloud map or semantic map of the external environment as a unified three-dimensional spatial model. Then, it uses 2D / 3D object detection algorithms (such as YOLOv8 combined with PointPillars) to identify external target objects (pedestrians, vehicles, trees, buildings, etc.) and assigns annotation information to each detected object.

[0035] Step S13: Intersect the gaze vector of each user with the three-dimensional space model, and determine the target object currently being gazed at by each user based on the target object at the intersection point.

[0036] The system extends the driver's and passenger's gaze vectors forward and intersects them with the target objects in the 3D space model using rays to determine the target objects currently being gazed at by the driver and passenger.

[0037] Step S14: Based on the spatial proximity and semantic consistency of the target objects currently being gazed at by the at least two users, when it is determined that the at least two users are gazing at the same target object, the target object is locked as a unified target being gazed at by the at least two users.

[0038] The system performs consistency determination on the target objects determined by the driver and passenger respectively: if it is determined that the two are looking at the same target object, the target object is locked as the unified target that the two are looking at and is assigned a unique unified target identifier (Object ID).

[0039] In this embodiment of the application, the aforementioned unified target is not limited to a single target object, but may include: (1) Static object class: a single static object (a building, a tree) or multiple static objects (a row of buildings, multiple parked vehicles). (2) Dynamic Objects: Moving objects (driving vehicles, pedestrians / cyclists), whose world coordinates are updated as the real-time pose of the associated target objects are updated; (3) Region type: A scene region composed of multiple target objects that are spatially adjacent / semantically related, such as a car accident scene. The target objects or unified target include: multiple vehicles involved + surrounding people + vehicle fragments, expressed in the form of a set of target objects, and given a unique unified target identifier. (4) Others: such as static and dynamic mixed (parked vehicles and pedestrians in the same accident area), aggregated landmarks (shops / bridges / monuments composed of multiple sub-objects), overall moving areas (congested road sections / walking groups), spatially dispersed but semantically related groups (an intersection containing traffic lights + signs + pedestrians and vehicles), etc. Regardless of the form of the unified target, it is expressed in a unified 3D spatial model and associated with one or more target object identifiers (Object IDs). When there are multiple target objects containing the unified target, each object's identifier is retained and assigned a unique unified target identifier. In step S15, the position of the unified target in the world coordinate system is mapped to the display plane of the HUD corresponding to the user gazing at the unified target through coordinate transformation, and the HUD is driven to synchronously lock and display the unified target.

[0040] The system establishes a rigid body transformation from the world coordinate system to the HUD coordinate system, projects the homogeneous coordinates of the unified target in the world coordinate system onto each HUD display plane to obtain the projected coordinates, and synchronously locks and displays the unified target on each HUD accordingly.

[0041] For example, if the front passenger looks at a tree outside the car, the system obtains their head posture and eye gaze direction to establish a gaze vector. This gaze vector is then intersected with a 3D spatial model to determine that the front passenger is currently looking at the "Ginkgo Tree" target object with ID=101. Simultaneously, if the driver is also looking at the same tree, the system determines that both are looking at the same target object and locks it as the unified target. The system then maps the tree's position in the world coordinate system to the display planes of the driver's and front passenger's HUDs. The front passenger's HUD renders a highlighted 3D wireframe, the "Ginkgo" label, and an encyclopedia card, while the driver's HUD only renders a minimalist outline highlight or an arrow indicator, thus achieving coordinated and consistent display of "this tree" by both passengers.

[0042] As can be seen from the above embodiment 1, since this application does not simply project according to each user's individual eye position, but instead establishes the gaze vector of each user, intersects it with a unified three-dimensional spatial model to determine the target object each user is looking at, then determines whether multiple users are looking at the same target object and locks onto a unified target, and finally performs coordinate transformation using the world coordinates of the unified target and synchronously maps it to each HUD, it can fundamentally eliminate the projection misalignment problem caused by differences in the occupants' perspectives. In other words, no matter how far apart the physical eye positions of the driver and passenger are, as long as the system determines that they are looking at the same target object, it locks onto the same unified target and presents the same spatial coordinates, thereby achieving the core technical effect of unifying the perspectives of multiple users and locking onto the same target, laying a unified spatial benchmark for subsequent multi-HUD collaborative display and active interaction.

[0043] Example 2 The following is a more detailed explanation of establishing the line-of-sight vector in step S11 above.

[0044] Accurate estimation of the user's eye position (head posture and eye gaze direction) is a prerequisite for subsequent gaze intersection and consistency determination. If there is a large error in the eye position estimation, the gaze vector will deviate from the true gaze direction, leading to misjudgment of the target object. Therefore, this embodiment introduces an eye position solution process based on facial key points and the PnP algorithm to ensure the accuracy of the gaze vector.

[0045] Figure 2 A flowchart illustrating a multi-user head-up display method according to another embodiment of this application is shown below. Figure 2 As shown, step S11 above includes: Step S111: Obtain the user's facial image and extract the facial key points from the facial image.

[0046] The system collects user facial images through in-vehicle cameras and uses deep learning models (such as MediaPipe or self-developed CNN) to extract 68 or 468 key points of the face in real time, focusing on the coordinates of the corners of the eyes, the tip of the nose, and the corners of the mouth.

[0047] Step S112: Using the PnP algorithm, determine the user's head pose based on the facial key points.

[0048] The PnP (Perspective-n-Point) algorithm takes a set of 2D image coordinates and a set of 3D face model coordinates as input, and outputs a rotation matrix R and a translation vector T describing the head orientation and position. Essentially, it seeks the R and T values ​​that minimize the overlap error between the key points of the 3D model, after being transformed by R and T, and projected onto the camera image, and the key points marked on the real face photo. In other words, it minimizes the sum of squared errors between the predicted 2D pixel coordinates of each key point and the actual detected 2D pixel coordinates.

[0049]

[0050] Here, argmin represents the parameter that minimizes the subsequent error and T, which is the optimal R and T to be found. ∑ represents summing over all key points i, that is, adding up the error of each point; The projection function projects the points of the 3D model. After R and T transformations, the image is projected onto the camera's imaging plane to obtain the predicted 2D pixel coordinates. This represents the coordinates of the i-th key point on the 3D face model (three-dimensional coordinates in the model coordinate system, such as the position of the tip of the nose, the corner of the eye, and the corner of the mouth). This represents the 2D pixel coordinates of the i-th key point extracted from the image (e.g., the pixel positions of the corner of the eye and the tip of the nose detected by MediaPipe). It represents the square of the Euclidean distance, which is the square of the pixel distance between the predicted 2D point and the actual detected 2D point, and is used to measure the magnitude of the error.

[0051] Step S113: Based on the head posture and the preset facial anatomy model, determine the coordinates of the user's eye center.

[0052] Based on head posture and a facial anatomy model, the coordinates E of the center of the left and right eyeballs (e.g., the center of the left eye) are calculated. Right eye center ).

[0053] Step S114: Calculate the user's gaze vector based on the eyeball center coordinates and head rotation matrix.

[0054] line-of-sight vector (Unit vector), for a single eye there is .

[0055] in, R is the gaze direction of the eye in the local coordinate system (usually pointing towards the center of the screen or the direction of the pupil reflection point), and R is the head rotation matrix.

[0056] The final line of sight originates from the center E of the eyeball and extends in the following direction: It can be expressed as a parametric equation:

[0057] L(t) is the coordinate of any point on the ray, which is the set of all points that the line of sight passes through in three-dimensional space. By taking different values ​​of t, we can obtain points at different positions on the ray.

[0058] E represents the coordinates of the eye's center, its position in three-dimensional space, such as the center of the left eye. Or the center of the right eye It is the starting point of our vision.

[0059] The gaze vector, representing the direction the eyes are looking, is a unit vector (of length 1) indicating the direction the gaze is directed. It is determined by both head posture and eye movement.

[0060] t is a ray parameter that controls the position of the point on the ray: when t=0, the point is the coordinate E of the eye center; when t>0, the point extends forward along the line of sight (the larger t is, the farther away from the eye).

[0061] Optionally, the left and right eye gaze vectors can be fused or one of them can be selected to obtain the gaze vector corresponding to the user.

[0062] This embodiment uses facial landmarks combined with the PnP algorithm to solve head pose, and further combines a facial anatomy model to estimate the eye center and calculate the gaze vector from the head rotation matrix. Therefore, it can transform the abstract gaze direction into a gaze vector (ray) that can be calculated in a unified coordinate system, providing a quantitative basis for subsequent accurate intersection with the 3D spatial model. Accurate gaze vectors make the target object obtained from gaze intersection closer to the user's actual gaze point, thereby reducing the probability of target misjudgment due to eye position estimation errors and improving the reliability of the overall system's consistency judgment.

[0063] Example 3 In an optional embodiment, the steps of constructing a three-dimensional spatial model and identifying annotations in step S12 above are further refined.

[0064] To achieve the recognition of the object being gazed at by the user, the prerequisite is the existence of a unified, accurate model of the vehicle's external environment that is in the same coordinate system (world coordinate system) as the gaze vector, and each object in the model is given annotation information that can be used for semantic matching.

[0065] Figure 3 A flowchart illustrating a multi-user head-up display method according to another embodiment of this application is shown below. Figure 3 As shown, step S12 above includes: Step S121: Acquire multimodal perception data of the external environment collected by the multimodal sensor.

[0066] The multimodal sensors include at least one or more of the following types of devices: vehicle-mounted image acquisition equipment, lidar, inertial measurement units, wheel speed encoders, and satellite positioning modules, etc. One or more of each type of device may be present.

[0067] Multimodal perception data of the external environment is collected through vehicle-mounted image acquisition equipment, LiDAR, inertial measurement unit, wheel speed encoder and satellite positioning module; in optional embodiments, millimeter-wave radar can also participate in environmental perception and be fused with camera and LiDAR to improve perception robustness in adverse weather conditions.

[0068] Step S122: Unify the multimodal perception data to the vehicle coordinate system with a preset origin; After acquiring the perception data collected by different sensors, it is necessary to perform time synchronization and spatial extrinsic parameter calibration to unify the data of each sensor into the vehicle coordinate system with the rear axle center or the lidar center as the origin.

[0069] For example, PTP or hardware triggering can be used to align the microsecond-level timestamps of each sensor; the rigid external parameters of the camera, LiDAR, IMU and vehicle body can be pre-calibrated and unified to the vehicle coordinate system with the "vehicle rear axle center or LiDAR center" as the origin.

[0070] Step S123: Based on the multimodal perception data in the vehicle coordinate system, generate a dense point cloud map of the external environment of the vehicle, and use the dense point cloud map as the three-dimensional spatial model. Laser-inertial SLAM (e.g., LIO-SAM or FAST-LIO2) is used to estimate the six-DOF pose of the vehicle in each frame, and visual-inertial odometry is fused to compensate for weak texture scenes. After eliminating accumulated drift through loop closure detection and global pose graph optimization, the pose sequence and depth information are fused with TSDF to generate a dense point cloud map of the external environment of the vehicle.

[0071] Step S124: Identify and segment the target objects in the dense point cloud map, assign a target identifier to each target object, fit a three-dimensional bounding box, and obtain a structured target representation containing the annotation information. Using 2D / 3D object detection algorithms (such as YOLOv8 combined with PointPillars), targets such as pedestrians, vehicles, trees, and buildings are identified, and 3D bounding boxes are fitted; each target object O in the scene can be structured as:

[0072] Where O stands for Object, representing any target detected outside the vehicle, such as pedestrians, vehicles, road signs, etc. ID stands for Object ID (Identity). It assigns a unique number to each target and is used to track the same object in consecutive frames (for example, if the car has ID=101 in frame 1, it will still have ID=101 in frame 10). The class refers to the object category, i.e., what type of object it is, such as "Pedestrian", "Car", "Truck", "Sign", etc., which is output by the object detection algorithm; The three-dimensional coordinates of the target center in the world coordinate system are the position of the geometric center of the object in the unified world coordinate system (Ow, usually with the center of the rear axle of the vehicle as the origin). The format is (x,y,z), which represents its distance from the front, back, left, right, up, and down of the vehicle.

[0073] Size is the three-dimensional dimensions of the object, including its length, width, and height. It is used to construct a 3D bounding box and determine how much space it occupies. θ represents the orientation of an object, i.e., the heading angle / yaw angle, such as the direction in which the front of a vehicle is facing. It is used to indicate whether it is going straight, turning, or making a U-turn on the road, and is usually the rotation angle around the vertical axis (Z-axis).

[0074] By labeling the target object O, any target object outside the vehicle can obtain a complete representation in the world coordinate system. This is equivalent to creating a structured data set for each detected object, which includes its identity, category, location, size, and orientation, making it easier to map to the HUD display plane later.

[0075] For dynamic targets such as pedestrians and vehicles, based on multi-view... Figure 1By separating it from the static map and tracking its real-time pose independently, the three-dimensional spatial model can reflect the current real state of the external environment, providing a unified and consistent spatial reference for the subsequent intersection of each user's line-of-sight vector with the model.

[0076] Step S125: The structured target representation is used as the target object in the three-dimensional spatial model.

[0077] This embodiment constructs a 3D spatial model in a unified world coordinate system using multimodal sensing and SLAM, and assigns structured annotations to each object, including identifier, category, center coordinates, size, and orientation. Therefore, the "3D spatial model" and "object annotations" upon which the line-of-sight vector intersection depends are fully usable in the unified coordinate system. In this way, the line-of-sight vectors of the driver and passenger can intersect with the same model in the same world coordinate system. The resulting target objects not only have spatial coordinates but also semantic categories, thus providing directly usable input for subsequent semantic matching and consistency determination, avoiding the problem of being unable to determine whether they are the same target due to inconsistent coordinate systems or missing annotations.

[0078] Example 4 After obtaining the gaze vector and three-dimensional spatial model of each user, the target object currently being gazed at by the user can be determined.

[0079] Figure 4 A flowchart illustrating a multi-user head-up display method according to another embodiment of this application is shown below. Figure 4 As shown, step S13 above includes: Step S131: Extend the gaze vector of each user to the three-dimensional space model, and obtain the intersection point between the gaze vector and the target object in the three-dimensional space model by ray intersection.

[0080] Taking the driver's seat as an example, the driver's line of sight vector ( The intersection of the object with each object in the 3D spatial model (such as each 3D bounding box or point cloud cluster) is calculated to obtain several intersection points.

[0081] Step S132: The target object corresponding to the intersection point closest to the user along the line of sight is determined as the target object that the user is currently looking at.

[0082] Since the actual target of human eye gaze is usually the object that is first seen in the direction of the line of sight, the object corresponding to the intersection along the line of sight (where t is the smallest, i.e., closest to the center of the eyeball) is selected as the target object currently being gazed at by the user. For example, if the driver's line of sight first intersects with the bounding box of the "ginkgo tree" numbered ID=101, then the tree is the target object currently being gazed at by the driver; the target object for the passenger's line of sight is obtained in the same way.

[0083] This embodiment uses the rule of obtaining intersection points through ray intersection and selecting the object corresponding to the nearest intersection point along the line of sight to determine the target object currently being viewed by each user. Therefore, it can transform the abstract line-of-sight vector into a specific and unique target object identifier, making "who is looking at what" clearly quantifiable. This clear mapping relationship is a necessary prerequisite for cross-view consistency determination.

[0084] Example 5 In step S14 above, it is determined that at least two users are looking at the same target object. This can be done in various ways. This embodiment provides an implementation method based on spatial clustering and semantic matching.

[0085] Figure 5 A flowchart illustrating a multi-user head-up display method according to another embodiment of this application is shown below. Figure 5 As shown, in step S14 above, determining that at least two users are gazing at the same target object specifically includes: Step S141: Spatial clustering is performed on the intersection points of at least two users' gaze vectors and the three-dimensional spatial model to obtain intersection point clusters.

[0086] For example, density-based clustering algorithms can be used to cluster the intersections of the driver's and passenger's lines of sight in a 3D spatial model into a single cluster.

[0087] Step S142: Perform semantic matching on the target objects corresponding to each intersection point within the intersection point cluster.

[0088] Compare the semantic categories of the objects belonging to each intersection point, such as whether they are all "vehicles" or all "ginkgo trees".

[0089] Step S143: Based on the relationship between the distance between each intersection point and a preset distance threshold, obtain the degree of proximity of the target objects currently being gazed at by the at least two users in the spatial dimension; based on the semantic category of the target objects corresponding to each intersection point, obtain the degree of consistency of the target objects currently being gazed at by the at least two users in the semantic dimension. The smaller the distance between each intersection point relative to the preset distance threshold, the higher the degree of proximity between the two in the spatial dimension; the higher the semantic category of the target objects corresponding to each intersection point is the same or the higher the degree of similarity, the higher the degree of consistency in the semantic dimension.

[0090] Step S144: If the distance between each intersection point is less than or equal to a preset distance threshold, and the semantic category similarity of the target object corresponding to each intersection point is greater than or equal to a preset similarity value, then it is determined that the at least two users are looking at the same target object.

[0091] In actual recognition scenarios, the preset example threshold can be set to 1 meter, and the preset similarity value can be set to 95%.

[0092] For example, if the object category corresponding to the intersection of the driver's line of sight is "ginkgo tree" and the object category corresponding to the intersection of the passenger's line of sight is also "ginkgo tree", and the similarity between the two is 100%, which is greater than the preset similarity value of 95%, and the world coordinate distance between the two intersection points is less than 1 meter, then it is determined that the driver and the passenger are looking at the same target object.

[0093] This embodiment uses two conditions—semantic category consistency and spatial distance between intersection points being less than a threshold—to determine whether multiple users are looking at the same target object. Therefore, this determination considers both the semantic identity of the target, avoiding misclassifying adjacent objects of different categories as the same target, and spatial proximity, avoiding misclassifying two objects of the same category but far apart as the same target. This dual constraint makes the consistency determination both intuitive and highly accurate, thereby reducing false triggers of unified target locking and improving the reliability of multi-HUD collaborative display.

[0094] Example 6 In response to step S14 above, which determines that at least two users are looking at the same target object, this embodiment also provides a weighted fusion determination method based on consistency confidence, which gives a comprehensive score between 0 and 1. This method is more suitable for real and complex scenarios such as slight deviations in the occupants' line of sight and partial occlusion of the target, and can smoothly reflect the degree to which two people are looking at the same target.

[0095] Figure 6 A flowchart illustrating a multi-user head-up display method according to another embodiment of this application is shown below. Figure 6 As shown, in step S14 above, determining that at least two users are gazing at the same target object specifically includes: Step S141a: Calculate the semantic category similarity of the target objects viewed by the at least two users as the degree of consistency of the target objects in the semantic dimension, and calculate the spatial distance proximity of the center points of the target objects viewed by the at least two users as the degree of proximity of the target objects in the spatial dimension.

[0096] Semantic category similarity S reflects the degree of semantic matching: when the semantic category of the target being looked at by the driver and passenger is the same (e.g., both are looking at "vehicle"), S takes the first preset value (e.g., 1); when the categories are different (e.g., one person is looking at "vehicle" and the other is looking at "pedestrian"), S takes the second preset value (e.g., 0).

[0097] The spatial proximity P is obtained by normalizing the Euclidean distance D between the world coordinates of the center points of the two targets.

[0098] Step S142a: Using a weighted fusion method, the consistency confidence is calculated based on the semantic category similarity and the normalized spatial proximity.

[0099]

[0100] For consistency confidence; The larger the value, the stronger the consistency between the two people looking at the same target; α is the weighting coefficient of the semantic category similarity term, controlling the influence of target category matching on confidence; β is the weighting coefficient of the spatial distance term, controlling the influence of spatial proximity on confidence; α + β = 1. For example, if α = 0.5 and β = 0.5, then C is the arithmetic mean of the two terms. This indicates the semantic category similarity of the target objects viewed by the user. For example, if both people are looking at "vehicles", the similarity is 1; if one person is looking at "vehicles" and the other is looking at "pedestrians", the similarity is 0. D represents the Euclidean distance between the center points of the target viewed by different users; This is a threshold for spatial distance, used to normalize the distance term. When D=0, the distance term is 1; when D= When the threshold is exceeded, the distance term is 0; the confidence level will decrease after the threshold is exceeded. .

[0101] Step S143a: Determine whether the at least two users are looking at the same target object based on the consistency confidence level.

[0102] when If the confidence level is greater than or equal to a preset confidence threshold (e.g., 0.7), it is determined that the two people are looking at the same target object; otherwise, they are determined to be different targets.

[0103] This embodiment combines semantic category similarity and spatial distance proximity in a weighted fusion manner into a continuous consistency confidence score. Therefore, compared to the binary hard judgment in Embodiment 5, it can more delicately characterize the consistency of the gazes of two occupants. For example, when the driver and passenger are looking at the same category of the target (S=1) but their center points are slightly different (P=0.8), C=0.9, and they can still be judged as the same target, thus tolerating slight deviations caused by eye position measurement errors or large target size in real-world scenarios. Conversely, when the categories are different (S=0), no matter how close the space is, The accuracy of the judgments is low, effectively avoiding misjudgments of adjacent but different objects. Therefore, this embodiment improves the system's robustness to real-world complex working conditions while maintaining judgment accuracy.

[0104] Example 7 In step S141a, calculating the spatial distance proximity between the center points of the target objects viewed by the at least two users includes: calculating the Euclidean distance between the world coordinates of the center points of the target objects viewed by the at least two users as the spatial distance between the center points of the target objects viewed by the at least two users; and calculating the spatial distance proximity between the center points of the target objects viewed by the at least two users based on the spatial distance and a preset spatial distance threshold.

[0105] For example, the target object that the driver is looking at is Its center point world coordinates The target object being gazed upon by the passenger is... Its center point world coordinates .

[0106]

[0107] D represents the Euclidean distance between the center points of the target being viewed by the driver and passenger, usually expressed in meters per millimeter. The smaller the value, the closer the target is to the target being viewed by the two people. This represents the L2 norm, i.e., the Euclidean distance. and All are three-dimensional points, in the format (x, y, z).

[0108] Spatial proximity P = ,in The preset spatial distance threshold is used; when D=0, P=1, and when... When P=0, the closer the distance, the larger P becomes.

[0109] Since this embodiment quantifies the spatial separation of the target center points gazed at by two occupants using Euclidean distance and normalizes it to a proximity P∈[0,1] using a threshold, the heterogeneous physical distances are unified into the same dimensional range as the semantic category similarity S, allowing the two to be directly added and fused in the weighted formula of Embodiment Six. Thus, the consistency of the spatial dimension and the consistency of the semantic dimension can be jointly expressed under the same confidence framework, ensuring the rigor and computability of the judgment logic.

[0110] Example 8 Once a unified target is locked, its position in the world coordinate system must be accurately projected onto each HUD display plane to ensure "consistent view across multiple HUDs". This embodiment presents a perspective projection implementation based on rigid body transformation and camera intrinsic parameters.

[0111] Step S15 above includes: Step S151: Establish a rigid body transformation from the world coordinate system to the HUD coordinate system, and project the homogeneous coordinates of the unified target in the world coordinate system onto the HUD camera intrinsic parameter matrix through the rigid body transformation to obtain the projected coordinates of the unified target on the HUD display plane.

[0112] Define the HUD projection coordinate system Its origin is located at the optical center of the HUD projection unit; Establish the rigid body transformation matrix from the world coordinate system to the HUD coordinate system. .

[0113] Project the target O in 3D space onto the 2D HUD image plane (u,v):

[0114] Where S is the scaling factor, the homogeneous coordinate scaling coefficient generated by perspective projection, used to normalize the projected coordinates. The final pixel coordinates are obtained using u / s and v / s during calculation. Let (u,v) be the homogeneous pixel coordinates on the HUD plane, where (u,v) is the 2D pixel position of the target on the HUD screen, and the third dimension 1 is the representation of the homogeneous coordinates. K is the intrinsic parameter matrix (3×3) of the HUD camera, used to describe the optical characteristics of the HUD projection unit itself, including focal length, principal point offset, etc. This is the extrinsic parameter matrix (3×4) for transforming points from the world coordinate system to the HUD coordinate system, and the rigid body transformation matrix for converting points from the world coordinate system to the HUD camera coordinate system. A 3×3 rotation matrix describing the orientation of the HUD camera relative to the world coordinate system; : 3×1 translation vector, describing the position of the HUD camera relative to the world coordinate system.

[0115] Let (Xw, Yw, Zw) be the homogeneous coordinates of the target in the world coordinate system. (Xw, Yw, Zw) is the 3D position of the target in the world coordinate system. The 1 at the end indicates the homogeneous coordinate representation, which is convenient for multiplying with the extrinsic parameter matrix.

[0116] Step S152: Based on the projection coordinates, synchronously lock and display the unified target on each of the HUDs.

[0117] like Figure 9As shown, the calculated projection coordinates are sent to the driver's HUD and the passenger's HUD respectively, driving each HUD to render augmented reality annotations (such as selection boxes, labels, and arrows) of the same target at the corresponding pixel positions, so as to realize the synchronous locking display of the same target by multiple HUDs.

[0118] This embodiment uses a unified rigid body transformation [R|t] and camera intrinsic parameter K to map the world coordinates of a unified target onto the pixel plane of each HUD using perspective projection. Therefore, regardless of the differences in the physical installation position and optical parameters of each HUD, the unified target projection coordinates on which the rendering is based all originate from the same world coordinates and the same transformation chain. This mathematically ensures that the display position of the same target on the driver's HUD and the passenger's HUD is strictly aligned with the same object in real space, achieving a consistent technical effect of multi-HUD collaborative display and avoiding target drift and misalignment caused by the individual projection of each HUD.

[0119] Example 9 Based on the aforementioned scheme of multiple HUDs displaying consistent content, rendering the same content to all users would result in either excessive information overwhelming the driver or insufficient information for the passenger. Therefore, this embodiment introduces a rendering strategy that differentiates based on user role.

[0120] In step S15 above, driving the HUD to synchronously lock and display the unified target includes: adopting differentiated HUD rendering strategies for different user roles, and synchronously locking and displaying the unified target on each HUD.

[0121] User roles can be categorized based on their location within the cockpit, such as the driver's seat user and the front passenger seat user.

[0122] For the passenger in the front seat (the initiator of the interaction or the information requester), render a unified target's 3D wireframe highlight, large-font label text (such as "Ginkgo"), and encyclopedia information cards on the passenger-side HUD. For the driver in the front seat (the safety-first party), only render a unified target's outline highlight or arrow indicator on the driver-side HUD to reduce noise and avoid obstructing the real road view.

[0123] In this application specification, the terms "user role" and "user" have the same meaning in the context of the rendering strategy, referring to passengers in different seats such as the driver's seat user or the front passenger seat user.

[0124] This embodiment employs differentiated rendering strategies for different user roles, presenting a simplified safety prompt on the driver's side and rich interactive information on the passenger's side for the same unified target. Therefore, based on the simultaneous locking of the same target by multiple HUDs, it further achieves differentiated display content according to the user's role. As a result, the driver's attention is not distracted by redundant information, ensuring driving safety; while passengers, such as the passenger in the front seat, still receive ample extended information such as encyclopedias and navigation, satisfying their information richness requirements. This application thus resolves the contradiction between "driving safety" and "information richness" that is difficult to balance in existing solutions.

[0125] Example 10 A rendering strategy differentiated by user role is adopted. Considering that the driver's eye position and the optical center of the HUD projection unit are usually not in the same position, if rendered directly using uniform projection coordinates, the actual labeled position seen by the driver will have a parallax offset from their true gaze point. This embodiment introduces driver parallax compensation to eliminate this offset.

[0126] In step S15 above, driving the HUD to synchronously lock and display the unified target includes: when at least two users include the driver's seat user, recalculating the projection position of the unified target on the HUD display plane corresponding to the driver's seat user based on the head posture and eye gaze direction of the driver's seat user, to obtain the parallax-compensated projection position for the driver's seat; and rendering the unified target on the HUD corresponding to the driver's seat user based on the parallax-compensated projection position for the driver's seat.

[0127] The system recalculates the projection position of the unified target on the driver's HUD based on the driver's real-time eye position (head posture and eye center), ensuring the label accurately falls where the driver's actual line of sight is pointing. In optional embodiments, it can also combine binocular or depth information to calculate the target depth using the depth formula Z = f·B / d (where Z is the target depth, f is the focal length, B is the baseline distance, i.e., the distance between the two cameras, and d is the parallax value), thereby achieving more accurate 3D stereoscopic display and parallax compensation. Based on the compensated projection coordinates, the outline of the unified target is highlighted or indicated by an arrow on the driver's HUD.

[0128] This embodiment recalculates the projection position of a unified target on the driver's HUD based on the driver's real-time eye position, thus correcting the projection offset caused by the misalignment between the driver's eye position and the HUD's optical center. This ensures that the rendered annotations on the driver's HUD are accurately aligned with their actual gaze point, achieving a precise "point-and-shoot" indication effect. Furthermore, this compensation is performed while ensuring safe rendering for the driver; therefore, it improves indication accuracy without increasing the driver's visual burden, achieving a balance between safety and accuracy.

[0129] Example 11 Even if all HUDs are rendered based on the same projection coordinates, if the display position does not adjust accordingly when the user moves their head, causing changes in head posture and eye gaze direction, annotation drift will still occur. This embodiment ensures continuous consistency through unified target identification and a dynamic correction mechanism.

[0130] In step S15 above, driving the HUD to synchronously lock and display the unified target includes: assigning a unified target identifier corresponding to the unified target to each HUD, and independently rendering the unified target on each HUD. When a change in the user's head posture and eye gaze direction is detected, the display position of the unified target on each HUD is dynamically corrected according to the change in the user's head posture and eye gaze direction.

[0131] The system assigns a unique object identifier (such as an Object ID) to each locked, unified target. Each HUD (driver's HUD, passenger's HUD) independently retrieves the geometric and semantic information of the target from a shared data source and renders it separately, but the target identifier and coordinate data source are unified. When the user's head moves, causing a change in eye position, the system re-estimates the user's gaze vector and projection coordinates in real time and fine-tunes the display position of the unified target on each HUD accordingly, ensuring that the annotation always follows the actual gaze point.

[0132] This embodiment introduces a unified target identifier to enable all HUDs to share the same data source and respond dynamically to changes in eye position. Therefore, during vehicle movement and changes in the natural posture of the occupants, the display position of the unified target can be continuously aligned with the actual eye position of each HUD user. That is, multiple HUDs not only maintain consistency at a certain static moment, but also maintain consistency during dynamic processes, thereby avoiding label drift caused by the user moving their head, improving the stability of collaborative display and the continuity of user experience.

[0133] Example 12 This application also provides a proactive interactive implementation method of "what you see is what you get". After enabling multiple people to view the same target and display the same information, it further enables interactive interaction on the target, building a closed loop from perception to interaction.

[0134] In an optional embodiment, the method further includes: receiving an interaction request initiated by the user via voice or gesture; and providing the user with data corresponding to the unified target based on the interaction request.

[0135] For example, after the front passenger looks at a tree outside the car, they can ask "What kind of tree is this?" via voice input or by pointing to the tree with a gesture, and the system will receive the interaction request. Based on the locked unified target, the system calls on the cloud knowledge base or local database to obtain the corresponding encyclopedic information (such as plant descriptions), navigation information (such as AR navigation POIs), and other extended services (such as AR navigation, teaching modes, interactive games for objects outside the vehicle, explanations of autonomous driving scenarios, etc.), and synchronizes them to each HUD controller for display via the communication bus.

[0136] Specifically, the above interaction can be expanded into the following closed-loop process: Step 1 (Trigger): The passenger asks "What is this?" or points with a gesture; Step 2 (Locking): The system performs the aforementioned consistency matching to lock the unified target O; Step 3 (Query): The vehicle system accesses the cloud knowledge base to obtain detailed encyclopedic data (Data) for the target O; Step 4 (Distribution): Data is synchronized to the driver's HUD controller and passenger's HUD controller via the vehicle's Ethernet or CAN FD communication bus in a shared data packet containing a unified target identifier, target world coordinates, semantic information and encyclopedia data address. Step 5 (Rendering): The AR labels are displayed simultaneously on both screens. The passenger receives detailed explanations (such as plant descriptions and AR navigation POI information), while the driver receives auxiliary confirmation information (such as minimalist outlines or arrows). The HUD display may include 3D border highlights, label text, and dynamic arrow indicators.

[0137] This embodiment, based on locking onto a unified target, further receives user voice or gesture interaction requests and provides data corresponding to the unified target accordingly. Therefore, it upgrades this application from "passive target recognition and display" to "active, see-and-get interaction." Combined with the aforementioned closed-loop and communication bus distribution mechanism, the driver and passenger can obtain their respective adapted information on the same unified target (passenger provides detailed answers, driver confirms), maintaining consistency across multiple HUDs while meeting the information needs of different roles. Thus, this application forms a complete technical closed loop of "multi-source perception—eye-to-eye intersection—consistency matching—cooperative display—active interaction," significantly enhancing the encyclopedic interactive experience integrating the vehicle's interior and exterior, and solving the problems of single interaction methods and low user participation in existing solutions.

[0138] Example 13 To facilitate productization and decoupling of software and hardware, this embodiment provides a device implementation that corresponds one-to-one with the method.

[0139] Figure 7 This application provides a schematic diagram of the structure of a head-up display device for multiple users, as shown in the embodiments of this application. Figure 7 As shown, the device includes: The gaze vector establishment module 71 is used to establish a gaze vector for each user based on the head posture and eye gaze direction of at least two users in the vehicle. The target object determination module 72 is used to construct a three-dimensional spatial model of the vehicle's external environment, perform target detection on target objects in the three-dimensional spatial model, and assign annotation information to each target object; wherein, the annotation information includes at least one of the following: object identifier, object category, center coordinates, object size, and object orientation; The target object determination module 73 is used to find the intersection of the gaze vector of each user with the three-dimensional space model, and determine the target object currently being gazed at by each user based on the target object at the intersection point; The target consistency judgment module 74 is used to determine, based on the spatial proximity and semantic consistency of the target objects currently being gazed at by the at least two users, that when the at least two users are gazing at the same target object, the target object is locked as a unified target being gazed at by the at least two users. The display driver module 75 is used to map the position of the unified target in the world coordinate system to the display plane of the HUD corresponding to the user who is looking at the unified target through coordinate transformation, and drive the HUD to synchronously lock and display the unified target.

[0140] Optionally, the gaze vector establishment module 71 is further configured to: acquire the user's facial image and extract facial key points from the facial image; use the PnP algorithm to determine the user's head posture based on the facial key points; determine the user's eye center coordinates based on the head posture and a preset facial anatomy model; and calculate the user's gaze vector based on the eye center coordinates and the head rotation matrix.

[0141] Optionally, the target object determination module 72 is also used to acquire multimodal perception data of the external environment collected by the multimodal sensor; Unify the multimodal perception data to a vehicle coordinate system with a preset origin; Based on the multimodal perception data in the vehicle coordinate system, a dense point cloud map of the external environment is generated, and the dense point cloud map is used as the three-dimensional spatial model. The target objects in the dense point cloud map are identified and segmented into instances. Each target object is assigned a target identifier and fitted with a 3D bounding box to obtain a structured target representation containing the annotation information. The structured target representation is used as the target object in the three-dimensional space model.

[0142] Optionally, the target object determination module 73 is further configured to extend the gaze vector of each user to the three-dimensional space model, obtain the intersection point of the gaze vector and the target object in the three-dimensional space model by ray intersection; and determine the target object corresponding to the intersection point closest to the user along the gaze direction as the target object currently being gazed at by the user.

[0143] Optionally, the target consistency judgment module 74 is further configured to perform spatial clustering on the intersection points of the gaze vectors of at least two users and the three-dimensional spatial model to obtain an intersection point cluster; perform semantic matching on the target objects corresponding to each intersection point within the intersection point cluster; obtain the spatial proximity of the target objects currently being gazed at by the at least two users based on the relationship between the distance between each intersection point and a preset distance threshold; obtain the semantic consistency of the target objects currently being gazed at by the at least two users based on the semantic category of the target objects corresponding to each intersection point; and determine that the at least two users are gazing at the same target object if the distance between each intersection point is less than or equal to the preset distance threshold and the semantic category similarity of the target objects corresponding to each intersection point is greater than or equal to a preset similarity value.

[0144] Optionally, the target consistency judgment module 74 is further configured to calculate the semantic category similarity of the target objects viewed by the at least two users as the degree of consistency of the target objects in the semantic dimension, and calculate the spatial distance proximity of the center points of the target objects viewed by the at least two users as the degree of proximity of the target objects in the spatial dimension; using a weighted fusion method, the consistency confidence is calculated based on the semantic category similarity and the normalized spatial proximity; wherein, the consistency confidence is equal to the product of the semantic category similarity weight and the semantic category similarity, plus the product of the spatial distance proximity weight and the spatial distance proximity, and the sum of the semantic category similarity weight and the spatial distance proximity weight is 1; and determine whether the at least two users are viewing the same target object based on the consistency confidence.

[0145] Optionally, the target consistency judgment module 74 is further configured to calculate the Euclidean distance between the world coordinates of the center points of the target objects viewed by the at least two users, as the spatial distance between the center points of the target objects viewed by the at least two users; and calculate the spatial distance proximity between the center points of the target objects viewed by the at least two users based on the spatial distance and a preset spatial distance threshold.

[0146] Optionally, the display driving module 75 is further configured to establish a rigid body transformation from the world coordinate system to the HUD coordinate system, and project the homogeneous coordinates of the unified target in the world coordinate system onto the HUD camera intrinsic parameter matrix through the rigid body transformation to obtain the projected coordinates of the unified target on the HUD display plane; and synchronously lock and display the unified target on each of the HUDs according to the projected coordinates.

[0147] Optionally, the display driver module 75 is further configured to adopt differentiated HUD rendering strategies for different users and synchronously lock and display the unified target on each HUD.

[0148] Optionally, the display driving module 75 is further configured to, when the at least two users include the driver's seat user, recalculate the projection position of the unified target on the HUD display plane corresponding to the driver's seat user based on the eye position of the driver's seat user, to obtain the parallax-compensated projection position for the driver's seat; and render the unified target on the HUD corresponding to the driver's seat user based on the parallax-compensated projection position for the driver's seat.

[0149] Optionally, the display driver module 75 is further configured to assign a unified target identifier corresponding to the unified target to each of the HUDs, and to render the unified target independently on each of the HUDs; when a change in the user's eye position is detected, the display position of the unified target on each of the HUDs is dynamically corrected according to the change in the user's eye position.

[0150] Optionally, the device further includes an interaction module for receiving an interaction request initiated by the user via voice or gesture; and providing the user with data corresponding to the unified target based on the interaction request.

[0151] The modular design of the device in this application corresponds completely to the method embodiment, enabling this application to be implemented purely in software or carried by specific hardware (processor, camera, HUD, bus, etc.), facilitating flexible deployment in different vehicle models and cabin designs. Furthermore, the functional division between each module of the device corresponds one-to-one with the method steps, ensuring consistency between the product implementation and the scope of protection of the claims.

[0152] Example 14 This embodiment provides a vehicle-level implementation of the above method so that the head-up display method can be implemented in the form of a whole vehicle electronic and electrical architecture.

[0153] This application provides a vehicle, including: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus. The processor is configured to execute the multi-user head-up display method described in the above embodiments of this application. The memory stores computer programs, external environment data, user eye position data, target labeling data, etc.; the bus (such as in-vehicle Ethernet or CAN FD) connects the driver's computing unit, the passenger's computing unit, and each HUD controller to distribute shared data packets such as unified target identifiers, world coordinates, and encyclopedia data.

[0154] This embodiment implements the head-up display method of this application in the form of a vehicle processor, and connects each HUD controller and computing unit through a bus. Therefore, the aforementioned cross-view target unified recognition and multi-HUD collaborative display capabilities are directly integrated into the whole vehicle. Users can obtain the technical effects in a real driving environment without additional external devices, which improves the engineering usability and mass production feasibility of the solution.

[0155] To recognize user gaze, in-vehicle sensing modules are deployed within the vehicle cabin, such as high-precision in-vehicle sensing camera arrays covering the driver and passenger areas respectively. These cameras are equipped with infrared illumination to ensure clear facial images are captured even at night or in backlit conditions. To recognize objects outside the vehicle, external sensing modules are installed, such as forward-facing and side-facing cameras and LiDAR (Light Detection and Ranging). When an in-vehicle user (e.g., the passenger) expresses an interaction intention, the system first obtains the user's head posture and eye gaze direction (eye position estimation) through the in-vehicle sensing modules, constructing the user's gaze vector. Subsequently, the system combines this with a 3D semantic map (spatial mapping) constructed by the external sensing modules, extending the gaze vector and calculating its intersection with point clouds or objects in 3D space. This precisely positions the user's "gaze" to a specific physical coordinate in the external environment.

[0156] Example 15 This embodiment provides a computer storage medium to enable the above-described method to be stored, distributed, and deployed as a software product. The computer storage medium stores computer-executable instructions for executing the multi-user head-up display method of any of the above embodiments of this application.

[0157] The computer storage medium may include read-only memory (ROM), random access memory (RAM), magnetic disk, optical disk, etc.

[0158] The computer storage medium of this embodiment stores computer-executable instructions for executing the head-up display method of this application. Therefore, this technical solution can be implemented in essence by means of "software + general hardware platform" and can be circulated and deployed in the form of software products, which reduces the portability cost of the solution between different computing platforms and facilitates the continuous iteration of the aforementioned cross-view consistency matching and collaborative display functions through subsequent firmware upgrades.

[0159] like Figure 8 As shown, this application provides an electronic device including a processor 111, a communication interface 112, a memory 113, and a communication bus 114. The processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114. The memory 113 is used to store computer programs. In one embodiment of this application, when the processor 111 executes the program stored in the memory 113, it implements the head-up display method for multiple users in any of the aforementioned method embodiments.

[0160] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0162] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also mean including the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0163] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features described herein.

Claims

1. A head-up display method for multiple users, characterized in that, include: Based on the head posture and eye gaze direction of at least two users in the vehicle, establish a gaze vector for each user. A three-dimensional spatial model of the vehicle's external environment is constructed, target objects in the three-dimensional spatial model are detected, and each target object is assigned annotation information; wherein, the annotation information includes at least one of the following: object identifier, object category, center coordinates, object size, and object orientation; The intersection of each user's gaze vector with the three-dimensional spatial model is calculated, and the target object currently being gazed at by each user is determined based on the target object at the intersection point; Based on the spatial proximity and semantic consistency of the target objects currently being gazed at by the at least two users, when it is determined that the at least two users are gazing at the same target object, the target object is locked as a unified target being gazed at by the at least two users. The position of the unified target in the world coordinate system is mapped to the display plane of the HUD corresponding to the user who is looking at the unified target through coordinate transformation, thereby driving the HUD to synchronously lock and display the unified target.

2. The method according to claim 1, characterized in that, The step of establishing a gaze vector for each user based on the head posture and eye gaze direction of at least two users inside the vehicle includes: Acquire the user's facial image and extract facial key points from the facial image; The PnP algorithm is used to determine the user's head pose based on the facial key points; Based on the head posture and the preset facial anatomy model, the coordinates of the user's eye center are determined; The user's gaze vector is calculated based on the eye center coordinates and the head rotation matrix.

3. The method according to claim 1, characterized in that, The construction of a three-dimensional spatial model of the vehicle's external environment, target detection of target objects in the three-dimensional spatial model, and assignment of annotation information to each target object include: Acquire multimodal perception data of the external environment collected by multimodal sensors; Unify the multimodal perception data to a vehicle coordinate system with a preset origin; Based on the multimodal perception data in the vehicle coordinate system, a dense point cloud map of the external environment is generated, and the dense point cloud map is used as the three-dimensional spatial model. The target objects in the dense point cloud map are identified and segmented into instances. Each target object is assigned a target identifier and fitted with a 3D bounding box to obtain a structured target representation containing the annotation information. The structured target representation is used as the target object in the three-dimensional space model.

4. The method according to claim 1, characterized in that, The step of intersecting the gaze vector of each user with the three-dimensional spatial model and determining the target object currently being gazed at by each user based on the target object at the intersection point includes: Extend the gaze vector of each user to the three-dimensional space model, and obtain the intersection point between the gaze vector and the target object in the three-dimensional space model by ray intersection; The target object corresponding to the intersection point closest to the user along the line of sight is determined as the target object that the user is currently looking at.

5. The method according to claim 1, characterized in that, When determining that the at least two users are looking at the same target object based on the spatial proximity and semantic consistency of the target objects currently being looked at by each of the at least two users, locking the target object as a unified target being looked at by the at least two users includes: Spatial clustering is performed on the intersection points of at least two users' gaze vectors and the three-dimensional spatial model to obtain intersection point clusters; semantic matching is performed on the target objects corresponding to each intersection point within the intersection point clusters; Based on the relationship between the distance between each intersection point and a preset distance threshold, the degree of proximity of the target objects currently being viewed by the at least two users in the spatial dimension is obtained; based on the semantic category of the target objects corresponding to each intersection point, the degree of consistency of the target objects currently being viewed by the at least two users in the semantic dimension is obtained. If the distance between each intersection point is less than or equal to a preset distance threshold, and the semantic category similarity of the target objects corresponding to each intersection point is greater than or equal to a preset similarity value, then it is determined that the at least two users are looking at the same target object.

6. The method according to claim 1, characterized in that, When determining that the at least two users are looking at the same target object based on the spatial proximity and semantic consistency of the target objects currently being looked at by each of the at least two users, locking the target object as a unified target being looked at by the at least two users includes: The semantic category similarity of the target objects viewed by the at least two users is calculated as the degree of consistency of the target objects in the semantic dimension, and the spatial distance between the center points of the target objects viewed by the at least two users is calculated as the degree of proximity of the target objects in the spatial dimension. A weighted fusion method is used to calculate the consistency confidence based on the semantic category similarity and the normalized spatial proximity; wherein, the consistency confidence is equal to the product of the semantic category similarity weight and the semantic category similarity, plus the product of the spatial proximity weight and the spatial proximity, and the sum of the semantic category similarity weight and the spatial proximity weight is 1; The consistency confidence level is used to determine whether the at least two users are looking at the same target object.

7. The method according to claim 6, characterized in that, Calculating the spatial proximity of the center points of the target objects viewed by at least two users includes: Calculate the Euclidean distance between the world coordinates of the center points of the target objects viewed by the at least two users, and use it as the spatial distance between the center points of the target objects viewed by the at least two users; Based on the spatial distance and the preset spatial distance threshold, the spatial distance proximity of the center points of the target objects being viewed by the at least two users is calculated.

8. The method according to claim 1, characterized in that, The step of mapping the position of the unified target in the world coordinate system to the display plane of the HUD corresponding to the user gazing at the unified target through coordinate transformation, and driving the HUD to synchronously lock and display the unified target, includes: A rigid body transformation is established from the world coordinate system to the HUD coordinate system, and the homogeneous coordinates of the unified target in the world coordinate system are projected onto the HUD camera intrinsic parameter matrix through the rigid body transformation to obtain the projected coordinates of the unified target on the HUD display plane. Based on the projection coordinates, the unified target is synchronously locked and displayed on each of the HUDs.

9. The method according to claim 1, characterized in that, The process of driving the HUD to synchronously lock and display the unified target includes: Different HUD rendering strategies are adopted for different user roles, and the unified target is synchronously locked and displayed on each HUD.

10. The method according to claim 1, characterized in that, The process of driving the HUD to synchronously lock and display the unified target includes: When at least two users include the driver's seat user, the projection position of the unified target on the HUD display plane corresponding to the driver's seat user is recalculated based on the head posture and eye gaze direction of the driver's seat user, so as to obtain the projection position after parallax compensation for the driver's seat. Based on the projection position after parallax compensation for the driver's seat, the unified target is rendered on the HUD corresponding to the driver's seat user.

11. The method according to claim 1, characterized in that, The process of driving the HUD to synchronously lock and display the unified target includes: Assign a unified target identifier corresponding to the unified target to each of the HUDs, and render the unified target independently on each of the HUDs; When a change in the user's head posture and eye gaze direction is detected, the display position of the unified target on each of the HUDs is dynamically corrected according to the change in the user's head posture and eye gaze direction.

12. The method according to claim 1, characterized in that, The method further includes: Receive the user's interaction request initiated by voice or gesture; Based on the interaction request, data corresponding to the unified goal is provided to the user.

13. A head-up display device for multiple users, characterized in that, include: The gaze vector establishment module is used to establish a gaze vector for each user based on the head posture and eye gaze direction of at least two users in the vehicle. The target object determination module is used to construct a three-dimensional spatial model of the vehicle's external environment, perform target detection on target objects in the three-dimensional spatial model, and assign annotation information to each target object; wherein, the annotation information includes at least one of the following: object identifier, object category, center coordinates, object size, and object orientation; The target object determination module is used to find the intersection of each user's gaze vector with the three-dimensional space model, and determine the target object currently being gazed at by each user based on the target object at the intersection point; The target consistency judgment module is used to determine that the at least two users are looking at the same target object based on the degree of proximity of the target objects currently being looked at by the at least two users in the spatial dimension and the degree of consistency in the semantic dimension, and to lock the target object as a unified target being looked at by the at least two users. The display driver module is used to map the position of the unified target in the world coordinate system to the display plane of the HUD corresponding to the user who is looking at the unified target through coordinate transformation, and drive the HUD to synchronously lock and display the unified target.

14. A vehicle comprising: At least one communication interface; At least one bus connected to the at least one communication interface; At least one processor connected to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to execute the head-up display method for multiple users as described in any one of claims 1-12.

15. A computer storage medium storing computer-executable instructions for performing the head-up display method for multiple users as described in any one of claims 1-12.

Images