Heads-up display methods and electronic devices
By transforming the three-dimensional coordinates of road targets into a two-dimensional HUD coordinate system for delay compensation and rendering, the problems of large computational load and error accumulation in AR-HUD are solved, the display accuracy and adaptability are improved, and the millisecond-level rendering requirements are met.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG CRYSTAL OPTECH
- Filing Date
- 2026-02-05
- Publication Date
- 2026-05-26
AI Technical Summary
In existing technologies, delay compensation for AR-HUDs involves large computational loads and progressively accumulating errors, making it difficult to meet millisecond-level rendering requirements, and it also heavily relies on vehicle attitude sensors.
The three-dimensional coordinates of road targets collected by vehicle sensors are transformed into a two-dimensional HUD coordinate system. Delay compensation and rendering are performed using a two-dimensional trajectory array to reduce error accumulation and computational complexity.
It improves the display accuracy and adaptability of AR-HUD, meets the requirements of millisecond-level rendering, and reduces the computational complexity of latency compensation.
Smart Images
Figure CN122089909A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of vehicle technology, and more specifically, to a head-up display method and electronic device. Background Technology
[0002] Augmented Reality Head-up Display (AR-HUD) is a core feature of smart cockpits, overlaying virtual AR markers of road targets such as vehicles and pedestrians onto the driver's field of vision, effectively reducing driving risks. However, due to inherent latency of 100-300 milliseconds in sensor acquisition, data transmission, and image processing, directly rendering the target's current position causes the AR markers to lag behind the real target, affecting display accuracy and driving safety.
[0003] Currently, delay compensation is generally based on a three-dimensional coordinate system. This involves first obtaining the target's three-dimensional position and the vehicle's attitude, then performing multiple three-dimensional coordinate transformations and three-dimensional predictions, and finally projecting the result onto the two-dimensional plane of the HUD. This delay compensation method involves a large amount of computation and the error accumulates step by step, making it difficult to meet the millisecond-level rendering requirements of AR-HUD, and it also heavily relies on the vehicle's attitude sensors.
[0004] Therefore, existing technologies for delay compensation in AR-HUD have certain limitations. Summary of the Invention
[0005] The purpose of this application is to address the shortcomings of the prior art by providing a head-up display method and electronic device to solve the practical problem of the limitations of the prior art in delay compensation for AR-HUD.
[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a head-up display method, the method comprising: The system acquires structured data of various road targets collected by vehicle sensors in real time. The structured data includes the identification, type, and three-dimensional coordinates of the road targets. Based on the structured data of each road target and the pre-built two-dimensional HUD coordinate system, a two-dimensional trajectory array of each road target is generated; Based on the two-dimensional trajectory array of each road target, delay compensation is performed on each road target to obtain the compensated data of each road target. The compensated data includes the road target's identifier, type, validity flag, and compensated two-dimensional coordinates. Based on the compensated data of each road target, the road targets are rendered in real time on the vehicle's AR-HUD.
[0007] As an optional implementation, before acquiring the structured data of each road target collected by vehicle sensors in real time, the following steps are included: Obtain the configuration information of the vehicle's AR-HUD, including resolution and field of view; Based on the configuration information, a two-dimensional HUD coordinate system is constructed.
[0008] As an optional implementation, the step of generating a two-dimensional trajectory array for each road target based on the structured data of each road target and a pre-built two-dimensional HUD coordinate system includes: According to preset conversion parameters, the current frame three-dimensional coordinates of the road target are converted into two-dimensional coordinates in the two-dimensional HUD coordinate system; A two-dimensional trajectory array of the road target is generated based on the road target's identifier and its current frame two-dimensional coordinates.
[0009] As an optional implementation, generating a two-dimensional trajectory array of the road target based on the road target's identifier and its current frame two-dimensional coordinates includes: Based on the identifier of the road target, determine the tracking type of the road target, wherein the tracking type is initial tracking or continuous tracking; Based on the tracking type of the road target and the current frame two-dimensional coordinates of the road target, a two-dimensional trajectory array of the road target is generated.
[0010] As an optional implementation, generating a two-dimensional trajectory array of the road target based on the tracking type of the road target and the current frame two-dimensional coordinates of the road target includes: If the tracking type of the road target is the initial tracking, then the tracking of the road target is initialized, a two-dimensional trajectory array of the road target is constructed, and the current frame two-dimensional coordinates of the road target are stored in the two-dimensional trajectory array of the road target.
[0011] As an optional implementation, generating a two-dimensional trajectory array of the road target based on the tracking type of the road target and the current frame two-dimensional coordinates of the road target includes: If the tracking type of the road target is continuous tracking, then obtain the pre-stored two-dimensional trajectory array of the road target before the update based on the identifier of the road target; Call the add trajectory function to insert the current frame two-dimensional coordinates of the road target into the head of the road target's pre-update two-dimensional trajectory array, and delete the historical frame two-dimensional coordinates of the road target located at the tail of the pre-update two-dimensional trajectory array, to obtain the road target's two-dimensional trajectory array.
[0012] As an optional implementation, the delay compensation for each road target based on the two-dimensional trajectory array of each road target includes: The target compensation algorithm is determined based on the number of frames in the two-dimensional trajectory array of the road target and the preset upper limit of the trajectory length of the two-dimensional trajectory array. Based on the two-dimensional coordinates of each frame in the two-dimensional trajectory array of the road target, the target compensation algorithm is used to perform delay compensation on the road target to obtain the compensated two-dimensional coordinates of the road target.
[0013] As an optional implementation, the step of determining the target compensation algorithm based on the number of frames in the two-dimensional trajectory array of the road target and a preset upper limit for the trajectory length of the two-dimensional trajectory array includes: If the number of frames in the two-dimensional trajectory array of the road target is less than the upper limit of the trajectory length, then the weighted linear extrapolation algorithm will be used as the target compensation algorithm. If the number of frames in the two-dimensional trajectory array of the road target is equal to the upper limit of the trajectory length, then the weighted linear combination algorithm, the two-dimensional kinematic compensation algorithm, or the two-dimensional Kalman filter algorithm will be used as the target compensation algorithm.
[0014] As an optional implementation, the step of performing delay compensation on each road target based on the two-dimensional trajectory array of each road target to obtain the compensated data of each road target includes: Based on the two-dimensional trajectory array of the road target, delay compensation is performed on the road target to obtain the compensated two-dimensional coordinates of the road target; Based on the compensated two-dimensional coordinates of the road target and the configuration information of the vehicle's AR-HUD, the validity flag of the road target is determined; The identifier, type, validity flag, and compensated two-dimensional coordinates of the road target are integrated into the compensated data of the road target.
[0015] Secondly, embodiments of this application provide a head-up display device, the device comprising: The acquisition module is used to acquire structured data of various road targets collected by vehicle sensors in real time. The structured data includes the identification, type and three-dimensional coordinates of the road targets. The generation module is used to generate a two-dimensional trajectory array for each road target based on the structured data of each road target and a pre-built two-dimensional HUD coordinate system. The compensation module is used to perform delay compensation on each road target based on the two-dimensional trajectory array of each road target to obtain the compensated data of each road target. The compensated data includes the road target's identifier, type, validity flag, and compensated two-dimensional coordinates. The rendering module is used to render each road target in real time on the vehicle's AR-HUD based on the compensated data of each road target.
[0016] As an optional implementation, the apparatus further includes: a construction module, the construction module being used for: Obtain the configuration information of the vehicle's AR-HUD, including resolution and field of view; Based on the configuration information, a two-dimensional HUD coordinate system is constructed.
[0017] As an optional implementation, the generation module is specifically used for: According to preset conversion parameters, the current frame three-dimensional coordinates of the road target are converted into two-dimensional coordinates in the two-dimensional HUD coordinate system; A two-dimensional trajectory array of the road target is generated based on the road target's identifier and its current frame two-dimensional coordinates.
[0018] As an optional implementation, the generation module is specifically used for: Based on the identifier of the road target, determine the tracking type of the road target, wherein the tracking type is initial tracking or continuous tracking; Based on the tracking type of the road target and the current frame two-dimensional coordinates of the road target, a two-dimensional trajectory array of the road target is generated.
[0019] As an optional implementation, the generation module is specifically used for: If the tracking type of the road target is the initial tracking, then the tracking of the road target is initialized, a two-dimensional trajectory array of the road target is constructed, and the current frame two-dimensional coordinates of the road target are stored in the two-dimensional trajectory array of the road target.
[0020] As an optional implementation, the generation module is specifically used for: If the tracking type of the road target is continuous tracking, then obtain the pre-stored two-dimensional trajectory array of the road target before the update based on the identifier of the road target; Call the add trajectory function to insert the current frame two-dimensional coordinates of the road target into the head of the road target's pre-update two-dimensional trajectory array, and delete the historical frame two-dimensional coordinates of the road target located at the tail of the pre-update two-dimensional trajectory array, to obtain the road target's two-dimensional trajectory array.
[0021] As an optional implementation, the compensation module is specifically used for: The target compensation algorithm is determined based on the number of frames in the two-dimensional trajectory array of the road target and the preset upper limit of the trajectory length of the two-dimensional trajectory array. Based on the two-dimensional coordinates of each frame in the two-dimensional trajectory array of the road target, the target compensation algorithm is used to perform delay compensation on the road target to obtain the compensated two-dimensional coordinates of the road target.
[0022] As an optional implementation, the compensation module is specifically used for: If the number of frames in the two-dimensional trajectory array of the road target is less than the upper limit of the trajectory length, then the weighted linear extrapolation algorithm will be used as the target compensation algorithm. If the number of frames in the two-dimensional trajectory array of the road target is equal to the upper limit of the trajectory length, then the weighted linear combination algorithm, the two-dimensional kinematic compensation algorithm, or the two-dimensional Kalman filter algorithm will be used as the target compensation algorithm.
[0023] As an optional implementation, the compensation module is specifically used for: Based on the two-dimensional trajectory array of the road target, delay compensation is performed on the road target to obtain the compensated two-dimensional coordinates of the road target; Based on the compensated two-dimensional coordinates of the road target and the configuration information of the vehicle's AR-HUD, the validity flag of the road target is determined; The identifier, type, validity flag, and compensated two-dimensional coordinates of the road target are integrated into the compensated data of the road target.
[0024] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the head-up display method as described in the first aspect above.
[0025] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the head-up display method described in the first aspect above.
[0026] The beneficial effects of this application are: This application provides a head-up display (HUD) method and electronic device that acquires structured data of various road targets collected by vehicle sensors in real time, including the road target's identifier, type, and three-dimensional coordinates. Based on the structured data of each road target and a pre-constructed two-dimensional HUD coordinate system, a two-dimensional trajectory array of each road target is generated. Delay compensation is performed on each road target based on its two-dimensional trajectory array to obtain compensated data, including the road target's identifier, type, validity flag, and compensated two-dimensional coordinates. Based on this compensated data, each road target is rendered in real time on the vehicle's AR-HUD. By mapping each road target from three-dimensional space to the two-dimensional HUD plane and representing the two-dimensional historical trajectory of the road target through the two-dimensional trajectory array, delay compensation and rendering can be performed within the two-dimensional HUD plane. This reduces error accumulation during delay compensation, improves the display accuracy of the AR-HUD, and reduces the computational complexity of delay compensation, adapting to the millisecond-level rendering requirements of the AR-HUD. Attached Figure Description
[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0028] Figure 1 Flowchart of the head-up display method provided in the embodiments of this application Figure 1 ; Figure 2 Flowchart of the head-up display method provided in the embodiments of this application Figure 2 ; Figure 3 Flowchart of the head-up display method provided in the embodiments of this application Figure 3 ; Figure 4 Flowchart of the head-up display method provided in the embodiments of this application Figure 4 ; Figure 5 Flowchart of the head-up display method provided in the embodiments of this application Figure 5 ; Figure 6 Flowchart of the head-up display method provided in the embodiments of this application Figure 6 ; Figure 7 Flowchart of the head-up display method provided in the embodiments of this application Figure 7 ; Figure 8 Flowchart of the head-up display method provided in the embodiments of this application Figure 8 ; Figure 9 This is a module structure diagram of the head-up display device provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.
[0030] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0031] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.
[0032] In the automotive field, AR-HUD can overlay virtual AR markers of road targets such as vehicles and pedestrians onto the driver's field of vision, effectively reducing driving risks. However, due to inherent delays in sensor acquisition, data transmission, and image processing, directly rendering the target's current position causes the AR markers to lag behind the real target, affecting display accuracy and driving safety. Currently, delay compensation is generally based on a three-dimensional coordinate system, which involves large computational loads and cumulative errors, making it difficult to meet the millisecond-level rendering requirements of AR-HUD, and heavily reliant on the vehicle's attitude sensors. In other words, existing technologies for delay compensation in AR-HUD have certain limitations.
[0033] Based on the above-mentioned problems, this application provides a head-up display method that transforms the three-dimensional coordinates of the target into a two-dimensional HUD coordinate system, completes target tracking and delay compensation in a two-dimensional plane, reduces computational complexity, avoids multi-step error accumulation, and improves the display accuracy of AR-HUD and its adaptability to vehicles.
[0034] Figure 1 Flowchart of the head-up display method provided in the embodiments of this application Figure 1 The execution subject of this method can be any electronic device with computing power, which can be integrated into in-vehicle equipment; there are no restrictions on this. For example... Figure 1 As shown, the method includes: S101. Real-time acquisition of structured data of various road targets collected by vehicle sensors. The structured data includes the identification, type and three-dimensional coordinates of the road targets.
[0035] Optionally, the electronic device communicates with the vehicle sensors via a Controller Area Network (CAN) or Ethernet to receive raw data of each road target collected by the vehicle sensors in real time, and performs structured processing on the raw data of each road target to obtain structured data of each road target. The vehicle sensors may include onboard radar, cameras, Global Positioning System (GPS), and other sensors.
[0036] Specifically, the raw data of each road target collected by vehicle sensors are fused to determine the identifier, type, and three-dimensional coordinates of each road target. These identifiers, types, and three-dimensional coordinates are then integrated into structured data of the road targets. The structured data of a road target includes its identifier, type, and three-dimensional coordinates. The type of the road target can be vehicles, pedestrians, etc., and the three-dimensional coordinates of the road target can meet certain requirements. , , In other words, the three-dimensional coordinates of road targets outside this range are invalid data and are removed during the structured processing.
[0037] S102. Based on the structured data of each road target and the pre-built two-dimensional HUD coordinate system, generate a two-dimensional trajectory array for each road target.
[0038] Optionally, the pre-built two-dimensional HUD coordinate system can cover the entire scene, ensuring that the three-dimensional coordinates of each road target can be mapped to the two-dimensional plane where the two-dimensional HUD coordinate system is located.
[0039] Based on the 3D coordinates of each road target and the 2D HUD coordinate system, a 3D-to-2D coordinate transformation is performed to map each road target from 3D space to a 2D plane. According to the identifier of each road target, a 2D trajectory array is generated for each road target. This 2D trajectory array represents the historical 2D trajectory of that road target.
[0040] S103. Based on the two-dimensional trajectory array of each road target, perform delay compensation on each road target to obtain the compensated data of each road target. The compensated data includes the road target's identifier, type, validity flag, and compensated two-dimensional coordinates.
[0041] Optionally, to address the delay caused by vehicle sensor acquisition and data transmission, the two-dimensional position of each road target after the delay is predicted based on the two-dimensional trajectory array of each road target, that is, delay compensation is performed on each road target to obtain the compensated two-dimensional coordinates of each road target.
[0042] The validity of the compensated two-dimensional coordinates of each road target is judged to obtain the validity flag of each road target. Combined with the identifier and type of each road target, the compensated data of each road target is obtained, so that the compensated data of the road target includes the identifier, type, validity flag and compensated two-dimensional coordinates of the road target.
[0043] S104. Based on the compensated data of each road target, render each road target in real time on the vehicle's AR-HUD.
[0044] Optionally, each road target to be rendered is determined based on the validity flag of each road target, and each road target to be rendered is rendered in real time on the vehicle's AR-HUD based on the identifier, type and compensated two-dimensional coordinates of each road target to be rendered.
[0045] For example, if the validity flag of a road target is 1, then the road target is determined to be a road target to be rendered. If the validity flag of a road target is 0, then the road target is determined not to be a road target to be rendered, and the road target cannot be rendered in real time on the vehicle's AR-HUD.
[0046] By mapping each road target from three-dimensional space to a two-dimensional HUD plane, and performing delay compensation and rendering based on the two-dimensional HUD plane, the accumulation of errors in the delay compensation process can be reduced, the display accuracy of AR-HUD can be improved, and the computational complexity of delay compensation can be reduced, thus adapting to the millisecond-level rendering requirements of AR-HUD.
[0047] In this embodiment, structured data of each road target collected by vehicle sensors is acquired in real time, including the road target's identifier, type, and three-dimensional coordinates. Based on the structured data of each road target and a pre-constructed two-dimensional HUD coordinate system, a two-dimensional trajectory array of each road target is generated. Delay compensation is performed on each road target based on its two-dimensional trajectory array to obtain compensated data, including the road target's identifier, type, validity flag, and compensated two-dimensional coordinates. Based on this compensated data, each road target is rendered in real time on the vehicle's AR-HUD. By mapping each road target from three-dimensional space to the two-dimensional HUD plane and representing the two-dimensional historical trajectory of the road target through the two-dimensional trajectory array, delay compensation and rendering can be performed within the two-dimensional HUD plane. This reduces error accumulation during delay compensation, improves the display accuracy of the AR-HUD, and reduces the computational complexity of delay compensation, adapting to the millisecond-level rendering requirements of the AR-HUD.
[0048] Figure 2 Flowchart of the head-up display method provided in the embodiments of this application Figure 2 ,like Figure 2 As shown, before acquiring the structured data of each road target collected by the vehicle sensors in real time in step S101 above, the following steps are included: S201. Obtain the configuration information of the vehicle's AR-HUD, including resolution and field of view.
[0049] Optionally, configuration information can be read from the vehicle's AR-HUD to obtain the resolution and field of view (FOV) of the vehicle's AR-HUD.
[0050] For example, the resolution of the vehicle's AR-HUD can be 746×387 pixels, and the field of view can be 10°×3°.
[0051] S202. Based on the configuration information, construct a two-dimensional HUD coordinate system.
[0052] Optionally, based on the resolution and field of view of the vehicle's AR-HUD, a hypothetical HUD is set that is adapted to the resolution and field of view of the vehicle's AR-HUD. The resolution of the hypothetical HUD can be twice that of the vehicle's AR-HUD, and the field of view of the hypothetical HUD can be six times that of the vehicle's AR-HUD, so as to achieve full scene coverage and adapt to real display.
[0053] If the vehicle's AR-HUD has a resolution of 746×387 pixels and a field of view of 10°×5°, then the hypothetical HUD would have a resolution of 1492×774 pixels and a field of view of 60°×30°.
[0054] When constructing a two-dimensional HUD coordinate system, the origin of the two-dimensional HUD coordinate system is set to the upper left corner of the imaginary HUD plane. middle, The axis is the horizontal pixel axis, with a value range of 0 to 1491 pixels. The axis is the vertical pixel axis, with a value range of 0~773 pixels, to form a complete two-dimensional coordinate mapping space, so that the two-dimensional HUD coordinate system can be used as the core coordinate system for target tracking and delay compensation, covering the two-dimensional mapping range of road targets in the entire scene.
[0055] In this embodiment, the configuration information of the vehicle's AR-HUD, including resolution and field of view, is obtained. Based on the configuration information, a two-dimensional HUD coordinate system is constructed. This ensures that the three-dimensional coordinates of all road targets can be mapped into the two-dimensional HUD coordinate system, covering the two-dimensional mapping range of all road targets in the scene.
[0056] Figure 3 Flowchart of the head-up display method provided in the embodiments of this application Figure 3 In step S102 above, a two-dimensional trajectory array for each road target is generated based on the structured data of each road target and the pre-built two-dimensional HUD coordinate system, including: S301. According to the preset conversion parameters, convert the current frame three-dimensional coordinates of the road target into two-dimensional coordinates in the two-dimensional HUD coordinate system.
[0057] Optionally, the `ar.convertSinglePiont` method of the `ArHudAlgorithm` class is called according to the preset conversion parameters. The method transforms the three-dimensional coordinates of the road target in the current frame (i.e., frame t). Convert to 2D coordinates in a 2D HUD coordinate system . Transformation of the current frame's 3D coordinates for the input road target , The output is the current frame's two-dimensional coordinates of the road target. .
[0058] The preset conversion parameters include the vehicle coordinate system reference, eyebox offset parameters, viewing angle parameters, and AR-HUD display parameters. Specifically, the vehicle coordinate system reference is located at the center of the front of the vehicle, 35 cm above the ground, with the positive X-axis pointing forward, the positive Y-axis pointing to the right, and the positive Z-axis pointing upward. The eyebox offset parameters include... , and This is used to translate the coordinates of road targets in the vehicle coordinate system to the eye-box coordinate system. Viewpoint parameters include the downward viewpoint and yaw angle, used to correct the driver's downward and yaw angles to match the actual field of vision of the human eye. AR-HUD display parameters are used to map the three-dimensional coordinates to a two-dimensional pixel plane, i.e., the two-dimensional HUD plane, through perspective projection.
[0059] S302. Generate a two-dimensional trajectory array of the road target based on the road target's identifier and its current frame two-dimensional coordinates.
[0060] Optionally, the road target identifier is used as the index of the two-dimensional trajectory array of the road target, establishing a mapping relationship between the road target identifier and the two-dimensional coordinate set of the road target, and the current frame two-dimensional coordinates of the road target are... Add it to the two-dimensional coordinate set of the road target to generate a two-dimensional trajectory array of the road target.
[0061] In other words, the two-dimensional trajectory array of a road target is a set of two-dimensional coordinates of a road target stored in a time sequence with the road target's identifier as the index. Each element in the two-dimensional trajectory array is the two-dimensional coordinate of the road target in the two-dimensional HUD coordinate system at each historical frame, so as to represent the historical two-dimensional trajectory of the road target.
[0062] In this embodiment, the current frame three-dimensional coordinates of the road target are converted into two-dimensional coordinates in a two-dimensional HUD coordinate system according to preset conversion parameters. Then, a two-dimensional trajectory array of the road target is generated based on the road target's identifier and its current frame two-dimensional coordinates. Trajectory tracking of the road target is achieved using this two-dimensional trajectory array, which facilitates improved accuracy in delay compensation for the road target.
[0063] Figure 4 Flowchart of the head-up display method provided in the embodiments of this application Figure 4 ,like Figure 4 As shown, in step S302 above, a two-dimensional trajectory array of the road target is generated based on the road target's identifier and its current frame two-dimensional coordinates, including: S401. Based on the identification of the road target, determine the tracking type of the road target, which is either initial tracking or continuous tracking.
[0064] Optionally, after obtaining the current frame two-dimensional coordinates of the road target, the identifier of the road target is compared with the identifiers of each historical road target in the stored set of historical road target identifiers one by one, the matching status of the road target is marked, and the tracking type of the road target is determined as initial tracking or continuous tracking based on the matching status.
[0065] Specifically, if the identifier of the road target exists in the stored set of historical road target identifiers, the matching status of the road target is marked as a successful match, indicating that the road target is a continuously tracked target. If the historical trajectory data of the road target has been stored, the tracking type of the road target is determined to be continuous tracking.
[0066] Correspondingly, if the identifier of the road target is not found in the stored set of historical road target identifiers, the matching status of the road target is marked as a matching failure, indicating that the road target is a newly appeared target, that is, a target being tracked for the first time. Since no historical trajectory data of the road target has been stored, the tracking type of the road target is determined to be a first-time tracking.
[0067] S402. Generate a two-dimensional trajectory array of the road target based on the tracking type of the road target and the two-dimensional coordinates of the road target in the current frame.
[0068] Optionally, based on the tracking type of the road target, a trajectory tracking strategy corresponding to the tracking type of the road target is adopted, combined with the current frame two-dimensional coordinates of the road target. Generate a two-dimensional trajectory array of the road target.
[0069] In this embodiment, the tracking type of the road target is determined as either initial tracking or continuous tracking based on the road target's identifier. Then, a two-dimensional trajectory array of the road target is generated based on the tracking type and the road target's current frame two-dimensional coordinates. This ensures the accuracy and timeliness of the road target's two-dimensional trajectory array.
[0070] As an optional implementation, step S402 above, which generates a two-dimensional trajectory array of the road target based on the tracking type of the road target and the current frame two-dimensional coordinates of the road target, includes: If the tracking type of the road target is initial tracking, then the tracking of the road target is initialized, a two-dimensional trajectory array of the road target is constructed, and the two-dimensional coordinates of the road target in the current frame are stored in the two-dimensional trajectory array of the road target.
[0071] Optionally, if the road target is being tracked for the first time (i.e., a newly appearing road target), then tracking initialization is performed on that road target. Based on the road target's identifier, a dedicated two-dimensional trajectory array is created for that road target, and the current frame's two-dimensional coordinates of the road target are entered into the tracking array. It is stored in a two-dimensional trajectory array of the road target.
[0072] In this embodiment, if the road target is being tracked for the first time, the tracking is initialized, a two-dimensional trajectory array of the road target is constructed, and the current frame two-dimensional coordinates of the road target are stored in the two-dimensional trajectory array. This facilitates subsequent continuous two-dimensional trajectory tracking of the initially tracked road target.
[0073] Figure 5 Flowchart of the head-up display method provided in the embodiments of this application Figure 5 ,like Figure 5 As shown, in step S402 above, a two-dimensional trajectory array of the road target is generated based on the tracking type of the road target and the current frame two-dimensional coordinates of the road target, including: S501. If the tracking type of the road target is continuous tracking, then obtain the pre-stored two-dimensional trajectory array of the road target before the update based on the road target's identifier.
[0074] Optionally, if the tracking type of the road target is continuous tracking, the identifier of the road target is used as an index to accurately retrieve the pre-stored two-dimensional trajectory array of the road target before the update, so as to update the two-dimensional trajectory array of the road target before the update.
[0075] S502. Call the add trajectory function to insert the current frame two-dimensional coordinates of the road target into the head of the road target's pre-update two-dimensional trajectory array, and delete the historical frame two-dimensional coordinates of the road target located at the tail of the pre-update two-dimensional trajectory array to obtain the road target's two-dimensional trajectory array.
[0076] Optionally, the addTrajectory function can be called to add the current frame's two-dimensional coordinates of the road target. Insert the header of the two-dimensional trajectory array of the road target before the update, shift the original two-dimensional coordinates of the historical frames in the array to the right in sequence, delete the two-dimensional coordinates of the historical frames of the road target located at the tail of the two-dimensional trajectory array before the update, and obtain the two-dimensional trajectory array of the road target.
[0077] In other words, the current frame's two-dimensional coordinates are used in the two-dimensional trajectory array of the road target. Overwrite / replace the 2D coordinates of the earliest frame in the array to ensure the timeliness of the 2D trajectory array of the road target.
[0078] In this embodiment, if the road target tracking type is continuous tracking, the pre-stored two-dimensional trajectory array of the road target before the update is obtained according to the road target's identifier. The "add trajectory" function is called to insert the current frame two-dimensional coordinates of the road target into the head of the two-dimensional trajectory array before the update, and delete the historical frame two-dimensional coordinates of the road target located at the tail of the two-dimensional trajectory array before the update, thus obtaining the two-dimensional trajectory array of the road target. This ensures the timeliness of the two-dimensional trajectory array of the road target.
[0079] Figure 6 Flowchart of the head-up display method provided in the embodiments of this application Figure 6 ,like Figure 6As shown, in step S103 above, delay compensation is performed on each road target based on the two-dimensional trajectory array of each road target, including: S601. Determine the target compensation algorithm based on the number of frames in the two-dimensional trajectory array of the road target and the preset upper limit of the trajectory length of the two-dimensional trajectory array.
[0080] Optionally, the number of frames in the two-dimensional trajectory array of the road target is determined, and the target compensation algorithm corresponding to that number of frames is determined based on the number of frames and a preset upper limit for the trajectory length of the two-dimensional trajectory array. The number of frames in the two-dimensional trajectory array of the road target can represent the tracking time for that road target.
[0081] For example, the maximum length of the preset two-dimensional trajectory array can be 5, that is, the two-dimensional trajectory array of the road target can store a maximum of 5 frames of the two-dimensional coordinates of the road target.
[0082] S602. Based on the two-dimensional coordinates of each frame in the two-dimensional trajectory array of the road target, a target compensation algorithm is used to perform delay compensation on the road target to obtain the compensated two-dimensional coordinates of the road target.
[0083] Optionally, based on the two-dimensional coordinates of each frame in the two-dimensional trajectory array of the road target, a target compensation algorithm corresponding to the frame number is used to perform delay compensation on the road target, thereby obtaining the compensated two-dimensional coordinates of the road target. .
[0084] Among them, the actual delay can be determined. With the preset compensation step size Determine the number of compensations Specifically, the actual delay The current rendering trigger timestamp The timestamps of road target acquisition output by the vehicle's Advanced Driver-Assistance Systems (ADAS) The difference, i.e. The timestamp uses a high-precision system clock at the millisecond level to ensure a calculation accuracy of less than or equal to 1ms. Compensation step size. Settings and AR-HUD rendering frame rate Matching, i.e. For example, if the AR-HUD rendering frame rate If the frequency is 10Hz, then the compensation step size is... The time is 100ms. Number of compensation attempts. Actual delay With compensation step size The quotient is rounded up to the nearest integer. For example, if the actual delay The compensation step size is 200ms. If the time is 100ms, then the number of compensation cycles is... The target compensation algorithm is executed twice in a loop to obtain the compensated two-dimensional coordinates of the road target. If the actual delay The delay is 260ms, and the compensation step size is... If the time is 100ms, then the number of compensation cycles is... The compensation two-dimensional coordinates of the road target are obtained by performing three iterations. To ensure that the compensation duration is as close as possible to the actual delay. This further improves the accuracy of compensation.
[0085] In this embodiment, a target compensation algorithm is determined based on the number of frames in the two-dimensional trajectory array of the road target and a preset upper limit for the trajectory length of the two-dimensional trajectory array. Then, based on the two-dimensional coordinates of each frame in the two-dimensional trajectory array of the road target, the target compensation algorithm is used to perform delay compensation on the road target, resulting in the compensated two-dimensional coordinates of the road target. Based on different tracking durations of the road target, differentiated delay compensation for the road target is achieved, improving the accuracy of the compensated two-dimensional coordinates of the road target.
[0086] Figure 7 Flowchart of the head-up display method provided in the embodiments of this application Figure 7 ,like Figure 7 As shown, in step S601 above, the target compensation algorithm is determined based on the number of frames in the two-dimensional trajectory array of the road target and the preset upper limit of the trajectory length of the two-dimensional trajectory array, including: S701. If the number of frames in the two-dimensional trajectory array of the road target is less than the upper limit of the trajectory length, then the weighted linear extrapolation algorithm will be used as the target compensation algorithm.
[0087] Optionally, if the number of frames in the two-dimensional trajectory array of the road target is less than the upper limit of the trajectory length, it indicates that the road target is in the initial stage of tracking and the two-dimensional trajectory data is insufficient. In this case, the weighted linear extrapolation algorithm is used as the target compensation algorithm.
[0088] Specifically, the compensated two-dimensional coordinates of the road target can be determined based on the following formula. :
[0089]
[0090] in, The number of frames in the two-dimensional trajectory array of the road target. and These are the current frame's two-dimensional coordinates of the road target. and These are the compensated two-dimensional coordinates of the road target. and These are the two-dimensional coordinates of the road target in the previous frame of the current frame.
[0091] S702. If the number of frames in the two-dimensional trajectory array of the road target is equal to the upper limit of the trajectory length, then the weighted linear combination algorithm, the two-dimensional kinematic compensation algorithm, or the two-dimensional Kalman filter algorithm shall be used as the target compensation algorithm.
[0092] Optionally, if the number of frames in the two-dimensional trajectory array of the road target is equal to the upper limit of the trajectory length, it indicates that the road target is in a stable tracking stage and there is sufficient two-dimensional trajectory data. In this case, the weighted linear combination algorithm, the two-dimensional kinematic compensation algorithm, or the two-dimensional Kalman filter algorithm can be used as the target compensation algorithm to make full use of the multi-frame trajectory data to improve the delay compensation accuracy.
[0093] Specifically, taking the weighted linear combination algorithm as an example, if the upper limit of the trajectory length of the two-dimensional trajectory array of the road target is 5, then the compensated two-dimensional coordinates of the road target are determined based on the following formula. :
[0094]
[0095] in, and These are the current frame's two-dimensional coordinates of the road target. and These are the compensated two-dimensional coordinates of the road target. and These are the two-dimensional coordinates of the road target in the previous frame of the current frame. and These are the two-dimensional coordinates of the road target in the current frame and the two frames preceding it. and These are the two-dimensional coordinates of the road target in the previous three frames of the current frame. and These are the two-dimensional coordinates of the road target in the previous four frames of the current frame.
[0096] In this embodiment, if the number of frames in the two-dimensional trajectory array of the road target is less than the upper limit of the trajectory length, the weighted linear extrapolation algorithm is used as the target compensation algorithm; if the number of frames in the two-dimensional trajectory array of the road target is equal to the upper limit of the trajectory length, the weighted linear combination algorithm, the two-dimensional kinematic compensation algorithm, or the two-dimensional Kalman filter algorithm is used as the target compensation algorithm. This achieves differentiated delay compensation for the road target based on trajectory duration.
[0097] Figure 8 Flowchart of the head-up display method provided in the embodiments of this application Figure 8 ,like Figure 8 As shown, in step S103 above, delay compensation is performed on each road target based on the two-dimensional trajectory array of each road target to obtain the compensated data of each road target, including: S801. Based on the two-dimensional trajectory array of the road target, perform delay compensation on the road target to obtain the compensated two-dimensional coordinates of the road target.
[0098] Optionally, based on the set of two-dimensional coordinates of the road target in the two-dimensional trajectory array of the road target, delay compensation is performed on the road target to obtain the compensated two-dimensional coordinates of the road target. .
[0099] S802. Determine the validity flag of the road target based on the compensated two-dimensional coordinates of the road target and the configuration information of the vehicle's AR-HUD.
[0100] Optionally, the resolution in the vehicle's AR-HUD configuration information is used as the criterion, based on the compensated two-dimensional coordinates of the road target. Determine the validity flag of the road target.
[0101] Specifically, if the compensated two-dimensional coordinates of the road target Satisfy 0 And 0 If the validity flag of the road target is set to 1, the road target is located within the AR-HUD field of view of the vehicle, and the road target can be rendered subsequently.
[0102] Accordingly, if the compensated two-dimensional coordinates of the road target Not satisfied with 0 And 0 If the validity flag of the road target is set to 0, the road target is not located within the AR-HUD field of view of the vehicle. Therefore, there is no need to render the road target subsequently, reducing rendering pressure and improving display efficiency.
[0103] S803. Integrate the road target's identifier, type, validity flag, and compensated two-dimensional coordinates into compensated data for the road target.
[0104] Optionally, the road target's identifier, type, validity marker, and compensated two-dimensional coordinates can be included. The data is structured and encapsulated according to the vehicle's AR-HUD data protocol to obtain the compensated data of the road target, so that the AR-HUD can determine whether to render the road target based on the validity flag of the road target.
[0105] In this embodiment, delay compensation is performed on the road targets based on their two-dimensional trajectory arrays to obtain their compensated two-dimensional coordinates. Based on these coordinates and the vehicle's AR-HUD configuration information, a validity flag is determined for each road target. The road target's identifier, type, validity flag, and compensated coordinates are then integrated into the compensated data. This facilitates accurate AR-HUD rendering of road targets within the vehicle's AR-HUD field of view, eliminates road targets outside the AR-HUD's field of view, reduces rendering load, and improves display efficiency and accuracy.
[0106] Based on the same inventive concept, this application also provides a head-up display device corresponding to the head-up display method. Since the principle of the device in this application is similar to that of the head-up display method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.
[0107] Figure 9 This is a module structure diagram of the head-up display device provided in the embodiments of this application, such as... Figure 9 As shown, the device includes: The acquisition module 901 is used to acquire structured data of various road targets collected by vehicle sensors in real time. The structured data includes the identification, type and three-dimensional coordinates of the road targets. The generation module 902 is used to generate a two-dimensional trajectory array for each road target based on the structured data of each road target and the pre-built two-dimensional HUD coordinate system. The compensation module 903 is used to perform delay compensation on each road target based on the two-dimensional trajectory array of each road target, and obtain the compensated data of each road target. The compensated data includes the road target's identifier, type, validity flag bit, and compensated two-dimensional coordinates. The rendering module 904 is used to render each road target in real time on the vehicle's AR-HUD based on the compensated data of each road target.
[0108] As an optional implementation, the device further includes: a construction module 905, which is configured to: Obtain the vehicle's AR-HUD configuration information, including resolution and field of view; Based on the configuration information, construct a two-dimensional HUD coordinate system.
[0109] As an optional implementation, the generation module 902 is specifically used for: According to the preset transformation parameters, the current frame three-dimensional coordinates of the road target are converted into two-dimensional coordinates in the two-dimensional HUD coordinate system; Based on the road target's identifier and its current frame 2D coordinates, generate a 2D trajectory array for the road target.
[0110] As an optional implementation, the generation module 902 is specifically used for: Based on the identification of the road target, determine the tracking type of the road target, which is either initial tracking or continuous tracking; Based on the tracking type of the road target and the current frame two-dimensional coordinates of the road target, generate a two-dimensional trajectory array of the road target.
[0111] As an optional implementation, the generation module 902 is specifically used for: If the tracking type of the road target is initial tracking, then the tracking of the road target is initialized, a two-dimensional trajectory array of the road target is constructed, and the two-dimensional coordinates of the road target in the current frame are stored in the two-dimensional trajectory array of the road target.
[0112] As an optional implementation, the generation module 902 is specifically used for: If the tracking type of the road target is continuous tracking, then obtain the pre-stored two-dimensional trajectory array of the road target before the update based on the road target's identifier; Call the Add Track function to insert the current frame 2D coordinates of the road target into the head of the road target's pre-update 2D trajectory array, and delete the historical frame 2D coordinates of the road target located at the tail of the pre-update 2D trajectory array, thus obtaining the road target's 2D trajectory array.
[0113] As an optional implementation, the compensation module 903 is specifically used for: The target compensation algorithm is determined based on the number of frames in the two-dimensional trajectory array of the road target and the preset upper limit of the trajectory length of the two-dimensional trajectory array. Based on the two-dimensional coordinates of each frame in the two-dimensional trajectory array of the road target, a target compensation algorithm is used to perform delay compensation on the road target, and the compensated two-dimensional coordinates of the road target are obtained.
[0114] As an optional implementation, the compensation module 903 is specifically used for: If the number of frames in the two-dimensional trajectory array of the road target is less than the upper limit of the trajectory length, then the weighted linear extrapolation algorithm will be used as the target compensation algorithm. If the number of frames in the two-dimensional trajectory array of the road target is equal to the upper limit of the trajectory length, then the weighted linear combination algorithm, the two-dimensional kinematic compensation algorithm, or the two-dimensional Kalman filter algorithm will be used as the target compensation algorithm.
[0115] As an optional implementation, the compensation module 903 is specifically used for: Based on the two-dimensional trajectory array of the road target, delay compensation is performed on the road target to obtain the compensated two-dimensional coordinates of the road target; The validity flag of the road target is determined based on the compensated two-dimensional coordinates of the road target and the configuration information of the vehicle's AR-HUD. The road target's identifier, type, validity flag, and compensated two-dimensional coordinates are integrated into the compensated data of the road target.
[0116] This application also provides an electronic device, which can be an in-vehicle device integrated into a vehicle, such as... Figure 10 The diagram shown is a schematic representation of the structure of an electronic device provided in an embodiment of this application, including a processor 101, a memory 102, and a bus 103. The memory 102 stores machine-readable instructions executable by the processor 101. When the electronic device is running, the processor 101 communicates with the memory 102 via the bus 103, and the processor 101 executes the machine-readable instructions to perform the steps of the head-up display method in the aforementioned embodiment.
[0117] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the head-up display method described in the foregoing embodiments.
[0118] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.
[0119] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.
[0120] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.
Claims
1. A head-up display method, characterized in that, include: The system acquires structured data of various road targets collected by vehicle sensors in real time. The structured data includes the identification, type, and three-dimensional coordinates of the road targets. Based on the structured data of each road target and the pre-built two-dimensional head-up display (HUD) coordinate system, a two-dimensional trajectory array of each road target is generated; Based on the two-dimensional trajectory array of each road target, delay compensation is performed on each road target to obtain the compensated data of each road target. The compensated data includes the road target's identifier, type, validity flag, and compensated two-dimensional coordinates. Based on the compensated data of each road target, the road targets are rendered in real time on the vehicle's augmented reality head-up display (AR-HUD).
2. The method according to claim 1, characterized in that, Before acquiring the structured data of each road target collected by the vehicle sensors in real time, the following steps are included: Obtain the configuration information of the vehicle's AR-HUD, including resolution and field of view; Based on the configuration information, a two-dimensional HUD coordinate system is constructed.
3. The method according to claim 2, characterized in that, The step of generating a two-dimensional trajectory array for each road target based on the structured data of each road target and a pre-built two-dimensional head-up display (HUD) coordinate system includes: According to preset conversion parameters, the current frame three-dimensional coordinates of the road target are converted into two-dimensional coordinates in the two-dimensional HUD coordinate system; A two-dimensional trajectory array of the road target is generated based on the road target's identifier and its current frame two-dimensional coordinates.
4. The method according to claim 3, characterized in that, The step of generating a two-dimensional trajectory array of the road target based on the road target's identifier and its current frame two-dimensional coordinates includes: Based on the identifier of the road target, determine the tracking type of the road target, wherein the tracking type is initial tracking or continuous tracking; Based on the tracking type of the road target and the current frame two-dimensional coordinates of the road target, a two-dimensional trajectory array of the road target is generated.
5. The method according to claim 4, characterized in that, The step of generating a two-dimensional trajectory array for the road target based on the tracking type of the road target and the current frame two-dimensional coordinates of the road target includes: If the tracking type of the road target is the initial tracking, then the tracking of the road target is initialized, a two-dimensional trajectory array of the road target is constructed, and the current frame two-dimensional coordinates of the road target are stored in the two-dimensional trajectory array of the road target.
6. The method according to claim 4, characterized in that, The step of generating a two-dimensional trajectory array for the road target based on the tracking type of the road target and the current frame two-dimensional coordinates of the road target includes: If the tracking type of the road target is continuous tracking, then obtain the pre-stored two-dimensional trajectory array of the road target before the update based on the identifier of the road target; Call the add trajectory function to insert the current frame two-dimensional coordinates of the road target into the head of the road target's pre-update two-dimensional trajectory array, and delete the historical frame two-dimensional coordinates of the road target located at the tail of the pre-update two-dimensional trajectory array to obtain the road target's two-dimensional trajectory array.
7. The method according to claim 1, characterized in that, The delay compensation for each road target based on the two-dimensional trajectory array of each road target includes: The target compensation algorithm is determined based on the number of frames in the two-dimensional trajectory array of the road target and the preset upper limit of the trajectory length of the two-dimensional trajectory array. Based on the two-dimensional coordinates of each frame in the two-dimensional trajectory array of the road target, the target compensation algorithm is used to perform delay compensation on the road target to obtain the compensated two-dimensional coordinates of the road target.
8. The method according to claim 7, characterized in that, The step of determining the target compensation algorithm based on the number of frames in the two-dimensional trajectory array of the road target and the preset upper limit of the trajectory length of the two-dimensional trajectory array includes: If the number of frames in the two-dimensional trajectory array of the road target is less than the upper limit of the trajectory length, then the weighted linear extrapolation algorithm will be used as the target compensation algorithm. If the number of frames in the two-dimensional trajectory array of the road target is equal to the upper limit of the trajectory length, then the weighted linear combination algorithm, the two-dimensional kinematic compensation algorithm, or the two-dimensional Kalman filter algorithm will be used as the target compensation algorithm.
9. The method according to claim 1, characterized in that, The step of performing delay compensation on each road target based on the two-dimensional trajectory array of each road target to obtain the compensated data of each road target includes: Based on the two-dimensional trajectory array of the road target, delay compensation is performed on the road target to obtain the compensated two-dimensional coordinates of the road target; Based on the compensated two-dimensional coordinates of the road target and the configuration information of the vehicle's AR-HUD, the validity flag of the road target is determined; The identifier, type, validity flag, and compensated two-dimensional coordinates of the road target are integrated into the compensated data of the road target.
10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the head-up display method as described in any one of claims 1 to 9.