Augmented reality HUD test method and device, vehicle and computer program product

Abstract

The invention discloses an augmented reality HUD testing method and device, a vehicle and a computer program product, and relates to the technical field of intelligent testing. The method comprises the steps of performing parameter calibration on a target camera configured on a target vehicle according to a to-be-calibrated parameter; acquiring initial video stream data acquired by the calibrated target camera in real time in the running process of the target vehicle; generating augmented reality element data by utilizing the environment perception information, the driving state information and the pose information of the target vehicle; performing data fusion on the augmented reality element data and the initial video stream data to obtain fused video stream data; and controlling a display device associated with the target vehicle to display the fused video stream data in real time. The technical problems of high test cost and low test efficiency in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the field of intelligent testing technology, and more specifically, to a testing method, device, vehicle, and computer program product for augmented reality HUD. Background Art

[0002] With the development of assisted driving technology, in order to improve driving safety, related technologies combine head-up display (HUD) technology with augmented reality (AR) display technology to project vehicle driving information in the form of images on the display screen to reduce safety risks caused by distraction.

[0003] Multiple tests are required during the development phase of ARHUD technology. However, the testing methods in related technologies usually rely on the ARHUD hardware system, resulting in high testing costs. In addition, the AR elements generated in related technologies can only be displayed in specific areas, making collaborative testing difficult during the development phase and resulting in low testing efficiency.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, vehicle, and computer program product for testing an augmented reality HUD to at least address the technical problems of high testing cost and low testing efficiency in related technologies.

[0006] According to one aspect of an embodiment of the present application, a method for testing an augmented reality HUD is provided, comprising: calibrating parameters of a target camera configured on a target vehicle according to parameters to be calibrated, wherein the parameters to be calibrated are determined according to properties of the augmented reality HUD of the vehicle model to be tested, and the target camera is used to simulate the driving observation perspective of the vehicle model to be tested to collect video streams; obtaining initial video stream data collected in real time by the calibrated target camera during the driving process of the target vehicle; generating augmented reality element data by utilizing the environmental perception information, driving status information, and posture information of the target vehicle; fusing the augmented reality element data with the initial video stream data to obtain fused video stream data, wherein the fused video stream data is used to evaluate and verify element display indicators of the augmented reality HUD; and controlling a display device associated with the target vehicle to display the fused video stream data in real time.

[0007] Optionally, the parameters to be calibrated include internal parameters and external parameters; calibrating the parameters of the target camera according to the parameters to be calibrated includes: obtaining the position coordinates of the target camera in the vehicle coordinate system of the target vehicle; determining the external parameters using the position coordinates and the attribute information of the augmented reality HUD; performing external parameter calibration on the target camera according to the external parameters and a preset external parameter calibration algorithm; determining the internal parameters using the factory characteristic parameters and attribute information of the target camera; and performing internal parameter calibration on the target camera according to the internal parameters and a preset internal parameter calibration algorithm.

[0008] Optionally, using environmental perception information, driving status information and posture information to generate augmented reality element data includes: obtaining environmental perception information, driving status information and posture information, wherein the environmental perception information is used to characterize environmental perception elements constructed by multiple sensor perceptions during the driving process of the target vehicle, the driving status information includes navigation data and vehicle control data read during the driving process of the target vehicle, and the posture information is used to characterize the real-time spatial posture of the target vehicle during the driving process; using a preset augmented reality algorithm, environmental perception information, driving status information and posture information to generate augmented reality element data, wherein the augmented reality element data is used to characterize the augmented reality elements to be displayed in the augmented reality HUD, and the preset augmented reality algorithm is used to construct augmented reality elements corresponding to the environmental perception elements.

[0009] Optionally, data fusion is performed on the augmented reality element data and the initial video stream data to obtain fused video stream data, including: using the current parameters of the target camera to perform coordinate conversion on the first display coordinate corresponding to the augmented reality element data to obtain a second display coordinate, wherein the first display coordinate is used to represent the display position of the augmented reality element in the display light machine of the augmented reality HUD, and the second display coordinate is used to represent the position to be displayed of the augmented reality element in the video frame image of the initial video stream data; according to the second display coordinate, data fusion is performed on the augmented reality element data and the initial video stream data, so that the augmented reality element is superimposed and displayed in the video frame image of the initial video stream data, to obtain fused video stream data.

[0010] Optionally, the testing method of the augmented reality HUD also includes: calculating an element display index based on the second display coordinate and the third display coordinate, wherein the third display coordinate is used to characterize the display position of the environmental perception element corresponding to the augmented reality element in the video frame image of the fused video stream data, and the element display index includes at least one of the following: element display fit, element display smoothness and element display delay.

[0011] Optionally, the testing method of the augmented reality HUD also includes: obtaining historical video stream data and historical element data based on the attribute information corresponding to the augmented reality HUD to be tested, wherein the historical video stream data is recorded by a target camera calibrated according to the attribute information during the historical travel of the target vehicle, and the historical element data is generated using the environmental perception information, driving status information and posture information recorded by the target vehicle during the historical travel; using the video frame time of the historical video stream data and the element to-be-played time of the historical element data, the historical video stream data and the historical element data are time-aligned and fused to obtain historical fusion data, wherein the historical fusion data is used to retrospectively verify the element display indicators of the augmented reality HUD.

[0012] Optionally, the display device is the central control screen of the target vehicle; controlling the display device to display the fused video stream data in real time includes: controlling the display device to display the fused video stream data in real time according to the target display mode, wherein the target display mode includes one of the following: pop-up display mode, split-screen display mode.

[0013] According to another aspect of an embodiment of the present application, a testing device for an augmented reality HUD is also provided, including: a calibration module for calibrating parameters of a target camera configured on a target vehicle according to parameters to be calibrated, wherein the parameters to be calibrated are determined according to the properties of the augmented reality HUD of the vehicle model to be tested, and the target camera is used to simulate the driving observation perspective of the vehicle model to be tested to acquire video stream; an acquisition module for acquiring initial video stream data acquired in real time by the calibrated target camera during the driving process of the target vehicle; a generation module for generating augmented reality element data by using the environmental perception information, driving status information and posture information of the target vehicle; a fusion module for performing data fusion on the augmented reality element data and the initial video stream data to obtain fused video stream data, wherein the fused video stream data is used to evaluate and verify the element display indicators of the augmented reality HUD; and a display module for controlling a display device associated with the target vehicle to display the fused video stream data in real time.

[0014] According to another aspect of an embodiment of the present application, a vehicle is further provided, comprising an on-board memory and an on-board processor, wherein a computer program is stored in the on-board memory, and the on-board processor is configured to run the computer program to execute any one of the above methods.

[0015] According to another aspect of an embodiment of the present application, a computer program product is further provided, including a computer program, which implements any of the above methods when executed by a processor.

[0016] It is easy to notice that in the embodiment of the present application, the parameters to be calibrated are determined according to the properties of the augmented reality HUD of the vehicle to be tested, the parameters of the target camera configured on the target vehicle are calibrated according to the parameters to be calibrated, and the calibrated target camera is used to simulate the driving observation angle of the vehicle to be tested to collect video streams, so as to support the verification of the element display effect of the augmented reality HUD on the target vehicle that is not equipped with the ARHUD hardware system; further, the initial video stream data collected in real time by the calibrated target camera during the driving process of the target vehicle is obtained, and the environmental perception information, driving status information and posture information of the target vehicle are used to generate augmented reality element data. , the augmented reality element data and the initial video stream data are fused to obtain fused video stream data, so as to realize the evaluation and verification of the element display indicators of the augmented reality HUD. In the above-mentioned process of evaluating and verifying the element display indicators of the augmented reality HUD, there is no need to rely on the ARHUD hardware system, which can reduce the cost of the test method of the augmented reality HUD; furthermore, the display device associated with the target vehicle is controlled to display the fused video stream data in real time, which overcomes the defect that the AR elements generated in the related technology can only be displayed in a specific area, facilitates multiple testers to observe the fused video stream data, helps collaborative testing, and improves test efficiency. Therefore, the present application achieves the purpose of using the calibrated target camera and the display device associated with the target vehicle, combined with data fusion technology, to efficiently evaluate and verify the element display indicators of the augmented reality HUD at a low cost, thereby achieving the technical effect of reducing test costs and improving test efficiency, and thus solving the technical problems of high test costs and low test efficiency in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and the related descriptions of the embodiments are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a computing terminal for implementing an optional testing method for an augmented reality HUD according to an embodiment of the present application;

[0019] Figure 2 is a flowchart of a method for testing an augmented reality HUD according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of an optional display device according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of another optional display device according to an embodiment of the present application;

[0022] Figure 5 This is a structural block diagram of a testing device for augmented reality HUD according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments only include some embodiments of the present invention, not all embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] According to an embodiment of the present invention, a method embodiment of a method for testing an augmented reality HUD is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] First, the operating environment of the above method embodiment is exemplarily described. Figure 1 This is a hardware structure block diagram of a computing terminal for implementing an optional test method for augmented reality HUD according to an embodiment of the present application, such as Figure 1As shown, a computing terminal 10 (e.g., a computer terminal, a mobile smart terminal, a vehicle terminal, or a cloud computing virtual terminal, etc.) may include: one or more processors 102 (e.g., processors 102a, 102b, ..., 102n), a memory 104 for storing data, and a transmission device 106 for implementing a communication function, wherein the processor 102 may include but is not limited to processing components such as a microprocessor (Microcontroller Unit, abbreviated as MCU) or a programmable logic device (Field Programmable Gate Array, abbreviated as FPGA).

[0027] The computing terminal 10 may further include a display device 110, an input / output interface 108, a Universal Serial Bus (USB) port (the USB port may be used as one of the ports of a computer bus, not shown in the figure), a network interface (not shown in the figure), a power supply (not shown in the figure), and a camera (not shown in the figure).

[0028] It should be noted that the one or more processors 102 and / or other data processing circuits in the computing terminal 10 described above may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computing terminal 10 (or mobile device).

[0029] The memory 104 can be used to store software programs and modules of application software, such as program instructions and data storage devices corresponding to the test method for the augmented reality HUD in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned test method for the augmented reality HUD. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the vehicle terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by a communications provider of the vehicle terminal. In one embodiment, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC) and a network interface. The network adapter can be connected to other network devices via a base station to communicate with the Internet. The transmission device 106 can use a wired and / or wireless network connection to communicate data. In one embodiment, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] The input / output interface can be connected to corresponding input / output devices of the computing terminal 10 to implement input / output functions. Such input / output devices may include, but are not limited to, cursor control devices, keyboards, displays, etc. These input / output devices may be built into the computing terminal 10 or external devices connected to the computing terminal 10.

[0032] It can be understood by those skilled in the art that Figure 1 The structure of the computing terminal 10 shown is only for illustration and does not impose a strict limitation on the structure of the computing terminal 10. For example, the computing terminal 10 may also include Figure 1 More or fewer components than those shown in FIG, or computing terminal 10 may have the same Figure 1 Different categories of components are shown.

[0033] In the aforementioned operating environment, embodiments of the present application provide a method for testing an augmented reality HUD. This method can be used to provide intelligent testing capabilities for pre-defined application scenarios. These pre-defined application scenarios may include the following in the automotive field: system integration testing, after-sales fault diagnosis and repair, and production quality control.

[0034] With the continuous evolution of technology, the test method for augmented reality HUD provided in the embodiment of the present application can also be applied to a wider range of scenarios. By providing an automatic testing solution, the element display indicators of the augmented reality HUD can be evaluated and verified more efficiently, supporting a variety of advanced functions and applications, and improving the efficiency of testing.

[0035] In the above operating environment, the embodiment of the present application provides the following Figure 2 The test method of augmented reality HUD shown in Figure 2 FIG. 1 is a flow chart of a method for testing an augmented reality HUD according to an embodiment of the present application. Figure 2 As shown, the method includes the following implementation steps:

[0036] Step S201: calibrating a target camera configured on a target vehicle according to parameters to be calibrated, wherein the parameters to be calibrated are determined based on the properties of the augmented reality HUD of the vehicle to be tested, and the target camera is used to simulate the driving observation perspective of the vehicle to be tested to capture a video stream;

[0037] Step S202, obtaining initial video stream data collected in real time by the calibrated target camera during the driving process of the target vehicle;

[0038] Step S203, generating augmented reality element data using the target vehicle's environmental perception information, driving state information, and posture information;

[0039] Step S204: performing data fusion on the augmented reality element data and the initial video stream data to obtain fused video stream data, wherein the fused video stream data is used to evaluate and verify element display indicators of the augmented reality HUD;

[0040] Step S205 , controlling the display device associated with the target vehicle to display the fused video stream data in real time.

[0041] The aforementioned vehicle model to be tested is equipped with an augmented reality HUD hardware system. It should be noted that the aforementioned target vehicle may be different from the vehicle model to be tested, and the target vehicle may not be equipped with an augmented reality HUD hardware system.

[0042] The properties of the above-mentioned augmented reality HUD can be used to characterize the working performance of the augmented reality HUD hardware system on the vehicle model to be tested. The properties of the augmented reality HUD may include but are not limited to: the tilt angle of the projection surface, the driver's eye point position, the projection distance, the image resolution, the field of view, the brightness and transparency settings, the color calibration, and the optical distortion correction coefficient. The properties of the augmented reality HUD can be determined based on the parameters of the vehicle model to be tested and the parameters of the augmented reality HUD hardware system. The properties of the augmented reality HUD are directly related to the display effect of the AR elements. By accurately setting the properties of the augmented reality HUD, it can be ensured that the displayed AR elements are accurately aligned with the real world.

[0043] The parameters to be calibrated may include, but are not limited to, focal length, pixel size, image center, camera position relative to the vehicle coordinate system, and camera pose relative to the vehicle coordinate system. These parameters are used to characterize the camera's imaging characteristics and its physical installation on the vehicle.

[0044] The process of calibrating the parameters of the target camera configured on the target vehicle can be achieved by utilizing a calibration algorithm. The calibration algorithm may include, but is not limited to: a dynamic calibration algorithm (for example, controlling the vehicle's driving in a specific scenario, automatically recording and analyzing the changing patterns of the camera-captured images, and then adjusting the parameters to be calibrated to improve the accuracy of the parameters to be calibrated), a machine learning calibration algorithm (for example, utilizing a dataset fused with AR element display positions and video stream data, training a deep learning model to predict and correct camera parameters, and determining the parameters to be calibrated), a lidar-assisted calibration algorithm, and an adaptive calibration algorithm.

[0045] In particular, the aforementioned calibration algorithms can be used individually or in combination, with the appropriate calibration algorithm selected based on test requirements to reduce calibration errors. By calibrating the target camera mounted on the target vehicle according to the parameters to be calibrated, the video stream captured by the target camera accurately reflects the driving observation angle of the test vehicle, providing a reliable data foundation for subsequent evaluation and verification.

[0046] The driving observation angle of the test vehicle model may refer to the driver's forward-looking perspective of the augmented reality display area, which displays elements of the augmented reality head-up display (HUD). This augmented reality display area may include, but is not limited to, a HUD display and an AR device display area. The augmented reality display area is configured within the vehicle, for example, within the vehicle's windshield. Accordingly, the driving observation angle refers to the perspective when looking forward through the vehicle's windshield.

[0047] The target camera may include, but is not limited to, an infrared camera, a fisheye camera, a binocular camera, and a color camera. The target camera may be disposed on an exterior surface of the vehicle (e.g., a roof, a front bumper, a rearview mirror, etc.).

[0048] It should be noted that for some target vehicles equipped with cameras that meet test requirements, this video stream data can be shared during the acquisition of initial video stream data and the subsequent collection of the target vehicle's environmental perception information and driving status information. For target vehicles not equipped with cameras that meet test requirements, a separate target camera can be configured. In this case, this target camera can be used solely to simulate the driver's observation angle of the test vehicle for video stream acquisition. In other words, the primary purpose of the target camera's video stream acquisition is to simulate the driver's observation angle of the test vehicle. It is sufficient to ensure that the initial video stream data captured by the camera can be used to simulate the driver's observation angle of the test vehicle.

[0049] The above-mentioned initial video stream data can refer to the original video signal of the road area around the vehicle (e.g., the road in front of the vehicle) captured in real time by the designated target camera. This initial video stream data can be used to ensure a realistic road condition image. This initial video stream data can be acquired by the target camera. This initial video stream data can include multiple image frames. Using the target camera, the road area around the vehicle is acquired in real time at a preset shooting frequency to obtain initial video stream data; at the same time, this initial video stream data can be transmitted to the on-board processing unit via a network interface. The above-mentioned shooting frequency can be adjusted according to actual needs. The shooting frequency can also be automatically adjusted according to the pitch angle of the vehicle.

[0050] The aforementioned environmental perception information can be used to characterize environmental perception elements constructed by various sensors during the target vehicle's driving process. These sensors may include, but are not limited to, visual sensors (e.g., onboard cameras), onboard radar sensors (e.g., visual radar, lidar), and ultrasonic sensors. In particular, as previously mentioned, if the onboard camera meets the testing requirements, the video stream data can be shared during the acquisition of the initial video stream data, as well as during the collection of environmental perception information and driving status information.

[0051] The driving state information can be used to characterize the real-time motion characteristics of the vehicle. The driving state information can include dynamic data of the target vehicle during driving. In particular, the driving state information can include navigation data and vehicle control data read during the driving process of the target vehicle.

[0052] The above-mentioned posture information may refer to the position and posture of the target vehicle in three-dimensional space. The posture information may be used to characterize the absolute position of the vehicle relative to the earth and the relative rotation relationship between the vehicle body coordinate system and the world coordinate system. The posture information may be obtained through a positioning system (e.g., Global Positioning System, abbreviated as GPS), an inertial measurement unit (IMU) and high-precision positioning technology (e.g., Real-Time Kinematic, abbreviated as RTK). The above-mentioned enhanced display element data may include but is not limited to: a virtual guide light carpet, a virtual arrow indication, a virtual road marking, and a virtual pedestrian.

[0053] The above-mentioned fused video data stream can be used to characterize the display effect observed from the driving observation perspective when the ARHUD technology is used to render virtual road elements. The fused video data stream may include AR virtual road elements, real road elements (such as environmental perception information, driving status information and posture information). The fused video data stream can be obtained using data fusion technology. The above-mentioned data fusion technology may include but is not limited to: coordinate transformation and matching algorithm, feature point matching algorithm, depth map generation algorithm, image fusion algorithm, motion compensation algorithm, point cloud and image fusion algorithm. By using data fusion technology to fuse the augmented reality element data and the initial video stream data, the augmented reality element data can be naturally integrated into the initial video stream data to obtain fused video stream data, which is convenient for evaluating and verifying the element display indicators of the augmented reality HUD.

[0054] These element display metrics may include, but are not limited to, element position accuracy, element size fit, element color fidelity, element display fit, element display smoothness, and element display latency. These element display metrics can characterize the display quality of AR elements from the driver's perspective. These element display metrics can be used to evaluate the integration of AR HUD elements with the real-world environment.

[0055] It is easy to understand that the above-mentioned target vehicle may not be equipped with an augmented reality HUD hardware system. Therefore, through the technical solution of the embodiment of the present application, the element display effect of the augmented reality HUD can be tested without relying on the ARHUD hardware system, which can reduce the testing cost.

[0056] The display device associated with the target vehicle may refer to an output device for presenting fused video stream data. The display device associated with the target vehicle may include, but is not limited to, a central control display screen, an instrument panel display, AR glasses, a tablet computer, a HUD display, and an AR device display area. The reality device can be used to display the fused video stream data to the driver or tester in real time, so that multiple testers can simultaneously and intuitively observe the fusion effect of AR elements (such as navigation arrows, warning information) with the actual driving environment, so as to more accurately and quickly evaluate the element display effect of the augmented reality HUD.

[0057] Through the above steps S201 to S205, in the embodiment of the present application, the parameters to be calibrated are determined according to the properties of the augmented reality HUD of the vehicle to be tested, the parameters of the target camera configured on the target vehicle are calibrated according to the parameters to be calibrated, and the calibrated target camera is used to simulate the driving observation angle of the vehicle to be tested to collect video streams, so as to support the verification of the element display effect of the augmented reality HUD on the target vehicle that is not equipped with the ARHUD hardware system; further, the initial video stream data collected in real time by the calibrated target camera during the driving process of the target vehicle is obtained, and the environmental perception information, driving state information and posture information of the target vehicle are used to generate the augmented reality HUD. Real element data, data fusion of augmented reality element data and initial video stream data, to obtain fused video stream data, so as to realize the evaluation and verification of element display indicators of augmented reality HUD. In the above process of evaluating and verifying element display indicators of augmented reality HUD, there is no need to rely on ARHUD hardware system, which can reduce the cost of the test method of augmented reality HUD; furthermore, the display device associated with the target vehicle is controlled to display the fused video stream data in real time, which overcomes the defect that the AR elements generated in the related technology can only be displayed in a specific area, facilitates multiple testers to observe the fused video stream data, helps collaborative testing, and improves test efficiency. Therefore, the present application achieves the purpose of using the calibrated target camera and the display device associated with the target vehicle, combined with data fusion technology, to efficiently evaluate and verify the element display indicators of augmented reality HUD at a low cost, thereby achieving the technical effect of reducing test costs and improving test efficiency, and thus solving the technical problems of high test costs and low test efficiency in related technologies.

[0058] As an optional implementation, the parameters to be calibrated include internal parameters and external parameters. In the above step S201, calibrating the parameters of the target camera according to the parameters to be calibrated includes the following implementation steps:

[0059] Step S211, obtaining the position coordinates of the target camera in the target vehicle's own vehicle coordinate system;

[0060] Step S212, determining external parameters using the position coordinates and attribute information of the augmented reality HUD;

[0061] Step S213, performing extrinsic calibration on the target camera according to the external parameters and a preset extrinsic calibration algorithm;

[0062] Step S214, using the factory characteristic parameters and attribute information of the target camera to determine the internal parameters;

[0063] Step S215 , performing internal calibration on the target camera according to the internal parameters and a preset internal calibration algorithm.

[0064] The ego-vehicle coordinate system can be a coordinate system based on the vehicle itself. Within this ego-vehicle coordinate system, the X-axis can be defined as pointing forward, the Y-axis to the left, and the Z-axis perpendicular to the ground, pointing upward. Data in the ego-vehicle coordinate system can directly associate the video stream data captured by the target camera with the vehicle. External parameters can be determined based on the position of the target camera within the ego-vehicle coordinate system of the target vehicle, facilitating the subsequent fusion of augmented reality element data with the initial video stream data.

[0065] The aforementioned external parameters may refer to the position and attitude information of the target camera relative to the target vehicle's own coordinate system. These external parameters may include, but are not limited to, the rotation matrix and translation vector of the target camera. These external parameters can be used to characterize the orientation and position of the target camera in space. The aforementioned preset external parameter calibration algorithm may include, but is not limited to, Zhang's calibration method and feature point-based automatic calibration technology. The preset external parameter calibration algorithm can be determined based on the type of target camera and the test accuracy requirements.

[0066] The aforementioned internal parameters may refer to the optical properties of the target camera itself. These internal parameters may include, but are not limited to, focal length, pixel size, image center point, principal point location, and lens distortion coefficient. These internal parameters can be used to ensure the camera's imaging characteristics. The accuracy of the target camera's internal parameters directly affects the position and form of AR elements in the video stream and is the foundation for achieving high-quality AR fusion.

[0067] The preset intrinsic parameter calibration algorithm may include, but is not limited to, a checkerboard calibration method, a distortion model algorithm, and a camera optical parameter calibration optimization algorithm. The preset intrinsic parameter calibration algorithm may be obtained from a pre-collected intrinsic parameter calibration algorithm set.

[0068] In an exemplary application scenario, the above-mentioned external parameters may include the rotation matrix and translation vector of the target camera, and the position coordinates of the target camera in the target vehicle's own vehicle coordinate system (including the coordinate values ​​on the X, Y, and Z axes) and the attribute information of the augmented reality HUD are obtained. According to the position coordinates of the target camera and the attribute information of the augmented reality HUD, the coordinate transformation matrix is ​​applied to convert the AR elements from the HUD coordinate system to the target camera coordinate system, and the rotation matrix and translation vector of the target camera are calculated; further, the target camera is used to shoot the calibration pattern from different angles and positions to obtain multi-view images, and the feature points of the calibration pattern (such as the corner points of the checkerboard) are identified and extracted from the shot images. Using the preset external parameter calibration algorithm, the accuracy of the external parameters is verified by comparing the actual collected image with the predefined calibration pattern, and the external parameter calibration of the target camera is completed.

[0069] Still in the above application scenario, based on the factory characteristic parameters and attribute information of the target camera, the focal length and principal point position in the internal parameters can be determined, and the lens distortion coefficient in the internal parameters can be obtained by shooting a standard chessboard pattern and using computer vision technology for analysis and calculation; further, the preset internal parameter calibration algorithm (such as the chessboard calibration method) is used to perform internal parameter calibration on the target camera according to the internal parameters.

[0070] Through the above steps S211 to S215, in the embodiment of the present application, by utilizing external parameters and internal parameters, combined with a preset calibration algorithm and attribute information of the augmented reality HUD, the target camera can be accurately calibrated, so that the target camera can simulate the driving observation perspective of the vehicle model to be tested to collect video streams, avoid display errors caused by inaccurate camera parameters, and provide reliable input data for subsequent data fusion and display.

[0071] As an optional implementation, in the above step S203, using the environmental perception information, driving state information and posture information to generate augmented reality element data includes the following implementation steps:

[0072] Step S231: Acquire environmental perception information, driving state information, and position information, wherein the environmental perception information is used to represent environmental perception elements constructed by multiple sensors during the driving process of the target vehicle, the driving state information includes navigation data and vehicle control data read during the driving process of the target vehicle, and the position information is used to represent the real-time spatial position of the target vehicle during the driving process;

[0073] Step S232: Generate augmented reality element data using a preset augmented reality algorithm, environmental perception information, driving status information, and posture information, wherein the augmented reality element data is used to represent the augmented reality elements to be displayed in the augmented reality HUD, and the preset augmented reality algorithm is used to construct augmented reality elements corresponding to the environmental perception elements.

[0074] The above-mentioned environmental perception elements may include but are not limited to: road markings (such as lane dividing lines, stop lines, zebra crossings, guide arrows, etc.), traffic signs, traffic lights, pedestrian guardrails, trees, street lights, pedestrians, surrounding vehicles, and weather information. The above-mentioned driving status information may include navigation data and vehicle control data read during the driving process of the target vehicle. The above-mentioned navigation data can be obtained by utilizing high-precision maps and standard-precision maps. The navigation data can also be obtained through the vehicle navigation system. The above-mentioned vehicle control data may include but are not limited to: vehicle speed, acceleration, steering angle, brake status, suspension system status. The vehicle control data can be obtained through the vehicle's electronic control unit.

[0075] The aforementioned posture information may include, but is not limited to, vehicle location information (e.g., vehicle longitude and latitude), altitude, heading angle, pitch angle, and roll angle. The aforementioned preset augmented reality algorithm may include, but is not limited to, a recognition module, a data fusion module, a virtual element modeling module, and a mapping module.

[0076] In an exemplary application scenario, the specific process of generating augmented reality element data can be as follows: using the recognition module in the preset augmented reality algorithm, the real element data corresponding to the augmented reality element data in the environmental perception information and driving status information is identified; using the data fusion module in the preset augmented reality algorithm to perform data fusion on the real element data; further, using the virtual element modeling module in the preset augmented reality algorithm to process the real element data to obtain initial augmented reality element data; then, using the mapping module in the preset augmented reality algorithm, the initial augmented reality element data is spatially mapped based on the posture information, and the coordinate information of the initial augmented reality element data in the ARHUD optical display area is determined, and the initial augmented reality element data and the coordinate information of the initial augmented reality element data in the ARHUD optical display area are used to generate augmented reality element data.

[0077] Through the above steps S231 to S232, in an embodiment of the present application, by utilizing a preset augmented reality algorithm, environmental perception information, driving status information and posture information, augmented reality element data corresponding to actual applications can be generated, which facilitates subsequent data fusion of the augmented reality element data and the initial video stream data to obtain fused video stream data, so as to truly simulate the augmented reality element data that can be observed by the driver in the test vehicle model using ARHUD technology, thereby enhancing the accuracy of the testing method.

[0078] As an optional implementation, in step S204, performing data fusion on the augmented reality element data and the initial video stream data to obtain fused video stream data includes the following implementation steps:

[0079] Step S241: Using the current parameters of the target camera, coordinate conversion is performed on the first display coordinates corresponding to the augmented reality element data to obtain second display coordinates, wherein the first display coordinates are used to represent the display position of the augmented reality element in the display light machine of the augmented reality HUD, and the second display coordinates are used to represent the position of the augmented reality element to be displayed in the video frame image of the initial video stream data;

[0080] Step S242: performing data fusion on the augmented reality element data and the initial video stream data according to the second display coordinates, so that the augmented reality element is superimposed and displayed in the video frame image of the initial video stream data, thereby obtaining fused video stream data.

[0081] The above-mentioned second display coordinates may include pixel horizontal coordinates and pixel vertical coordinates. In particular, in some applications, the above-mentioned second display coordinates may also include depth information for displaying occlusion and perspective effects. The above-mentioned second display coordinates can be obtained through a rendering mapping relationship. The above-mentioned rendering mapping relationship may be a conversion rule between the first display coordinates and the second display coordinates. The rendering mapping relationship may be stored in an on-board memory. The rendering mapping relationship may be obtained by performing a mapping test on the current parameters of the target camera and the first display coordinates. The rendering mapping relationship may also be obtained by utilizing a mapping network model, which may include but is not limited to: a fully connected neural network model, a convolutional neural network model, a recurrent neural network model, a long short-term memory network model, a generative adversarial network model, and a deep residual network model.

[0082] The display position of the AR HUD display in the display light machine, the AR device display area. The AR element data may include but is not limited to: the brightness of the AR element, the color of the AR element, the transparency of the AR element, the type of the AR element, and the shape of the AR element.

[0083] The video frame image may refer to a static image captured by a target camera, and the video frame image may be a basic unit constituting a video stream. The video frame image may include visual information of the environment in front of the vehicle, such as the road, signs, other vehicles, etc.

[0084] In an exemplary application scenario, the pixel coordinate system of the target camera's video frame (i.e., the coordinate system corresponding to the second display coordinate) is determined using the target camera's current parameters. Furthermore, for any first display coordinate corresponding to augmented reality element data, a preset rendering mapping relationship in the vehicle's memory is invoked, and the rendering mapping relationship is used to perform coordinate conversion processing on the first display coordinate corresponding to the aforementioned augmented reality element data to obtain the second display coordinate corresponding to the augmented reality element data. In this way, the augmented reality element data can be converted from the first display coordinate in the HUD coordinate system corresponding to the display optical mechanism of the augmented reality HUD to the second display coordinate in the pixel coordinate system of the target camera's video frame.

[0085] Still in the above application scenario, according to the obtained second display coordinates, the augmented reality element data is rendered to the position corresponding to the second display coordinates in the initial video stream data, and the video frame image of the initial video stream data is adjusted (including adjustment of brightness, color, and transparency) to achieve data fusion of the augmented reality element data and the initial video stream data, so that the augmented reality element is superimposed and displayed in the video frame image of the initial video stream data to obtain fused video stream data.

[0086] Through the above steps S241 to S242, in an embodiment of the present application, the first display coordinates corresponding to the augmented reality element data are converted by utilizing the current parameters of the target camera, so that the first display coordinates in the HUD coordinate system can be converted into the second display coordinates in the pixel coordinate system of the target camera video frame, thereby ensuring the degree of fit between the augmented reality element data and the initial video stream data, so as to obtain more realistic fused video stream data.

[0087] As an optional implementation, the above-mentioned augmented reality HUD testing method further includes the following implementation steps:

[0088] Step S206: Calculate the element display index based on the second display coordinate and the third display coordinate, wherein the third display coordinate is used to characterize the display position of the environmental perception element corresponding to the augmented reality element in the video frame image of the fused video stream data, and the element display index includes at least one of the following: element display fit, element display smoothness, and element display delay.

[0089] The above-mentioned environmental perception elements can be used to represent real road elements. Accordingly, the above-mentioned third display coordinates are the display positions of the real road elements corresponding to the augmented reality elements in the video frame image of the fused video stream data.

[0090] The element display index is used to quantitatively evaluate the fusion effect of the augmented reality element and the corresponding environmental perception element in the video frame image of the fused video stream data. This element display index can be obtained by utilizing an image processing algorithm. Such image processing algorithms may include, but are not limited to, calculation methods based on deep learning models and image feature matching algorithms.

[0091] The element display fit can represent the degree of fit between the augmented reality element and the corresponding environmental perception element in the video frame image of the fused video stream data. The element display fit can include but is not limited to: position fit, size fit, and visual overlap.

[0092] The element display smoothness described above can represent the smoothness and coherence of the display of the augmented reality element and its corresponding environmental perception element within the video frame image of the fused video stream data. This element display smoothness may include, but is not limited to, temporal smoothness and spatial smoothness. This element smoothness can be determined by comparing the amplitude of changes in the augmented reality elements of adjacent video frames in the video stream data.

[0093] The element display delay can represent the time lag between the display of an augmented reality element in a video frame image of the fused video stream data. This element display delay can be used to evaluate the real-time response of the augmented reality element to environmental changes or user operations.

[0094] In an exemplary application scenario, the above-mentioned element display indicators include element display fit, element display smoothness, and element display delay. The element display indicators are calculated based on the second display coordinate and the third display coordinate using an image processing algorithm. Specifically, the process of calculating the element display fit can be: by analyzing the correspondence between the augmented reality element and the environmental perception element corresponding to the augmented reality element, using an image feature matching algorithm to calculate the second display coordinate of the augmented reality element and the position difference between the third display coordinate of the environmental perception element corresponding to the augmented reality element, the element display fit is calculated. It can be understood that the higher the element display fit, the more natural and accurate the fusion of the augmented reality element and the environmental perception element is.

[0095] Still in the above application scenario, the process of calculating the element display smoothness can be: by using the second display coordinates between adjacent frames, calculating the position change of the augmented reality element between adjacent frames (such as Euclidean distance, Manhattan distance, etc.), to calculate the temporal smoothness in the element display smoothness, by using the second display coordinates between adjacent frames, determining the curve of the augmented reality element display position, and determining the spatial smoothness in the element display smoothness by calculating the smoothness index of the curve (such as the curvature change rate). It can be understood that the higher the element display smoothness, the smoother the display effect of the augmented reality element. The process of calculating the element display delay can be: recording timestamps respectively when the augmented reality element data is generated, coordinates are converted, data is fused, and the augmented reality element is displayed, and quantifying the element display delay by calculating the timestamp difference.

[0096] Through the above step S206, in the embodiment of the present application, the element display index is obtained by calculation, and the display effect of the augmented reality element in the video frame image of the fused video stream data can be quantified to improve the accuracy of the augmented reality HUD testing method.

[0097] As an optional implementation, the above-mentioned augmented reality HUD testing method further includes the following implementation steps:

[0098] Step S207: Acquire historical video stream data and historical element data based on the attribute information corresponding to the augmented reality HUD to be tested. The historical video stream data is recorded by a target camera calibrated according to the attribute information during the target vehicle's historical travels, and the historical element data is generated using environmental perception information, driving status information, and posture information recorded by the target vehicle during its historical travels.

[0099] Step S208, using the video frame time of the historical video stream data and the element waiting time of the historical element data, time-align and fuse the historical video stream data and the historical element data to obtain historical fusion data, wherein the historical fusion data is used to retrospectively verify the element display indicators of the augmented reality HUD.

[0100] The historical video stream data may be video stream data recorded by the target camera during the vehicle's past travels. This historical video stream data may include, but is not limited to, views of the environment through which the vehicle traveled, such as road conditions, traffic signs, and weather conditions. The historical video stream data and the historical element data may be stored in an onboard memory.

[0101] It is easy to understand that the above historical video stream data is similar to the initial video stream data in the previous article. The main difference between the two is that the initial video stream data in the previous article is obtained in real time during the current driving process of the vehicle, while the above historical video stream data is recorded by the target camera during the vehicle's past driving process.

[0102] Similarly, the above-mentioned historical element data is similar to the augmented reality element data mentioned above. The main difference between the two is that the augmented reality element data mentioned above is obtained in real time during the current driving process of the vehicle, while the above-mentioned historical element data is calculated during the past driving process of the vehicle. Therefore, the above-mentioned historical video stream data and the above-mentioned historical element data can refer to the relevant description of the initial video stream data and augmented reality element data in the previous article. The stored historical video stream data and historical element data can be used to achieve scene reproduction (i.e., scene playback) to retrospectively verify the element display indicators of the augmented reality HUD.

[0103] The above historical fusion data can be obtained by using an alignment fusion algorithm. In the above process of performing time alignment fusion processing on the historical video stream data and the historical element data, the video frame time of the historical video stream data can be aligned with the element waiting time of the historical element data to ensure that accurate historical fusion data can be obtained.

[0104] In an exemplary application scenario, the specific process of obtaining historical fusion data can be: extracting corresponding timestamp information from each video frame of the historical video stream data and the historical element data, and using these timestamp information to determine the video frame time of the historical video stream data and the element waiting time of the historical element data; then, using the video frame time of the historical video stream data and the element waiting time of the historical element data, aligning the historical video stream data with the historical element data (i.e., determining the correspondence between the historical video stream data and the historical element data) to ensure data synchronization; further, for any video frame, the historical video stream data and the historical element data are fused, and the historical element data corresponding to the video frame is superimposed on the video frame to obtain historical fusion data.

[0105] Through the above steps S207 to S208, in the embodiment of the present application, by obtaining historical video stream data and historical element data, combined with time alignment fusion processing technology to obtain historical fusion data, the historical fusion data can be used to analyze the display effect of the elements of the augmented reality HUD in the vehicle's past journey, thereby realizing the long-term stability evaluation and verification of the display effect of the elements of the augmented reality HUD.

[0106] As an optional implementation, the display device is a central control screen of the target vehicle; in the above step S205, controlling the display device to display the fused video stream data in real time includes the following implementation steps:

[0107] Step S251: Control the display device to display the fused video stream data in real time according to a target display mode, wherein the target display mode includes one of the following: a pop-up display mode and a split-screen display mode.

[0108] The display device may be an onboard display device. The display device may also be an image output device in a test host computer associated with the target vehicle (eg, a smart phone, an external display, a smart watch, etc.).

[0109] In an exemplary application scenario, Figure 3 As shown, the above display device is the central control screen of the target vehicle. The above target display mode is set to pop-up display mode. The fused video stream data (including augmented reality element 1 and augmented reality element 2) can be displayed in real time in the display interface of the real device in the form of a pop-up window. Specifically, the display device is controlled to pop up the pop-up display area. Figure 3 The element 1 marked by the middle triangle represents the augmented reality element 1 currently to be displayed in the driving scene. Figure 3Element 2 marked with a middle circle represents the augmented reality element 2 currently to be displayed in the driving scene, and displays the fused video stream data in real time in the pop-up display area. In particular, the pop-up display area can also include a control component (such as closing the pop-up component); in addition to the pop-up display area, the display interface of the display can also include real-time navigation information (such as map road data, navigation signs, etc.), and can also include voice assistant data (such as the three-dimensional virtual model corresponding to the voice assistant, voice recognition results, etc.).

[0110] In another exemplary application scenario, Figure 4 As shown, the target display mode is set to split-screen display mode, and the fused video stream data (including augmented reality element 1 and augmented reality element 2) can be displayed in real time in the display interface of the real device in the form of split screen. Specifically, the display device is controlled to divide the display interface into multiple display areas (such as Figure 4 As shown, including split-screen display area 1 and split-screen display area 2), Figure 4 The element 1 marked by the middle triangle represents the augmented reality element 1 currently to be displayed in the driving scene. Figure 4 The element 2 marked with a circle in the middle represents the augmented reality element 2 to be displayed in the driving scene, which displays the fused video stream data in real time in one of the display areas (such as Figure 4 In the allocated display area 2 shown, other data content (such as real-time navigation information) can be displayed in the remaining display areas. In particular, the display area corresponding to the fused video stream data can also be adjusted. For example, the display area corresponding to the fused video stream data can be adjusted from split-screen display area 2 to split-screen display area 1.

[0111] Through the above-mentioned step S251, in the embodiment of the present application, the display device associated with the target vehicle is controlled to display the fused video stream data in real time, which overcomes the defect that the AR elements generated in the related technology can only be displayed in a specific area, facilitates multiple testers to observe and evaluate the fused video stream data, helps collaborative testing, and improves test efficiency.

[0112] According to another aspect of the present invention, a test device for augmented reality HUD is provided. Figure 5The test device for the augmented reality HUD includes: a calibration module 501, which is used to calibrate the parameters of a target camera configured on a target vehicle according to parameters to be calibrated, wherein the parameters to be calibrated are determined according to the properties of the augmented reality HUD of the vehicle to be tested, and the target camera is used to simulate the driving observation perspective of the vehicle to be tested to collect video streams; an acquisition module 502, which is used to obtain initial video stream data collected in real time by the calibrated target camera during the driving process of the target vehicle; a generation module 503, which is used to generate augmented reality element data using the environmental perception information, driving state information and posture information of the target vehicle; a fusion module 504, which is used to fuse the augmented reality element data with the initial video stream data to obtain fused video stream data, wherein the fused video stream data is used to evaluate and verify the element display indicators of the augmented reality HUD; and a display module 505, which is used to control a display device associated with the target vehicle to display the fused video stream data in real time.

[0113] It should be noted here that the above-mentioned calibration module 501, acquisition module 502, generation module 503, fusion module 504 and display module 505 correspond to steps S201 to S205 in the embodiment. These five modules are the same as the instances and application scenarios implemented by the corresponding steps, but are not limited to the contents disclosed in the above-mentioned embodiments.

[0114] Optionally, the parameters to be calibrated include internal parameters and external parameters; the above-mentioned calibration module 501 is also used to: obtain the position coordinates of the target camera in the vehicle coordinate system of the target vehicle; determine the external parameters using the position coordinates and the attribute information of the augmented reality HUD; perform external parameter calibration on the target camera based on the external parameters and the preset external parameter calibration algorithm; determine the internal parameters using the factory characteristic parameters and attribute information of the target camera; perform internal parameter calibration on the target camera based on the internal parameters and the preset internal parameter calibration algorithm.

[0115] Optionally, the above-mentioned generation module 503 is also used to: obtain environmental perception information, driving status information and posture information, wherein the environmental perception information is used to characterize the environmental perception elements constructed by multiple sensor perceptions during the driving process of the target vehicle, the driving status information includes navigation data and vehicle control data read during the driving process of the target vehicle, and the posture information is used to characterize the real-time spatial posture of the target vehicle during the driving process; and generate augmented reality element data using a preset augmented reality algorithm, environmental perception information, driving status information and posture information, wherein the augmented reality element data is used to characterize the augmented reality elements to be displayed in the augmented reality HUD, and the preset augmented reality algorithm is used to construct augmented reality elements corresponding to the environmental perception elements.

[0116] Optionally, the above-mentioned fusion module 504 is also used to: use the current parameters of the target camera to perform coordinate conversion on the first display coordinates corresponding to the augmented reality element data to obtain second display coordinates, wherein the first display coordinates are used to represent the display position of the augmented reality element in the display light machine of the augmented reality HUD, and the second display coordinates are used to represent the position to be displayed of the augmented reality element in the video frame image of the initial video stream data; according to the second display coordinates, the augmented reality element data and the initial video stream data are data-fused, so that the augmented reality element is superimposed and displayed in the video frame image of the initial video stream data to obtain fused video stream data.

[0117] Optionally, the above-mentioned augmented reality HUD testing device method also includes a calculation module (not shown in the figure) for calculating an element display index based on the second display coordinate and the third display coordinate, wherein the third display coordinate is used to characterize the display position of the environmental perception element corresponding to the augmented reality element in the video frame image of the fused video stream data, and the element display index includes at least one of the following: element display fit, element display smoothness and element display delay.

[0118] Optionally, the above-mentioned augmented reality HUD testing device also includes a playback module (not shown in the figure) for: obtaining historical video stream data and historical element data according to the attribute information corresponding to the augmented reality HUD to be tested, wherein the historical video stream data is recorded by a target camera calibrated according to the attribute information during the historical journey of the target vehicle, and the historical element data is generated using the environmental perception information, driving status information and posture information recorded by the target vehicle during the historical journey; using the video frame time of the historical video stream data and the element to-be-played time of the historical element data, the historical video stream data and the historical element data are time-aligned and fused to obtain historical fused data, wherein the historical fused data is used to retrospectively verify the element display indicators of the augmented reality HUD.

[0119] Optionally, the display device is the central control screen of the target vehicle; the above-mentioned display module 505 is also used to: control the display device to display the fused video stream data in real time according to the target display mode, wherein the target display mode includes one of the following: pop-up display mode, split-screen display mode.

[0120] According to another aspect of an embodiment of the present application, a vehicle is further provided, comprising an on-board memory and an on-board processor, wherein a computer program is stored in the on-board memory, and the on-board processor is configured to run the computer program to execute any one of the above methods.

[0121] According to another aspect of an embodiment of the present application, a computer program product is further provided, including a computer program, which implements any of the above methods when executed by a processor.

[0122] Optionally, the computer program product may provide an augmented reality HUD testing service based on the augmented reality HUD testing method.

[0123] Alternatively, in this embodiment, the computer program product may be a set of instructions and codes pre-written according to the aforementioned augmented reality HUD testing method. The computer program product may be run on various computer platforms, including personal computers, servers, mobile devices, and the like.

[0124] Optionally, in this embodiment, the instructions and codes corresponding to the computer program product are used to implement the following method steps: calibrating the parameters of the target camera configured on the target vehicle according to the parameters to be calibrated, wherein the parameters to be calibrated are determined according to the properties of the augmented reality HUD of the vehicle model to be tested, and the target camera is used to simulate the driving observation perspective of the vehicle model to be tested to collect video streams; obtaining the initial video stream data collected in real time by the calibrated target camera during the driving process of the target vehicle; generating augmented reality element data by using the environmental perception information, driving status information and posture information of the target vehicle; fusing the augmented reality element data and the initial video stream data to obtain fused video stream data, wherein the fused video stream data is used to evaluate and verify the element display indicators of the augmented reality HUD; and controlling the display device associated with the target vehicle to display the fused video stream data in real time.

[0125] It should be noted that the instructions and codes corresponding to the above-mentioned computer program products can be deployed in a cloud server in the cloud, and a series of data processing and calculations can be performed using the cloud server. With the help of the computing power of the cloud server, the testing efficiency of the augmented reality HUD can be further improved, while also reducing the demand for computing resources of the target vehicle.

[0126] According to another aspect of an embodiment of the present application, a computer-readable storage medium is further provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the storage medium is located is controlled to execute any of the above methods.

[0127] Optionally, the computer storage medium may include but is not limited to: a hard disk drive (HDD), a solid state drive (SSD), a USB flash drive, an optical disc, a memory card, a cloud storage medium, and a network storage device (NAS).

[0128] Optionally, the computer-readable storage medium may be configured to store a computer program for executing the following steps: calibrating the parameters of a target camera configured on a target vehicle according to parameters to be calibrated, wherein the parameters to be calibrated are determined according to the properties of the augmented reality HUD of the vehicle model to be tested, and the target camera is used to simulate the driving observation perspective of the vehicle model to be tested to acquire video streams; obtaining initial video stream data acquired in real time by the calibrated target camera during the driving process of the target vehicle; generating augmented reality element data using the environmental perception information, driving status information, and posture information of the target vehicle; performing data fusion on the augmented reality element data and the initial video stream data to obtain fused video stream data, wherein the fused video stream data is used to evaluate and verify the element display indicators of the augmented reality HUD; and controlling a display device associated with the target vehicle to display the fused video stream data in real time.

[0129] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0130] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, the technical solutions in the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions in the described action combinations, because according to this application, some of the above-mentioned steps can be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in this application specification are preferred embodiments, and the actions and modules involved are not necessarily required to implement the technical solutions of this application.

[0131] In the above-mentioned multiple embodiments of the present application, the description of each embodiment has different emphases. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. For example, the device embodiments described above are only exemplary. For example, the division of multiple modules can be a logical function division, and there can be any other possible division methods when actually implemented in the application scenario; for example, multiple modules (or units or components in the modules) can be combined with each other and can be integrated into another system. For example, some features of the method embodiments described above can be ignored or skipped.

[0133] It should be noted that in the above embodiments, the modules, components or units described as separate parts can be physically separated or physically integrated. The parts displayed as modules or units can be physical modules or physical units, or virtual modules or virtual units. In other words, multiple modules or multiple units can be in the same position, or distributed in multiple positions or multiple spaces. In the application scenario, according to the actual needs of the scenario, some or all of the multiple modules or multiple units can be selected to implement the technical solutions of the embodiments of the present application, thereby achieving the corresponding technical purpose.

[0134] In particular, for an integrated functional module or functional unit, if it is implemented in the form of a software functional unit and sold or used as an independent product, the module or functional unit can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0135] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for testing an augmented reality HUD, characterized in that: include: Calibrate the target camera configured on the target vehicle according to the parameters to be calibrated, wherein the parameters to be calibrated are determined based on the properties of the augmented reality HUD of the test vehicle model, and the target camera is used to simulate the driving observation perspective of the test vehicle model to collect video streams; Acquire initial video stream data collected in real time by the calibrated target camera during the driving process of the target vehicle; Generate augmented reality element data using environmental perception information, driving status information, and posture information of the target vehicle; Performing data fusion on the augmented reality element data and the initial video stream data to obtain fused video stream data, wherein the fused video stream data is used to evaluate and verify element display indicators of the augmented reality HUD; Control a display device associated with the target vehicle to display the fused video stream data in real time.

2. The method for testing augmented reality HUD according to claim 1, wherein: The parameters to be calibrated include internal parameters and external parameters; calibrating the target camera according to the parameters to be calibrated includes: Obtaining the position coordinates of the target camera in the vehicle coordinate system of the target vehicle; Determining the external parameters using the position coordinates and attribute information of the augmented reality HUD; Performing extrinsic calibration on the target camera according to the extrinsic parameters and a preset extrinsic calibration algorithm; Determining the internal parameters using the factory characteristic parameters of the target camera and the attribute information; The target camera is calibrated with internal parameters according to the internal parameters and a preset internal parameter calibration algorithm.

3. The method for testing augmented reality HUD according to claim 1, wherein: Generating the augmented reality element data using the environmental perception information, the driving state information, and the posture information includes: Acquiring the environmental perception information, the driving state information, and the posture information, wherein the environmental perception information is used to represent environmental perception elements constructed by multiple sensor perceptions during the driving process of the target vehicle, the driving state information includes navigation data and vehicle control data read during the driving process of the target vehicle, and the posture information is used to represent the real-time spatial posture of the target vehicle during the driving process; The augmented reality element data is generated using a preset augmented reality algorithm, the environmental perception information, the driving status information and the posture information, wherein the augmented reality element data is used to characterize the augmented reality element to be displayed in the augmented reality HUD, and the preset augmented reality algorithm is used to construct the augmented reality element corresponding to the environmental perception element.

4. The method for testing an augmented reality HUD according to claim 1, wherein: Performing data fusion on the augmented reality element data and the initial video stream data to obtain the fused video stream data includes: Using the current parameters of the target camera, performing coordinate conversion on the first display coordinates corresponding to the augmented reality element data to obtain second display coordinates, wherein the first display coordinates are used to represent the display position of the augmented reality element in the display light machine of the augmented reality HUD, and the second display coordinates are used to represent the position of the augmented reality element to be displayed in the video frame image of the initial video stream data; According to the second display coordinates, the augmented reality element data and the initial video stream data are fused so that the augmented reality element is superimposed and displayed in the video frame image of the initial video stream data, thereby obtaining the fused video stream data.

5. The method for testing augmented reality HUD according to claim 4, wherein: The test method for the augmented reality HUD further includes: The element display index is calculated based on the second display coordinate and the third display coordinate, wherein the third display coordinate is used to characterize the display position of the environmental perception element corresponding to the augmented reality element in the video frame image of the fused video stream data, and the element display index includes at least one of the following: element display fit, element display smoothness and element display delay.

6. The method for testing an augmented reality HUD according to any one of claims 1 to 5, wherein: The test method for the augmented reality HUD further includes: Obtaining historical video stream data and historical element data based on attribute information corresponding to the augmented reality HUD to be tested, wherein the historical video stream data is recorded by the target camera calibrated according to the attribute information during the historical travel of the target vehicle, and the historical element data is generated using the environmental perception information, the driving state information, and the posture information recorded by the target vehicle during the historical travel; The video frame time of the historical video stream data and the element waiting time of the historical element data are used to perform time alignment and fusion processing on the historical video stream data and the historical element data to obtain historical fusion data, wherein the historical fusion data is used to retrospectively verify the element display indicators of the augmented reality HUD.

7. The method for testing an augmented reality HUD according to any one of claims 1 to 5, wherein: The display device is the central control screen of the target vehicle; Controlling the display device to display the fused video stream data in real time includes: The display device is controlled to display the fused video stream data in real time according to a target display mode, wherein the target display mode includes one of the following: a pop-up display mode and a split-screen display mode.

8. A test device for augmented reality HUD, characterized in that: include: a calibration module, configured to calibrate a target camera configured on a target vehicle according to parameters to be calibrated, wherein the parameters to be calibrated are determined based on the properties of the augmented reality HUD of the vehicle to be tested, and the target camera is configured to simulate the driving observation perspective of the vehicle to be tested to capture a video stream; An acquisition module, configured to acquire initial video stream data collected in real time by the calibrated target camera during the driving process of the target vehicle; A generation module, configured to generate augmented reality element data using environmental perception information, driving state information, and posture information of the target vehicle; a fusion module, configured to fuse the augmented reality element data and the initial video stream data to obtain fused video stream data, wherein the fused video stream data is used to evaluate and verify element display indicators of the augmented reality HUD; The display module is used to control the display device associated with the target vehicle to display the fused video stream data in real time.

9. A vehicle, characterized in that: include: a memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 7 when running.

10. A computer program product, characterized in that The method comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images