Automobile instrument test method, system, device and medium based on digital twinning
By generating structured test cases using digital twin technology and combining them with multi-sensor fusion, the problems of low efficiency and insufficient virtual-real interaction in existing automotive instrument testing methods are solved, enabling efficient and accurate self-optimization of testing strategies.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHANGSHA YIFENG AUTOMOBILE TECHNOLOGY CO LTD
- Filing Date
- 2026-01-09
- Publication Date
- 2026-07-14
AI Technical Summary
Existing automotive instrument testing methods are inefficient, have insufficient coverage of test scenarios, lack virtual-real interaction capabilities, cannot self-optimize test strategies, and are costly.
A digital twin-based testing approach is adopted, which utilizes a large language model to generate structured test cases, combines a high-fidelity rendering engine and a real-time simulation engine to build a virtual instrument model, optimizes the pose of the physical entity and the virtual model through multi-sensor fusion, and optimizes the testing strategy through reinforcement learning.
It achieves efficient test case generation, reduces test preparation time, improves test efficiency and accuracy, can self-optimize test strategies, and enhances the consistency of virtual and real interactions and the accuracy of test results.
Smart Images
Figure CN121898803B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of automotive testing technology, specifically relating to a method, system, equipment, and medium for testing automotive instruments based on digital twins. Background Technology
[0002] With the continuous improvement of the digitalization and intelligence of automobiles, the human-machine interface of automobile instruments has evolved from traditional mechanical pointer instruments to highly integrated digital intelligent interactive terminals. Its functions cover multiple dimensions such as core driving information display, vehicle status monitoring, navigation assistance, and entertainment control, which puts forward higher requirements for the comprehensiveness, accuracy and intelligence of testing.
[0003] Existing automotive instrument cluster (HMI) testing methods mainly include hardware-in-the-loop (HIL) testing, image recognition automated testing, and software simulation testing. HIL testing simulates the vehicle's environmental signals to test the actual instrument cluster hardware, but it is limited by physical hardware dependencies, resulting in insufficient test scenario coverage and high testing costs. Image recognition automated testing uses cameras or screen screenshots to capture images of the instrument cluster display and then uses traditional image processing algorithms to identify UI elements.
[0004] In existing automotive instrument cluster testing technologies, most automated testing methods write test scripts for fixed scenarios, making it difficult to adapt to frequent version iterations of HMI software. At the same time, the virtual-physical interaction capability is insufficient, the data synchronization accuracy between the virtual test environment and the physical entity in traditional testing methods is low, and existing digital twin technology has performance bottlenecks in multi-user access and large-scale scene loading, affecting testing efficiency. Furthermore, existing testing systems are limited to the judgment of preset rules, and cannot perform root cause analysis on anomalies found during the testing process, nor can they self-optimize testing strategies based on historical test data. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the low efficiency of existing automotive instrument testing methods, thereby providing an automotive instrument testing method, system, equipment and medium based on digital twins.
[0006] A digital twin-based automotive instrument testing method includes the following steps:
[0007] Step S1: Receive test requirements and design documents described in natural language through the large language model, and generate structured test cases containing test case names, preconditions, test steps and expected results;
[0008] Step S2: Construct a virtual instrument model that matches the visual effect of the physical instrument based on a high-fidelity rendering engine; construct a vehicle dynamics model, an environment model, and a sensor model based on a real-time simulation engine; and achieve accurate mapping and data fusion of the dual-engine coordinate system through a scene transformation matrix.
[0009] Step S3: The test management platform schedules the real-time simulation engine to generate vehicle status parameters based on the test cases, and synchronizes them to the high-fidelity rendering engine to generate the expected virtual instrument screen; at the same time, the vehicle status parameters are converted into CAN bus signals through the signal simulation device and injected into the physical instrument to trigger the physical instrument to update the display.
[0010] Step S4: Using sensor arrays such as industrial cameras, inertial measurement units, and lidar, collect real-time display images, status data, and environmental data from physical instruments, and combine them with virtual scene data to form a multimodal dataset;
[0011] Step S5: Perform fusion processing on multi-source sensor data to achieve pose optimization of physical entities and virtual models;
[0012] Step S6: Compare the expected image of the virtual instrument with the actual image of the physical instrument. Analyze the image differences, CAN signal data, and system logs using a large language model to determine the cause of the defect and the level of the defect.
[0013] Step S7: Record test data, optimize the test strategy based on historical data through reinforcement learning, and update the test case library and large language model parameters.
[0014] Furthermore, the large language model mentioned in step S1 is fine-tuned and optimized using data from the automotive instrument testing field. The fine-tuning methods include full parameter fine-tuning, supervised fine-tuning, and reinforcement learning based on human feedback. During the test case generation process, the prompts received by the large language model include system roles, task contexts, specific requirements, and output formats. The system role is set as a professional automotive HMI test engineer, and the task context provides the version of the instrument HMI under test and key parameters.
[0015] Furthermore, the high-fidelity rendering engine mentioned in step S2 is built on Unreal Engine 5 and is responsible for the visual presentation of the virtual instrument HMI, including lighting effects, material textures, and animation transitions; the real-time simulation engine is built on a GIS engine or professional simulation software and is responsible for running vehicle dynamics models, environmental models, and sensor models, and processing physical logic and real-time data; the scene transformation matrix is a homogeneous coordinate transformation matrix; the parameters of the scene transformation matrix are determined through a hand-eye calibration process, collecting multiple sets of corresponding points in physical space and virtual space, and solving for the optimal parameters through a singular value decomposition algorithm.
[0016] Furthermore, in step S5, pose optimization is achieved through a combination of multi-sensor fusion and vision-assisted absolute positioning. Specifically, this includes: synchronously collecting data through industrial cameras, LiDAR, and inertial measurement units deployed around the physical instrument; the industrial camera provides visual texture information for identifying instrument UI elements; the LiDAR provides high-precision depth and geometric information; the inertial measurement unit provides equipment motion attitude data to assist in pose estimation; and using Kalman filtering or extended Kalman filtering algorithms to fuse multi-source heterogeneous data and output six-degree-of-freedom pose data.
[0017] Predefined calibration targets are detected from acquired images using a pre-trained deep learning model. Based on a digital twin model that stores the precise three-dimensional coordinates of the calibration targets, the correspondence between the two-dimensional pixel coordinates of the images and the three-dimensional world coordinates is established. The optimal pose result is solved based on a least squares estimation optimization framework.
[0018] Furthermore, sensor data, control commands, and model status are synchronized with high-precision timestamps via the PTP precision clock synchronization protocol or hardware triggers. The data processing and synchronization modules sort, align, and fuse the data based on these timestamps.
[0019] Furthermore, the vehicle status parameters mentioned in step S3 include vehicle motion status, body system status, power system status, navigation and ADAS information, alarm and fault information; the CAN bus signal conversion process is completed according to the predefined CAN database file. First, the identifier, data length code, data field position and encoding rules of each CAN signal are defined in the DBC file. Then, the physical values of the vehicle status parameters are converted into digital quantities according to the encoding rules. The digital quantities are then converted into hexadecimal bytes and filled into the CAN message data field in the specified position. Finally, a complete CAN data frame containing the identifier, data length code, data field and check bit is assembled and sent to the CAN bus in a specific cycle through the CAN card or USB-CAN adapter.
[0020] Furthermore, the closed-loop self-optimization process in step S7 includes data collection and feature extraction, pattern recognition and root cause analysis, strategy optimization and knowledge base update, and model parameter iterative update. In data collection and feature extraction, multi-dimensional raw data such as CAN bus signal timing, pixel-level differences between virtual and actual screens, system log information, and test step execution time are recorded for each test. The large language model extracts key features from the data and associates log error information with the test context. In the strategy optimization and knowledge base update stage, the system autonomously adjusts the test process, increasing the test priority of high-frequency failure or edge scenarios, increasing the number of stress test cycles, automatically generating new test scenarios or supplementing existing test cases, and inputting the analysis results, optimization strategies, and new test cases into the knowledge base. In the model parameter iterative update stage, historical test data is organized into structured instruction-response pairs, and the large language model is supervised and fine-tuned. Through reinforcement learning, improved defect detection rate and shortened test cycle are used as rewards to optimize model parameters.
[0021] A digital twin-based automotive instrument testing system, used to implement the aforementioned digital twin-based automotive instrument testing method, is characterized by comprising a physical entity layer, a virtual twin layer, an intelligent analysis layer, a control core, and a test management platform.
[0022] The physical entity layer includes automotive instrument hardware, signal simulation equipment, and data acquisition equipment. The signal simulation equipment includes CAN bus tools, CAN signal simulation test boxes, and programmable power supplies. The data acquisition equipment includes industrial cameras, lidar, and IMU.
[0023] The virtual twin layer includes a high-fidelity rendering engine and a real-time simulation engine. The high-fidelity rendering engine is based on Unreal Engine 5 to realize the rendering of the instrument HMI's lighting, materials and animations. The real-time simulation engine realizes the real-time calculation of vehicle dynamics and environment.
[0024] The intelligent analysis layer integrates a domain-fine-tuned large language model for test case generation, multimodal data fusion analysis, defect root cause localization, and automatic test report generation.
[0025] The control core includes a digital twin synchronization module and a pose optimization module. The digital twin synchronization module is used for dual-engine data synchronization and virtual-real state calibration. The pose optimization module uses Kalman filtering and visual-assisted positioning technology to solve the pose.
[0026] The test management platform provides a graphical user interface to control the test process, monitor the test process, store data, and display test reports.
[0027] A computer device includes a processor and a memory connected to the processor. The memory stores one or more programs that are executed by the processor to perform the steps in the above-described digital twin-based automotive instrument testing method.
[0028] A computer-readable storage medium storing one or more programs, which are executed by a processor to implement the steps in the above-described digital twin-based automotive instrument testing method.
[0029] Beneficial effects:
[0030] This invention employs a dual-engine architecture where a high-fidelity rendering engine and a real-time simulation engine work in tandem, solving the problem that traditional single engines cannot simultaneously meet the needs of large-scale scenes and detailed rendering. Seamless switching and data fusion between the two engines are achieved through a scene transformation matrix, ensuring consistency between the virtual environment and physical entities. A large language model is introduced into the test case generation and result analysis stages, enabling natural language understanding-level test script generation and anomaly analysis. Simultaneously, leveraging the code generation capabilities of the large language model, test requirements are automatically transformed into executable test scripts, significantly reducing the workload of test script development. Furthermore, the system can continuously optimize test strategies based on historical test data, achieving self-improvement in testing efficiency. Attached Figure Description
[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0032] Figure 1 This is a schematic flowchart illustrating the main method steps of the present invention. Detailed Implementation
[0033] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.
[0034] In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Unless otherwise explicitly specified and limited, the terms "installed," "connected," "linked," "fixed," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly specified. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.
[0035] Reference Figure 1 As shown, this embodiment provides a method for testing automotive instruments based on digital twins, including the following steps:
[0036] Step S1: Receive test requirements and design documents described in natural language through the large language model, and generate structured test cases containing test case names, preconditions, test steps and expected results;
[0037] Step S2: Construct a virtual instrument model that matches the visual effect of the physical instrument based on a high-fidelity rendering engine; construct a vehicle dynamics model, an environment model, and a sensor model based on a real-time simulation engine; and achieve accurate mapping and data fusion of the dual-engine coordinate system through a scene transformation matrix.
[0038] Step S3: The test management platform schedules the real-time simulation engine to generate vehicle status parameters based on the test cases, and synchronizes them to the high-fidelity rendering engine to generate the expected virtual instrument screen; at the same time, the vehicle status parameters are converted into CAN bus signals through the signal simulation device and injected into the physical instrument to trigger the physical instrument to update the display.
[0039] Step S4: Using sensor arrays such as industrial cameras, inertial measurement units, and lidar, collect real-time display images, status data, and environmental data from physical instruments, and combine them with virtual scene data to form a multimodal dataset;
[0040] Step S5: Perform fusion processing on multi-source sensor data to achieve pose optimization of physical entities and virtual models;
[0041] Step S6: Compare the expected image of the virtual instrument with the actual image of the physical instrument. Analyze the image differences, CAN signal data, and system logs using a large language model to determine the cause of the defect and the level of the defect.
[0042] Step S7: Record test data, optimize the test strategy based on historical data through reinforcement learning, and update the test case library and large language model parameters.
[0043] Specifically, in step S1, the test requirements and design documents described in natural language are received through a large language model fine-tuned using data from the automotive instrument testing domain, generating structured and executable test cases. Fine-tuning the large language model is crucial for achieving accurate test case generation. It employs a combination of full-parameter fine-tuning, supervised fine-tuning, and reinforcement learning based on human feedback: Full-parameter fine-tuning is suitable for core tasks with tens of thousands of high-quality domain data points, abundant computing resources, and a pursuit of ultimate performance; it involves training the model on domain data by unfreezing all model parameters. Supervised fine-tuning uses high-quality instruction responses paired with data to train the model's ability to follow test-related instructions, enabling the model to learn to output compliant formats and content based on instructions such as test case generation; this is the foundational step of fine-tuning. Reinforcement learning based on human feedback utilizes preference data to further align the model's output with human expert judgment, improving the quality of generated test cases and the accuracy of anomaly analysis, making the model's judgments more closely resemble those of senior test engineers.
[0044] In this embodiment, during the test case generation process, the prompts received by the large language model must include four key elements: the system role is clearly defined as a professional automotive HMI test engineer; the task context provides background information such as the version of the instrument HMI under test and key parameters; the specific requirements clearly describe the test intent in accordance with the Given-When-Then specification; and the output format is strictly defined as a JSON structure containing fields such as test_case_id, test_objective, preconditions, test_steps, and expected_results. For example, the prompt for speedometer testing is: "You are a professional automotive HMI test engineer. Please generate test cases based on the following requirements. Background: The current test instrument HMI version is v3.1.0, supporting a maximum speed display of 260km / h. Test requirements: Verify the accuracy of the digital speed display and the smoothness of the acceleration animation when the vehicle accelerates from a standstill to 100km / h. Output requirements: Please output in strict JSON format, including the following fields: test_case_id, test_objective, preconditions, test_steps, expected_results." Test cases generated by the large language model based on this prompt can be directly used for subsequent test execution without manual editing, significantly reducing test preparation time.
[0045] In step S2, the high-fidelity rendering engine is built on Unreal Engine 5. Its core function is to achieve high-fidelity visual presentation of the virtual instrument HMI, including simulating real lighting effects, material textures, animation transitions, etc., to ensure that the virtual instrument and the physical instrument are extremely close in visual effect, providing a reliable expected benchmark for subsequent comparison of virtual and real images.
[0046] The real-time simulation engine, built upon a GIS engine or specialized simulation software, is responsible for running vehicle dynamics models, environmental models, and sensor models: the vehicle dynamics model simulates the vehicle's motion under different operating conditions; the environmental model simulates the environmental conditions of different test scenarios; and the sensor model simulates the signal output characteristics of physical sensors. By processing the physical logic and real-time data of these models, the real-time simulation engine ensures that the dynamic changes of the virtual environment are consistent with the real situation, providing virtual scenario support that conforms to actual operating conditions for the execution of test cases.
[0047] The two engines achieve data fusion and synchronization through a scene transformation matrix. This scene transformation matrix is a 4x4 homogeneous coordinate transformation matrix, expressed as TranM = T × S × R, where: T is the translation transformation matrix, describing the offset of the origin of one coordinate system relative to the origin of the other, determined by the offset amount; R is the rotation transformation matrix, describing the rotation of the coordinate systems around the X, Y, and Z axes, obtained from the rotation angles via Euler angles or quaternions; and S is the scaling transformation matrix, describing the scaling ratio along the coordinate axes, determined by the scaling factor. The parameters of the scene transformation matrix are determined through a rigorous hand-eye calibration process: first, multiple sets of corresponding points of known physical and virtual space coordinates are collected; then, the optimal transformation matrix parameters are solved using a singular value decomposition algorithm. This ultimately achieves precise mapping between the virtual camera coordinate system of the high-fidelity rendering engine and the real industrial camera coordinate system, and between the world coordinate system of the real-time simulation engine and the virtual world coordinate system of the rendering engine. This ensures seamless integration and consistency of the visualization scenes between the two engines, resolving the contradiction that a single engine cannot simultaneously satisfy large-scale scene simulation and refined rendering.
[0048] In step S3, the test management platform schedules the real-time simulation engine to calculate the corresponding vehicle state parameters based on the test cases generated in step S1. These parameters include vehicle motion state, including vehicle speed, engine speed, transmission gear, vehicle longitudinal / lateral acceleration, body system state, including door open / closed state, window position, headlight state, handbrake state, power system state, including battery charge, engine coolant temperature, remaining fuel level, navigation and ADAS information, including current speed limit, steering navigation prompts, adaptive cruise control status, forward collision warning, and alarm and fault information, including engine malfunction indicator lamp, airbag warning, low tire pressure warning, braking system fault, etc., comprehensively covering all kinds of core information that the vehicle instrument panel needs to display.
[0049] The real-time simulation engine synchronizes the calculated vehicle state parameters to the high-fidelity rendering engine. The high-fidelity rendering engine updates the display state of the virtual instrument based on these parameters and generates the expected screen of the virtual instrument. This screen includes UI element display, animation effects, etc., corresponding to the test cases, and serves as the standard benchmark for subsequent comparisons.
[0050] Simultaneously, the test management platform converts vehicle status parameters into CAN bus signals, which are then injected into the physical instruments via signal simulation equipment. The CAN bus signal conversion process is completed according to a predefined CAN database file (DBC file): the DBC file clearly defines the identifier, data length code, data field position, and encoding rules for each CAN signal; the conversion module first converts the physical values of the vehicle status parameters into digital quantities according to the encoding rules, then converts the digital quantities into hexadecimal bytes and fills them into the CAN message data field according to the positions specified in the DBC file, and finally assembles a complete CAN data frame containing the identifier, data length code, data field, and check bit, which is then sent to the CAN bus cyclically at a specific period through a CAN card or USB-CAN adapter. After receiving the CAN bus signals, the physical instruments update their display status.
[0051] In step S4, multimodal data from the physical instruments is acquired in real time using a sensor array, forming a complete dataset with the virtual scene data. The sensor array includes a high-definition industrial camera, an inertial measurement unit (IMU), a lidar sensor, and various auxiliary sensors: the high-definition industrial camera acquires display images of the physical instruments, obtaining visual presentation information of UI elements; the inertial measurement unit acquires motion attitude data of the equipment to assist in pose estimation; the lidar provides high-precision depth and geometric information, compensating for the shortcomings of visual information in depth perception; and the auxiliary sensors acquire environmental data and the status data of the physical instruments. The acquired data corresponds to the virtual scene data generated by the virtual twin layer, forming a multimodal dataset containing images, text, and numerical data, providing data support for subsequent pose optimization and intelligent analysis.
[0052] In step S5, a combination of multi-sensor fusion and vision-assisted absolute positioning is used to fuse multi-source sensor data, thereby achieving centimeter-level pose optimization of the physical entity and the virtual model and ensuring spatial consistency of virtual-real comparison.
[0053] The first stage involves multi-source data sensing and fusion. After industrial cameras, LiDAR, and inertial measurement units simultaneously acquire data, the system uses Kalman filtering or extended Kalman filtering algorithms to fuse this multi-source heterogeneous data. Kalman filtering can effectively estimate the optimal state of a dynamic system through a predict-update recursive process: first, it predicts the state estimate for the current moment based on the optimal state of the previous moment and the system model; then, it corrects the predicted value using the sensor observation data at the current moment to obtain the optimal state estimate for the current moment. This process can significantly reduce noise interference in single-sensor data and perform online correction of pose accumulation errors generated by devices such as odometers, thereby outputting smoother and more accurate six-degree-of-freedom pose data.
[0054] The second stage is visual-assisted absolute positioning. To further improve absolute positioning accuracy, the system introduces visual location recognition technology: using a pre-trained YOLO-pose deep learning model, predefined calibration targets are detected from images captured by industrial cameras; based on a pre-built digital twin model, the correspondence between the two-dimensional pixel coordinates of the target in the image and the three-dimensional world coordinates is established; by solving the PnP problem, the precise pose of the camera at the current moment is calculated. This visual pose, along with the odometry pose initially optimized by Kalman filtering, is input into a least-squares estimation optimization framework to solve for the optimal pose result, ultimately achieving centimeter-level pose optimization between the physical entity and the virtual model, laying the foundation for accurate comparison of virtual and real images.
[0055] In step S6, the test management platform compares the expected virtual instrument image generated by the high-fidelity rendering engine with the actual physical instrument image captured by the industrial camera at the pixel level. When a difference is found, the platform uses a large language model to comprehensively analyze multi-source data to achieve intelligent identification, root cause analysis, and level determination of defects.
[0056] The analysis process of the large language model is based on multimodal data fusion: First, it receives multi-source information such as image difference data, CAN signal data, and system log data; then, it performs in-depth analysis according to a preset analysis framework to determine the nature of the difference—whether it is the expected dynamic response or a real functional defect; if it is determined to be a defect, the large language model further traces the root cause of the defect, such as analyzing whether it is an error in the parameter settings of the virtual simulation model, a logical loophole in the rendering engine, or a hardware failure of the physical instrument, CAN signal transmission delay, etc.; finally, it determines the defect based on factors such as the scope of the defect's impact and the frequency of occurrence, thus significantly reducing the false alarm rate and false negative rate of traditional testing methods.
[0057] In addition, the large language model can automatically generate test reports based on the test results. The report content includes the overall pass rate, statistics of failed test cases, description of major defects, root cause analysis, risk assessment and improvement suggestions. The report is written in professional and concise language and has a clear structure, which makes it easy for the team to quickly locate key issues and formulate countermeasures.
[0058] In step S7, the system records complete data for each test, including test cases, execution process data, defect analysis results, and judgment conclusions. Based on this historical data, the system uses reinforcement learning to achieve closed-loop self-optimization of test strategies, test case libraries, and large language model parameters, thereby continuously improving the system's testing capabilities.
[0059] The self-optimization process specifically includes four stages:
[0060] The first stage is data collection and feature extraction. After the system collects multi-dimensional raw data, the large language model, like a seasoned testing expert, extracts key features from the massive amount of data. For example, it categorizes image differences into types such as "rendering delay," "color deviation," "element misalignment," and "texture loss," associates log error information with specific test contexts, and categorizes CAN signal anomalies into types such as "signal delay," "data packet loss," and "numerical error."
[0061] The second stage is pattern recognition and root cause analysis. The large language model uses deep pattern recognition to discover the correlation between faults and test scenarios and parameters. For example, it found that 80% of the phenomenon of "stuttering navigation interface switching animation" occurs when the vehicle speed exceeds 100km / h, and the blurring of specific icons is always strongly correlated with the version number of a specific UI component. At the same time, it traces the complete chain of faults to accurately locate the root cause of the problem whether it is in the simulation model, rendering logic, physical signal transmission or the hardware itself.
[0062] The third stage involves strategy optimization and knowledge base updates. Based on pattern recognition results, the system autonomously adjusts its testing strategies: it automatically increases the testing priority for high-frequency failures or edge scenarios and increases the number of stress test cycles; based on its understanding of failure modes, it automatically generates new test scenarios or supplements existing test cases; and it stores all analysis results, optimization strategies, and newly generated test cases in the system's knowledge base, enriching the system's testing experience.
[0063] The fourth stage involves iterative updates to the model parameters. Historical test data is organized into structured instruction-response pairs for supervised fine-tuning of the large language model, making its output more closely match actual testing needs. Reinforcement learning is employed, using "improving defect detection rate," "shortening testing cycle," and "reducing false positive rate" as reward signals, allowing the model to learn how to formulate better testing strategies through continuous testing iterations, thereby achieving continuous optimization of model parameters.
[0064] This embodiment also provides a digital twin-based automotive instrument testing system for implementing the aforementioned digital twin-based automotive instrument testing method. Its features include a physical entity layer, a virtual twin layer, an intelligent analysis layer, a control core, and a test management platform.
[0065] The physical entity layer includes automotive instrument hardware, signal simulation equipment, and data acquisition equipment. The signal simulation equipment includes CAN bus tools, CAN signal simulation test boxes, and programmable power supplies. The data acquisition equipment includes industrial cameras, lidar, and IMU.
[0066] The virtual twin layer includes a high-fidelity rendering engine and a real-time simulation engine. The high-fidelity rendering engine is based on Unreal Engine 5 to realize the rendering of the instrument HMI's lighting, materials and animations. The real-time simulation engine realizes the real-time calculation of vehicle dynamics and environment.
[0067] The intelligent analysis layer integrates a domain-fine-tuned large language model for test case generation, multimodal data fusion analysis, defect root cause localization, and automatic test report generation.
[0068] The control core includes a digital twin synchronization module and a pose optimization module. The digital twin synchronization module is used for dual-engine data synchronization and virtual-real state calibration. The pose optimization module uses Kalman filtering and visual-assisted positioning technology to solve the pose.
[0069] The test management platform provides a graphical user interface to control the test process, monitor the test process, store data, and display test reports.
[0070] This embodiment also provides a computer device, which includes a processor and a memory. The memory is connected to the processor and stores one or more programs. The one or more programs are executed by the processor to implement the steps in the above-described digital twin-based automotive instrument testing method.
[0071] This embodiment also provides a computer-readable storage medium storing one or more programs that are executed by a processor to implement the steps in the above-described digital twin-based automotive instrument testing method.
[0072] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0073] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for testing automotive instruments based on digital twins, characterized in that, Includes the following steps: Step S1: Receive test requirements and design documents described in natural language through the large language model, and generate structured test cases containing test case names, preconditions, test steps and expected results; Step S2: Construct a virtual instrument model that matches the visual effect of the physical instrument based on a high-fidelity rendering engine; construct a vehicle dynamics model, an environment model, and a sensor model based on a real-time simulation engine; and achieve accurate mapping and data fusion of the dual-engine coordinate system through a scene transformation matrix. Step S3: The test management platform schedules the real-time simulation engine to generate vehicle status parameters based on the test cases, and synchronizes them to the high-fidelity rendering engine to generate the expected virtual instrument screen; at the same time, the vehicle status parameters are converted into CAN bus signals through the signal simulation device and injected into the physical instrument to trigger the physical instrument to update the display. Step S4: Using a sensor array including industrial cameras, inertial measurement units, and lidar, the displayed images, status data, and environmental data of the physical instruments are collected in real time, and combined with the virtual scene data to form a multimodal dataset; Step S5: Perform fusion processing on multi-source sensor data to achieve pose optimization of physical entities and virtual models; Step S6: Compare the expected image of the virtual instrument with the actual image of the physical instrument. Analyze the image differences, CAN signal data, and system logs using a large language model to determine the cause of the defect and the level of the defect. Step S7: Record test data, optimize the test strategy based on historical data through reinforcement learning, and update the test case library and large language model parameters.
2. The automotive instrument testing method based on digital twins according to claim 1, characterized in that, The large language model mentioned in step S1 is fine-tuned and optimized using data from the automotive instrument testing field. The fine-tuning methods include full parameter fine-tuning, supervised fine-tuning, and reinforcement learning based on human feedback. During the test case generation process, the prompts received by the large language model include system role, task context, specific requirements and output format. The system role is set as a professional automotive HMI test engineer, and the task context includes the version of the instrument HMI under test and key parameters.
3. The automotive instrument testing method based on digital twins according to claim 1, characterized in that, The high-fidelity rendering engine described in step S2 is built on Unreal Engine 5 and is responsible for the visual presentation of the virtual instrument HMI, including lighting effects, material textures, and animation transitions. The real-time simulation engine is built on a GIS engine or professional simulation software and is used to run vehicle dynamics models, environmental models, and sensor models, and to process physical logic and real-time data. The scene transformation matrix is a homogeneous coordinate transformation matrix. The parameters of the scene transformation matrix are determined through a hand-eye calibration process, by collecting multiple sets of corresponding points in physical space and virtual space, and solving for the optimal parameters through a singular value decomposition algorithm.
4. The automotive instrument testing method based on digital twins according to claim 1, characterized in that, In step S5, pose optimization is achieved by combining multi-sensor fusion with vision-assisted absolute positioning. Specifically, it includes: synchronously collecting data through industrial cameras, LiDAR, and inertial measurement units deployed around the physical instrument. The industrial camera provides visual texture information for identifying instrument UI elements, the LiDAR provides high-precision depth and geometric information, and the inertial measurement unit provides equipment motion posture data to assist in pose estimation. The multi-source heterogeneous data is fused using Kalman filtering or extended Kalman filtering algorithms to output six-degree-of-freedom pose data. Predefined calibration targets are detected from acquired images using a pre-trained deep learning model. Based on a digital twin model that stores the precise three-dimensional coordinates of the calibration targets, the correspondence between the two-dimensional pixel coordinates of the images and the three-dimensional world coordinates is established. The optimal pose result is solved based on a least squares estimation optimization framework.
5. The automotive instrument testing method based on digital twins according to claim 1, characterized in that, Sensor data, control commands, and model status are synchronized with high-precision timestamps via the PTP precision clock synchronization protocol or hardware triggers. The data processing and synchronization modules sort, align, and fuse the data based on the timestamps.
6. The automotive instrument testing method based on digital twins according to claim 1, characterized in that, The vehicle status parameters mentioned in step S3 include vehicle motion status, body system status, power system status, navigation and ADAS information, alarm and fault information; the CAN bus signal conversion process is completed according to the predefined CAN database file. First, the identifier, data length code, data field position and encoding rules of each CAN signal are defined in the DBC file. Then, the physical values of the vehicle status parameters are converted into digital quantities according to the encoding rules. The digital quantities are converted into hexadecimal bytes and filled into the CAN message data field in the specified position. Finally, a complete CAN data frame containing the identifier, data length code, data field and check bit is assembled and sent to the CAN bus in a specific cycle through the CAN card or USB-CAN adapter.
7. The automotive instrument testing method based on digital twins according to claim 1, characterized in that, The closed-loop self-optimization process in step S7 includes data collection and feature extraction, pattern recognition and root cause analysis, strategy optimization and knowledge base update, and model parameter iterative update. In data collection and feature extraction, multi-dimensional raw data including CAN bus signal timing, pixel-level differences between virtual and actual screens, system log information, and test step execution time are recorded for each test. The large language model extracts key features from the data and associates log error information with the test context. In strategy optimization and knowledge base updates, we will increase the test priority of high-frequency failure or edge scenarios, increase the number of stress test cycles, generate new test scenarios, and input the analysis results, optimization strategies and new test cases into the knowledge base. During the iterative update of model parameters, historical test data is organized into structured instruction-response pairs, and the large language model is fine-tuned under supervision. Through reinforcement learning, the improvement of defect detection rate and the shortening of test cycle are used as rewards to optimize model parameters.
8. A digital twin-based automotive instrument testing system, used to implement the digital twin-based automotive instrument testing method according to any one of claims 1-7, characterized in that, It includes a physical entity layer, a virtual twin layer, an intelligent analysis layer, a control core, and a test management platform. The physical entity layer includes automotive instrument hardware, signal simulation equipment, and data acquisition equipment. The signal simulation equipment includes CAN bus tools, CAN signal simulation test boxes, and programmable power supplies. The data acquisition equipment includes industrial cameras, lidar, and IMU. The virtual twin layer includes a high-fidelity rendering engine and a real-time simulation engine. The high-fidelity rendering engine is based on Unreal Engine 5 to realize the rendering of the instrument HMI's lighting, materials and animations. The real-time simulation engine realizes the real-time calculation of vehicle dynamics and environment. The intelligent analysis layer integrates a domain-fine-tuned large language model for test case generation, multimodal data fusion analysis, defect root cause localization, and automatic test report generation. The control core includes a digital twin synchronization module and a pose optimization module. The digital twin synchronization module is used for dual-engine data synchronization and virtual-real state calibration. The pose optimization module uses Kalman filtering and visual-assisted positioning technology to solve the pose. The test management platform provides a graphical user interface to control the test process, monitor the test process, store data, and display test reports.
9. A computer device, characterized in that, The computer device includes a processor and a memory connected to the processor. The memory stores one or more programs that are executed by the processor to perform the steps in the digital twin-based automotive instrument testing method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs that are executed by a processor to implement the steps in the digital twin-based automotive instrument testing method as described in any one of claims 1-7.