Vehicle-mounted intelligent equipment testing method and system based on multi-modal large model

Abstract

The invention provides a vehicle-mounted intelligent equipment test method and system based on a multi-modal large model, the method and system have the characteristics of comprehensive test, high efficiency and intelligent data analysis, all to-be-tested functions are excited in parallel in a single road test through an integrated real vehicle test design, basic performance, dynamic alarm and complex scene verification are covered, and the test efficiency is improved. The problems of traditional test isolation and fragmentation are solved; through a large model intelligent analysis engine, automatic cleaning, association and diagnosis of multi-source data are realized, and the data credibility and the problem positioning accuracy are improved; meanwhile, the consistency and repeatability of test results are ensured by standardized test processes and quantitative indexes, the test period is remarkably shortened, the manpower and material resource cost is reduced, and the reliability and adaptability of the vehicle-mounted intelligent equipment in a real environment are enhanced.

Description

Technical Field

[0001] This invention relates to the field of testing technology, and in particular to a testing method and system for in-vehicle intelligent devices based on a multimodal large model. Background Technology

[0002] With the rapid development of intelligent driving technology, in-vehicle intelligent devices such as Advanced Driver Assistance Systems (ADAS) and Driver Monitoring Systems (DMS) have become core components of vehicle safety. These devices achieve environmental perception, behavior analysis, and hazard warning through multi-sensor fusion (such as cameras, GPS, and inertial measurement units), and their reliability is directly related to driving safety. Currently, industry testing of in-vehicle intelligent devices mainly relies on functional verification methods, which involve testing each functional module item by item according to a checklist, such as individually verifying GPS positioning accuracy, ADAS forward collision warning, or DMS fatigue detection functions. Such tests are usually conducted in laboratory simulated environments or simple road test conditions, focusing on the performance indicators of single functions and lacking an evaluation of the device's comprehensive performance in real-world, complex scenarios.

[0003] However, existing testing methods have several shortcomings: First, the problems of isolated and fragmented testing are prominent. Each functional module is tested separately, failing to reflect the performance of multiple modules working collaboratively in dynamic driving environments. For example, GPS positioning tests are only conducted in open areas, without considering the linkage with other functions in signal-obstructed scenarios such as underground parking garages; ADAS and DMS alarm tests are performed separately, making it difficult to verify the system response in complex scenarios (such as sharp turns while the driver is distracted). Second, there is a lack of verification of the "human-vehicle-road" closed-loop scenario. Traditional methods only focus on the single dimension of "vehicle-road" or "human-vehicle," failing to systematically test the device's data processing capabilities and consistency when multiple events occur concurrently in a real driving environment. Third, testing efficiency is low. Due to the dispersed test cases, multiple repeated tests are required to cover all functional points, resulting in long testing cycles, high costs, and results that rely on the experience of testers, leading to poor reproducibility. Fourth, the data analysis capabilities are weak. The large amount of heterogeneous data generated during testing (such as GPS tracks and alarm records) lacks effective correlation and analysis methods. When problems such as track drift and missed alarms occur, it is difficult to quickly locate whether the cause is hardware, software or environmental factors, resulting in low problem-solving efficiency.

[0004] Therefore, there is an urgent need in this field to develop a testing method and system for in-vehicle intelligent devices that can achieve multi-functional integrated testing, cover complex real-world scenarios, and have intelligent data analysis capabilities, in order to solve the problems of incomplete testing, low efficiency, and insufficient data analysis in existing technologies. Summary of the Invention

[0005] The purpose of this invention is to provide a testing method and system for in-vehicle intelligent devices based on a multimodal large model, so as to solve the problems existing in the prior art.

[0006] To achieve the above objectives, the present invention provides the following solution: This invention provides a testing method for in-vehicle intelligent devices based on a multimodal large model, comprising the following steps: S1. Test preparation: Configure test equipment, including front and rear dual cameras, data acquisition terminal and cloud test platform, and set multiple test scenarios, including open space, underground garage, urban road and intersection, to simulate real driving environment; S2. Test execution: Through one or more real vehicle drives, GPS positioning test, combined driving behavior alarm trigger test, scenario-based integration test of ADAS and DMS functions, and data quality and integrity monitoring are executed simultaneously. The test execution is based on the integrated scenario design and all functions to be tested are triggered in parallel during a single trip. S3. Data Analysis: Import the multi-source heterogeneous data generated during the testing process into the large model data processing module for data preprocessing, parallel analysis, intelligent association reasoning, and automated report generation. The large model data processing module uses machine learning algorithms for data cleaning, anomaly detection, and event association.

[0007] Preferably, in step S2, the GPS positioning test includes testing the cold start positioning time and hot start positioning time of the device under different signal environments in open areas and underground garages, and verifying the automatic sleep and wake-up mechanism of the device in combination with vehicle ignition, shutdown and automatic start-stop scenarios; the combined driving behavior alarm triggering test includes designing to simultaneously trigger overspeed alarm, sharp turn alarm and DMS smoking alarm in a single driving action to verify the system's concurrent processing capability.

[0008] Preferably, in step S3, the data preprocessing step includes: Time standardization, unifying the timestamp format and time zone of all data sources; Coordinate transformation converts location data into a unified coordinate system; Data cleaning involves smoothing GPS trajectories by screening for satellite numbers, analyzing velocity mutations, and using Kalman filtering algorithms to remove invalid data points. Automatic labeling: Based on the test scenario configuration file, alarm data is automatically labeled as "expected" or "abnormal" through time window matching and spatial location verification.

[0009] Preferably, in step S3, the parallel analysis step adopts a multi-threaded architecture, including: The GPS quality analysis thread is used for track continuity checks, positioning accuracy assessments, and signal loss statistics. The alarm logic verification thread is used to trigger condition compliance checks, alarm timeliness verification, and duplicate alarm detection. The behavior pattern recognition thread is used for driving habit analysis, alarm pattern mining, and abnormal behavior detection. The analysis results of each thread are imported into the database for comprehensive evaluation.

[0010] Preferably, in step S3, the intelligent association reasoning step includes: Spatiotemporal correlation discovery involves aligning GPS trajectories, alarm events, and vehicle status on a timeline and map by constructing a unified spatiotemporal reference system, thereby identifying event clustering patterns. Causal chain construction, based on temporal sequence and spatial proximity, applies domain knowledge to verify event correlation and generates a "cause-effect" event chain; Root cause identification distinguishes between equipment hardware failure, algorithm misjudgment, or environmental interference, and provides improvement suggestions.

[0011] Preferably, the spatiotemporal correlation discovery employs a density clustering algorithm to cluster events in a four-dimensional spatiotemporal space. The clustering parameters include a temporal neighborhood threshold, a spatial neighborhood threshold, and a minimum event cluster size, in order to identify spatiotemporally close event groups.

[0012] Preferably, in step S3, the automated report generation step includes: integrating the analysis results to generate a diagnostic report containing an execution summary, key findings, anomaly details, and visualization charts, wherein the visualization charts include a spatiotemporal event graph and a performance trend graph, and support the generation of natural language descriptions.

[0013] This invention also provides a vehicle-mounted intelligent device testing system based on a multimodal large model, comprising: The test equipment module includes front and rear integrated dual cameras and a data acquisition terminal, used to collect vehicle status data, GPS trajectory data, ADAS alarm data and DMS alarm data; A cloud-based testing platform is used to receive and store test data, configure test scenarios, and coordinate test execution. The large model data processing module is used to analyze test data, including a data cleaning and automatic labeling unit, an anomaly detection unit, a correlation analysis unit, and a report generation unit. The module uses machine learning models to achieve intelligent processing of multi-source data.

[0014] Preferably, the anomaly detection unit implements multi-dimensional anomaly detection through a logical consistency verification algorithm, including: The sharp turn alarm verification is performed by calculating the rate of change of the direction angle of the GPS trajectory points and the vehicle speed to verify the alarm triggering conditions. DMS and ADAS alarm missed detection involves analyzing video data frame by frame to identify driver behavior or vehicle driving characteristics and comparing them with reported alarm records.

[0015] Preferably, the correlation analysis unit adopts real-time incremental processing to dynamically generate a spatiotemporal event map during the test and display the event correlation through an interactive visualization interface. The map is constructed based on a unified spatiotemporal reference system and adopts a layered design including a base map layer, a trajectory layer and an event marker layer.

[0016] The present invention achieves the following beneficial technical effects compared to the prior art: This invention provides a testing method and system for in-vehicle intelligent devices based on a multimodal large model. It features comprehensive testing, high efficiency, and intelligent data analysis. Through an integrated real-vehicle test design, all functions under test are activated in parallel during a single road test, covering basic performance, dynamic alarms, and complex scenario verification, thus solving the problems of isolation and fragmentation in traditional testing. The large-model intelligent analysis engine enables automatic cleaning, correlation, and diagnosis of multi-source data, improving data reliability and the accuracy of problem localization. Simultaneously, standardized testing procedures and quantitative indicators ensure the consistency and repeatability of test results, significantly shortening the testing cycle, reducing manpower and material costs, and enhancing the reliability and adaptability of in-vehicle intelligent devices in real-world environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 The flowchart of the vehicle-mounted intelligent device testing method based on a multimodal large model provided by the present invention is shown. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The purpose of this invention is to provide a testing method and system for in-vehicle intelligent devices based on a multimodal large model, aiming to solve the problems of isolated testing, low efficiency, and insufficient data analysis capabilities in the existing technology.

[0021] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] Example 1: This embodiment provides a testing method and system for in-vehicle intelligent devices based on a multimodal large model, such as... Figure 1 As shown, thorough preparation is required before the test begins. The test equipment includes a front-facing dual-camera setup with a resolution of at least 1920×1080 and a rear-facing camera with a frame rate of at least 30fps, terminal equipment for data acquisition, and a test platform deployed in the cloud. The test scenario must be carefully designed, including at least an open field (for positioning benchmark testing), an underground parking garage (to simulate GPS signal loss and recovery scenarios), urban road sections with speed limits of 40km / h and 60km / h, and intersections with automatic start-stop functionality. The test vehicle must undergo one or more real-vehicle drives with a total mileage of at least 5 kilometers and at least three ignition operations to ensure coverage of the device's entire lifecycle from startup, operation to hibernation.

[0023] Furthermore, the test execution phase is the core of the entire methodology. Unlike traditional serial testing, this method uses carefully designed fusion scenarios to trigger and test all functions in parallel within a single road test. For example, in GPS positioning testing, not only are the positioning times for cold starts (time points a to b) and warm starts (time points c to d) recorded in open areas, but the vehicle is also deliberately arranged to shuttle between underground parking garages and open roads multiple times to rigorously test the device's positioning continuity during GPS signal loss and recovery, as well as the accompanying sleep and wake-up mechanisms and power consumption. In combined driving behavior alarm triggering tests, test engineers design compound actions, such as the vehicle speeding (triggering a speeding alarm) while simultaneously making a sharp turn (triggering a sharp turn alarm), and having the driver smoke (triggering a DMS smoking alarm), to verify the system's data processing and reporting capabilities under concurrent event pressure. For scenario-based fusion testing of ADAS and DMS functions, dangerous scenarios such as "the driver looking down at their phone when the vehicle in front brakes suddenly" are designed to trigger sequentially within a short period of time, rigorously testing the system's response logic and the integrity of data recording under compound situations.

[0024] The multi-source heterogeneous data generated during the test, including high-frequency GPS location data, vehicle CAN bus signals, ADAS alarm data, DMS alarm data, and device status logs, are reported to the cloud platform in real time via wireless network.

[0025] Furthermore, the data analysis phase is led by a large-scale intelligent analysis engine. First, data preprocessing is performed, unifying the timestamps of all data sources to UTC millisecond precision and converting all location information to the WGS84 coordinate system, establishing a unified spatiotemporal grid with 100-millisecond intervals. The data cleaning stage is particularly crucial. The system performs multi-dimensional quality assessments on GPS data: automatically filtering positioning points with fewer than four satellites; calculating the distance between adjacent points using the Haversine formula and combining it with the time difference to derive instantaneous velocity, marking physically unreachable points with speeds exceeding 200 km / h as drift points; and applying a Kalman filter algorithm to smooth the trajectory, suppressing noise interference while preserving realistic driving characteristics. The automatic labeling engine, based on the preset execution time window and geographical location in the test script, calculates the time matching degree (70% weight), spatial matching degree (20% weight), and event type matching degree (10% weight) for each alarm event. Finally, events with a confidence score higher than 0.8 are marked as "expected," and the rest are marked as "abnormal."

[0026] The preprocessed data enters a parallel analysis architecture. The system starts three independent analysis threads: the GPS quality analysis thread is responsible for evaluating trajectory continuity, positioning accuracy, and counting signal loss duration; the alarm logic verification thread checks whether the alarm triggering conditions meet preset rules (such as whether the speed exceeds the threshold), whether the alarm reporting is timely, and whether there are duplicate alarms; the behavior pattern recognition thread analyzes historical data to uncover driving habits and potential abnormal alarm patterns. The analysis results from each thread are finally merged into a central database for comprehensive evaluation.

[0027] Intelligent correlation reasoning is a key step embodying the intelligence of this invention. Spatiotemporal correlation discovery employs a density clustering algorithm to cluster events in a four-dimensional spatiotemporal space. A feature vector [timestamp, latitude, longitude, event type, confidence level] is constructed for each event, with a temporal neighborhood threshold of 30 seconds, a spatial neighborhood threshold of 100 meters, and a minimum event cluster size of two related events. Through clustering, groups of events that are spatiotemporally close can be identified, such as the pattern of "DMS alarms occurring intensively in areas with unstable GPS signals." Based on this, the system constructs causal chains, using temporal sequence and spatial proximity, and applying domain knowledge to verify the rationality of the correlations, thereby constructing a "cause-effect" event chain, such as identifying the behavioral sequence of "rapid acceleration often following prolonged distraction." Finally, the root cause localization module can distinguish whether the problem stems from equipment hardware failure, algorithm model misjudgment, or external environmental interference, and provides specific improvement suggestions such as "optimizing the positioning recovery strategy at the parking garage exit."

[0028] Finally, the report generator integrates all intermediate results and outputs an assessment report with diagnostic conclusions. The report not only includes an execution summary and key data-supported findings (such as "the system performed normally in 85% of complex scenarios"), but also presents structured details, evidence, and improvement recommendations for each anomaly. The system automatically generates visual charts, including a spatiotemporal event map overlaid with color-coded trajectory lines and alarm icons of different shapes on a base map, as well as performance trend charts. The maps are interactive; clicking on event markers displays detailed information and can dynamically play the event sequence. Simultaneously, the system utilizes natural language processing technology to automatically generate intuitive natural language descriptions such as "At the XX intersection, the driver was smoking while the vehicle was speeding," making the test conclusions immediately clear.

[0029] Example 2: Corresponding to the above method, the system implementation of the present invention includes a test equipment module, a cloud test platform, and a large model data processing module. The front and rear integrated dual cameras and data acquisition terminal in the test equipment module are responsible for raw data acquisition. The cloud test platform is responsible for scene configuration, task scheduling, and data aggregation. The large model data processing module integrates multiple intelligent units: a data cleaning and automatic labeling unit executes the aforementioned cleaning and labeling process; an anomaly detection unit performs deep checks using a logical consistency verification algorithm; for example, in sharp turn alarm verification, it extracts GPS trajectory points 30 seconds before and after the alarm time, calculates the rate of change of direction angle between consecutive points to determine the maximum turning angle, and obtains the real-time vehicle speed to verify whether the triggering condition of "speed > 30 km / h and turning angle > 45 degrees" is simultaneously met; if not, an anomaly report is generated; the correlation analysis unit supports real-time incremental processing, can dynamically construct and update spatiotemporal event maps, and clearly displays complex correlations to test engineers through an interactive visualization interface.

[0030] Through the collaborative work of the above methods and systems, this invention achieves a leapfrog improvement in the testing of in-vehicle intelligent devices, transforming it from isolated, serial, and human experience-driven processes to integrated, parallel, and data-driven processes, greatly enhancing the depth, efficiency, and reliability of testing.

[0031] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0032] It should be noted that the components mentioned in the above embodiments are all general standard parts or components known to those skilled in the art. Their structures and principles can be learned by those skilled in the art through technical manuals or conventional experimental methods.

[0033] This invention has illustrated its principles and implementation methods using specific examples. The descriptions of these embodiments are merely illustrative of the method and its core ideas; furthermore, those skilled in the art will recognize that modifications may be made to the specific implementation methods and application scope based on the principles of this invention. Therefore, the content of this specification should not be construed as limiting the invention.

Claims

1. A testing method for in-vehicle intelligent devices based on a multimodal large model, characterized in that, Includes the following steps: S1. Test preparation: Configure test equipment, including front and rear dual cameras, data acquisition terminal and cloud test platform, and set multiple test scenarios, including open space, underground garage, urban road and intersection, to simulate real driving environment; S2. Test execution: Through one or more real vehicle drives, GPS positioning test, combined driving behavior alarm trigger test, scenario-based integration test of ADAS and DMS functions, and data quality and integrity monitoring are executed simultaneously. The test execution is based on the integrated scenario design and all functions to be tested are triggered in parallel during a single trip. S3. Data Analysis: Import the multi-source heterogeneous data generated during the testing process into the large model data processing module for data preprocessing, parallel analysis, intelligent association reasoning, and automated report generation. The large model data processing module uses machine learning algorithms for data cleaning, anomaly detection, and event association.

2. The testing method for in-vehicle intelligent devices based on a multimodal large model according to claim 1, characterized in that, In step S2, the GPS positioning test includes testing the cold start positioning time and hot start positioning time of the device under different signal environments in open areas and underground garages, and verifying the automatic sleep and wake-up mechanism of the device in combination with vehicle ignition, shutdown and automatic start-stop scenarios; the combined driving behavior alarm triggering test includes designing to simultaneously trigger overspeed alarm, sharp turn alarm and DMS smoking alarm in a single driving action to verify the system's concurrent processing capability.

3. The testing method for in-vehicle intelligent devices based on a multimodal large model according to claim 1, characterized in that, In step S3, the data preprocessing steps include: Time standardization, unifying the timestamp format and time zone of all data sources; Coordinate transformation converts location data into a unified coordinate system; Data cleaning involves smoothing GPS trajectories by screening for satellite numbers, analyzing velocity mutations, and using Kalman filtering algorithms to remove invalid data points. Automatic labeling: Based on the test scenario configuration file, alarm data is automatically labeled as "expected" or "abnormal" through time window matching and spatial location verification.

4. The testing method for in-vehicle intelligent devices based on a multimodal large model according to claim 1, characterized in that, In step S3, the parallel analysis step adopts a multi-threaded architecture, including: The GPS quality analysis thread is used for track continuity checks, positioning accuracy assessments, and signal loss statistics. The alarm logic verification thread is used to trigger condition compliance checks, alarm timeliness verification, and duplicate alarm detection. The behavior pattern recognition thread is used for driving habit analysis, alarm pattern mining, and abnormal behavior detection. The analysis results of each thread are imported into the database for comprehensive evaluation.

5. The testing method for in-vehicle intelligent devices based on a multimodal large model according to claim 1, characterized in that, In step S3, the intelligent associative reasoning steps include: Spatiotemporal correlation discovery involves aligning GPS trajectories, alarm events, and vehicle status on a timeline and map by constructing a unified spatiotemporal reference system, thereby identifying event clustering patterns. Causal chain construction, based on temporal sequence and spatial proximity, applies domain knowledge to verify event correlation and generates "cause-effect" event chains; Root cause identification distinguishes between equipment hardware failure, algorithm misjudgment, or environmental interference, and provides improvement suggestions.

6. The testing method for in-vehicle intelligent devices based on a multimodal large model according to claim 5, characterized in that, The spatiotemporal correlation discovery employs a density clustering algorithm to cluster events in a four-dimensional spatiotemporal space. The clustering parameters include a temporal neighborhood threshold, a spatial neighborhood threshold, and a minimum event cluster size, in order to identify groups of events that are spatiotemporally close.

7. The testing method for in-vehicle intelligent devices based on a multimodal large model according to claim 1, characterized in that, In step S3, the automated report generation step includes: integrating the analysis results to generate a diagnostic report containing an execution summary, key findings, anomaly details, and visualization charts, wherein the visualization charts include spatiotemporal event graphs and performance trend graphs, and support the generation of natural language descriptions.

8. A vehicle-mounted intelligent device testing system based on a multimodal large model, characterized in that, include: The testing equipment module includes front and rear integrated dual cameras and a data acquisition terminal, used to collect vehicle status data, GPS trajectory data, ADAS alarm data and DMS alarm data; A cloud-based testing platform is used to receive and store test data, configure test scenarios, and coordinate test execution. The large model data processing module is used to analyze test data, including a data cleaning and automatic labeling unit, an anomaly detection unit, a correlation analysis unit, and a report generation unit. The module uses machine learning models to achieve intelligent processing of multi-source data.

9. The vehicle-mounted intelligent device testing system based on a multimodal large model according to claim 8, characterized in that, The anomaly detection unit implements multi-dimensional anomaly detection through a logical consistency verification algorithm, including: The sharp turn alarm verification is performed by calculating the rate of change of the direction angle of the GPS trajectory points and the vehicle speed to verify the alarm triggering conditions. DMS and ADAS alarm missed detection involves analyzing video data frame by frame to identify driver behavior or vehicle driving characteristics and comparing them with reported alarm records.

10. The vehicle-mounted intelligent device testing system based on a multimodal large model according to claim 8, characterized in that, The correlation analysis unit adopts real-time incremental processing to dynamically generate a spatiotemporal event map during the test and displays the event correlation through an interactive visualization interface. The map is constructed based on a unified spatiotemporal reference system and adopts a layered design including a base map layer, a trajectory layer, and an event marker layer.

Images