Vehicle function automatic test method based on multi-dimensional decision
By constructing a unified cross-domain testing framework, and combining ADB operation commands, machine vision, and CAN bus analysis, the problems of vehicle screen display and insufficient bus data monitoring have been solved, enabling efficient and low-cost automatic testing of vehicle functions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-19
- Publication Date
- 2026-03-10
AI Technical Summary
Existing technologies cannot effectively detect physical display problems on vehicle infotainment screens, nor can they analyze in real time whether bus data meets expectations, resulting in low testing efficiency and high costs.
By constructing a unified cross-domain testing framework, combining ADB operation commands, machine vision, and CAN bus analysis, we can achieve automated testing of vehicle infotainment functions with multi-dimensional decision-making capabilities. This includes visual detection of screen display issues and real-time monitoring of CAN bus signals, forming a complete closed-loop testing system.
It enables real-time detection of display problems on vehicle infotainment screens and synchronous monitoring of bus data, improving the reliability and automation of testing, shortening the problem localization time, and reducing hardware costs.
Smart Images
Figure CN121635283A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of vehicle infotainment testing technology, and in particular to an automatic testing method for vehicle infotainment functions based on multi-dimensional decision-making. Background Technology
[0002] With the rapid development of new energy vehicles, traditional physical buttons are no longer sufficient to meet the functional requirements of real-world vehicles, necessitating the integration of rapidly evolving intelligent technologies into the automotive industry. Intelligent instrument panels and central control screens have become the core carriers of human-machine interaction in automobiles, directly impacting the user's actual driving experience and the realization of intelligent functions. To cope with increasing market competition, intelligent cockpits serve as a crucial breakthrough for automakers to build innovation and competitiveness. More efficient automated cockpit testing technologies have become essential for automakers to shorten development cycles and achieve rapid product iteration. However, automated cockpit testing technologies geared towards the development end cannot fully expose potential problems encountered by users in real-world scenarios, and have limitations for vehicle-level testing. Therefore, a vehicle-level, user-centric, efficient, and low-cost automated cockpit testing system and method has become a necessity.
[0003] One industry approach involves automating interaction through ADB (Android Debug Bridge) commands. A host computer connects to the vehicle's central control screen via USB. The host computer software uses scripts to automatically send ADB commands (such as clicks and swipes) to control the large in-vehicle screen. Another approach uses a six-axis robotic arm paired with an industrial camera to mimic human hand movements for automated, precise clicks and swipes, while a CAN bus analyzer monitors bus interaction signals in real time.
[0004] However, ADB-based operation commands cannot achieve physical-level display judgment (such as black screen, distorted screen, flickering screen, etc.), and if the system does not completely freeze, causing interface lag, the operation is still effective, but the system itself cannot recognize it. Furthermore, the approach of using a six-axis robotic arm + industrial camera + CAN bus analyzer is highly complex to develop, has high hardware costs, and the accuracy of the robotic arm operation depends entirely on the image acquisition quality of the industrial camera and the analysis capabilities of the vision system. It is also easily affected by ambient light, and one set of equipment can only operate one device at a time. None of the above-mentioned existing technical solutions can achieve automated real-time analysis of whether the bus data meets expectations. Summary of the Invention
[0005] In view of the above, the present invention aims to provide an automatic testing method for vehicle infotainment functions based on multi-dimensional decision-making, so as to solve the aforementioned technical problems.
[0006] The technical solution adopted in this invention is as follows:
[0007] This invention provides an automatic testing method for vehicle infotainment functions based on multi-dimensional decision-making, including:
[0008] According to the established testing process, ADB operation commands are issued one by one to control the vehicle screen, and the interface image of the current execution result is automatically captured and saved.
[0009] The system uses machine vision to detect in real time whether there are display problems on the vehicle's infotainment screen that executes ADB operation commands;
[0010] After determining that no display problem has occurred, the display image captured by the camera is matched with the interface image stored by the vehicle system for similarity.
[0011] After determining that the similarity matching degree meets the preset standard, it is determined whether the current operation command involves the CAN bus layer and whether the CAN signal has undergone the expected change;
[0012] If the current operation command does not involve the CAN bus layer, or if the current operation command involves the CAN bus layer and the expected change in the CAN signal is triggered normally, then determine whether the current operation involves power-on or power-off.
[0013] After determining that the current operation involves power-on or power-off, a power-off operation command is sent via ADB, and then the power-on control is executed.
[0014] After confirming that the power-down and power-on operations have been performed normally, execute the next ADB operation instruction and perform a loop test.
[0015] In at least one of the possible implementations, detecting whether a display problem occurs on the screen using visual detection methods includes:
[0016] After image acquisition, image preprocessing, and region of interest extraction, the fault detection algorithm determines in real time whether the current vehicle screen is experiencing black screen, distorted screen, or flickering faults.
[0017] If so, end the current test and report the fault timestamp and fault cause.
[0018] In at least one of the possible implementations, if the similarity matching degree does not meet the preset standard, the current test is terminated directly and the fault timestamp and fault cause are reported.
[0019] In at least one of the possible implementations, if the current operation instruction involves the CAN bus layer but fails to correctly trigger the expected change in the CAN bus signal, the current test is terminated directly and the fault timestamp and fault cause are reported.
[0020] In at least one of the possible implementations, if it is determined that the current operation does not involve power-on or power-off, then the next ADB operation instruction is executed.
[0021] In at least one of the possible implementations, when an abnormality is determined to occur during power-down or power-on operation, the current test is terminated directly and the fault timestamp and fault cause are reported.
[0022] Compared with existing technologies, the main design concept of this invention lies in building a unified cross-domain testing framework that seamlessly connects the human-machine interaction layer (UI), operating system layer (Android), vehicle network layer (CAN), and physical hardware layer (screen) to synchronously complete verification. Specifically, a high-precision unified timestamp can be introduced to align and merge asynchronous data streams (UI events, CAN messages, image frames) from different sources and of different natures on the same timeline. Based on this, a decision-making mechanism for multi-source information fusion can be constructed to perform correlation analysis on the results of each test module. Furthermore, corresponding test modules can be developed according to actual testing needs and integrated into the main control program through reserved interfaces. This invention integrates visual inspection with ADB simulated click control and CAN bus analysis to form a complete closed-loop testing architecture covering software instructions → hardware response → physical presentation, greatly improving the reliability and automation of vehicle system testing. For example, but not limited to, it can be used in test cases where the vehicle status is set in the whole-vehicle cockpit UI interface and whether the changes in the whole vehicle meet expectations, as well as in automated UI stress testing. Attached Figure Description
[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings, wherein:
[0024] Figure 1 This is a schematic diagram of an automatic testing method for vehicle infotainment functions based on multi-dimensional decision-making, provided in an embodiment of the present invention. Detailed Implementation
[0025] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0026] This invention is based on a pre-built test architecture, including, for example, a display, an industrial computer, a camera (with a sunshade as needed), and a CAN card. More preferably, it may also include external devices (pedal robots / various sensors), etc. The camera and CAN card are connected to the industrial computer via a USB interface; the other end of the CAN card is connected to the vehicle's OBD port, accessing the vehicle's CAN bus; other external devices communicate with the industrial computer via serial ports according to test requirements; the display is connected to the industrial computer via HDMI to display the real-time running status of the automation program and the real-time video stream acquired by the vision module.
[0027] Therefore, the industrial control computer sends ADB operation commands to the vehicle in the form of Python programs to realize programmed simulation of automatic click operations. ADB feeds back the simulated click results through the program and collects real-time video stream data through a vision detection module (camera + OpneCV library to implement the function of this module) to automatically detect physical display faults on the vehicle screen: it compares the images of the previous and next frames in real time according to the expected display results and outputs the vision decision results (including but not limited to: simulated click results, black screen, screen distortion, screen flickering, etc.) as one of the comprehensive decision judgment conditions. In addition, the vehicle bus is connected to the industrial control computer through a CAN card to complete the hardware construction; the bus performance of the whole vehicle status is pre-entered into the expected result library; after the program runs, it collects, monitors, analyzes and compares the bus data in real time. After the simulated vehicle status event occurs, if the bus data of the relevant ECU components shows unexpected results, the abnormal results are reported, such as recording and reporting the timestamp of the bus signal not changing as expected after the click. In addition, a serial communication interface for external devices is reserved: In the automated testing of the vehicle-level cockpit, power-on and power-off operations may be involved, and automatic power-on after power-off is required. In some better embodiments, in order to achieve unmanned operation, external devices can be connected to assist, such as pedal robots, which can simulate human stepping on the brake to power on through programmed control; for example, in the automated testing of the vehicle's air conditioning settings, an external air volume sensor / temperature sensor can be connected through the reserved interface to realize the actual status detection and reporting of each air outlet, ensuring that the actual operating status of the hardware matches the expectations.
[0028] More specifically, this invention proposes an embodiment of an automatic testing method for vehicle infotainment functions based on multi-dimensional decision-making, specifically, as follows: Figure 1 As shown, it includes:
[0029] Step S1: Following the established test procedure, issue ADB operation commands one by one to control the vehicle screen, and automatically capture and save the interface image of the current execution result.
[0030] Step S2: Use machine vision to detect in real time whether there are display problems on the vehicle screen that executes ADB operation commands;
[0031] Regarding visual detection methods, this section may refer to, but is not limited to, methods that, after image acquisition, image preprocessing, and Region of Interest (ROI) extraction, use fault detection algorithms to determine in real time whether the current vehicle screen is experiencing hardware display problems such as black screen, distorted screen, or flickering screen; if so, end the current test and report the fault timestamp and cause of the fault.
[0032] Step S3: After determining that no display problem has occurred, perform a similarity match between the display image captured by the camera and the interface image retained by the vehicle system;
[0033] If the similarity is low, it indicates that the vehicle screen did not display the expected results. The current test will end directly and the fault timestamp and fault cause will be reported. Otherwise, proceed to step S4.
[0034] Step S4: After determining that the similarity matching degree meets the preset standard (indicating that it is displayed normally as expected), determine whether the current operation command involves the CAN bus layer and whether the underlying CAN signal has changed as expected.
[0035] If bus layer signals are involved, but the expected change in the bus signals is not triggered correctly, end the current test directly and report the fault timestamp and fault cause.
[0036] Step S5: If the current operation instruction does not involve the CAN bus layer, or if the current operation instruction involves the CAN bus layer and the expected change in the CAN signal is triggered normally, then determine whether the current operation involves power-on or power-off.
[0037] If it is determined that the current operation does not involve power on or off, then execute the next ADB operation instruction.
[0038] Step S6: After determining that the current operation involves power-on or power-off, send a power-off operation command through ADB and then execute power-on control;
[0039] The power-on control here includes sending action commands from the external device via the serial port to simulate the operation of pressing the brake to power on.
[0040] Step S7: After confirming that the power-down and power-on operations are performed normally, execute the next ADB operation instruction and perform a loop test according to the above steps.
[0041] Otherwise, if an abnormality is detected during power-down or power-on operation, the current test should be terminated immediately and the fault timestamp and fault cause should be reported.
[0042] During the above-mentioned cyclic test, when the test program is running, the sending of ADB operation commands, visual inspection, and the acquisition and analysis of CAN bus signals can all be carried out synchronously. For example, while executing step S1, the camera is called to acquire screen images, and the CAN card is called to load the DBC file to acquire and parse the raw CAN data.
[0043] In summary:
[0044] ① This invention realizes a complete closed-loop testing system from software instructions → hardware response → physical presentation, which makes up for the disadvantage of ADB being unable to detect hardware-level problems, and can monitor bus data in real time with a wide test coverage.
[0045] ②When a test fails, this invention can immediately pinpoint the problem to a specific step, significantly shortening debugging and root cause analysis time and greatly improving the accuracy and efficiency of problem localization.
[0046] ③ This invention, through a multi-verification mechanism, avoids the drawbacks of a single detection method being prone to misjudgment, making the judgment results more reliable.
[0047] ④ Each module of this invention is independent and highly expandable. Modules can be developed according to actual testing needs, and then scheduled and exchanged through the main control program.
[0048] In this invention, when directional terms are mentioned, they are relative concepts based on the embodiments. Furthermore, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.
[0049] The above description of the structure, features, and effects of the present invention is based on the embodiments shown in the figures. However, the above are only preferred embodiments of the present invention. It should be noted that the technical features involved in the above embodiments and their preferred methods can be reasonably combined and matched by those skilled in the art to form a variety of equivalent solutions without departing from or changing the design concept and technical effects of the present invention. Therefore, the present invention is not limited to the scope of implementation shown in the figures. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments, that do not exceed the spirit covered by the specification and figures, should be within the protection scope of the present invention.
Claims
1. A method for automatically testing vehicle-machine functions based on multi-dimensional decision-making, characterized in that, The method comprises the following steps: According to the established test process, the ADB operation instructions are issued one by one to control the vehicle screen, and the interface picture of the current execution result is automatically intercepted and reserved; In a machine vision manner, it is detected in real time whether the vehicle screen executing the ADB operation instruction has display problems; After determining that there are no display problems, the display picture taken by the camera is matched with the interface picture reserved by the vehicle; After determining that the similarity matching degree meets the preset standard, it is determined whether the current operation instruction involves the CAN bus layer and whether the CAN signal changes as expected; If the current operation instruction does not involve the CAN bus layer, or the current operation instruction involves the CAN bus layer and the CAN signal changes as expected after being normally triggered, it is determined whether the current operation involves power-on and power-off; After determining that the current operation involves power-on and power-off, the ADB sends a power-off operation instruction, and then performs power-on control; After determining that the power-off and power-on operations are normally executed, the next ADB operation instruction is executed, and the test is cycled.
2. The multi-dimensional decision based automatic testing method of car-machine functions according to claim 1, characterized in that, Detecting whether the screen has display problems in a visual detection manner comprises: After image acquisition, image preprocessing and region of interest extraction, it is determined in real time whether the current vehicle screen has black screen, flower screen and flash screen faults according to a fault detection algorithm; If yes, the current test is ended and the fault timestamp and fault reason are reported.
3. The multi-dimensional decision based automatic testing method of car-machine functions according to claim 1, characterized in that, If the similarity matching degree does not meet the preset standard, the current test is directly ended and the fault timestamp and fault reason are reported.
4. The multi-dimensional decision based automatic testing method of car-machine functions according to claim 1, characterized in that, If the current operation instruction involves the CAN bus layer, but does not correctly trigger the CAN bus signal to change as expected, the current test is directly ended and the fault timestamp and fault reason are reported.
5. The multi-dimensional decision based automatic testing method of car-machine functions according to claim 1, characterized in that, If it is determined that the current operation does not involve power-on and power-off, the next ADB operation instruction is executed.
6. The multi-dimensional decision based automatic testing method of car-machine functions according to claim 1, wherein, When it is determined that the power-off or power-on operation has an abnormality, the current test is directly ended and the fault timestamp and fault reason are reported.