Man-machine interaction automatic test system and method for vehicle-mounted instrument

Abstract

The invention relates to the field of automatic testing, and provides a man-machine interaction automatic testing system and method for a vehicle-mounted instrument, and the system comprises a configuration management module which is used for storing and managing structured configuration data; the communication connection module is used for establishing and maintaining a two-way communication link with the vehicle-mounted instrument to be tested through a standardized communication interface; the protocol adaptation module is used for encoding the test operation instruction into a protocol format supported by the vehicle-mounted instrument and decoding response data received from the vehicle-mounted instrument into a format capable of being processed by a system; the instruction execution and verification module is used for calling a corresponding test operation instruction based on the test case data, sending the test operation instruction to the vehicle-mounted instrument through the protocol adaptation module and the communication connection module, and verifying a test result based on the response data or state information obtained from the vehicle-mounted instrument; and the test process control module is used for controlling the execution process of the test instruction. According to the system and the method provided by the invention, the test automation level is improved.

Description

Technical Field

[0001] This invention relates to the field of automated testing technology, and in particular to an automated testing system and method for human-machine interaction of vehicle instruments. Background Technology

[0002] With the accelerated development of intelligent and connected vehicles, modern vehicle instrument panels have been transformed from traditional mechanical types to fully digital LCD displays, integrating various human-machine interface (HMI) functions such as navigation, multimedia, vehicle status monitoring, and driver assistance information display. Their interface logic is complex, interaction paths are diverse, and real-time performance and reliability requirements are high.

[0003] Current mainstream testing methods still rely on manual operation: testers trigger commands through physical buttons, visually observe the screen response, and manually record the results; some manufacturers use simple script tools based on USB or serial ports, or rely on vehicle diagnostic protocols to send basic control commands, but none of them have built closed-loop verification capabilities for HMI visual feedback and multimodal interaction behavior.

[0004] This manual and semi-automated approach has inherent limitations: the testing process cannot automatically trigger screenshots and image comparisons, making it difficult to cover visual changes such as icon lighting, text switching, and animation transitions; key parameters such as operation timing, pressure intensity, and response latency are uncontrollable, resulting in poor test repeatability; scripts need to be rewritten and personnel retrained for new car models or UI iterations, resulting in weak scalability; and it cannot simulate stable operating scenarios under high concurrency, long cycles, and boundary conditions, leading to insufficient test coverage and a high rate of missed defects. Summary of the Invention

[0005] This invention provides an automated testing system and method for human-machine interaction of vehicle instruments, which solves the technical problems of low efficiency, poor repeatability, error susceptibility, weak scalability and incomplete coverage caused by the high dependence on manual labor in the testing of human-machine interaction of vehicle instruments in the prior art.

[0006] This invention provides an automated testing system for human-machine interaction in vehicle instruments, comprising: The configuration management module is used to store and manage structured configuration data, which includes test case data, connection parameters of the vehicle instrument under test, and user interface element definition information. The communication connection module is used to establish and maintain a two-way communication link with the on-board instrument under test through a standardized communication interface; The protocol adaptation module is communicatively coupled to the communication connection module and is used to encode test operation commands into a protocol format supported by the vehicle instrument and decode response data received from the vehicle instrument into a format that the system can process. The instruction execution and verification module is communicatively coupled with the protocol adaptation module. It is used to call the corresponding test operation instructions based on the test case data, send them to the vehicle instrument through the protocol adaptation module and the communication connection module, and verify the test results based on the response data or the status information obtained from the vehicle instrument. The test process control module is communicatively coupled with the instruction execution and verification module, and is used to control the execution process of test instructions based on the test case data in the configuration management module.

[0007] According to the automated testing system for human-machine interaction of vehicle instrument provided by the present invention, the user interface element definition information includes: a user interface element template image that matches the resolution of the display screen of the vehicle instrument under test, and its position information in the screen coordinate system. Each template image corresponds to the icon, text area or indicator light display area that needs to be verified in the instrument HMI interface. The instruction execution and verification module is specifically used to: acquire the screen image of the vehicle instrument panel, and compare the screen image with the user interface element template image to verify whether the displayed content meets expectations.

[0008] According to the automated testing system for human-machine interaction of vehicle instrument provided by the present invention, the instruction execution and verification module is specifically used to: control the vehicle instrument to take a screenshot to obtain the screen image.

[0009] According to the automated testing system for human-machine interaction of vehicle instruments provided by the present invention, the structured configuration data adopts YAML format, the communication connection module establishes a Socket connection with the vehicle instrument through TCP / IP protocol, and the protocol adaptation module is specifically used to encode the test operation instructions into data packets in JSON format.

[0010] The automated testing system for human-machine interaction of vehicle instruments provided by the present invention further includes: The test report generation module is used to summarize the execution results of each test case and generate a test report that includes the test pass rate, error logs, and associated screenshots.

[0011] This invention also provides an automated testing method for human-machine interaction of vehicle instruments, comprising: Load structured configuration data to obtain the connection parameters and test case set of the on-board instrument under test; Based on the connection parameters, a communication connection is established with the vehicle instrument under test; Based on the test case set, the test cases are executed sequentially, wherein executing a single test case includes: Based on the operation defined in the test case, generate the corresponding control command and send the control command to the vehicle instrument panel; receive the response data returned by the vehicle instrument panel and parse the response data to obtain the parsing result; based on the parsing result, verify whether the response of the vehicle instrument panel meets expectations.

[0012] According to the automated testing method for human-machine interaction of vehicle instruments provided by the present invention, the step of generating corresponding control commands based on the operations defined in the test cases includes: The operations defined in the test cases are converted into predefined structured data; According to the protocol specifications for communicating with the vehicle instrument cluster, the structured data is encapsulated into data packets that the vehicle instrument cluster can recognize.

[0013] According to the automated human-machine interaction testing method for vehicle instrument provided by the present invention, after verifying whether the response of the vehicle instrument meets expectations, the method further includes: Based on the execution results of all test cases, generate and output a test report that includes the test pass rate, error logs, and associated screenshots; After all test cases have been executed, disconnect the communication connection with the vehicle instrument panel.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the human-machine interaction automated testing method for vehicle instruments as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the human-machine interaction automated testing method for vehicle instruments as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the human-computer interaction automated testing method for vehicle instruments as described above.

[0017] The automated testing system and method for human-machine interaction in vehicle-mounted instruments provided by this invention offer several advantages. The configuration management module provides structured and versionable test assets, freeing use case writing from code constraints. The communication connection module shields underlying connection details, enabling rapid system migration to different network topologies. The protocol adaptation module decouples instruction semantics from physical protocols, allowing the same set of test cases to be reused on MXC protocol instruments and CAN FD protocol instruments. The instruction execution and verification module integrates protocol response verification and visual content verification, extending test coverage from "functional accessibility" to "display correctness." The test flow control module provides flexible execution orchestration capabilities, enabling precise modeling of complex interaction logic. This effectively solves the technical problems of low efficiency, poor repeatability, error susceptibility, weak scalability, and incomplete coverage associated with manual testing, thereby improving test automation, enhancing system adaptability, ensuring result consistency, and expanding into higher-level verification scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the structural schematic diagrams of the automated testing system for human-machine interaction of vehicle instruments provided by the present invention.

[0020] Figure 2 This is one of the flowcharts of the automated testing method for human-computer interaction of vehicle instruments provided by the present invention.

[0021] Figure 3 This is the second structural schematic diagram of the automated testing system for human-machine interaction of vehicle instruments provided by the present invention.

[0022] Figure 4 This is the second flowchart of the automated testing method for human-computer interaction of vehicle instruments provided by the present invention.

[0023] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] With the development of automotive intelligence and connectivity, modern vehicle instrument clusters integrate navigation, multimedia, vehicle status display, and driver assistance functions. Their display interfaces have evolved from traditional mechanical types to full LCD screens, and HMI interfaces are becoming increasingly complex. Currently, vehicle instrument cluster testing mainly relies on manual operation: testers manually press buttons, observe the screen, and record the results. This method suffers from low efficiency, poor repeatability, error-proneness, poor scalability, and limited coverage. Therefore, this invention provides an automated human-machine interaction testing system for vehicle instrument clusters, such as... Figure 1 As shown, the system includes: The configuration management module 110 is used to store and manage structured configuration data, which includes test case data, connection parameters of the on-board instrument under test, and user interface element definition information. The communication connection module 120 is used to establish and maintain a two-way communication link with the on-board instrument under test through a standardized communication interface; The protocol adaptation module 130 is communicatively coupled with the communication connection module and is used to encode test operation commands into a protocol format supported by the vehicle instrument and decode the response data received from the vehicle instrument into a format that the system can process. The instruction execution and verification module 140 is communicatively coupled with the protocol adaptation module. It is used to call the corresponding test operation instructions based on the test case data, send them to the vehicle instrument through the protocol adaptation module and the communication connection module, and verify the test results based on the response data or the status information obtained from the vehicle instrument. The test process control module 150 is communicatively coupled with the instruction execution and verification module. It is used to control the execution process of test instructions based on the test case data in the configuration management module.

[0026] The automated testing system for human-machine interaction (HMI) of in-vehicle instruments provided in this invention adopts a modular, layered architecture design, with each module working together to complete a closed-loop testing task. The system is configuration-driven at its core, achieving controllable communication, pluggable protocols, programmable operations, and scalable verification through standardized interfaces. This allows for stable, efficient, and reproducible verification of in-vehicle instrument HMI functions without relying on manual intervention. Its core lies in constructing an automated testing execution framework with clearly defined responsibilities, sufficient decoupling, and protocol generalization and UI awareness capabilities. This framework not only meets the testing needs of current mainstream in-vehicle instruments (such as full LCD instruments based on AUTOSAR or QNX platforms and supporting TCP / IP or CAN FD communication), but also reserves expansion interfaces for future new interaction forms (such as voice feedback, gesture recognition linkage, and multi-screen collaboration).

[0027] The configuration management module serves as the system's data hub, responsible for the unified loading, parsing, and maintenance of structured configuration data. This module supports standard formats such as YAML, JSON, and XML, and manages three types of data: test case data (e.g., defining a "vehicle speed display test" including fields such as input operation sequence, expected interface changes, and timeout thresholds), connection parameters of the vehicle's instrument cluster under test (e.g., IP address, port number, reconnection strategy, heartbeat interval), and user interface element definition information (e.g., the coordinate rectangle area of ​​a certain alarm icon at 1280×480 resolution, the corresponding template image file path, and the matching confidence threshold). The configuration management module does not participate in business logic execution, only providing read-only data services; internally, it can employ a memory caching + file monitoring mechanism to trigger hot reload during configuration updates, avoiding system restarts.

[0028] The communication connection module acts as a bridge between the system and physical devices, establishing and maintaining a reliable, low-latency, and monitorable bidirectional communication link. This module encapsulates the underlying network communication details, providing unified send() and receive() interfaces. Internally, it can implement long-lived Socket connections based on the TCP / IP protocol stack, and can also be extended to support UDP multicast, WebSocket, serial ports (RS232 / RS485), or vehicle bus protocols. This module ensures the reliable transmission of test commands and the accurate reception of responses from vehicle instruments.

[0029] The protocol adaptation module resides above the communication layer and below the business logic layer, undertaking the function of protocol semantic conversion. The module takes high-level commands as input (such as "screenshot", "turn on left turn signal", "set current vehicle speed to 60km / h") and outputs data packets natively parsable by the instrument cluster. Conversely, it takes the instrument cluster's raw response byte stream as input and outputs structured objects (such as {"status":"success","timestamp":1733456789,"data":{"screenshot_path":" / tmp / cap_20251205_142301.png"}}). The module defines a base class for protocol processing (such as `base_protocol.py`) and provides specific implementations for specific protocols (such as the MXC protocol) (such as `mxc_protocol.py`). This module supports multiple protocols, such as control commands (button presses, screenshots) and pass-through mode protocols, enabling the framework to easily adapt to instrument clusters from different manufacturers and models.

[0030] The instruction execution and verification module is the main body for executing test behaviors and the central decision-making mechanism. It receives test case scheduling signals from the test flow control module, and based on the operation type declared in the test case data, calls the protocol adaptation module to generate instructions and sends them out via the communication connection module. Subsequently, it actively polls or passively listens for responses, performing multi-dimensional verification on the returned results: for status-based responses (such as a button success flag), it performs Boolean checks; for numerical responses (such as the current RPM value), it performs range checks; for visual responses (such as whether the instrument displays "ESC OFF"), it triggers image acquisition and template matching. This module itself does not embed any hardware interaction logic; all operations are injected through plug-in operation libraries (such as MxcOperations and CanOperations), ensuring separation of business logic and protocol implementation.

[0031] The instruction execution and verification module, based on the capabilities provided by the protocol adaptation module, transforms test logic (such as "set vehicle speed to 40km / h" and "light up all indicator lights") into specific protocol instructions. It contains a base operation class (`base_operations.py`) and implementations for specific business logic (`mxc_operations.py`), encapsulating atomic operations such as startup, shutdown, screenshotting, parameter setting, and result verification for use by the test flow control module.

[0032] The test flow control module is the scheduling engine of the entire system, responsible for transforming static test case data into dynamic execution sequences. Based on the test case set provided by the configuration management module, it triggers action calls from the instruction execution and verification module sequentially or conditionally. It supports control paradigms such as sequential execution, parallel execution (for independent sub-test cases), conditional branching (e.g., "skip the UI verification step if network connection fails"), and loop retries (e.g., "wait for 'Ready' status up to 5 times, with 200ms intervals"). The test flow control module combines complete test scenarios by calling interfaces provided by the instruction execution and verification module. For example, a test case can sequentially call operations such as "connect to instrument panel," "set vehicle speed," "screenshot verification," and "disconnect" to complete a functional test.

[0033] The five modules described above follow a strict unidirectional dependency relationship: the test process control module depends on the instruction execution and verification module, the instruction execution and verification module depends on the protocol adaptation module, the protocol adaptation module depends on the communication connection module, and the communication connection module and configuration management module have a parallel dependency relationship, jointly providing data and channel support to the upper layer. The modules interact through well-defined interface contracts, do not share memory or global variables, and all data transfer is completed in the form of immutable objects or message queues, ensuring system stability and testability.

[0034] Through the above technical solution, the embodiments of the present invention realize the fully automated closed-loop process of vehicle instrument HMI testing. Taking "verifying the consistency between the pointer position and the digital display of the speedometer at 40km / h" as an example: the test process control module reads the TC-001 test case in the configuration and calls the instruction execution and verification module to execute the "set vehicle speed = 40km / h" operation; the latter passes the semantic instruction to the protocol adaptation module, which encapsulates it into a JSON format data packet {"command":"set_speed","value":40,"unit":"km / h"}; the communication connection module sends the packet to the instrument via TCP Socket; after the instrument executes, it returns a confirmation response and the current interface snapshot path; the instruction execution and verification module parses the response to know that the operation is successful, and immediately calls the screenshot interface to obtain the real-time screen; then it loads the pre-stored "vehicle speed digital area" template image in the configuration, locates the area in the screen coordinate system and performs a structural similarity comparison to confirm that the "40" character is clearly distinguishable and has no afterimage; after all verifications are passed, the test process control module records the test case as "PASS" and starts the next test case.

[0035] Throughout the process, the configuration management module provides structured, versionable test assets, freeing use case writing from code constraints; the communication connection module shields the underlying connection details, enabling the system to be quickly migrated to different network topologies; the protocol adaptation module decouples command semantics from physical protocols, allowing the same set of test cases to be reused on MXC and CANFD protocol instruments; the command execution and verification module integrates both protocol response verification and visual content verification, extending test coverage from "functionally achievable" to "displaying correctly"; and the test flow control module provides flexible execution orchestration capabilities, enabling precise modeling of complex interactive logic (such as long-press + swipe combination operations). This effectively solves the technical problems of low efficiency, poor repeatability, error-proneness, weak scalability, and incomplete coverage in manual testing, thereby improving test automation, enhancing system adaptability, ensuring result consistency, and expanding into higher-level verification scenarios.

[0036] Based on the above embodiments, the user interface element definition information includes: a user interface element template image that matches the resolution of the vehicle instrument display screen under test, and its position information in the screen coordinate system. Each template image corresponds to the icon, text area, or indicator light display area that needs to be verified in the instrument HMI interface. The instruction execution and verification module is used to: acquire the screen image of the vehicle instrument, compare the screen image with the user interface element template image, and verify whether the displayed content meets expectations.

[0037] This embodiment constructs an automated verification mechanism for HMI display status based on image recognition. Specifically, instead of relying solely on status codes or data fields returned by the communication protocol to determine functional correctness, it achieves objective, reproducible, and high-precision verification of graphical output results such as icon visibility, text content, and indicator light status through spatial positioning and pixel / feature-level comparison between real screen images and pre-stored standard templates. This effectively bridges the testing blind spot between "protocol layer reachability" and "interface layer visibility," significantly improving the completeness and reliability of HMI testing.

[0038] The user interface element definition information includes two key elements: first, user interface element template images that strictly match the resolution of the vehicle instrument display screen under test; and second, the position information of each template image in the screen coordinate system. The template images are static image files (such as PNG and BMP formats), and their size, color depth, and DPI are calibrated and cropped according to the actual display parameters of the target instrument to ensure pixel-level alignment with the actual screenshot. The position information is described using a two-dimensional Cartesian coordinate system, with the upper left corner of the screen as the origin (0,0), the X-axis to the right, and the Y-axis downward. It records the vertex coordinates (x_min, y_min, x_max, y_max) or center point coordinates and width and height (x_c, y_c, w, h) of the rectangular area corresponding to the template image in the entire screen image.

[0039] Each template image corresponds to an icon, text area, or indicator light display area that needs to be verified in the instrument cluster HMI interface. Icons can be graphic controls with clear semantics, such as function buttons, status indicators, and navigation arrows; text areas are text content blocks such as fixed-position text labels, numerical displays, and menu items; indicator light display areas specifically refer to luminous areas with binary status (on / off), such as vehicle speed warning lights, engine malfunction lights, and seatbelt reminder lights.

[0040] Based on this, the instruction execution and verification module is used to: acquire the screen image of the vehicle instrument panel, compare the screen image with the user interface element template image to verify whether the displayed content meets expectations. As the main body for visual verification, this module's workflow is as follows: First, it triggers the screen image acquisition action (which can be achieved through controlled screenshots, a dedicated screenshot command issued by the communication layer, or by calling the instrument panel's built-in SDK interface); then, based on the location information loaded in the configuration management module, it precisely crops the sub-region to be verified from the entire screen image; next, it loads the corresponding template image and executes the preset comparison algorithm; finally, it generates a Boolean verification conclusion based on the comparison result.

[0041] This embodiment achieves automated, standardized, and quantifiable verification of the graphical output of the vehicle instrument HMI interface. For example, when executing the test case of "checking whether the engine malfunction indicator lamp is off after starting the vehicle," the system automatically captures the image of that area based on the malfunction indicator lamp template image and its screen coordinates defined in the configuration. Brightness analysis confirms that its average grayscale value is lower than the preset extinguishing threshold, thus determining that the display status meets expectations. Similarly, when verifying "whether the channel number text is correctly displayed when switching to a radio channel in the multimedia interface," the system crops the corresponding text area, calls OCR to recognize "FM 98.7," and compares it with the preset expected string in the test case. This achieves high-precision, high-reliability, and fully automated verification of the visual output status of the HMI interface.

[0042] Based on the above embodiments, the instruction execution and verification module is used to control the vehicle instrument to take screenshots to obtain screen images.

[0043] In existing technologies, most automotive instrument clusters lack standard video output interfaces (such as HDMI or LVDS external test ports) and do not support screen mirroring protocols such as USB Display or Miracast. While some instrument clusters have debugging serial ports or JTAG interfaces, these are only used for firmware burning and low-level diagnostics, and cannot export frame buffer images in real time. Using a camera to capture the screen is susceptible to problems such as ambient light, reflections, resolution distortion, difficulties in frame rate synchronization, and OCR recognition errors, resulting in unreliable and poorly repeatable visual verification results. Furthermore, it fails to meet the requirements of traceability and determinism of image data sources in ISO 26262 functional safety testing. Therefore, there is an urgent need for a screen image acquisition mechanism that does not rely on external hardware intervention and allows the device under test to actively respond and provide raw image data.

[0044] In this embodiment, the instruction execution and verification module is specially configured to generate and trigger "screenshot" type control instructions, rather than simply passively receiving external image input.

[0045] Controlling the vehicle's instrument cluster to take a screenshot refers to the following: The instruction execution and verification module, based on preset business rules (e.g., after completing a parameter setting operation or at a specific state transition moment), submits a semantically clear screenshot request object to the protocol adaptation module. This object includes at least a screenshot trigger condition identifier, the desired image format (e.g., PNG / JPEG), compression quality parameters (0–100), target display area coordinates (full screen / partial ROI), and a timeout threshold (default 3000ms). The protocol adaptation module then serializes this into a protocol data packet recognizable by the vehicle's instrument cluster. This instruction is then sent to the vehicle's instrument cluster's communication service process via an established TCP socket link through the communication connection module.

[0046] After receiving the screenshot command, the vehicle instrument panel calls the graphics subsystem API (such as the Linux DRM / KMS interface) to perform an atomic screenshot operation, generating raw pixel data (such as a memory buffer in RGBA8888 format), which is then compressed by the internal encoder into an image file or streaming data block in the specified format, and returned to the test system through the same communication channel.

[0047] This invention enables the acquisition of a realistic instrument panel rendering in a low-intrusion, high-fidelity, and reproducible manner in an in-vehicle embedded system without external video output capabilities.

[0048] Based on the above embodiments, the structured configuration data adopts YAML format, the communication connection module establishes a Socket connection with the vehicle instrument through TCP / IP protocol, and the protocol adaptation module encodes the test operation instructions into JSON format data packets.

[0049] Specifically, YAML, as a plain text data serialization format with strong human readability, clear structure, and support for comments and nesting, is chosen as the carrier of structured configuration data; TCP / IP (Transmission Control Protocol / Internet Protocol) forms the underlying network communication foundation, ensuring the reliability, orderliness, and integrity of command transmission; Socket, as an inter-process communication abstraction interface provided by the operating system, establishes an end-to-end connection between the application layer and the transport layer; JSON (JavaScript Object Notation), as a lightweight, language-independent, and easy-to-parse and generate data exchange format, undertakes the task of semantic encapsulation of test operation commands. Together, these three constitute a general technology stack for automotive HMI automated testing scenarios, taking into account maintainability, stability, and cross-platform compatibility.

[0050] YAML-formatted configuration data provides the system with an understandable and maintainable input source; TCP / IP socket connections provide a stable and monitorable channel for command transmission; and JSON encoding transforms abstract operational semantics into standard message units that this channel can carry. Together, these three elements form a complete and reliable mapping link from "human-editable test intentions" to "device-executable binary instructions."

[0051] Based on the above embodiments, the system further includes a test report generation module, which summarizes the execution results of each test case and generates a test report containing the test pass rate, error logs and associated screenshots.

[0052] Specifically, the test report generation module provided in this embodiment focuses on transforming the discrete test execution process into structured, traceable, and auditable quantitative output. This module does not participate in real-time control processes such as test command issuance or response parsing. Instead, after test case execution, its basic functions are data aggregation and formatted presentation, achieving the solidification of results and quality measurement throughout the entire testing process. This frees test results from reliance on manual recording, supports multi-dimensional statistical analysis, enhances defect reproducibility, and provides objective evidence for software quality assessment.

[0053] This module can be deployed on a local test host or on a centralized test management platform via a network service, and supports batch report merging and comparison with historical versions.

[0054] In this embodiment of the invention, after the automated testing system for human-machine interaction in vehicle-mounted instruments completes the execution of all test cases, the test report generation module automatically collects, structures, and visualizes the test results. This unifies discrete execution states, text logs, and image evidence within the same test context, solving the problems of difficulty in systematically archiving manual test results, incomplete fault evidence chains, and the inability to quantitatively assess quality trends.

[0055] Based on any of the above embodiments Figure 2 This is one of the flowcharts illustrating the automated testing method for human-machine interaction of vehicle-mounted instruments provided by this invention. For example... Figure 2 As shown, the method includes: loading structured configuration data to obtain the connection parameters and test case set of the vehicle instrument under test; establishing a communication connection with the vehicle instrument under test based on the connection parameters; executing test cases sequentially according to the test case set, wherein executing a single test case includes: generating corresponding control commands according to the operation defined in the test case and sending the control commands to the vehicle instrument; receiving response data returned by the vehicle instrument and parsing the response data to obtain the parsing result; and verifying whether the response of the vehicle instrument meets expectations based on the parsing result.

[0056] Specifically, the method provided in this embodiment of the invention takes structured configuration as the core driver, strictly separates test intent (use cases) from execution carrier (instructions); ensures reliable transmission of instructions and accurate capture of responses through a layered communication mechanism; achieves semantic unification of cross-vendor instrument instructions by relying on the protocol adaptation layer; and finally, the verification module transforms abstract expectations into calculable multi-dimensional objective indicators such as images, text, and audio.

[0057] Based on the above embodiments, corresponding control instructions are generated according to the operations defined in the test cases, including: converting the operations defined in the test cases into predefined structured data; and encapsulating the structured data into data packets that can be recognized by the vehicle instrument according to the protocol specifications for communication with the vehicle instrument.

[0058] Specifically, high-level interactive behaviors described in natural language or domain semantics in test cases (such as "press and hold the 'menu' button for 3 seconds", "switch to driving mode interface", "turn on the left turn indicator") are uniformly mapped by the test case parser into an intermediate representation (IR) data structure with fixed fields, types and hierarchical relationships.

[0059] Subsequently, the protocol adaptation module calls the corresponding wrapper according to the pre-registered protocol driver, serializing and formatting the structured data according to the communication protocol rules supported by the target instrument. The protocol specifications include, but are not limited to, MXC (Mitsubishi eXtended Communication) protocol, CAN FD (Controller Area Network with Flexible Data-Rate) application layer protocol, TCP / IP-based proprietary JSON-RPC protocol, or AUTOSAR-compliant Some / IP (Scalable service-Oriented Middleware over IP) message format.

[0060] Taking the MXC protocol as an example, the encapsulation process includes: extracting the action and parameters from the structured data, mapping them to MXC command codes (such as 0x0A indicating button trigger) and data field length; encoding the target as a combination of device address and function sub-index; converting parameters.duration_ms to a 16-bit unsigned integer and filling it into the data field; adding a CRC16 checksum and frame start / end identifiers, and finally generating a binary data packet conforming to the MXC physical frame format (such as 0x55 0x0A 0x00 0x01 0x00 0x64 0xXX XX 0xAA).

[0061] The encapsulation process features protocol pluggability—the system dynamically loads the corresponding protocol driver through factory mode. When the instrument under test is replaced with a new model that supports CAN FD, only the canfd_protocol.py driver needs to be deployed and the protocol type configured as "can_fd", without modifying the test case script or operation layer code. Similarly, if the instrument is upgraded to a domain controller architecture that supports Some / IP, a smooth migration can be achieved by accessing the someip_packager module.

[0062] Understandably, structured data, as an abstraction layer, carries the human-understandable HMI interaction intent; protocol encapsulation, as an adaptation layer, precisely translates it into physical signals that the instrument hardware can parse. Through these steps, a two-way decoupling is achieved between test case definition and the underlying communication protocol: on the one hand, test engineers can focus on writing HMI functional logic (such as "enter settings → select language → switch to German → verify title bar text") without needing to understand the MXC frame format or CAN message ID allocation rules; on the other hand, protocol developers can independently complete the development and integration of new protocol-driven systems without modifying any test scripts. This avoids the drawbacks of high maintenance costs, error-proneness, and difficulty in reusability caused by manually hard-coded instructions, significantly improving the maintainability, cross-platform adaptability, and long-term evolution capability of the test system.

[0063] Based on the above embodiments, after verifying whether the response of the vehicle instrument meets expectations, the method further includes: generating and outputting a test report containing the test pass rate, error logs and associated screenshots based on the execution results of all test cases; and disconnecting the communication connection with the vehicle instrument after all test cases have been executed.

[0064] Specifically, the execution results of all test cases refer to the structured data set collected by the system after all preset test cases have been fully run in this test session. This includes the unique identifier (ID) of each test case, its execution status (pass / fail / skip), execution time (in milliseconds), raw response data (such as a JSON-formatted return body), parsed key field values ​​(such as vehicle speed and indicator light status codes), image comparison confidence level (0.0–1.0 floating-point number), failure reason classification (protocol timeout, incorrect response format, UI element recognition failure, threshold exceeding limits, etc.), and the local storage path of the corresponding screenshot (such as / reports / 20251205_142301 / case_007_screenshot.png). This data set is written to the memory cache in real time by the test flow control module after each test case is executed, and is submitted to the test report generation module after all test cases have been traversed.

[0065] The test flow control module completes a full traversal of the test case list defined in the configuration management module, and the state of the last test case has been persisted to the execution result set. Simultaneously, the test report generation module confirms that the report file has been successfully written to disk and verifies its MD5 integrity. At this point, the system enters the final stage, where the communication connection module proactively initiates a connection termination process.

[0066] This embodiment achieves final-state control over the entire automated testing process of vehicle instruments: on the one hand, through a structured report generation mechanism, discrete testing behaviors are transformed into technical documents with statistical analysis value, fault location capabilities, and compliance proof validity, solving the problems of fragmented, unreproducible, and difficult-to-archive test results; on the other hand, through a deterministic connection release mechanism, the risk of subsequent test failures caused by long-term idle port occupation of Socket connections, memory leaks, or unresolved timeout sessions on the instrument side is eliminated, improving the stability and resource utilization of the system under continuous multi-round testing tasks.

[0067] Based on the above embodiments, such as Figure 3 As shown, an automated testing system for human-machine interaction in vehicle instruments is provided. This system adopts a layered architecture and includes: Data and Configuration Layer: As the top layer of the system, this layer is used to uniformly manage the data and configurations required for testing. This layer stores test case data (such as `test_cases.yaml`), system configuration files (such as `config.yaml`, containing target device IPs, ports, etc.), and UI element location libraries (such as `elements.yaml`, defining interface areas and template image paths for image recognition). This design achieves separation of data and logic, making testing more flexible and supporting both configuration-driven and data-driven testing.

[0068] Communication Layer: Responsible for establishing and maintaining communication connections with the vehicle's instrument cluster under test. This layer provides a standardized communication client interface (such as `base_client.py`) and can be implemented according to specific communication methods, for example, using `socket_client.py` to implement TCP / IP-based socket communication. This layer ensures the reliable sending of test commands and the accurate reception of instrument cluster responses.

[0069] Protocol Layer: Acting as a bridge between the communication layer and the operation layer, this layer is responsible for encoding upper-layer operation commands into data packets conforming to a specific instrument protocol format, and decoding data packets received from the instrument into a format understandable by the upper layer. This layer defines the base class for protocol processing (e.g., `base_protocol.py`) and provides specific implementations for specific protocols (e.g., `mxc_protocol.py`). It supports multiple protocols, such as control commands (button presses, screenshots) and pass-through mode protocols, allowing the framework to easily adapt to instruments from different manufacturers and models.

[0070] The operation layer defines all executable test operations. Based on the capabilities provided by the protocol layer, this layer translates test logic (such as "setting the vehicle speed to 40km / h" or "illuminating all indicator lights") into specific protocol instructions. It contains a base operation class (`base_operations.py`) and implementations for specific business logic (`mxc_operations.py`), encapsulating atomic operations such as startup, shutdown, screenshotting, parameter setting, and result verification for use by the test case layer.

[0071] Test Case Layer: Located at the lowest level of the system, this layer is responsible for orchestrating and executing specific test processes. Test cases are combined into complete test scenarios by calling interfaces provided by the operation layer. For example, a test case can sequentially call operations such as "connect to instrument panel," "set vehicle speed," "screenshot verification," and "disconnect" to complete a functional test.

[0072] like Figure 4 As shown, this embodiment of the invention also provides a corresponding automated testing method for human-computer interaction. This method is based on the aforementioned system, and its testing process includes the following steps: 1. Initialization: When the test system starts, it first loads the configuration file from the data and configuration layer to obtain the connection parameters and test case set of the target instrument.

[0073] 2. Establish connection: The communication layer establishes a communication connection with the instrument under test according to the configuration parameters.

[0074] 3. Execute the test: The system iterates through the test case set and executes them one by one.

[0075] a. For each test case, the test case layer invokes one or more operations defined in the operation layer.

[0076] b. The operation layer converts human-readable operations (such as pressing the OK button) into specific commands and parameters.

[0077] c. After receiving the command, the protocol layer encapsulates it into a binary or text data packet according to a predefined protocol format (such as JSON). For example, a key press command is encapsulated as `{"command": "key", "parameters": {"action": "enter", "status": "1"}}`.

[0078] d. The communication layer sends the encapsulated data packets to the instrument via methods such as Socket.

[0079] 4. Receiving and parsing the response: a. The communication layer receives the response data returned by the instrument.

[0080] b. The protocol layer parses the response data and extracts the execution status (success / failure) and result data (such as the screenshot path).

[0081] 5. Result Verification: The operation layer verifies the results based on the response. For scenarios requiring visual verification, a screenshot operation can be invoked, and the image can be compared with the data and the UI element templates defined in the configuration layer to determine whether the display meets expectations.

[0082] 6. Generate reports: Record the execution results of each test case and generate a detailed test report after all tests are completed.

[0083] 7. Disconnect: After all tests are completed, the communication layer disconnects from the instrument.

[0084] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute an automated human-machine interaction testing method for an in-vehicle instrument cluster. This method includes: loading structured configuration data to obtain connection parameters and a test case set for the in-vehicle instrument cluster under test; establishing a communication connection with the in-vehicle instrument cluster under test based on the connection parameters; and executing test cases sequentially according to the test case set. Executing a single test case includes: generating a corresponding control command based on the operation defined in the test case and sending the control command to the in-vehicle instrument cluster; receiving response data returned by the in-vehicle instrument cluster and parsing the response data to obtain a parsing result; and verifying whether the response of the in-vehicle instrument cluster meets expectations based on the parsing result.

[0085] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0086] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the human-machine interaction automated testing method for vehicle instruments provided by the above methods. The method includes: loading structured configuration data to obtain connection parameters and a test case set for the vehicle instrument under test; establishing a communication connection with the vehicle instrument under test based on the connection parameters; and executing test cases sequentially according to the test case set. Executing a single test case includes: generating a corresponding control command based on the operation defined in the test case and sending the control command to the vehicle instrument; receiving response data returned by the vehicle instrument and parsing the response data to obtain a parsing result; and verifying whether the response of the vehicle instrument meets expectations based on the parsing result.

[0087] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements an automated human-machine interaction testing method for vehicle-mounted instruments provided by the methods described above. This method includes: loading structured configuration data to obtain connection parameters and a test case set for the vehicle-mounted instrument under test; establishing a communication connection with the vehicle-mounted instrument under test based on the connection parameters; and sequentially executing test cases according to the test case set. Executing a single test case includes: generating a corresponding control command based on the operation defined in the test case and sending the control command to the vehicle-mounted instrument; receiving response data returned by the vehicle-mounted instrument and parsing the response data to obtain a parsing result; and verifying whether the response of the vehicle-mounted instrument meets expectations based on the parsing result.

[0088] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0089] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automated testing system for human-machine interaction of vehicle-mounted instruments, characterized in that, include: The configuration management module is used to store and manage structured configuration data, which includes test case data, connection parameters of the vehicle instrument under test, and user interface element definition information. The communication connection module is used to establish and maintain a two-way communication link with the on-board instrument under test through a standardized communication interface; The protocol adaptation module is communicatively coupled to the communication connection module and is used to encode test operation commands into a protocol format supported by the vehicle instrument and decode response data received from the vehicle instrument into a format that the system can process. The instruction execution and verification module is communicatively coupled with the protocol adaptation module. It is used to call the corresponding test operation instructions based on the test case data, send them to the vehicle instrument through the protocol adaptation module and the communication connection module, and verify the test results based on the response data or the status information obtained from the vehicle instrument. The test process control module is communicatively coupled with the instruction execution and verification module, and is used to control the execution process of test instructions based on the test case data in the configuration management module.

2. The automated testing system for human-machine interaction of vehicle-mounted instruments according to claim 1, characterized in that, The user interface element definition information includes: a user interface element template image that matches the resolution of the vehicle instrument display screen under test, and its position information in the screen coordinate system. Each template image corresponds to the icon, text area or indicator light display area that needs to be verified in the instrument HMI interface. The instruction execution and verification module is specifically used to: acquire the screen image of the vehicle instrument panel, and compare the screen image with the user interface element template image to verify whether the displayed content meets expectations.

3. The automated testing system for human-machine interaction of vehicle-mounted instruments according to claim 2, characterized in that, The instruction execution and verification module is specifically used to: control the vehicle instrument panel to take a screenshot to obtain the screen image.

4. The automated testing system for human-machine interaction of vehicle-mounted instruments according to claim 1, characterized in that, The structured configuration data is in YAML format. The communication connection module establishes a Socket connection with the vehicle instrument via TCP / IP protocol. The protocol adaptation module is specifically used to encode the test operation instructions into JSON format data packets.

5. The automated testing system for human-machine interaction of vehicle-mounted instruments according to any one of claims 1 to 4, characterized in that, The system also includes: The test report generation module is used to summarize the execution results of each test case and generate a test report that includes the test pass rate, error logs, and associated screenshots.

6. An automated testing method for human-machine interaction of vehicle-mounted instruments, characterized in that, include: Load structured configuration data to obtain the connection parameters and test case set of the on-board instrument under test; Based on the connection parameters, a communication connection is established with the vehicle instrument under test; Based on the test case set, the test cases are executed sequentially, wherein executing a single test case includes: Based on the operation defined in the test case, generate the corresponding control command and send the control command to the vehicle instrument panel; receive the response data returned by the vehicle instrument panel and parse the response data to obtain the parsing result; based on the parsing result, verify whether the response of the vehicle instrument panel meets expectations.

7. The automated testing method for human-machine interaction of vehicle-mounted instruments according to claim 6, characterized in that, The step of generating corresponding control instructions based on the operations defined in the test cases includes: The operations defined in the test cases are converted into predefined structured data; According to the protocol specifications for communicating with the vehicle instrument cluster, the structured data is encapsulated into data packets that the vehicle instrument cluster can recognize.

8. The automated testing method for human-machine interaction of vehicle-mounted instruments according to claim 6, characterized in that, After verifying whether the response of the vehicle instrument panel meets expectations, the method further includes: Based on the execution results of all test cases, generate and output a test report that includes the test pass rate, error logs, and associated screenshots; After all test cases have been executed, disconnect the communication connection with the vehicle instrument panel.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the human-machine interaction automated testing method for vehicle instruments as described in any one of claims 6 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the human-machine interaction automated testing method for vehicle instruments as described in any one of claims 6 to 8.

Images