Ui testing method for in-vehicle infotainment system and related device

Abstract

This application discloses a UI testing method and related equipment for an in-vehicle infotainment system, relating to the field of vehicle testing technology. The method includes: acquiring natural language testing requirements for the in-vehicle infotainment system; parsing the natural language testing requirements based on a preset dual-mode knowledge base to obtain a test task sequence composed of atomic steps; executing test operations according to the test task sequence through multiple preset intelligent agents to obtain UI layer execution results and vehicle signal layer execution results corresponding to each atomic step; and determining the test results for the in-vehicle infotainment system based on the UI layer execution results and the vehicle signal layer execution results. This application, through natural language requirement parsing, dual-mode knowledge base support, automatic execution by multiple intelligent agents, and comprehensive determination of the UI layer and vehicle signal layer, can automate the UI testing process of an in-vehicle infotainment system and improve the accuracy of test task generation and the reliability of test result judgment.

Description

Technical Field

[0001] This application relates to the field of vehicle testing technology, and more specifically, to a UI testing method and related equipment for an in-vehicle infotainment system. Background Technology

[0002] With the continuous development of automotive electronics and intelligent connectivity technologies, in-vehicle infotainment (IVI) systems are becoming increasingly sophisticated in their functions. They not only provide navigation, media playback, and vehicle settings, but also integrate various vehicle control functions such as climate control, seat adjustment, and driving mode settings. Therefore, the stability and reliability of IVI systems have a significant impact on the overall vehicle user experience and functional safety. During product development and verification, efficient and accurate UI testing of IVI systems has become a crucial technical means to ensure system quality.

[0003] In related technologies, UI testing for in-vehicle infotainment systems typically relies on manual testing or script-based automated testing. For example, functional verification is performed through manual interface operation, or pre-written test scripts simulate user actions and detect interface changes. However, these methods often primarily focus on the display state of the UI itself or page transitions, making it difficult to effectively verify whether interface operations truly trigger changes in the vehicle's underlying control signals. This results in test results remaining largely at the interface level, lacking verification of the actual execution of vehicle functions. Furthermore, as the functions and interface structures of in-vehicle infotainment systems become increasingly complex, traditional testing methods also have limitations in test task planning, step breakdown, and test execution automation, making it difficult to efficiently handle complex testing requirements. In other words, related technologies suffer from low levels of automation in UI testing for in-vehicle infotainment systems, insufficient test task parsing capabilities, and incomplete verification of test results. Summary of the Invention

[0004] In the summary section of this application, the relevant technical solutions are described in general terms, and a series of simplified concepts are introduced. These concepts will be further elaborated in the detailed embodiments section. This summary section should not be construed as limiting the key or essential technical features of the claimed solutions, nor is it intended to limit the scope of protection of the claimed solutions.

[0005] The UI testing method and related equipment for in-vehicle infotainment systems provided in this application can automate the UI testing process of in-vehicle infotainment systems through natural language requirement parsing, dual-mode knowledge base support, automatic execution of multiple agents, and comprehensive judgment of the UI layer and vehicle signal layer, thereby improving the accuracy of test task generation and the reliability of test result judgment.

[0006] In a first aspect, this application provides a UI testing method for an in-vehicle infotainment system, comprising: acquiring natural language testing requirements for the in-vehicle infotainment system; parsing the natural language testing requirements based on a preset dual-mode knowledge base to obtain a test task sequence composed of atomic steps, wherein the dual-mode knowledge base includes a structured relational database storing the mapping relationship between UI operations and vehicle signals and a UI knowledge graph storing UI semantics and operation paths; executing test operations according to the test task sequence through multiple preset intelligent agents to obtain UI layer execution results and vehicle signal layer execution results corresponding to each atomic step; and determining the test results for the in-vehicle infotainment system based on the UI layer execution results and the vehicle signal layer execution results.

[0007] In some implementations, the step of parsing the natural language testing requirements based on a preset dual-mode knowledge base to obtain a test task sequence composed of atomic steps includes: performing intent recognition on the natural language testing requirements using a preset large language model to obtain multiple candidate steps; determining the operation type of each candidate step based on the dual-mode knowledge base, wherein the operation type includes UI operation type and signal operation type; associating each candidate step with its corresponding operation type to obtain multiple atomic steps; and sorting and combining the multiple atomic steps to obtain the test task sequence.

[0008] In some implementations, determining the operation type of each candidate step among the plurality of candidate steps based on the dual-mode knowledge base includes: for each candidate step, performing a matching query in the structured relational database and the UI knowledge base respectively; if a vehicle signal mapping relationship corresponding to the candidate step is matched in the structured relational database, then the operation type of the candidate step is determined as the signal operation type; if an interface element or operation path corresponding to the candidate step is matched in the UI knowledge graph, then the operation type of the candidate step is determined as the UI operation type.

[0009] In some implementations, the step of executing test operations according to the test task sequence through multiple preset intelligent agents to obtain the UI layer execution result and vehicle signal layer execution result corresponding to each atomic step includes: for atomic steps in the test task sequence whose operation type is UI operation, executing UI operations based on the UI knowledge graph through a UI operation intelligent agent to obtain the UI layer execution result corresponding to the atomic step; for atomic steps in the test task sequence whose operation type is signal operation, executing signal operations based on the structured relational database through a data signal intelligent agent to obtain the vehicle signal layer execution result corresponding to the atomic step; and monitoring the execution process of each atomic step in parallel through an execution monitoring intelligent agent to verify the UI layer execution result and vehicle signal layer execution result of each atomic step.

[0010] In some implementations, the step of executing UI operations based on the UI knowledge graph through a UI operation agent to obtain a UI layer execution result corresponding to the atomic step includes: obtaining a semantic description of the target element corresponding to the atomic step through the UI operation agent; querying the UI knowledge graph for an operation path and expected interface state that match the semantic description of the target element; generating a UI automated execution instruction based on the operation path and the real-time interface state of the in-vehicle infotainment system, wherein the UI automated execution instruction includes at least one of clicking, swiping, and input; and determining the UI layer execution result based on a comparison between a screenshot of the interface after the UI automated execution instruction is executed and the expected interface state.

[0011] In some implementations, the step of performing signal operations based on the structured relational database using a data signal agent to obtain the vehicle signal layer execution result corresponding to the atomic step includes: obtaining the signal semantic description corresponding to the atomic step through the data signal agent; querying the structured relational database for vehicle signal identifiers and expected signal values ​​that match the signal semantic descriptions; reading or setting the vehicle bus signal corresponding to the vehicle signal identifier through the vehicle communication interface to obtain the actual signal value; and comparing the actual signal value with the expected signal value to obtain the vehicle signal layer execution result.

[0012] In some implementations, the step of monitoring the execution process of each atomic step in parallel by an execution monitoring agent to verify the UI layer execution result and the vehicle signal layer execution result of each atomic step includes: for each atomic step in the test task sequence, after executing the corresponding UI operation or signal operation, synchronously triggering a first verification operation or a second verification operation; verifying the interface state after execution using the first verification operation to obtain a UI layer verification result; verifying the vehicle bus signal after execution using the second verification operation to obtain a signal layer verification result; and verifying the UI layer execution result and the vehicle signal layer execution result based on the UI layer verification result and the signal layer verification result.

[0013] Secondly, this application also provides a UI testing device for an in-vehicle infotainment system, comprising: a requirement acquisition unit for acquiring natural language testing requirements for the in-vehicle infotainment system; a requirement parsing unit for parsing the natural language testing requirements based on a preset dual-mode knowledge base to obtain a test task sequence composed of atomic steps, wherein the dual-mode knowledge base includes a structured relational database storing the mapping relationship between UI operations and vehicle signals and a UI knowledge graph storing UI semantics and operation paths; a test execution unit for executing test operations according to the test task sequence through multiple preset intelligent agents to obtain UI layer execution results and vehicle signal layer execution results corresponding to each atomic step; and a result acquisition unit for determining the test results for the in-vehicle infotainment system based on the UI layer execution results and the vehicle signal layer execution results.

[0014] Thirdly, this application also provides an electronic device, including: a memory and a processor, wherein the processor is configured to execute a computer program stored in the memory to implement the UI testing method of the in-vehicle infotainment system described in the first aspect.

[0015] Fourthly, this application also provides a computer-readable storage medium storing computer-executable instructions or a computer program, wherein when the computer-executable instructions or the computer program are executed by a processor, the steps of the UI testing method for the in-vehicle infotainment system described in the first aspect are implemented.

[0016] Fifthly, this application also provides a computer program product, including a computer program or computer executable instructions, wherein when the computer program or computer executable instructions are executed by a processor, the steps of the UI testing method for the in-vehicle infotainment system provided in the embodiments of this application are implemented.

[0017] In summary, this application obtains natural language test requirements for in-vehicle infotainment systems, enabling these requirements to be expressed in natural language. This reduces reliance on manually written complex test scripts. Directly receiving natural language test requirements allows testers to more easily describe test objectives and scenarios, improving the flexibility of test requirement input and providing a foundation for the generation of subsequent automated test tasks. By parsing the natural language test requirements based on a pre-set dual-mode knowledge base and generating a sequence of test tasks composed of atomic steps, complex test requirements can be broken down into multiple minimally executable steps, facilitating step-by-step execution by the system. The structured relational database provides the mapping relationship between UI operations and vehicle signals, while the UI knowledge graph provides UI semantic information and operation path information. Their synergistic effect facilitates the solution of test tasks. The method provides support for analysis and step generation, improving the accuracy and executability of test task generation. By having multiple pre-set agents execute test operations according to the test task sequence, different atomic steps can be automatically completed by the corresponding agents, thus achieving automated execution of the test process. The agents sequentially complete corresponding operations according to the task sequence and output corresponding execution results, reducing manual intervention, improving test execution efficiency, and making the test process more standardized and controllable. After executing the test operations, the method obtains the UI layer execution results and the vehicle signal layer execution results respectively, and obtains the final test result based on these two types of results. This allows for judgment of the test operation results from both the interface performance and vehicle signal levels. By comprehensively analyzing the execution results of the UI layer and the vehicle signal layer, the judgment of test results becomes more comprehensive, thereby improving the reliability of the in-vehicle infotainment system test results. In summary, the UI testing method for in-vehicle infotainment systems provided in this application, through natural language requirement parsing, dual-mode knowledge base support, multi-agent automatic execution, and comprehensive judgment of the UI layer and vehicle signal layer, can achieve automated execution of the UI testing process for in-vehicle infotainment systems and improve the accuracy of test task generation and the reliability of test result judgment. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit this specification. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a UI testing method for an in-vehicle infotainment system provided in an embodiment of this application; Figure 2 A schematic diagram of the composition structure of a UI testing device for an in-vehicle infotainment system provided in an embodiment of this application; Figure 3 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The terms used in the specification, claims, and drawings of this application, such as "first," "second," "third," "fourth," etc. (if any), are used to distinguish similar objects and not to describe a specific order or sequence. Therefore, it is to be understood that these terms can be used interchangeably where appropriate, allowing the described embodiments to be used in different orders, unless specifically required by the illustrations or description. Furthermore, the terms "is" and "has," and any variations thereof, are intended to cover, non-exclusively, all possible constituent elements. For example, a process, method, system, product, or apparatus comprising several steps or units is not necessarily limited to the steps or units explicitly listed, but may also include other steps or units not explicitly listed, or steps or units inherent to the process, method, product, or apparatus.

[0020] In this application, a "module" or "unit" refers to a computer program or part of a computer program that has a specific function and works in conjunction with other related parts to achieve a predetermined goal. These modules or units can be implemented by software, hardware (e.g., processing circuitry or memory), or a combination of both. One or more processors or memories can implement one or more modules or units. Furthermore, each module or unit can also be part of a larger module or unit.

[0021] The technical solutions of this application will be described in detail below with reference to the accompanying drawings of the embodiments. It should be noted that the described embodiments are only a part of this application, and not all embodiments. In the following description, the "some embodiments" mentioned are only a subset of all possible embodiments, which may be the same or different subsets, and different embodiments can be combined with each other without conflict.

[0022] Figure 1 This is a flowchart illustrating a UI testing method for an in-vehicle infotainment system provided in an embodiment of this application. For example, see [link to example]. Figure 1 The UI testing method for an in-vehicle infotainment system provided in this application embodiment may include the following steps 101 to 104: Step 101: Obtain the natural language testing requirements for the in-vehicle infotainment system.

[0023] In some examples, the in-vehicle infotainment system is an integrated in-vehicle human-machine interaction system installed in the vehicle, which is the test object of this UI test. This system integrates information display, entertainment experience and interactive functions related to the core control of the vehicle. It is the main carrier of the vehicle user interface operation and does not require additional acquisition operations. The in-vehicle infotainment system to be tested may include functional modules such as in-vehicle navigation, Bluetooth media playback, in-vehicle air conditioning control, seat heating adjustment, and driving mode switching. For example, the in-vehicle infotainment system installed in a certain pure electric new energy vehicle can realize multiple operations such as adjusting the vehicle air conditioning temperature, setting the heating level of the driver and passenger seats, and playing online music from the system interface. All of these operations are related to the underlying signals of the vehicle. Natural language testing requirements are UI testing requirements for in-vehicle infotainment systems presented in the form of everyday natural language. These requirements include information such as the functional objects to be tested, operational actions, and test objectives. They serve as the input source for subsequent test task analysis based on a dual-mode knowledge base. These requirements can be obtained in several ways: firstly, through the visual input interface of the test management terminal deployed in the test environment, where testers manually input the natural language testing requirements in text form; secondly, through the voice acquisition component of the test management terminal receiving the test requirements in voice form, which is then converted into standardized text format by the terminal's voice recognition program; and thirdly, by retrieving pre-edited natural language testing requirement text from a pre-set test requirement text library on the test management terminal. Examples of natural language testing requirements include testing the UI operation of automatically cooling the air conditioning on the in-vehicle infotainment system's homepage, verifying the activation process of the Bluetooth music playback function in the in-vehicle infotainment system, and testing the effectiveness of the UI operation of the seat ventilation function in the in-vehicle infotainment system.

[0024] For example, testers can directly input natural language test requirements for the vehicle's in-vehicle infotainment system through the text input box on the test management terminal, which establishes a communication connection with the vehicle under test. After receiving the natural language test requirements, the test management terminal will standardize and store the requirements and synchronously transmit them to the corresponding test parsing module, providing complete and standardized input data for the subsequent test requirement parsing steps based on the dual-mode knowledge base. In batch testing or rapid testing scenarios, testers can also directly retrieve matching natural language test requirements from the preset text library of the test management terminal, completing the acquisition step without manual input.

[0025] By implementing step 101, natural language test requirements for in-vehicle infotainment systems are obtained, enabling testers to directly describe test objectives or test scenarios in natural language without having to write complex automated test scripts or detailed operating procedures in advance. This improves the flexibility of test requirement expression, facilitates the unified reception and processing of test requirements, and provides a foundation for subsequent test task parsing and automated execution.

[0026] Step 102: Based on the preset dual-mode knowledge base, the natural language testing requirements are analyzed to obtain a test task sequence composed of atomic steps.

[0027] The dual-mode knowledge base includes a structured relational database that stores the mapping relationship between UI operations and vehicle signals, and a UI knowledge graph that stores UI semantics and operation paths.

[0028] In some examples, the pre-built dual-mode knowledge base is a dual knowledge storage architecture that is pre-built and deployed for UI testing of in-vehicle infotainment systems. This architecture integrates two types of knowledge carriers: structured relational databases and UI knowledge graphs, and serves as the knowledge foundation for parsing natural language testing requirements. It can be built offline before testing. By collecting, analyzing, and modeling multi-dimensional data from the in-vehicle infotainment system, the structured relational database and UI knowledge graph are built separately. The two are then linked and integrated to form a dual-mode knowledge base, which is deployed to the dedicated database server of the testing system and directly called by the testing system during testing. For example, for the in-vehicle infotainment system of a pure electric new energy vehicle, the pre-built dual-mode knowledge base fully covers the vehicle signal mapping relationships corresponding to all UI operations of the system, as well as the UI semantics, functional operation paths, and other related knowledge of the entire interface.

[0029] The offline construction process of this pre-defined dual-mode knowledge base is divided into three stages: multimodal data collection, multimodal model training, and dual-mode knowledge base generation. It relies on a dedicated intelligent program and deep learning model to complete data collection, relational learning, and knowledge carrier construction. The multimodal data acquisition phase utilizes two specialized tools to collect and initially process multi-dimensional vehicle data. First, it employs a breadth-first search (BFS) approach to traverse the in-vehicle infotainment system using a user interface traversal agent. This approach visits system interface nodes hierarchically, starting with the closest nodes and progressing to the next level. During this traversal, it simultaneously collects three types of user interface data: screenshots of the user interface (UI), control trees, and operation paths. Furthermore, it ensures that all data timestamps are aligned and associated with each other, guaranteeing data relevance and validity. Second, it uses signal analysis tools to professionally analyze the database file (Database CAN, DBC) and signal matrix-related documents of the vehicle controller's local area network. This extracts core fields such as DBC signal description information and signal key-value correspondences, which are then uniformly stored in the database, completing the initial storage of vehicle bus signal data. In the multimodal model training phase, a deep learning model is established. Through model training, it learns and masters three core relationships within the in-vehicle infotainment system: the mapping relationship between visual features of user interface elements and functional semantics, the causal relationship between user operations and vehicle signal changes, and the paths and constraints of interface state transitions. This deep learning model is built on a Vision-Language Model (VLM) architecture and uses multimodal data from the automotive domain for targeted training and fine-tuning. The specific training strategy consists of two steps: first, pre-training model selection, using general-purpose visual-language models such as CLIP and BLIP, which have strong vision-language alignment capabilities, as the basic architecture; second, professional domain data injection, using the collected in-vehicle multimodal data, namely, integrated data of user interface screenshots, control trees, operation paths, and vehicle signal data, as training data to adapt the general-purpose visual-language model for the automotive domain.The training of the three core associations each has clear input data and training objectives, and corresponding training examples: For the mapping relationship between visual features of user interface elements and functional semantics, the input is a screenshot of the user interface and functional description text. The training objective is to enable the model to understand the meaning of icons and controls unique to the vehicle system. For example, if the input is a screenshot of the vehicle system's air conditioning interface, the model can output "Air conditioning control interface, including temperature adjustment, airflow control, mode selection and other functional areas". If the input is a screenshot of the seat heating button, the model can output "Seat heating switch, current state: On". For the causal relationship between user operation and vehicle signal changes, the input is a description of the signal operation and signal data. The training objective is to establish the causal relationship between the operation and the vehicle signal. For example, if the input is "Set driving mode sport mode", the model can learn that this operation corresponds to executing the signal setting instruction setsignaldrive_mode=2. For the path and constraints of interface state transition, the input is a screenshot of the current interface, a description of the target function, and a sequence of reachable paths. The training objective is to enable the model to learn the optimal operation path to complete a specific function. For example, if the input is "Currently on the main interface, play Bluetooth music", the model can output the corresponding operation sequence: click the media icon - select the audio source - select Bluetooth - play music. In the dual-mode knowledge base generation phase, based on the various relational results obtained from multimodal model training, a structured relational database and a user interface knowledge graph are built, which together constitute the core knowledge carrier of the dual-mode knowledge base. The structured relational database stores precise mapping relationships between user interface elements, operations, and vehicle signals, providing accurate numerical and identifier basis for signal operation judgment and execution. The user interface knowledge graph stores semantic relationships and functional operation path information of the in-vehicle infotainment system's user interface, providing semantic and logical basis for user interface operation path planning and execution. After the independent construction of the two knowledge bases is completed, an association index and collaborative query mechanism are established between them to achieve linked querying and data exchange between the structured relational database and the user interface knowledge graph. This ensures that the two knowledge bases can be collaboratively invoked during subsequent test requirement analysis, providing comprehensive and professional knowledge support for the accurate analysis of natural language testing requirements.

[0030] The UI operation and vehicle signal mapping relationship is a one-to-one correspondence between various UI operations performed on the in-vehicle infotainment system and the specific changes in the underlying vehicle bus signals triggered by these operations. For example, clicking the 25-degree Celsius temperature adjustment button on the in-vehicle infotainment system's air conditioning control interface sets the target air conditioning temperature signal in the vehicle controller area network to 25. The structured relational database storing this UI operation and vehicle signal mapping relationship is a database system that organizes data in a standardized tabular format. The database stores the UI operation and vehicle signal mapping relationship, and the data organization is deterministic and unique, supporting precise querying and retrieval by the test system. For example, in this structured relational database, data is stored in rows and columns. One row of data has the operation name "Turn on passenger seat heating at level 1," the corresponding signal identifier is "Seat_Heat_Passenger," and the expected signal value is "1," forming a precise one-to-one correspondence between the fields.

[0031] UI semantics refers to the specific functional meanings and standardized expressions corresponding to the various interfaces and functional elements within an in-vehicle infotainment system. UI operation paths, starting from an initial interface of the in-vehicle infotainment system, represent the ordered sequence of interface jumps and element clicks required to complete a specific UI operation. Before testing, a full traversal of all interfaces of the in-vehicle infotainment system can be performed to systematically analyze the UI semantics corresponding to each interface and functional element. Simultaneously, various functional operations can be actually executed, recording the complete operation path for each function, thus forming standardized UI semantics and operation path knowledge. For example, in terms of UI semantics, the "Air Conditioning" icon on the in-vehicle infotainment system homepage corresponds to the air conditioning control function, the "Media" icon corresponds to the audio / video playback function, and the "Vehicle" icon corresponds to the vehicle control function. In terms of UI operation paths, the operation path from the in-vehicle infotainment system homepage to the air conditioning control interface is to click the "Air Conditioning" icon on the homepage; the operation path from the homepage to the driving mode adjustment interface is to click the "Vehicle" icon on the homepage and then click the "Driving Mode" option on the sub-interface. A UI knowledge graph that stores UI semantics and operation paths is a technical carrier that organizes knowledge in the form of a graph structure. This knowledge graph is specifically used to store the UI semantics and operation paths of in-vehicle infotainment systems. Nodes represent interfaces, functional elements, and functional concepts, while edges represent semantic and operation path relationships between various nodes, supporting semantic querying and reasoning in the test system. For example, this UI knowledge graph contains nodes such as "In-vehicle Infotainment System Homepage," "Air Conditioning Control Interface," and "Temperature Adjustment Button." The edge between "In-vehicle Infotainment System Homepage" and "Air Conditioning Control Interface" represents "Clicking the air conditioning icon," and the edge between "Temperature Adjustment Button" and "Air Conditioning Temperature Adjustment Function" represents "Corresponding function execution," forming both a complete operation path relationship and a clear functional semantic relationship.

[0032] An atomic step is the smallest executable test operation unit obtained after parsing a natural language testing requirement. This unit cannot be further decomposed and is the basic unit constituting a test task sequence. Based on various knowledge stored in a pre-defined dual-mode knowledge base, semantic parsing and step decomposition of natural language testing requirements can be performed to break down complex, non-standardized testing requirements into several indivisible operation units, thus obtaining atomic steps. A test task sequence is an ordered sequence of test operations formed by arranging several atomic steps according to the actual logical execution order of completing the natural language testing requirement. For example, for the natural language testing requirement "Test the automatic cooling function of the air conditioning from the home page of the vehicle infotainment system," two atomic steps are obtained. The test task sequence formed by arranging them according to the actual operation logic is: first, click the air conditioning icon on the home page of the vehicle infotainment system; second, click the automatic cooling button on the air conditioning control interface.

[0033] By implementing step 102, the natural language testing requirements are parsed based on the preset dual-mode knowledge base, and a test task sequence composed of atomic steps is generated. This can break down complex testing requirements into multiple minimum executable steps, thereby facilitating the system to execute specific test operations step by step. Among them, the structured relational database is used to provide the mapping relationship between UI operations and vehicle signals, and the UI knowledge graph is used to provide UI semantic information and operation path information. The synergy between the two can provide data support for test task parsing, and can improve the accuracy and executability of test task generation.

[0034] Step 103: Through multiple preset intelligent agents, perform test operations according to the test task sequence to obtain the UI layer execution results and vehicle signal layer execution results corresponding to each atomic step.

[0035] In some examples, multiple pre-defined intelligent agents are intelligent software modules developed and functionally debugged in advance for UI testing of in-vehicle infotainment systems and deployed in the testing system. Each module has its own dedicated test operation execution capability and can complete the entire process of the test task sequence through collaborative cooperation. Before the test, corresponding intelligent software modules can be developed according to the actual needs of UI testing of in-vehicle infotainment systems, such as operation execution and data collection. After the module functions are debugged and integrated, all modules are deployed to the execution end of the testing system. During the test, the testing system uniformly schedules and calls them according to the execution requirements of the test task sequence. For example, for the test requirements of interface operation execution, vehicle underlying signal collection, and interface status recognition of in-vehicle infotainment systems, multiple intelligent software modules developed together constitute multiple pre-defined intelligent agents. Each module can independently complete dedicated test-related operations such as interface clicks, signal reading, and image acquisition.

[0036] The UI layer execution result is the actual interface state information presented by the in-vehicle infotainment system after executing the test operation corresponding to a certain atomic step according to the test task sequence. It is data reflecting the execution effect of the atomic step at the interface level and is also a direct basis for verifying whether the UI operation is effective at the interface level. After executing the test operation of the atomic step according to the test task sequence, the actual interface state of the in-vehicle infotainment system can be collected through image acquisition, interface element recognition and other technologies. After standardizing and organizing the collected interface information, the UI layer execution result corresponding to the atomic step can be obtained. For example, after executing the atomic step of "clicking the media icon on the home page of the in-vehicle infotainment system", the corresponding UI layer execution result is that the in-vehicle infotainment system interface successfully jumps to the media playback control interface. The vehicle signal layer execution result is the actual change data of the vehicle's underlying bus signals after executing the test operation corresponding to a certain atomic step according to the test task sequence. It reflects whether the operation corresponding to the atomic step triggers the actual effect of the vehicle's underlying function and is also a direct basis for verifying whether the UI operation is effectively implemented from the interface layer to the vehicle function layer. After executing the test operation of the atomic step according to the test task sequence, the actual signal value of the vehicle's underlying bus can be read through the communication interface adapted to the vehicle bus. After standardizing, parsing, and organizing the read signal data, the vehicle signal layer execution result corresponding to the atomic step can be obtained. The vehicle's underlying bus includes types such as Controller Area Network (CAN) and Local Interconnect Network (LIN). For example, after executing the atomic step of "clicking the 26-degree Celsius temperature adjustment button on the air conditioning control interface of the in-vehicle infotainment system", the corresponding vehicle signal layer execution result is that the actual value of the air conditioning target temperature signal in the Controller Area Network signal is 26.

[0037] The process of obtaining UI layer and vehicle signal layer execution results corresponding to each atomic step by executing test operations according to a test task sequence using multiple pre-set intelligent agents involves the following steps: First, the scheduling module of the test system assigns corresponding operation execution and data acquisition tasks to the multiple pre-set intelligent agents based on the atomic step type and execution order of the test task sequence. Then, the multiple pre-set intelligent agents execute the test operations according to the assigned tasks and the order of the test task sequence. During the operation execution, the corresponding pre-set intelligent agents simultaneously complete the acquisition and processing of UI layer state and vehicle signal layer data. Finally, the acquired and processed dual-layer results are sent back to the test system, which then completes the comparison and processing of the results with the original data. Matching and associating sub-steps; for example, for the test task sequence "first step, click the vehicle icon on the home page of the in-vehicle infotainment system, second step, click the driver's seat heating level 1 button on the vehicle control interface", the test system schedules multiple preset intelligent agents to first execute the first step operation. After execution, the UI layer execution result corresponding to the first step is that the interface jumps to the vehicle control interface, and the vehicle signal layer execution result is that there is no change in the bus signal value. Then, the second step operation is executed. After execution, the UI layer execution result corresponding to the second step is that the driver's seat heating level 1 button is lit, and the vehicle signal layer execution result is that the driver's seat heating signal value in the controller area network signal is set to 1.

[0038] By implementing step 103, multiple preset agents are used to execute test operations according to the test task sequence, which enables different steps in the test task to be automatically completed by the corresponding agents, thereby realizing the automated execution of the test process. During the execution of each atomic step, the corresponding UI layer execution results and vehicle signal layer execution results are obtained simultaneously, which can record the changes in the interface state and the changes in the vehicle signal after the interface operation, thereby providing basic data for the subsequent test result determination.

[0039] Step 104: Based on the execution results of the UI layer and the vehicle signal layer, determine the test results for the in-vehicle infotainment system.

[0040] In some examples, the test result is the final judgment obtained after comprehensively analyzing the entire testing process of the corresponding UI function of the in-vehicle infotainment system based on the input natural language testing requirements. This conclusion integrates the information of the UI layer execution results and vehicle signal layer execution results of each atomic step, which can clearly reflect whether the UI operation within the test scope meets expectations and whether the corresponding vehicle function is actually effective. The UI layer execution results and vehicle signal layer execution results corresponding to each atomic step can be retrieved from the previous storage. According to the unified test judgment rules preset in the system, the two-layer execution results are comprehensively analyzed, logically judged and integrated to finally form a standardized judgment conclusion, that is, the test result for the in-vehicle infotainment system. This result will be stored and visualized in a standardized manner simultaneously. For example, the test results are divided into two categories: qualified and unqualified. One is the test result for the natural language testing requirement "turn on the driver's seat heating level 1 from the in-vehicle infotainment system home page". The judgment basis is that the UI layer execution results of each atomic step are consistent with expectations and the vehicle signal layer execution results also meet the numerical requirements for the function to be effective.

[0041] Exemplarily, a unified test determination rule can be preset in the test system in advance. Then, the result determination module of the test system can completely retrieve the UI layer execution results and vehicle signal layer execution results of each atomic step from the result storage module. Independently determine the double-layer execution results of each atomic step in sequence according to the order of the test task sequence. Subsequently, summarize the independent determination results of all atomic steps for overall analysis, eliminate invalid data during the test execution process, integrate valid determination conclusions, and finally form a standardized UI test result of the in-vehicle infotainment system while completing the storage and display of the test result. For example, for the natural language test requirement "Test the function of turning on the automatic air conditioning cooling function from the home page of the in-vehicle infotainment system", its corresponding test task sequence includes two atomic steps. The test system first independently determines the first step "Click the air conditioning icon on the home page of the in-vehicle infotainment system". The UI layer execution result is that the interface jumps to the air conditioning control interface as expected, and there is no change requirement for the vehicle signal layer in this step, so it is determined to be effectively executed. Then, independently determine the second step "Click the automatic cooling button on the air conditioning control interface". The UI layer execution result is that the automatic cooling button is lit as expected, and the vehicle signal layer execution result is that the air conditioning cooling mode signal in the controller area network is set to the corresponding value of automatic cooling, which is in line with expectations, so it is determined to be effectively executed. The test system summarizes the independent determination results of the two atomic steps, both of which are effectively executed, and comprehensively obtains that the test result for the in-vehicle infotainment system under this test requirement is qualified. Also, for the same test requirement, the UI layer execution result of the second step is that the automatic cooling button is lit as expected, but the vehicle signal layer execution result is that there is no change in the air conditioning cooling mode signal in the controller area network. It is determined that the second step is not effectively executed. After summarization by the test system, the comprehensive test result for the in-vehicle infotainment system is unqualified, and the determination basis is clearly that the interface display is in line with expectations but the underlying vehicle function has not taken effect.

[0042] Through the implementation of step 104, the test result for the in-vehicle infotainment system is obtained based on the UI layer execution result and the vehicle signal layer execution result, and the execution situation of the test operation can be comprehensively analyzed from two aspects of the interface performance and vehicle signals. Among them, the UI layer execution result is used to reflect the change of the interface state, and the vehicle signal layer execution result is used to reflect the change of the vehicle function-related signals. By comprehensively determining the two types of execution results, the comprehensiveness and reliability of the test result judgment can be improved.

[0043] In summary, this application's embodiments, by acquiring natural language test requirements for in-vehicle infotainment systems, enable these requirements to be expressed in natural language, reducing reliance on manually written complex test scripts. Directly receiving natural language test requirements allows testers to more easily describe test objectives and scenarios, improving the flexibility of test requirement input and providing a foundation for subsequent automated test task generation. By parsing the natural language test requirements based on a pre-set dual-mode knowledge base and generating a test task sequence composed of atomic steps, complex test requirements can be broken down into multiple minimally executable operation steps, facilitating step-by-step execution by the system. The structured relational database provides the mapping relationship between UI operations and vehicle signals, while the UI knowledge graph provides UI semantic information and operation path information. Their synergistic effect provides a foundation for test task generation. The parsing and step generation provide support, improving the accuracy and executability of test task generation. By having multiple pre-set agents execute test operations according to the test task sequence, different atomic steps can be automatically completed by the corresponding agents, thus achieving automated execution of the test process. The agents sequentially complete corresponding operations according to the task sequence and output corresponding execution results, reducing manual intervention, improving test execution efficiency, and making the test process more standardized and controllable. After executing the test operations, the UI layer execution results and vehicle signal layer execution results are obtained separately, and the final test result is derived based on these two types of results. This allows for judgment of the test operation results from both the interface performance and vehicle signal perspectives. By comprehensively analyzing the execution results of the UI layer and the vehicle signal layer, the judgment of test results becomes more comprehensive, thereby improving the reliability of the in-vehicle infotainment system test results. In summary, the UI testing method for in-vehicle infotainment systems provided in this application, through natural language requirement parsing, dual-mode knowledge base support, multi-agent automatic execution, and comprehensive judgment of the UI layer and vehicle signal layer, can achieve automated execution of the UI testing process for in-vehicle infotainment systems and improve the accuracy of test task generation and the reliability of test result judgment.

[0044] In some embodiments, step 102 may include: performing intent recognition on natural language testing requirements using a preset large language model to obtain multiple candidate steps; determining the operation type of each candidate step based on a dual-mode knowledge base, wherein the operation type may include UI operation type and signal operation type; associating each candidate step with its corresponding operation type to obtain multiple atomic steps; and sorting and combining the multiple atomic steps to obtain a test task sequence.

[0045] In some examples, the pre-set large language model is a language model that has been fine-tuned and optimized for the automotive domain in advance and deployed in the test system. It has the professional capabilities of natural language understanding and intent recognition for automotive scenarios and is the model for parsing the specific natural language test requirements of automotive IVI. A general large language model can be selected as the basic architecture, and fine-tuned and trained using automotive domain-specific data such as natural language test requirement text, UI operation description text, and vehicle signal association description text related to automotive IVI. After training is completed and functional verification is passed, the model is deployed to the model inference end of the test system, and the test system directly calls it according to the parsing requirements during the test. For example, the pre-set large language model, after being fine-tuned for the automotive domain, can accurately understand the core intent of automotive-specific natural language test requirements such as "testing the activation of driver's seat heating level 1 from the IVI home page" and "verifying the operational effectiveness of the IVI air conditioning automatic cooling function". The intent recognition process involves a pre-set large language model performing professional semantic analysis and key information extraction on the input natural language test requirements. This process uncovers key information such as the core test objectives, the actions to be performed, and the functional verification objects contained within the requirements. The pre-set large language model can autonomously parse the standardized natural language test requirements based on its natural language understanding capabilities, which have been fine-tuned for the automotive field. This allows for the discovery and recognition of the core intent of the requirements without human intervention. For example, when performing intent recognition on the natural language test requirement "verify the process of enabling Bluetooth music playback in the IVI," the core intent can be accurately identified as "execute the operation to enable the IVI Bluetooth music playback function and verify the validity of the process," while also extracting key information such as "IVI homepage," "Bluetooth music," and "enable operation."

[0046] Multiple candidate steps are several test operation steps to be verified and classified, obtained by breaking down the intent after the pre-set large language model completes intent recognition, based on the mined intent and the operation logic of the in-vehicle IVI; for example, after performing intent recognition on the natural language test requirement "test to turn on the driver's seat heating level 1 from the IVI home page", the multiple candidate steps obtained are "click the vehicle icon on the IVI home page", "enter the vehicle control interface", and "click the driver's seat heating level 1 button".

[0047] Type identification relies on the knowledge support of the dual-mode knowledge base to professionally analyze, judge, and classify the operational attributes of each candidate step. It can access the structured relational database and UI knowledge graph within the dual-mode knowledge base, accurately matching each candidate step with the stored content. Based on the matching results, the operation type of the candidate step is determined. The entire process is executed autonomously by the testing system without manual intervention. For example, for the candidate step "clicking the driver's seat heating level 1 button," the type identification is performed, and based on the signal mapping relationships stored in the dual-mode knowledge base, it is determined to be an operation related to the vehicle's underlying signals. For the candidate step "clicking the air conditioning icon on the IVI homepage," it is determined to be a pure UI operation involving only interface navigation. Operation type is the classification of candidate steps based on the operational attributes and associated objects of the in-vehicle IVI test. For example, the pre-set operation type system for in-vehicle IVI testing in the testing system only includes UI operation types and signal operation types; all candidate steps will be classified into one of these categories after identification. UI operation types involve only clicking, swiping, and navigating on the in-vehicle IVI interface, without directly triggering the reading, writing, or changes of the vehicle's underlying bus signals. For example, candidate steps such as "clicking the media icon on the IVI homepage," "swiping the IVI air conditioning control interface to adjust the temperature scale," and "entering the IVI vehicle control sub-interface" are all identified as UI operation types. Signal operation types are directly related to the reading, writing, and verification of the vehicle's underlying bus signals. Triggered by operations on the in-vehicle IVI interface, they result in changes to the vehicle's bus signals, directly reflecting the actual effective state of vehicle functions. For example, candidate steps such as "clicking the driver's seat heating level 1 button," "clicking the economy driving mode switch button," and "clicking the air conditioning automatic heating button" are all identified as signal operation types.

[0048] For example, a dedicated input-output link can be established between natural language testing requirements and a pre-defined large language model. Standardized natural language testing requirements, after removing redundant information, are input into the model. The model autonomously executes intent recognition and step decomposition processes, outputting several decomposed operation steps. The testing system defines this set of steps as multiple candidate steps and temporarily stores them. Then, a precise matching query link is established between the candidate steps and a dual-model knowledge base. Multiple candidate steps are traversed one by one, with each candidate step simultaneously input into a structured relational database and a UI knowledge graph for matching. If a match is successful with the structured relational database, it is determined to be a signal operation type; if a match is successful with the UI knowledge graph, it is determined to be a UI operation type. After completing the type determination of all candidate steps, a temporary association between candidate steps and operation types is established. Then, all candidate steps that have completed type determination are traversed, and the specific operation content of each candidate step is bound one-to-one with the determined operation type, assigning a unique step to each bound unit. The identifier, the independent operation unit after binding, is the atomic step, and the collection of all atomic steps is the multiple atomic steps; finally, relying on the UI knowledge graph in the dual-mode knowledge base, the common interface operation path and function execution logic of the vehicle IVI are extracted, the sequential execution relationship and operation dependency between multiple atomic steps are analyzed, and all atomic steps are ordered according to the relationship. After the ordering is completed, a continuous operation sequence is formed, which is the test task sequence that can be directly used for subsequent test execution; for example, multiple atomic steps are "UI operation - click the vehicle icon on the IVI home page", "UI operation - enter the vehicle control interface", "signal operation - click the driver's seat heating level 1 button", according to the actual operation logic of the vehicle IVI, the atomic steps are ordered and combined in the above order to obtain the test task sequence as follows: the first step is to execute "UI operation - click the vehicle icon on the IVI home page", the second step is to execute "UI operation - enter the vehicle control interface", and the third step is to execute "signal operation - click the driver's seat heating level 1 button".

[0049] By implementing the above embodiments, the intent of natural language testing requirements is identified using a preset large language model, and multiple candidate steps are generated. Then, the dual-mode knowledge base is used to determine the type of each candidate step and form atomic steps. This can automatically decompose complex natural language testing requirements into structured and executable test steps, and further sort and combine them according to the execution logic to form a test task sequence. This improves the test task parsing capability, reduces the dependence on manual understanding and decomposition of test requirements, and improves the efficiency and accuracy of test process generation.

[0050] In some embodiments, the aforementioned method of determining the operation type of each candidate step among multiple candidate steps based on a dual-mode knowledge base may include: performing matching queries in a structured relational database and a UI knowledge base for each candidate step; if a vehicle signal mapping relationship corresponding to the candidate step is matched in the structured relational database, the operation type of the candidate step is determined to be a signal operation type; if an interface element or operation path corresponding to the candidate step is matched in the UI knowledge graph, the operation type of the candidate step is determined to be a UI operation type.

[0051] In some examples, all candidate steps to be judged can be traversed one by one. For each candidate step, independent standardized matching queries are initiated in the structured relational database and the UI knowledge graph to retrieve whether there is associated data corresponding to the candidate step in the two knowledge bases, and the matching results of the two databases are obtained simultaneously, providing direct data support for the subsequent determination of the operation type. For example, for the candidate step "click the driver's seat heating level 1 button on the IVI vehicle control interface", the test system generates corresponding query keywords and searches for the existence of the vehicle signal mapping relationship corresponding to this operation in the structured relational database and the existence of the interface element or operation path corresponding to this operation in the UI knowledge graph. For the candidate step "click the media icon on the IVI homepage", the test system also generates query keywords and completes independent matching queries in the two knowledge bases.

[0052] When a candidate step finds a valid matching UI operation and vehicle signal mapping relationship in the database, the operation type of the candidate step is directly determined as a signal operation type, because the execution of this type of candidate step will directly trigger changes in the vehicle's underlying bus signals and relate to the actual effective state of the vehicle's core functions; if a valid matching interface element information or operation path information is found in the graph, the operation type of the candidate step is determined as a UI operation type, because the execution of this type of candidate step only involves IVI interface element operations, interface jumps, and other behaviors, and will not directly trigger changes in the vehicle's underlying bus signals.

[0053] For example, all candidate steps to be judged can be retrieved, and each candidate step can be traversed in the order of receipt. Standardized semantic query keywords adapted to dual-database queries can be generated for each candidate step. For a single candidate step to be judged, the test system inputs the query keywords into the structured relational database and the UI knowledge graph respectively, initiates two independent precise matching queries, and waits for and receives the matching results and related data returned by the two knowledge bases. The test system first judges the matching results of the structured relational database. If a valid matching vehicle signal mapping relationship is detected, the operation type of the candidate step is immediately marked as a signal operation type and the relevant mapping information is recorded. If the matching result returned by the structured relational database is no valid match, the test system then judges the matching results of the UI knowledge graph. If a valid matching interface element or operation path is detected, the operation type of the candidate step is immediately marked as a UI operation type and the relevant interface information is recorded.

[0054] By implementing the above embodiments, candidate steps are matched and queried in the structured relational database and the UI knowledge graph, respectively, and the operation type of the candidate steps is determined based on the matching results. This can accurately distinguish between UI operations and vehicle signal operations, thereby improving the accuracy of test step type identification. By clearly distinguishing between signal-related operations and interface operations, the subsequent test execution process can be made clearer, and a basis can be provided for assigning corresponding execution modules to different types of operations, thereby improving the accuracy and reliability of test execution.

[0055] In some embodiments, the aforementioned step 103 may include: for atomic steps in the test task sequence whose operation type is UI operation, performing UI operations based on a UI knowledge graph by a UI operation agent to obtain the UI layer execution result corresponding to the atomic step; for atomic steps in the test task sequence whose operation type is signal operation, performing signal operations based on a structured relational database by a data signal agent to obtain the vehicle signal layer execution result corresponding to the atomic step; and monitoring the execution process of each atomic step in parallel by an execution monitoring agent to verify the UI layer execution result and the vehicle signal layer execution result of each atomic step.

[0056] In some examples, the UI operation agent is a dedicated intelligent software module developed in advance to execute atomic steps of UI operation types for in-vehicle IVIs, debugged in-vehicle scenarios, and deployed in the test system. This module has the ability to access the UI knowledge graph, execute in-vehicle IVI interface operations, and collect and process UI layer states. Before the test implementation, based on the test requirements such as the interface operation type, interface element characteristics, and operation path logic of the in-vehicle IVI, a software module with the ability to execute operations such as interface clicks, swipes, and jumps can be developed. After completing the communication adaptation between the module and the in-vehicle IVI system and the query adaptation with the UI knowledge graph, functional integration and testing verification are carried out. After the verification is passed, the module is deployed to the execution end of the test system. During the test, the test system uniformly schedules and calls the module according to the operation type of the atomic steps. For example, the UI operation agent developed for the touch interface of the in-vehicle IVI can retrieve the operation path "click the media icon on the IVI homepage to jump to the playback interface" from the UI knowledge graph, automatically execute the click operation, collect the interface state after the operation, and complete the preliminary processing of the UI layer execution results. Atomic steps of all UI operation types can be assigned to UI operation agents. These agents access the UI knowledge graph to retrieve the UI semantics, interface elements, and operation path information corresponding to the atomic steps. Based on this information, standardized UI operations are performed in the in-vehicle IVI system. Then, the actual interface state of the in-vehicle IVI after the operation is collected through technologies such as image acquisition and interface element recognition. After standardization processing, the UI layer execution result corresponding to each atomic step is obtained.

[0057] The data signal intelligent agent is a dedicated intelligent software module developed in advance for executing atomic steps of signal operation types in vehicle IVIs, completing in-vehicle scenario function debugging, and deployed in the test system. This module has core capabilities such as access to structured relational databases, adaptation to vehicle bus communication interfaces, and vehicle signal reading, writing, acquisition, and processing. Before testing, based on the vehicle bus communication protocol, the UI operation of the vehicle IVI, vehicle signal mapping rules, and signal reading and writing acquisition requirements, a software module with the ability to query vehicle signal identifiers, read and set signal values ​​can be developed. After adapting the module to vehicle bus communication interfaces such as controller LANs and local interconnections, and after adapting to the query of the structured relational database, functional integration and testing verification are performed. After successful verification, the module is deployed to the execution end of the test system. During the test, the test system uniformly schedules and calls the module according to the operation type of the atomic steps. For example, a data signal intelligent agent adapted to the CAN bus communication interface can retrieve the mapping relationship of "setting the seat heating signal value to 1 when the driver's seat heating level 1 button is clicked" from the structured relational database. After executing the operation, it reads the actual value of the signal through the CAN bus interface, completing the preliminary processing of the vehicle signal layer execution result. Atomic steps of all signal operation types can be assigned to data signal agents. These agents access a structured relational database to retrieve the mapping relationship between UI operations and vehicle signals corresponding to the atomic steps, obtain core information such as vehicle signal identifiers and expected signal values, and perform signal read / write operations through the vehicle bus communication interface based on this information. They then read the actual signal values ​​of the underlying vehicle bus after the operations and obtain the vehicle signal layer execution results corresponding to the atomic steps after standardized parsing and processing.

[0058] The execution monitoring agent is a dedicated intelligent software module developed in advance to monitor the execution process of in-vehicle IVI testing, verify the results of dual-layer execution, complete the functional debugging of in-vehicle scenarios, and deploy it in the test system. This module has real-time communication capabilities with the UI operation agent and the data signal agent, and can acquire the execution process data and result data of the two types of agents in parallel. It also performs validity verification of the UI layer execution results and the vehicle signal layer execution results according to preset rules. Before the test, based on the monitoring requirements of the test execution and the verification rules of dual-layer execution results, a software module with real-time data reception, execution status monitoring, and result validity verification capabilities can be developed. After completing the communication adaptation between the module and the UI operation agent and the data signal agent, and after the preset verification rules are implanted, functional integration and testing verification are performed. After successful verification, the module is deployed to the execution end of the test system and starts up synchronously and runs in parallel with the other two types of agents during the test. For example, the execution monitoring agent can receive the interface acquisition data of the UI operation agent and the signal reading data of the data signal agent in real time, and can verify the clarity of the interface image and the validity of the signal data format according to preset rules, and remove invalid execution result data. The execution monitoring agent can be started synchronously and run in parallel with the UI operation agent and the data signal agent. It acquires the full-process time sequence data and status data of each atomic step executed by the two types of agents through a real-time communication link. At the same time, it receives the UI layer execution results and vehicle signal layer execution results returned by the two types of agents. According to the result verification rules preset in the test system, it performs validity and rationality verification on the two-layer execution results corresponding to each atomic step, eliminates invalid data, and ensures that the execution results returned to the test system are true and usable. For example, after the UI operation agent executes "UI operation - slide the IVI media interface to switch music categories", the returned UI layer execution result is a blurry interface screenshot. The execution monitoring agent determines that the result is invalid and eliminates it according to the verification rules. After the data signal agent executes "signal operation - click the economy driving mode button", the returned vehicle signal layer execution result is a signal value with an incorrect format. The execution monitoring agent determines that the result is invalid and eliminates it, and only the valid results are returned to the test system.

[0059] By implementing the above embodiments, a UI operation agent, a data signal agent, and an execution monitoring agent are set up, enabling different types of test operations to be processed by the corresponding agents, thus achieving automated execution of test tasks. Among them, the UI operation agent is responsible for executing interface operations, the data signal agent is responsible for executing vehicle signal-related operations, and the execution monitoring agent monitors and verifies the execution process of each atomic step. It can simultaneously obtain the execution results of the UI layer and the vehicle signal layer during test execution, improving the automation level of test execution and enhancing the reliability of test results.

[0060] In some embodiments, the aforementioned execution of UI operations based on a UI knowledge graph by a UI operation agent to obtain the UI layer execution result corresponding to the atomic step may include: obtaining the semantic description of the target element corresponding to the atomic step through the UI operation agent; querying the operation path and expected interface state that match the semantic description of the target element in the UI knowledge graph; generating UI automated execution instructions based on the operation path and the real-time interface state of the in-vehicle infotainment system, wherein the UI automated execution instructions may include at least one of clicking, swiping, and input; and determining the UI layer execution result based on the comparison result between the interface screenshot after the execution of the UI automated execution instructions and the expected interface state.

[0061] In some examples, the semantic description of the target element is a standardized functional meaning expression of the in-vehicle IVI interface functional element corresponding to the atomic step of the UI operation type. It serves as the retrieval basis for the UI operation agent to perform precise queries in the UI knowledge graph. Its description includes information such as the interface where the element is located and the element's functional attributes. The UI operation agent can extract key information such as the operation object and function direction from the atomic step of the UI operation type issued by the test system, and then process the extracted information through the semantic standardization rules of the in-vehicle IVI domain to generate a standardized and unified semantic description of the target element. For example, for the atomic step "UI operation - click the air conditioner icon on the IVI homepage", the semantic description of the target element generated by the UI operation agent after extracting key information is "air conditioner function icon on the homepage of the in-vehicle infotainment system"; for the atomic step "UI operation - click the automatic cooling button on the air conditioner control interface", the semantic description of the target element is "automatic cooling function button on the air conditioner control interface of the in-vehicle infotainment system".

[0062] The operation path starts from the current interface of the in-vehicle IVI, reaches the interface of the target element, and completes the corresponding UI operation in an ordered sequence of interface jumps and element operations. This sequence is consistent with the actual interface interaction logic of the in-vehicle IVI and serves as the basis for the UI operation agent to generate automated execution instructions. The UI operation agent can use the generated semantic description of the target element as a search keyword to perform precise matching queries in the UI knowledge graph. The corresponding ordered operation sequence is extracted from the relationship between nodes and edges in the graph, which is the operation path to complete the UI operation. For example, for the semantic description of the target element "automatic cooling function button of the air conditioning control interface of the in-vehicle infotainment system", the operation path obtained by matching and querying in the UI knowledge graph is "click the air conditioning function icon on the home page of the in-vehicle infotainment system - jump to the air conditioning control interface of the in-vehicle infotainment system"; for the semantic description of the target element "driving mode button of the vehicle control interface of the in-vehicle infotainment system", the operation path obtained by matching is "click the vehicle function icon on the home page of the in-vehicle infotainment system - jump to the vehicle control interface of the in-vehicle infotainment system".

[0063] The expected interface state is the standard state that the in-vehicle IVI interface should present after executing the UI automation command corresponding to the target element. It includes the interface jump result, the display status of functional elements, and changes in operation icons. The UI operation agent can simultaneously retrieve the preset standard interface state information corresponding to the operation path from the UI knowledge graph while matching the semantic description of the target element and retrieving the operation path in the UI knowledge graph. This information is pre-entered based on the standard functional logic of the in-vehicle IVI before testing. For example, for the operation path "click the air conditioning function icon on the home page of the in-vehicle infotainment system - jump to the air conditioning control interface of the in-vehicle infotainment system", the corresponding expected interface state is "successfully jump from the home page of the in-vehicle infotainment system to the air conditioning control interface, the interface fully displays the temperature adjustment, mode selection, fan speed control and other functional elements, and there are no abnormal pop-ups or interface lag". For the operation path "click the Bluetooth play button on the media interface of the in-vehicle infotainment system", the corresponding expected interface state is "the Bluetooth play button is lit up, the interface displays the Bluetooth audio source icon and playback progress bar, and there are no error prompts in the media function module".

[0064] Real-time interface status refers to the actual current interface state presented by the in-vehicle IVI system before the UI operation agent generates and executes UI automation commands. This includes the current interface, the display status of interface functional elements, and the availability status of functional modules. The UI operation agent can collect the current interface image information of the in-vehicle IVI system in real time through built-in image acquisition and interface element recognition technologies. Then, the collected information is standardized and parsed to extract content such as interface position and element status, forming the real-time interface status. For example, before the UI operation agent executes the operation of clicking the air conditioner icon, the real-time interface status collected and parsed is "Currently on the home page of the in-vehicle infotainment system, media, air conditioner, vehicle and other function icons are displayed normally, and all functional modules are in an operable state."

[0065] UI automation execution instructions are standardized operation instructions generated by the UI operation agent based on the matched operation path and the parsed real-time interface state. These instructions can be automatically executed in an in-vehicle IVI system and are the direct instructions that drive the in-vehicle IVI to complete the corresponding UI operation. They include at least one operation type: click, swipe, and input. The UI operation agent can combine the matched operation path and the parsed real-time interface state, and according to the interface operation communication protocol of the in-vehicle IVI system, generate standardized executable instructions adapted to the actual interface state. The instructions clearly specify the operation type, operation object, and operation location. For example... For example, combining the operation path "click the air conditioning function icon on the in-vehicle infotainment system homepage" with the real-time interface status "on the in-vehicle IVI homepage, the air conditioning icon is displayed normally", the generated UI automated execution command is "click operation - coordinates of the air conditioning function icon on the in-vehicle infotainment system homepage"; combining the operation path "adjust the air conditioning control interface temperature to 25 degrees Celsius" with the real-time interface status "on the air conditioning control interface, the current temperature is 30 degrees Celsius", the generated UI automated execution command is "slide operation - slide from the 30 degrees Celsius scale to the 25 degrees Celsius scale in the temperature scale area of ​​the air conditioning control interface".

[0066] The screenshot of the interface after the UI automation execution command is an image of the actual interface of the vehicle IVI system collected in real time after the UI operation intelligent agent has fully executed the UI automation execution command in the vehicle IVI system. It is a visual basis reflecting the actual execution effect of the UI operation. The UI operation intelligent agent can use interface element recognition and state feature extraction technologies to perform multi-dimensional feature comparison between the screenshot of the interface after the execution of the UI automated execution command and the expected interface state retrieved from the UI knowledge graph. Then, it determines the comparison result according to the matching threshold preset by the testing system, and finally generates the standardized UI layer execution result corresponding to the atomic step of the UI operation type based on the comparison result. For example, if the feature extracted from the interface screenshot is "successfully jumped to the air conditioning control interface, all functional elements are displayed normally", which completely matches the feature of the expected interface state, the comparison result is determined to be a match, and the corresponding UI layer execution result is "execution successful, the in-vehicle infotainment system interface state meets expectations". If the feature extracted from the interface screenshot is "did not jump to the air conditioning control interface, still staying on the in-vehicle infotainment system homepage", which does not match the feature of the expected interface state, the comparison result is determined to be a mismatch, and the corresponding UI layer execution result is "execution failed, the in-vehicle infotainment system interface did not complete the jump as expected".

[0067] Through the implementation of the above embodiments, the UI operation agent obtains the semantic description of the target element according to the atomic steps, and queries the corresponding operation path and expected interface state in the UI knowledge graph. This enables the automatic generation of UI automated execution instructions and the completion of interface operations, allowing UI operations to be automatically located and executed based on semantic information. By comparing the screenshot of the executed interface with the expected interface state, the interface operation results can be verified, thereby improving the accuracy of UI test execution and the degree of automation in the test process.

[0068] In some embodiments, the aforementioned method of performing signal operations based on a structured relational database using a data signal agent to obtain the vehicle signal layer execution result corresponding to the atomic step may include: obtaining the signal semantic description corresponding to the atomic step through the data signal agent; querying the structured relational database for vehicle signal identifiers and expected signal values ​​that match the signal semantic descriptions; reading or setting the vehicle bus signal corresponding to the vehicle signal identifier through the vehicle communication interface to obtain the actual signal value; and comparing the actual signal value with the expected signal value to obtain the vehicle signal layer execution result.

[0069] In some examples, the signal semantic description is a standardized functional meaning representation of the vehicle bus signal corresponding to the atomic step of the signal operation type. It serves as the retrieval basis for the data signal agent to perform precise queries in the structured relational database. The description includes key information such as the vehicle function and operation behavior corresponding to the signal. The data signal agent can extract key information such as the operation object, function direction, and signal operation behavior from the atomic step of the signal operation type issued by the test system, and then process the extracted information through the semantic standardization rules of the vehicle bus signal domain to generate a standardized and unified signal semantic description. For example, for the atomic step "signal operation - click the 26 degrees Celsius temperature adjustment button on the IVI air conditioning control interface", the signal semantic description generated by the data signal agent after extracting key information is "vehicle bus signal that sets the target temperature of the vehicle air conditioning to 26 degrees Celsius".

[0070] Vehicle signal identifiers are unique, standardized identifiers corresponding to each signal in the vehicle bus. They are used to distinguish different vehicle bus signals and serve as the basis for data signal agents to locate and read specific vehicle bus signals. Each identifier corresponds one-to-one with the actual function of the vehicle and is stored in a structured relational database. Data signal agents can use the generated signal semantic descriptions as search keywords to perform precise matching queries in the structured relational database, extracting the corresponding unique vehicle signal identifiers from the UI operation and vehicle signal mapping relationships stored in the database. For example, for the signal semantic description "vehicle bus signal that sets the target temperature of the vehicle air conditioning to 26 degrees Celsius," the matching query in the structured relational database yields the vehicle signal identifier "HVAC_Target_Temp"; for the signal semantic description "vehicle bus signal that sets the driver's seat heating to level 1," the matching vehicle signal identifier is "Seat_Heat_Driver."

[0071] The expected signal value is the standard value that the corresponding vehicle signal identifier in the vehicle bus should reach after the execution of the UI operation corresponding to the atomic step of the signal operation type. This value is the standard quantitative basis for the actual effectiveness of the vehicle function and is stored in a structured relational database bound to the vehicle signal identifier. The data signal agent can simultaneously retrieve the preset standard value bound to the vehicle signal identifier from the database while matching the signal semantic description in the structured relational database and retrieving the vehicle signal identifier. This value is pre-entered based on the standard logic of the vehicle's core functions before testing. For example, for the vehicle signal identifier "HVAC_Target_Temp", the corresponding expected signal value retrieved from the structured relational database is "26", which represents the air conditioning target temperature of 26 degrees Celsius; for the vehicle signal identifier "Seat_Heat_Driver", the corresponding expected signal value retrieved is "1", which represents the driver's seat heating level 1.

[0072] The vehicle communication interface is a standardized hardware communication port that enables data interaction between the data signal agent and the vehicle bus. It serves as the physical bridge for the data signal agent to read and set vehicle bus signals. This interface is compatible with different types of vehicle bus communication protocols, such as CAN and LIN. The actual signal value is the value of the corresponding vehicle signal identifier actually collected from the vehicle bus after the data signal agent performs read or set operations on the vehicle bus signals through the vehicle communication interface. It is a quantitative basis reflecting the actual execution effect of the vehicle signal operation, and its value type is consistent with the expected signal value. For example, after the data signal agent sets the value of the vehicle signal identifier "HVAC_Target_Temp" through the vehicle communication interface, if the current value of this identifier collected from the vehicle CAN bus is "26", this value is the actual signal value; if the current value of the vehicle signal identifier "Seat_Heat_Driver" is "0", this value is also the actual signal value.

[0073] The data signal agent can directly compare the actual signal value with the expected signal value in the same type and dimension. If the two values ​​are completely identical, the comparison result is considered a match; if the two values ​​differ, the comparison result is considered a mismatch. Subsequently, a standardized vehicle signal layer execution result is generated based on the comparison result. When there is a match, the result is "execution successful and the vehicle signal meets expectations." When there is a mismatch, the result is "execution failed" and the specific difference between the actual signal value and the expected signal value is marked. For example, if the actual signal value "26" and the expected signal value "26" are completely identical, the comparison result is considered a match, and the corresponding vehicle signal layer execution result is "execution successful, vehicle bus signal value meets expectations." If the actual signal value "0" and the expected signal value "1" differ, the comparison result is considered a mismatch, and the corresponding vehicle signal layer execution result is "execution failed, actual vehicle bus signal value is 0, inconsistent with expected value 1."

[0074] Through the implementation of the above embodiments, the data signal intelligent agent obtains the signal semantic description according to the atomic steps, queries the corresponding vehicle signal identifier and expected signal value in the structured relational database, and then reads or sets the vehicle bus signal through the vehicle communication interface, thereby enabling automated operation and detection of vehicle signals; by comparing the actual signal value with the expected signal value, it can verify whether the interface operation correctly triggers the vehicle signal change, thereby improving the accuracy and completeness of the test result verification.

[0075] In some embodiments, the aforementioned method of monitoring the execution process of each atomic step in parallel by executing a monitoring agent to verify the UI layer execution result and the vehicle signal layer execution result of each atomic step may include: executing a monitoring agent to synchronously trigger a first verification operation or a second verification operation after executing the corresponding UI operation or signal operation for each atomic step in the test task sequence; verifying the interface state after execution by the first verification operation to obtain the UI layer verification result; verifying the vehicle bus signal after execution by the second verification operation to obtain the signal layer verification result; and verifying the UI layer execution result and the vehicle signal layer execution result based on the UI layer verification result and the signal layer verification result.

[0076] In some examples, the first verification operation is a pre-set standardized verification process for the interface state after the UI operation is executed. It is specifically used to verify the validity, completeness, and rationality of the UI layer execution results from multiple dimensions. Before the test is implemented, a standardized verification process can be formulated based on the actual needs of the in-vehicle IVI UI test. This process includes verification rules such as the clarity of the interface screenshot, the completeness of the interface features, and the rationality of the interface state. This process is then embedded into the program module of the execution monitoring agent. During the test, the execution monitoring agent automatically triggers the execution based on the operation execution state. For example, the first verification operation may include three core rules: verifying whether the interface screenshot after the UI operation is clear, complete, and free of noise; verifying whether the features extracted from the interface screenshot completely cover the core interface state corresponding to the operation; and verifying whether the actual interface state conforms to the conventional operation logic of the in-vehicle IVI. The UI layer verification result is a standardized judgment result generated by the execution monitoring agent after completing the first verification operation, based on preset verification rules. It directly reflects the validity of the UI layer execution result returned by the UI operation agent and serves as the basis for the final verification of the UI layer execution result. The execution monitoring agent can verify the interface state data after the UI operation one by one according to the preset rules of the first verification operation, and generate a standardized judgment result based on the verification results. The result is divided into two categories: "valid" and "invalid". If all verifications meet the standards, it is judged as valid; if any indicator fails to meet the standards, it is judged as invalid. For example, if the UI screenshot of the UI layer execution result is clear and complete, the feature extraction is complete, and the interface state conforms to the operation logic, the UI layer verification result generated by the execution monitoring agent is "valid". If the interface screenshot is blurry and incomplete, or the core features are not extracted, the generated UI layer verification result is "invalid".

[0077] The second verification operation is a pre-set standardized verification process for vehicle bus signals after signal operation execution. It is specifically used to verify the validity, accuracy, and timing of the vehicle signal layer execution results from multiple dimensions. Before the test, a standardized verification process can be developed based on the professional needs of vehicle bus signal testing, including verification rules for signal data format, signal value rationality, and signal reading timing. This process is then embedded into the program module of the execution monitoring agent, which automatically triggers execution based on the operation execution status during the test. For example, the second verification operation may include three core rules: verifying whether the actual value of the vehicle bus signal is a standard number or function identifier format, verifying whether the actual signal value is within the reasonable value range corresponding to the signal, and verifying whether the signal reading time is within the valid timing interval after the signal operation is executed. The signal layer verification result is a standardized judgment result generated by the execution monitoring agent after completing the second verification operation, based on preset verification rules. It directly reflects the validity of the vehicle signal layer execution result returned by the data signal agent and serves as the basis for the final verification of the vehicle signal layer execution result. The execution monitoring agent can verify the vehicle bus signal data after the signal operation one by one according to the preset rules of the second verification operation, and generate a standardized judgment result based on the verification results. The result is divided into two categories: "valid" and "invalid". If all verifications meet the standards, it is judged as valid; if any indicator fails to meet the standards, it is judged as invalid. For example, if the actual signal value format of the vehicle signal layer execution result is standard, the value is within a reasonable range, and the reading timing is compliant, the signal layer verification result generated by the execution monitoring agent is "valid"; if the actual signal value format is incorrect or the reading timing exceeds the valid range, the generated signal layer verification result is "invalid".

[0078] The execution monitoring agent can maintain real-time communication and run in parallel with the UI operation agent and data signal agent. For each atomic step in the test task sequence, after the corresponding UI operation or signal operation is fully executed, the execution monitoring agent immediately triggers the corresponding first or second verification operation based on the operation type of the atomic step, achieving real-time linkage between operation execution and verification triggering without any time delay. This logic is implemented by the execution monitoring agent acquiring the operation type of each atomic step and the operation completion status of the UI operation agent and data signal agent through a dedicated real-time communication link. The program presets triggering rules such as "UI operation triggers the first verification operation, signal operation triggers the second verification operation." When the completion of an atomic step is detected, the corresponding verification operation is immediately triggered based on the operation type. For example, for a UI operation type atomic step in the test task sequence, after the UI operation agent completes the interface click and returns the UI layer execution result, the execution monitoring agent immediately triggers the first verification operation synchronously; for a signal operation type atomic step, after the data signal agent completes the signal read / write and returns the vehicle signal layer execution result, the execution monitoring agent immediately triggers the second verification operation synchronously.

[0079] After generating the hierarchical verification results, the execution monitoring agent can establish a one-to-one binding relationship between the verification results and the original execution results. If the verification result is "valid", the corresponding original execution result is determined to be valid and included in the valid data range for subsequent comprehensive judgment of test results. If the verification result is "invalid", the corresponding original execution result is determined to be invalid and removed from the valid data range, and the specific reason for the failure to meet the verification standard is marked simultaneously. For example, if the UI layer verification result of a certain atomic step is "valid", the execution monitoring agent determines that the UI layer execution result is valid and retains it; if its vehicle signal layer verification result is "invalid", the vehicle signal layer execution result is determined to be invalid and removed, and the reason for removal is marked as "incorrect actual signal value format".

[0080] Through the implementation of the above embodiments, the execution monitoring agent synchronously triggers UI layer verification and signal layer verification after each atomic step is executed. This enables simultaneous verification of the interface state and vehicle bus signals after the interface operation is executed, thus forming a dual-layer verification mechanism. By comprehensively verifying the execution result by combining the UI layer verification result and the signal layer verification result, the reliability of the test result judgment can be further improved, and the functional performance of the in-vehicle infotainment system in actual operation can be more comprehensively reflected.

[0081] Furthermore, as an implementation of the aforementioned method embodiments, this application also provides a UI testing device for an in-vehicle infotainment system, used to implement the aforementioned method embodiments. This device embodiment corresponds to the aforementioned method embodiments. For ease of reading, this UI testing device embodiment for an in-vehicle infotainment system will not repeat the details of the aforementioned method embodiments one by one, but it should be clear that the device in this application embodiment can correspondingly implement all the contents of the aforementioned method embodiments. For example... Figure 2 As shown, the UI testing device 20 for the in-vehicle infotainment system includes: a requirement acquisition unit 201, a requirement parsing unit 202, a test execution unit 203, and a result acquisition unit 204. The requirement acquisition unit 201 acquires natural language test requirements for the in-vehicle infotainment system. The requirement parsing unit 202 parses the aforementioned natural language test requirements based on a preset dual-mode knowledge base to obtain a test task sequence composed of atomic steps. The dual-mode knowledge base may include a structured relational database storing the mapping relationship between UI operations and vehicle signals, and a UI knowledge graph storing UI semantics and operation paths. The test execution unit 203 executes test operations according to the test task sequence through multiple preset intelligent agents to obtain UI layer execution results and vehicle signal layer execution results corresponding to each atomic step. The result acquisition unit 204 determines the test results for the in-vehicle infotainment system based on the UI layer execution results and the vehicle signal layer execution results.

[0082] In some embodiments, the requirement parsing unit 202 is further configured to perform intent recognition on natural language test requirements through a preset large language model to obtain multiple candidate steps; based on a dual-mode knowledge base, perform type discrimination on each candidate step among the multiple candidate steps to determine the operation type of the candidate steps, wherein the operation type includes UI operation type and signal operation type; associate each candidate step with the corresponding operation type to obtain multiple atomic steps; and sort and combine the multiple atomic steps to obtain a test task sequence.

[0083] In some embodiments, the requirement parsing unit 202 is further configured to perform matching queries in the structured relational database and the UI knowledge base for each candidate step; if a vehicle signal mapping relationship corresponding to the candidate step is matched in the structured relational database, the operation type of the candidate step is determined to be a signal operation type; if an interface element or operation path corresponding to the candidate step is matched in the UI knowledge graph, the operation type of the candidate step is determined to be a UI operation type.

[0084] In some embodiments, the test execution unit 203 is further configured to, for atomic steps in the test task sequence whose operation type is UI operation, execute UI operations based on a UI knowledge graph through a UI operation agent to obtain the UI layer execution result corresponding to the atomic step; for atomic steps in the test task sequence whose operation type is signal operation, execute signal operations based on a structured relational database through a data signal agent to obtain the vehicle signal layer execution result corresponding to the atomic step; and monitor the execution process of each atomic step in parallel through an execution monitoring agent to verify the UI layer execution result and the vehicle signal layer execution result of each atomic step.

[0085] In some embodiments, the test execution unit 203 is further configured to obtain the semantic description of the target element corresponding to the atomic step through the UI operation agent; query the operation path and expected interface state that match the semantic description of the target element in the UI knowledge graph; generate UI automated execution instructions based on the operation path and the real-time interface state of the in-vehicle infotainment system, wherein the UI automated execution instructions include at least one of clicking, swiping and input; and determine the UI layer execution result based on the comparison result between the interface screenshot after the UI automated execution instructions are executed and the expected interface state.

[0086] In some embodiments, the test execution unit 203 is further configured to obtain the signal semantic description corresponding to the atomic step through the data signal agent; query the vehicle signal identifier and expected signal value that match the signal semantic description in the structured relational database; read or set the vehicle bus signal corresponding to the vehicle signal identifier through the vehicle communication interface to obtain the actual signal value; and compare the actual signal value with the expected signal value to obtain the vehicle signal layer execution result.

[0087] In some embodiments, the test execution unit 203 is further configured to, by executing a monitoring agent, synchronously trigger a first verification operation or a second verification operation after executing the corresponding UI operation or signal operation for each atomic step in the test task sequence; verify the interface state after execution through the first verification operation to obtain the UI layer verification result; verify the vehicle bus signal after execution through the second verification operation to obtain the signal layer verification result; and verify the UI layer execution result and the vehicle signal layer execution result based on the UI layer verification result and the signal layer verification result.

[0088] This application also provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, will cause the processor to perform any step of the UI testing method for the in-vehicle infotainment system provided in this application.

[0089] In some embodiments, the computer-readable storage medium may be a random access memory (RAM), a read-only memory (ROM), flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); or it may be a variety of devices that include one or any combination of the above-mentioned memories.

[0090] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0091] In some embodiments, computer-executable instructions may, but do not necessarily, correspond to files in a file system, and may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0092] In some embodiments, computer-executable instructions may be deployed to execute on an electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0093] like Figure 3 As shown, this application also provides an electronic device 30, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor. When the processor 320 executes the computer program 311, it implements any step of the UI testing method of the above-mentioned in-vehicle infotainment system.

[0094] This application also provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program or computer-executable instructions from the computer-readable storage medium and executes the computer program or computer-executable instructions, causing the electronic device to perform any step of the UI testing method for an in-vehicle infotainment system described above.

[0095] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A UI testing method for an in-vehicle infotainment system, characterized in that, include: Obtain natural language testing requirements for in-vehicle infotainment systems; Based on a pre-set dual-mode knowledge base, the natural language testing requirements are parsed to obtain a test task sequence composed of atomic steps. The dual-mode knowledge base includes a structured relational database that stores the mapping relationship between UI operations and vehicle signals and a UI knowledge graph that stores UI semantics and operation paths. By using multiple preset intelligent agents, test operations are executed according to the test task sequence to obtain the UI layer execution results and vehicle signal layer execution results corresponding to each atomic step; Based on the execution results of the UI layer and the vehicle signal layer, the test results for the in-vehicle infotainment system are determined.

2. The method according to claim 1, characterized in that, The natural language testing requirements are parsed based on a pre-set dual-mode knowledge base to obtain a test task sequence consisting of atomic steps, including: The intent of the natural language testing requirements is identified by a pre-set large language model, resulting in multiple candidate steps; Based on the dual-mode knowledge base, the type of each candidate step in the plurality of candidate steps is determined to identify the operation type of the candidate step, wherein the operation type includes UI operation type and signal operation type; Each candidate step is associated with its corresponding operation type to obtain multiple atomic steps; The multiple atomic steps are sorted and combined to obtain the test task sequence.

3. The method according to claim 2, characterized in that, The step of determining the operation type of each candidate step based on the dual-mode knowledge base includes: For each candidate step, a matching query is performed in the structured relational database and the UI knowledge base, respectively; If a vehicle signal mapping relationship corresponding to the candidate step is matched in the structured relation database, then the operation type of the candidate step is determined as the signal operation type; If an interface element or operation path corresponding to the candidate step is matched in the UI knowledge graph, then the operation type of the candidate step is determined as the UI operation type.

4. The method according to claim 1, characterized in that, The process involves multiple pre-set intelligent agents executing test operations according to the test task sequence to obtain the UI layer execution results and vehicle signal layer execution results corresponding to each atomic step, including: For atomic steps of UI operation type in the test task sequence, the UI operation agent performs UI operation based on the UI knowledge graph to obtain the UI layer execution result corresponding to the atomic step. For atomic steps in the test task sequence whose operation type is signal operation, the data signal agent performs signal operations based on the structured relational database to obtain the vehicle signal layer execution result corresponding to the atomic step. The execution process of each atomic step is monitored in parallel by a monitoring agent to verify the execution results of the UI layer and the vehicle signal layer of each atomic step.

5. The method according to claim 4, characterized in that, The step of executing UI operations through a UI operation agent based on the UI knowledge graph to obtain the UI layer execution result corresponding to the atomic step includes: The semantic description of the target element corresponding to the atomic step is obtained through the UI operation agent. Query the UI knowledge graph for operation paths and expected interface states that match the semantic description of the target element; Based on the operation path and the real-time interface status of the in-vehicle infotainment system, UI automated execution instructions are generated, wherein the UI automated execution instructions include at least one of clicking, swiping, and input; The execution result of the UI layer is determined by comparing the screenshot of the interface after the execution of the UI automation command with the expected interface state.

6. The method according to claim 4, characterized in that, The step of performing signal operations based on the structured relational database through a data signal agent to obtain the vehicle signal layer execution result corresponding to the atomic step includes: The signal semantic description corresponding to the atomic step is obtained through the data signal agent; Query the structured relational database for vehicle signal identifiers and expected signal values ​​that match the signal semantic description; The actual signal value is obtained by reading or setting the vehicle bus signal corresponding to the vehicle signal identifier through the vehicle communication interface; The actual signal value is compared with the expected signal value to obtain the vehicle signal layer execution result.

7. The method according to claim 4, characterized in that, The method of monitoring the execution process of each atomic step in parallel by executing a monitoring agent to verify the execution results of the UI layer and the vehicle signal layer of each atomic step includes: Through the execution monitoring agent, for each atomic step in the test task sequence, after executing the corresponding UI operation or signal operation, the first verification operation or the second verification operation is triggered synchronously. The UI layer verification result is obtained by verifying the interface state after the first verification operation. The vehicle bus signal after execution is verified by the second verification operation to obtain the signal layer verification result; Based on the UI layer verification results and the signal layer verification results, the UI layer execution results and the vehicle signal layer execution results are verified.

8. A UI testing device for an in-vehicle infotainment system, characterized in that, include: The requirement elicitation unit is used to obtain natural language testing requirements for in-vehicle infotainment systems. The requirement parsing unit is used to parse the natural language test requirements based on a preset dual-mode knowledge base to obtain a test task sequence composed of atomic steps. The dual-mode knowledge base includes a structured relational database that stores the mapping relationship between UI operations and vehicle signals and a UI knowledge graph that stores UI semantics and operation paths. The test execution unit is used to execute test operations according to the test task sequence through multiple preset intelligent agents to obtain the UI layer execution result and the vehicle signal layer execution result corresponding to each atomic step. The result acquisition unit is used to determine the test results for the in-vehicle infotainment system based on the execution results of the UI layer and the execution results of the vehicle signal layer.

9. An electronic device, comprising: The memory and processor are characterized in that the processor, when executing a computer program stored in the memory, implements the steps of the UI testing method for an in-vehicle infotainment system as described in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or the computer program are executed by a processor, the steps of the UI testing method for the in-vehicle infotainment system as described in any one of claims 1 to 7 are implemented.

Images