Automated testing method for in-vehicle buttons and related devices

Abstract

This invention discloses an automated testing method and related equipment for in-vehicle buttons, relating to the field of automotive electronics technology, primarily addressing the problems of poor adaptability and low stability in current automated testing frameworks. The method includes: generating a test script based on test requirements, wherein the test requirements are used to perform interactive testing of the in-vehicle buttons, and the test script includes operation steps and corresponding initial strategies; executing test operations on the in-vehicle system based on the test script, wherein the test operations are performed based on the attribute location and / or image recognition of the in-vehicle buttons; and recording the execution log of the test operations to optimize the initial strategies of the operation steps, wherein the execution log includes process data and diagnostic information of the test operations. This invention is used for the automated testing process of in-vehicle buttons.

Description

Technical Field

[0001] This invention relates to the field of automotive electronics technology, and in particular to an automated testing method and related equipment for in-vehicle buttons. Background Technology

[0002] With the rapid advancement of automotive intelligence, in-vehicle systems have become complex human-machine interaction terminals integrating multiple functions. Their interfaces contain numerous standard and custom controls, placing high demands on the comprehensiveness of testing. Currently, the testing methods primarily relied upon for automated testing of such systems all have inherent limitations. In actual testing, testers typically need to pre-select a fixed testing method for different controls based on experience, or write complex fault-tolerant code. This static testing mode, relying on manual pre-setting and a fixed testing strategy once execution begins, cannot effectively handle various unexpected situations that arise during testing (such as control loading delays or slight UI offsets). This results in significant technical problems with poor adaptability and low stability when facing the diversity and uncertainty of the in-vehicle environment. Summary of the Invention

[0003] In view of the above problems, the present invention provides an automated testing method and related equipment for vehicle buttons, the main purpose of which is to solve the problems of poor adaptability and low stability of the current automated testing framework.

[0004] To address at least one of the aforementioned technical problems, in a first aspect, the present invention provides an automated testing method for vehicle-mounted buttons, the method comprising: Test scripts are generated based on test requirements, wherein the test requirements are used to perform interactive tests on in-vehicle buttons, and the test scripts include operation steps and corresponding initial strategies; Test operations are performed on the in-vehicle system based on the test script, wherein the test operations are performed based on the attribute positioning and / or image recognition of the in-vehicle buttons; The execution log of the test operation is recorded to optimize the initial strategy of the operation steps. The execution log includes process data and diagnostic information of the test operation.

[0005] Optionally, generating test scripts based on test requirements includes: Obtain test requirements, wherein the test requirements are described in natural language; The test requirements are analyzed to extract test parameters, which include screen specifications, interaction type, target object, and associated application. A test script is generated based on the test requirements and test parameters.

[0006] Optionally, performing test operations on the in-vehicle system based on the test script includes: Perform environment initialization based on the test parameters; The operation is executed based on the type of the operation steps described.

[0007] Optionally, the invocation of the operation based on the type of the operation step includes: When the operation step involves hardware interaction, the test operation is performed by calling the vehicle hardware driver interface. When the operation step involves UI interaction, the UI testing framework is invoked to execute the test operation.

[0008] Optionally, when the target object is a UI operation, invoking the UI testing framework to execute the test operation includes: In the case of the test operation being attribute location, the attribute location assistant is invoked to find the target object through the control attribute information; In the case of image recognition, the image recognition assistant is invoked to find the target object through image matching. If the search operation fails, an optimization strategy is executed.

[0009] Optionally, if the search operation fails, the optimization strategy is executed, including: If the time taken by the attribute location assistant to search for the target object using control attribute information exceeds a preset time, the search is determined to have failed. If the confidence level of the object found by the image recognition assistant through image matching is lower than the preset confidence level, the search is determined to have failed.

[0010] Optionally, the step of recording the execution log of the test operation to optimize the initial strategy of the operation steps includes: If the test operation fails, the process data and diagnostic information of the test operation are obtained, wherein the diagnostic information includes the error type, failure screenshot and scenario context. The cause of the operation failure was determined based on the process data and diagnostic information. An optimization strategy is generated based on the reasons for the operation failure, wherein the optimization strategy includes parameter correction; The initial strategy for optimizing the operation steps is based on the optimization strategy.

[0011] Secondly, embodiments of the present invention also provide an automated testing device for vehicle-mounted buttons, comprising: A generation unit is used to generate test scripts based on test requirements, wherein the test requirements are used to perform interactive tests on in-vehicle buttons, and the test scripts include operation steps and corresponding initial strategies. An execution unit is configured to perform test operations on the in-vehicle system based on the test script, wherein the test operations are performed based on the attribute positioning and / or image recognition of the in-vehicle buttons; An optimization unit is used to record the execution log of the test operation in order to optimize the initial strategy of the operation steps, wherein the execution log includes process data and diagnostic information of the test operation.

[0012] To achieve the above objectives, according to a third aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium comprising a stored program, wherein, when the program is executed by a processor, the steps of the above-described automated testing method for vehicle buttons are implemented.

[0013] To achieve the above objectives, according to a fourth aspect of the present invention, an electronic device is provided, including at least one processor and at least one memory connected to the processor; wherein the processor is configured to invoke program instructions in the memory to execute the steps of the above-described automated testing method for vehicle buttons.

[0014] By employing the above technical solution, the automated testing method and related equipment for vehicle buttons provided by this invention address the problems of poor adaptability and low stability of current automated testing frameworks. This invention generates test scripts based on test requirements, whereby the test requirements are used to perform interactive testing of vehicle buttons. The test scripts include operation steps and corresponding initial strategies. Based on the test scripts, test operations are executed on the vehicle system, whereby the test operations are performed based on the attribute positioning and / or image recognition of the vehicle buttons. Execution logs of the test operations are recorded to optimize the initial strategies of the operation steps. The execution logs include process data and diagnostic information of the test operations. In this solution, an initial strategy containing preliminary judgments is first generated. The system records detailed execution logs throughout the process, making the testing process no longer a simple repetition of one-way instructions, but a learning process that can be analyzed and diagnosed. When the test fails due to interface changes, hardware differences, or control recognition issues, the recorded log information can be used to backtrack and analyze the fault, thereby understanding the specific context and cause of the failure. Based on this analysis, we proactively adjusted and optimized subsequent action strategies, enabling the system to dynamically respond to various fluctuations and anomalies that occur in the actual testing environment. This reduced the risk of test interruption caused by environmental uncertainty and the limitations of a single method, and improved the adaptability and robustness of the overall testing process.

[0015] Correspondingly, the automated testing device, equipment, and computer-readable storage medium for vehicle buttons provided in the embodiments of the present invention also have the above-mentioned technical effects.

[0016] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0017] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating an automated testing method for vehicle buttons provided in an embodiment of the present invention is shown. Figure 2 This diagram illustrates the composition of an automated testing device for vehicle buttons according to an embodiment of the present invention. Figure 3 This diagram illustrates the composition of an automated testing electronic device for vehicle buttons provided in an embodiment of the present invention. Detailed Implementation

[0018] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0019] To address the issues of poor adaptability and low stability in current automated testing frameworks, this invention provides an automated testing method for vehicle-mounted buttons, such as... Figure 1 As shown, the method includes: S101. Generate a test script based on the test requirements, wherein the test requirements are used to perform interactive testing of the vehicle buttons, and the test script includes operation steps and corresponding initial strategies; For example, test requirements refer to test instructions provided by users in natural language to guide the interaction testing of in-vehicle buttons; the interaction testing of in-vehicle buttons refers to the activity of automating the testing of various button elements in the in-vehicle system; test scripts are executable code sequences automatically generated by the system, containing a series of operation steps; operation steps are specific action sequences defined in the test script, such as simulating voice input or control clicks; initial strategies are the preset positioning or interaction methods for each operation step, such as prioritizing attribute positioning or image recognition.

[0020] In this application, test requirements described in natural language are first obtained. By analyzing these requirements, key test parameters are automatically extracted, including screen specifications, interaction type, target object, and related application. Then, a structured test script is generated based on these parameters. This script not only contains a detailed sequence of operation steps, but also integrates a corresponding initial strategy for each step to ensure that the test process has clear execution guidelines.

[0021] By employing the above technical solutions, ambiguous natural language instructions are automatically converted into precise executable test scripts, reducing manual intervention and pre-set workload. By integrating initial strategies, preliminary adaptation directions are provided for test operations, thereby reducing the risk of script incompatibility caused by the diversity of test environments and improving the efficiency and reliability of test preparation.

[0022] In one embodiment, generating test scripts based on test requirements includes: Obtain test requirements, wherein the test requirements are described in natural language; The test requirements are analyzed to extract test parameters, which include screen specifications, interaction type, target object, and associated application. A test script is generated based on the test requirements and test parameters.

[0023] For example, the test parameters include screen specifications, interaction type, target object, and associated application. The screen specifications specify the size information of the display device, the interaction type describes how the user interacts with the vehicle system, such as voice input or control clicks, the target object identifies the specific button element to be tested, and the associated application specifies the vehicle software module involved in the test.

[0024] In this embodiment, the test requirements described in natural language are first obtained. By parsing the requirements, test parameters such as screen specifications, interaction type, target object, and associated application are automatically extracted. Then, the test requirements and the extracted parameters are combined to generate an executable test script. For example, when the test requirement is to operate the target button through voice interaction on a specified screen, the system will parse the relevant parameters and generate a script containing the corresponding operation steps and initial strategy to ensure that the test instructions are accurately converted into an automated operation sequence.

[0025] By employing the above technical solutions, this step enables the automatic conversion and parameterized configuration of test cases, reducing reliance on human experience and the need for manual coding. The structured parameter extraction enhances the accuracy and consistency of script generation, providing a flexible and reliable starting point for addressing diverse vehicle testing scenarios.

[0026] S102. Perform test operations on the vehicle system based on the test script, wherein the test operations are performed based on the attribute positioning and / or image recognition of the vehicle buttons; In the current field of automated testing for automotive systems, attribute localization (uiautomator2) and image recognition (airtest) are two mainstream interface element recognition technologies. Attribute localization relies on acquiring and parsing standardized attribute information of controls for location, but its recognition capability significantly weakens when faced with the numerous non-standard custom controls present in automotive systems. Image recognition methods locate targets by comparing screenshots with preset image templates, but its recognition stability is easily affected by factors such as screen resolution, ambient lighting, and differences in interface rendering, leading to significant fluctuations in matching success rates. Both of these technical approaches have inherent limitations, and in practical applications, they often employ single, static execution strategies, failing to dynamically adjust based on real-time feedback during test execution. This results in insufficient adaptability to dynamic changes in the automotive testing environment, constraining the overall stability and reliability of the testing process.

[0027] This application establishes an execution log analysis mechanism to continuously record the actual effects of the two recognition technologies. When attribute localization fails due to timeout caused by interface element loading delays, the system can automatically adjust the waiting time or switch to the image recognition scheme. When image recognition suffers from insufficient confidence due to changes in lighting, the system can either optimize image matching parameters or attempt attribute localization as a supplementary solution. This dynamic adjustment strategy enables the test system to autonomously select the optimal recognition path based on actual execution conditions.

[0028] By employing the aforementioned technical solutions, the collaborative application of attribute localization and image recognition establishes a multi-layered protection mechanism, significantly reducing the risk of test interruptions due to the limitations of a single recognition technology. The complementary advantages of these two technologies retain the efficient processing capability for standardized controls while enhancing adaptability to non-standard interfaces. This maintains the continuity and reliability of the testing process under the diverse and uncertain conditions of the automotive testing environment, ultimately achieving a dual improvement in the test framework's adaptability and stability.

[0029] In one embodiment, performing test operations on the in-vehicle system based on the test script includes: Perform environment initialization based on the test parameters; The operation is executed based on the type of the operation steps described.

[0030] For example, the type of operation step refers to the classification of operation steps based on the interactive object, such as hardware interaction involving physical device control or UI interaction involving graphical interface elements. This classification is used to determine the specific interface or framework called when performing the operation.

[0031] In this application, the test environment is first initialized based on test parameters such as screen specifications and associated application execution environment to ensure that the test environment matches the specified conditions. Then, the corresponding execution module is called according to the type of operation step. For example, when the operation step is identified as hardware interaction, the system calls the vehicle hardware driver interface to simulate physical operation. When it is identified as UI interaction, the UI test framework is called to execute interface element operations, thereby realizing the step-by-step execution of the test script.

[0032] By employing the above technical solutions, the consistency of test conditions is ensured through environment initialization, reducing the risk of execution deviations caused by differences in environment configuration; resource scheduling efficiency is optimized by distributing operation instructions by type, enabling the system to flexibly adapt to various interaction scenarios and improving the coordination and overall stability of the testing process.

[0033] In one embodiment, the type-based invocation of the operation based on the operation steps includes: When the operation step involves hardware interaction, the test operation is performed by calling the vehicle hardware driver interface. When the operation step involves UI interaction, the UI testing framework is invoked to execute the test operation.

[0034] In this embodiment, the execution method is dynamically selected according to the type of operation step. When the operation step is identified as a hardware interaction, the vehicle hardware driver interface is called to simulate physical operations such as knob rotation or button pressing. When the operation step is identified as a UI interaction, the system calls the UI testing framework to perform interface element operations such as control clicking or sliding, ensuring that the test action and interaction type are accurately matched.

[0035] By employing the above technical solution, this step optimizes the selection of execution paths by distributing operation instructions according to type, reduces the risk of execution errors caused by mismatched interaction methods, improves the adaptability of test operations to the actual interaction scenarios of the vehicle system, and thus enhances the accuracy and efficiency of the test process.

[0036] In one embodiment, when the target object is a UI operation, invoking a UI testing framework to execute the test operation includes: In the case of the test operation being attribute location, the attribute location assistant is invoked to find the target object through the control attribute information; In the case of image recognition, the image recognition assistant is invoked to find the target object through image matching. If the search operation fails, an optimization strategy is executed.

[0037] For example, an attribute location assistant is a tool for locating a target object through control attribute information, and an image recognition assistant is a tool for locating a target object through image matching. Control attribute information includes data such as the identifier or text description of the target control. Image matching is the process of identifying the target object by comparing the current screenshot with a pre-stored image template. The search operation is the action performed to locate the target object. The operation result is the output status of the search operation. Search failure indicates that the target object was not successfully located. The optimization strategy is the improvement plan adopted by the system when the search fails.

[0038] In this embodiment, when the target object is a UI operation, the corresponding assistant is invoked to perform the search operation according to the specific type of the test operation; when the test operation is attribute positioning, the attribute positioning assistant is invoked to find the target object by parsing the control attribute information; when the test operation is image recognition, the system invokes the image recognition assistant to find the target object by image matching technology; if the search operation fails, an optimization strategy is automatically triggered to adjust the positioning method.

[0039] By integrating multiple positioning methods, the system enhances its adaptability to different interface elements and reduces the probability of test interruption due to the failure of a single positioning method. By automatically activating optimization strategies in case of failure, the system's self-adjustment capability is improved, thereby enhancing the robustness and continuity of the test execution process.

[0040] In one embodiment, the step of executing an optimization strategy when the search operation fails includes: If the time taken by the attribute location assistant to search for the target object using control attribute information exceeds a preset time, the search is determined to have failed. If the confidence level of the object found by the image recognition assistant through image matching is lower than the preset confidence level, the search is determined to have failed.

[0041] For example, the preset time is the maximum time limit set for the attribute location assistant to complete the search operation, and the preset reliability refers to the minimum similarity threshold set by the system for the image matching results of the image recognition assistant. These two parameters together constitute the objective judgment criteria for the search operation results.

[0042] In this application, the system determines the result by continuously monitoring the execution status of the search operation. When the time consumed by the attribute positioning assistant to search for the target object through the control attribute information exceeds the preset time limit, the search operation is automatically determined to have failed. Similarly, when the similarity of the object found by the image recognition assistant through image matching technology is lower than the preset confidence threshold, the search is also determined to have failed. These two determination mechanisms provide clear triggering conditions for subsequent optimization strategies.

[0043] By using the above technical solution, precise monitoring of the search operation status is achieved by setting quantitative judgment criteria, which reduces the risk of resource waste caused by long-term no response or low-quality matching, and provides a decision basis for the system to switch operation strategies in a timely manner, thereby improving the response speed and execution efficiency of the testing process.

[0044] Considering that virtual buttons are common touchscreen controls in automotive systems (such as music play buttons and menu icons), their testing relies on software-level positioning and operation. This application achieves automated testing of virtual buttons through the following methods: When the test operation is attribute positioning, the system calls an attribute positioning assistant to find the target object through control attribute information (such as ID and text description). This is suitable for standardized controls (such as native Android buttons) and can quickly and accurately locate them. When attribute positioning fails, the system switches to an image recognition assistant to identify custom controls through image matching (such as screenshot comparison). This method can handle non-standard UI elements (such as irregularly shaped icons) and improve adaptability. Execution logs are loaded, and when virtual button recognition fails (such as due to interface offset), the system automatically adjusts its strategy (such as correcting the matching threshold) based on diagnostic information (such as error type and screenshot) to ensure test continuity. The test process of this application includes UI interaction steps from script generation to execution to complete the virtual button test.

[0045] Considering that physical buttons are physical hardware in the in-vehicle environment (such as steering wheel buttons and air conditioning knobs), their testing needs to simulate real physical operations. This application extends support for physical buttons through a hardware interaction module: when the operation step is a hardware interaction, the system calls the in-vehicle hardware driver interface to execute test operations (such as simulating knob rotation or button pressing). This directly converts physical actions into electrical signals, triggering a response from the in-vehicle system. Test parameters include "interaction type," which can be defined as a hardware operation (such as adjusting volume with a knob). The execution layer simulates physical button events through the driver interface and links them with UI changes (such as progress bar updates). When a physical button operation fails (such as a driver timeout), logs are recorded (such as process data and error type), and optimization strategies are generated (such as adjusting driver parameters or retry mechanisms) to improve stability.

[0046] This application uses a single framework to handle mixed scenarios involving virtual and physical buttons, avoiding the drawbacks of traditional testing that require switching tools. Test scripts are generated based on test requirements (e.g., natural language descriptions of "testing the volume knob and screen mute button"), and these scripts can include both UI operation steps (clicking the virtual mute button) and hardware operation steps (rotating the physical knob). The execution path is automatically selected based on the operation type—UI interactions call the test framework, and hardware interactions call the driver interface. When one method fails, the optimization unit can adjust the strategy across modalities (e.g., replacing the test with a virtual button after a physical button malfunctions). Taking an in-vehicle entertainment system as an example, a user might first adjust the volume using a physical knob and then switch songs using a virtual button. The framework fusion layer of this application coordinates both types of interactions, ensuring end-to-end test integrity.

[0047] S103. Record the execution log of the test operation to optimize the initial strategy of the operation steps, wherein the execution log includes process data and diagnostic information of the test operation.

[0048] In one embodiment, the step of recording the execution log of the test operation to optimize the initial strategy of the operation steps includes: If the test operation fails, the process data and diagnostic information of the test operation are obtained, wherein the diagnostic information includes the error type, failure screenshot and scenario context. The cause of the operation failure was determined based on the process data and diagnostic information. An optimization strategy is generated based on the reasons for the operation failure, wherein the optimization strategy includes parameter correction; The initial strategy for optimizing the operation steps is based on the optimization strategy.

[0049] For example, process data includes various raw records generated during test execution, diagnostic information includes error types, failure screenshots, and scenario contexts for analyzing problems, error types identify the specific category of operation failure, failure screenshots record the interface state when the operation fails, scenario context describes the environmental conditions when the test operation is executed, the cause of operation failure refers to the root cause of the test operation's failure, optimization strategy is a new solution developed to improve the test operation, and correction parameters are the specific numerical settings in the optimization strategy used to adjust the operation.

[0050] In this embodiment, when the result of the test operation is determined to be an operation failure, the system will automatically collect the process data and diagnostic information of the operation; then, based on the collected error type, failure screenshots and scene context and other multi-dimensional information, it will perform comprehensive analysis to accurately determine the specific reason for the operation failure; then, based on the analyzed failure reason, it will generate a targeted optimization strategy, which includes executable correction parameters; finally, the system will apply the generated optimization strategy to adjust and update the initial strategy of the original operation steps in real time.

[0051] By employing the above technical solutions, a complete self-optimization cycle is established through a systematic data collection and analysis mechanism, reducing the risk of repeated failures caused by fixed strategies being unsuitable for dynamic environments. Furthermore, the strategy adjustment mechanism based on actual execution results enhances the test system's adaptability to external changes, thereby improving the intelligence and sustainable improvement capabilities of the test process.

[0052] Furthermore, the system architecture of this application comprises five layers: a test scheduling layer responsible for parsing natural language test cases and generating execution plans; a framework fusion layer integrating the positioning and operation interfaces of uiautomator2 and airtest; an AI enhancement layer providing multimodal recognition and intelligent decision-making based on LLM (Large Language Model); an execution layer driving in-vehicle hardware interaction; and a results analysis layer generating diagnostic reports through LLM. Each layer synchronously calls itself via the JSON-RPC 2.0 protocol.

[0053] Understandably, LLM is the core engine driving the intelligence of automotive automated testing systems. It is not a standalone tool but deeply integrated into the system architecture, addressing issues like poor adaptability and low stability in automotive testing environments through multimodal data processing, dynamic decision-making, and self-learning capabilities. LLM is deployed in the AI ​​enhancement layer, located between the framework fusion layer and the execution layer, serving as the intelligent brain of the testing process. The AI ​​enhancement layer communicates with other layers through standardized interfaces (such as JSON-RPC 2.0): receiving test requests (such as control location requirements) from the framework fusion layer and outputting decision instructions (such as optimal location strategies) to the execution layer. This application employs a multimodal large model because it can simultaneously process text, images, and contextual information. For example, during control recognition, the model input includes control attributes (text), screenshots (images), and scene descriptions (text), outputting a comprehensive judgment result. To improve adaptability to automotive scenarios, LLM is fine-tuned using domain data. For example, models (such as CodeLlama-7B) can be fine-tuned using in-vehicle control image libraries and interaction logs to better familiarize them with in-vehicle UI elements (such as irregularly shaped buttons) and hardware interaction patterns (such as knob operations). This construction approach ensures that the LLM is not a general-purpose model, but rather a domain expert specifically designed for in-vehicle testing. The LLM's input data comes from multi-source information uniformly encapsulated by the framework fusion layer: Attribute data: control IDs, text descriptions, etc., provided by uiautomator2. Image data: screenshots or control templates captured by airtest. Context data: descriptions of the test scenario (such as the current application state and user operation history). The output is structured decisions, such as JSON-formatted policy instructions (e.g., {"framework": "airtest", "params": {"threshold":0.85}}), directly guiding execution layer operations. This construction makes the LLM an intelligent bridge connecting "perception" (framework fusion) and "execution" (hardware / UI operations).

[0054] LLM directly translates natural language testing requirements into executable code. For example, given the input "On a 12.3-inch screen, say 'Open music' and click the play button," LLM generates a Python script (integrating uiautomator2 and airtest calls), reducing manual coding costs.

[0055] When locating in-vehicle buttons (such as virtual touch buttons or physical knobs), LLM comprehensively analyzes information from multiple sources and outputs the optimal location strategy. For example, for a custom play button, LLM receives attribute data (such as contentDescription), screenshots, and context (such as "music app just launched"), determines that image recognition is more reliable than attribute location, and recommends matching parameters (such as a confidence threshold of 0.8).

[0056] When traditional localization times out or fails, LLM (Multimodal Learning) is invoked for multimodal recognition. It mentions executing an optimization strategy when a search fails—LLM is the core of this intelligent implementation.

[0057] The LLM analyzes the error types, screenshots, and context in the test failure logs to determine the root cause (e.g., resolution differences causing image matching failure). Next, the LLM generates optimization strategies (e.g., switching to attribute localization or adjusting matching thresholds). Finally, for new requirements, the LLM directly converts natural language test cases into test scripts, improving automation efficiency. The uiautomator2 and airtest interfaces are uniformly called through an adaptation plugin. The AI ​​decision module matches according to priority rules: for example, uiautomator2 is selected when control attributes are complete, and LLM multimodal recognition is triggered when timeouts or low confidence levels occur. The decision results optimize the localization strategy in real time, ensuring cross-vehicle adaptation.

[0058] For hardware interaction types (such as knobs or voice commands), the in-vehicle hardware driver interface is invoked to simulate operations: for example, turning a knob triggers a progress bar update, and voice input "open navigation" launches the corresponding application. Hardware events and UI testing are synchronized through the framework fusion layer, covering full-scenario verification.

[0059] Input natural language test cases (such as "verify air conditioner knob temperature adjustment"), and LLM automatically extracts parameters and generates test scripts. The optimization unit implements closed-loop repair: when an operation fails, LLM is invoked to analyze logs, dynamically correct strategies (such as increasing wait time or offset coordinates), and retry until success or the maximum number of attempts.

[0060] Furthermore, as a response to the above Figure 1 In addition to the implementation of the method shown, this embodiment of the invention also provides an automated testing device for vehicle-mounted buttons, used for testing the aforementioned... Figure 1 The method shown is implemented accordingly. This device embodiment corresponds to the foregoing method embodiment. For ease of reading, this device embodiment will not repeat the details of the foregoing method embodiment, but it should be clear that the device in this embodiment can implement all the contents of the foregoing method embodiment. Figure 2 As shown, the device includes: a generation unit 21, an execution unit 22, and an optimization unit 23, wherein... The generation unit 21 is used to generate test scripts based on test requirements, wherein the test requirements are used to perform interactive tests on the vehicle buttons, and the test scripts include operation steps and corresponding initial strategies. The execution unit 22 is used to perform test operations on the vehicle system based on the test script, wherein the test operations are performed based on the attribute positioning and / or image recognition of the vehicle buttons; The optimization unit 23 is used to record the execution log of the test operation in order to optimize the initial strategy of the operation steps, wherein the execution log includes the process data and diagnostic information of the test operation.

[0061] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured, and by adjusting kernel parameters, an automated testing method for vehicle buttons can be implemented, addressing the poor adaptability and instability issues of current automated testing frameworks.

[0062] This invention provides a computer-readable storage medium including a stored program that, when executed by a processor, implements an automated testing method for the vehicle-mounted button.

[0063] This invention provides a processor for running a program, wherein the program executes an automated testing method for the vehicle button.

[0064] This invention provides an electronic device, which includes at least one processor and at least one memory connected to the processor; wherein the processor is used to call program instructions in the memory to execute an automated testing method for vehicle buttons as described above. This invention provides an electronic device 30, such as... Figure 3 As shown, the electronic device includes at least one processor 301, and at least one memory 302 and bus 303 connected to the processor; wherein, the processor 301 and the memory 302 communicate with each other through the bus 303; the processor 301 is used to call program instructions in the memory to execute the above-mentioned automated testing method for vehicle buttons.

[0065] The smart electronic devices mentioned in this article can be PCs, tablets, mobile phones, etc.

[0066] This application also provides a computer program product that, when executed on a process management electronic device, is suitable for executing the steps of an automated test method that initializes the aforementioned vehicle-mounted buttons.

[0067] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0068] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0070] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0071] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0072] This application also provides a computer program product, which includes computer software instructions that, when executed on a processing device, cause the processing device to perform actions such as... Figure 1 The control flow of the memory in the corresponding embodiment.

[0073] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0074] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0075] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0076] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0077] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0078] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0079] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An automated testing method for vehicle-mounted buttons, characterized in that, include: Test scripts are generated based on test requirements, wherein the test requirements are used to perform interactive tests on in-vehicle buttons, and the test scripts include operation steps and corresponding initial strategies; Test operations are performed on the in-vehicle system based on the test script, wherein the test operations are performed based on the attribute positioning and / or image recognition of the in-vehicle buttons; The execution log of the test operation is recorded to optimize the initial strategy of the operation steps. The execution log includes process data and diagnostic information of the test operation.

2. The method according to claim 1, characterized in that, The generation of test scripts based on test requirements includes: Obtain test requirements, wherein the test requirements are described in natural language; The test requirements are analyzed to extract test parameters, which include screen specifications, interaction type, target object, and associated application. A test script is generated based on the test requirements and test parameters.

3. The method according to claim 2, characterized in that, The execution of test operations on the vehicle system based on the test script includes: Perform environment initialization based on the test parameters; The operation is executed based on the type of the operation steps described.

4. The method according to claim 3, characterized in that, The operation performed by invoking the type based on the operation steps includes: When the operation step involves hardware interaction, the test operation is performed by calling the vehicle hardware driver interface. When the operation step involves UI interaction, the UI testing framework is invoked to execute the test operation.

5. The method according to claim 4, characterized in that, When the target object is a UI operation, the step of calling the UI testing framework to execute the test operation includes: In the case of the test operation being attribute location, the attribute location assistant is invoked to find the target object through the control attribute information; In the case of image recognition, the image recognition assistant is invoked to find the target object through image matching. If the search operation fails, an optimization strategy is executed.

6. The method according to claim 5, characterized in that, When the search operation fails, the optimization strategy is executed, including: If the time taken by the attribute location assistant to search for the target object using control attribute information exceeds a preset time, the search is determined to have failed. If the confidence level of the object found by the image recognition assistant through image matching is lower than the preset confidence level, the search is determined to have failed.

7. The method according to claim 1, characterized in that, The execution log of the test operation, used to optimize the initial strategy of the operation steps, includes: If the test operation fails, the process data and diagnostic information of the test operation are obtained, wherein the diagnostic information includes the error type, failure screenshot and scenario context. The cause of the operation failure was determined based on the process data and diagnostic information. An optimization strategy is generated based on the reasons for the operation failure, wherein the optimization strategy includes parameter correction; The initial strategy for optimizing the operation steps is based on the optimization strategy.

8. An automated testing device for vehicle-mounted buttons, characterized in that, Also includes: A generation unit is used to generate test scripts based on test requirements, wherein the test requirements are used to perform interactive tests on in-vehicle buttons, and the test scripts include operation steps and corresponding initial strategies. An execution unit is configured to perform test operations on the in-vehicle system based on the test script, wherein the test operations are performed based on the attribute positioning and / or image recognition of the in-vehicle buttons; An optimization unit is used to record the execution log of the test operation in order to optimize the initial strategy of the operation steps, wherein the execution log includes process data and diagnostic information of the test operation.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed by a processor, it implements the steps of the automated testing method for vehicle buttons as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The electronic device includes at least one processor and at least one memory connected to the processor; wherein the processor is configured to call program instructions in the memory to execute the steps of the automated testing method for vehicle buttons as described in any one of claims 1 to 7.

Images