Test method, system and device for hmi application

Abstract

This application discloses a testing method and apparatus for an HMI application. The method includes: acquiring a set of test cases for a target HMI application; performing automated triggering operations on the target HMI application based on triggering information to cause the target HMI application to output a testable output corresponding to a target HMI interaction event; collecting the testable output to obtain a testable data set corresponding to the testable output; constructing at least one test input data pair based on the expected test content and the testable data set; inputting the at least one test input data pair into a multimodal large language model to obtain a test result characterizing the consistency between the testable output and the expected test content; and generating automated test conclusions for the corresponding test cases based on the test result. This elevates the testing from traditional text similarity or pixel-level comparison to semantic-level consistency verification oriented towards real user perception.

Description

Technical Field

[0001] This application relates to the field of software testing technology, and in particular to a testing method, system and device for HMI applications. Background Technology

[0002] In smart cockpits, pop-up text, text-to-speech (TTS) prompts, and pop-up icons of the Human-Machine Interface (HMI) are high-frequency and high-risk interaction elements. Once problems such as "inconsistent text, inconsistent prompts, or semantic errors in icons" occur, they often directly lead to driver distraction, affect the understanding of warnings, and affect the usability of functions. Therefore, scalable and repeatable automated verification capabilities are required.

[0003] Currently, automated testing frameworks are used to control the interaction flow of HMI applications and collect verifiable data during test case execution. For example, the Appium automated testing tool can be used to obtain pop-up text, save screenshots of pop-up areas, and extract or listen to TTS audio files for subsequent consistency verification.

[0004] However, traditional verification and testing methods have structural shortcomings in terms of "semantic consistency" and "user's actual perception." First, TTS often uses ASR (Automatic Speech Recognition) to transcribe the text and then compares it with the expected text for similarity. This process introduces recognition errors (such as pronunciation of polyphonic characters or unclear pronunciation), makes it difficult to evaluate auditory details such as phrasing, stress, and speech rate, and is affected by the stability of the ASR service. Second, icon verification often uses pixel-level comparison, which is heavily dependent on resolution and rendering consistency. It is prone to misjudgment under scaling and rendering differences, often requires manual masking of irrelevant areas, and lacks robust recognition capability for icon versions with different pixels but the same meaning. Summary of the Invention

[0005] This application provides a testing method, device, storage medium, and program product for HMI applications, which aims to solve at least one of the above-mentioned technical problems.

[0006] In a first aspect, embodiments of this application provide a testing method for an HMI application, comprising: acquiring a set of test cases for a target HMI application, wherein each test case in the set includes at least: triggering information for triggering a target HMI interaction event, and expected verification content corresponding to the target HMI interaction event; performing an automated triggering operation on the target HMI application based on the triggering information, so that the target HMI application outputs an output to be verified corresponding to the target HMI interaction event; wherein the output to be verified includes output information of at least one modality; collecting the output to be verified to obtain a set of data to be verified corresponding to the output to be verified; constructing at least one verification input data pair based on the expected verification content and the set of data to be verified, wherein the verification input data pair includes: expected verification data for the same target HMI interaction event and data to be verified corresponding to the expected verification data; inputting the at least one verification input data pair into a multimodal large language model to obtain a verification result for characterizing the consistency between the output to be verified and the expected verification content, and generating automated test conclusions for corresponding test cases based on the verification result.

[0007] Secondly, embodiments of this application provide a storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform the steps of the method described in any of the above claims of this application.

[0008] Thirdly, a computer device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any of the preceding claims of this application.

[0009] Fourthly, embodiments of this application also provide a computer program product, the computer program product including a computer program stored on a storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the steps of any of the methods described above.

[0010] Fifthly, embodiments of this application also provide a mobile platform, including: a platform body; and a control system disposed on the platform body; wherein the control system is configured to execute the steps of any of the above methods to manage intelligent driving scenarios during the operation of the mobile platform.

[0011] The beneficial effects of the embodiments of this application are as follows: By constructing a verification input data pair between the expected verification content of the same interaction event and the actual collected verification data (such as pop-up text, pop-up screenshots / icons, or TTS audio) in the HMI automated testing process, and inputting it into a multimodal large language model for unified consistency judgment, the test is upgraded from traditional text similarity or pixel-level comparison to semantic-level consistency verification oriented towards the user's real perception. This enables robust identification of semantic deviations and conflicts between cross-modal text / broadcast / icons without relying on ASR transcription and pixel alignment, reducing misjudgments caused by recognition errors, resolution and rendering differences, improving the accuracy, stability and repeatability of test conclusions, and enhancing the scalable automation capability of HMI interaction consistency verification. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating an example of a testing method for an HMI application according to an embodiment of this application is shown; Figure 2 This document shows an example of an operation flowchart illustrating the method for obtaining consistency verification results using a multimodal large language model according to an embodiment of this application. Figure 3 A schematic diagram of the architecture for TTS verification based on a specific multimodal large model according to an embodiment of this application is shown; Figure 4 This document shows a flowchart illustrating another example of obtaining consistency verification results using a multimodal large language model in the method according to embodiments of this application. Figure 5 A schematic diagram of the architecture for icon verification based on a specific multimodal large model according to an embodiment of this application is shown; Figure 6 A flowchart illustrating an example of constructing a validation input data pair in a method according to an embodiment of this application is shown. Figure 7 The diagram illustrates an example of an automated test conclusion generated based on the verification result in the method according to an embodiment of this application. Figure 8 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0015] It should also be noted that, in this document, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0016] It should be noted that in current technologies, in the automated verification scenario of smart cockpit HMI applications, testers typically need to verify the consistency of pop-up interaction elements (such as pop-up text, TTS broadcast content, and pop-up icons) during batch test case execution. To improve test coverage and regression efficiency, automated testing frameworks are commonly used to drive the HMI application, reproducing the interaction process through scripts and collecting verifiable data after the interaction event is triggered. In some engineering practices, mobile automation frameworks can be used to locate pop-up controls (e.g., locating the pop-up text area through control identification information), simultaneously saving screenshots of the pop-up area, and extracting or listening to TTS audio files from the application's audio output path to form an automated chain of test case triggering, data collection, and consistency verification.

[0017] However, in the aforementioned automated process, data collection does not equate to verification reflecting the consistency of actual interactions. Regarding TTS (Text-to-Speech) broadcast verification, current technologies often employ a method of first converting audio to text and then comparing it. Specifically, TTS audio is converted to text using an automatic speech recognition service, and then compared with the expected text for text similarity, using a similarity threshold as the pass / fail criterion. The inventors' research found that this verification method, using ASR (Automatic Speech Recognition) transcription as an intermediate step, is prone to introducing structural errors: First, the transcription results are affected by factors such as polyphonic characters, unclear pronunciation, and changes in speech rate, leading to deviations at the text level and thus unstable verification conclusions; second, text similarity comparisons typically only cover "whether the broadcast content is close to the expectation," but fail to reflect details closely related to the driver's actual listening experience, such as sentence rhythm, stress distribution, and intonation coherence; third, the verification process depends on the availability and stability of external recognition services, and is prone to fluctuations in large-scale regression or offline environments, thereby affecting the consistency and reproducibility of automated test results.

[0018] In pop-up icon verification, the most common approach in current technologies is pixel-level similarity comparison: comparing the current screenshot with a standard icon pixel by pixel or matching a template, and determining whether it passes or fails based on a similarity threshold. However, the inventors' research found that this verification method is highly sensitive to rendering consistency. When there are differences in screen scaling, font and graphics rendering strategies, anti-aliasing effects, or theme resources, even if the icons are semantically identical, significant differences may appear at the pixel level, leading to misjudgments. Meanwhile, pop-up icons often coexist with bottom text, buttons, background textures, and other elements. To achieve high pixel matching accuracy, testers often need to manually mask irrelevant areas around the icon or manually crop the screenshot. This adds extra configuration costs to the automated process and lacks robustness when the interface layout changes or when adapting to multiple devices. Furthermore, for icon versions that are not completely pixel-identical but semantically identical (e.g., different graphic representations of the same alarm meaning in different design iterations), pixel-level comparison struggles to provide stable semantic consistency judgments, thus limiting the scalability of icon verification in long-term iterations and multi-version management.

[0019] Therefore, in the automated testing of smart cockpit HMIs, how to make the verification mechanism closer to the real perception of human announcements and icons while maintaining the scalable execution capability of the automation framework, and reducing the reliance on external recognition services, pixel consistency and manual masking operations, has become a key issue that the industry urgently needs to solve.

[0020] It should be understood that the above description of the relevant technologies is intended only to help the public better understand the inventive spirit and motivation of this application, and is not intended to limit this application. Furthermore, the technical solutions described in the above-mentioned relevant technologies are not prior art, and may also be undisclosed technical solutions, such as those under research or in the laboratory stage.

[0021] The technical solutions in this application, including the collection, storage, use, processing, transmission, provision, and disclosure of users' personal information, comply with relevant laws and regulations and do not violate public order and good morals.

[0022] Figure 1 A flowchart illustrating an example of a testing method for an HMI application according to an embodiment of this application is shown.

[0023] Regarding the execution subject of the method in the embodiments of this application, it can be any controller or processor with computing or processing capabilities. In some examples, the method in the embodiments of this application can be integrated and configured in an electronic device or terminal through software, hardware or a combination of software and hardware, and the type of terminal or electronic device can be diverse, such as mobile phone, tablet computer, desktop computer or vehicle terminal, etc.

[0024] For example, the execution entity of the method in this application embodiment may be an automated testing system integrated into the vehicle cockpit domain controller (CDC) or infotainment controller. The system uses a pre-built test case management and execution engine (e.g., UI automation driver for the vehicle HMI automation interface), a multimodal data acquisition module (for acquiring pop-up text, saving pop-up area screenshots / icons, and recording / extracting TTS audio, etc.), and a multimodal large language model inference verification module to perform automated triggering operations on the target HMI application, collect data to be verified, and output consistency verification results, thereby generating automated test conclusions for the corresponding test cases.

[0025] like Figure 1 As shown, in step S110, a set of test cases for the target HMI application is obtained.

[0026] The target HMI application is the specific vehicle model or version of the HMI application to be tested. It should be noted that the initialization phase of the test system first involves loading and parsing test data. Specifically, the system reads a set of test cases for the specific vehicle model or HMI version from a pre-set test management database or configuration file (such as a JSON or YAML configuration document). The test case set includes at least one test case, each targeting a specific HMI interaction event. Each test case is designed to strictly adhere to an "incentive-response" logical structure; therefore, each test case must include at least the trigger information for the target HMI interaction event and the correct response information after the trigger.

[0027] For example, each test case in the test case set includes at least two core data components: one component is trigger information used to trigger the target HMI interaction event, which defines how to drive the HMI system into a specific state, such as simulating a vehicle speed signal on the CAN bus, sending a specific UI automated click command, or simulating a voice assistant wake-up command; the other component is the expected verification content corresponding to the target HMI interaction event, which defines the correct information that the HMI should provide to the user under this triggering condition. This is usually text described in natural language (e.g., "a right turn warning should be played ahead") or a reference image (e.g., "the dashboard should display a speed limit sign"). Thus, by obtaining a standardized test case set, the system can clearly define the boundaries and objectives of the test.

[0028] In step S120, an automated triggering operation is performed on the target HMI application based on the triggering information, so that the target HMI application outputs a verification output corresponding to the target HMI interaction event, and the verification output includes output information of at least one modality.

[0029] In some implementations, after entering the execution phase, the test execution engine parses the aforementioned trigger information and interacts with the target HMI application through the corresponding communication interface. Specifically, if the trigger information is a vehicle signal (such as a door opening or a battery alarm), the system can send a simulated broadcast to the operating system where the HMI resides through a virtual signal generator; in addition, if the trigger information is a user action (such as clicking a screen button), the system can call a UI automation framework (such as Appium) to simulate the user's touch events, which aims to simulate real user driving scenarios or vehicle driving states, forcing the HMI application under test to respond.

[0030] As the triggered operation is executed, the target HMI application can generate feedback in real time; this feedback is the output to be verified. In the smart cockpit scenario, this output can be unimodal or multimodal. For example, it can include not only visual interface changes displayed on the screen (such as pop-ups, icon flashing, page jumps) but also voice prompts or alarm sounds played through the speakers. This achieves the transformation from static test cases to dynamic system responses, covering not only single interface jumps but also triggering complex audiovisual linkage scenarios, ensuring the completeness of test coverage.

[0031] In step S130, the output to be verified is collected to obtain the set of data to be verified corresponding to the output to be verified.

[0032] In some implementations, the system can capture and archive the instantaneous feedback generated by the HMI application in the physical world or digital interface into analyzable data files. Specifically, the system initiates a multi-channel data acquisition service: for visual output, it captures screen image data at the current moment using a screenshot tool or video stream recording interface; for auditory output, it captures audio stream data using the system's internal audio recording interface or analog microphone channel. After acquisition and processing, the acquired screen light and / or sound signals are converted into a digitized set of data to be verified (such as PNG image files, WAV audio files, etc.), thereby achieving digital preservation of the HMI testing site.

[0033] In step S140, at least one verification input data pair is constructed based on the expected verification content and the set of data to be verified. The verification input data pair includes: expected verification data for the same target HMI interaction event and the data to be verified corresponding to the expected verification data.

[0034] In some implementations, since multimodal large language models need to understand the context of the task, the system can construct structured verification input data pairs after obtaining the raw data to be verified. Specifically, the system maps and binds the expected verification content (as the truth standard) to the collected data to be verified (as the actual observation) one-to-one according to the test case ID. For example, the expected text describing "navigation prompt tone" is paired with the collected actual audio file, or the expected text and / or expected image describing "warning pop-up" is paired with the collected screenshot.

[0035] By integrating data from the same modality (such as expected images and screenshots) or different modalities (such as expected text and measured images, or expected text and measured audio) into a single input structure, the system prepares all the contextual information needed for large models to perform semantic alignment tasks. This construction of data pairs is no longer limited by the traditional requirement of same-modal comparison (such as image-to-image or text-to-text), but fully utilizes the cross-modal understanding capabilities of large models and supports inputs from different modalities, greatly improving the flexibility of data preparation.

[0036] In step S150, at least one verification input data pair is input into the multimodal large language model to obtain a verification result that characterizes the consistency between the output to be verified and the expected verification content, and an automated test conclusion for the corresponding test case is generated based on the verification result.

[0037] Here, the system sends the assembled verification input data pairs to a pre-trained multimodal large language model. It's worth noting that the multimodal large language model can incorporate massive amounts of cross-modal alignment data during pre-training or fine-tuning, enabling it to specifically learn acoustic features in the audio modality, including contextual pronunciation of polyphonic characters, pause rhythm, and intonation stress; and in the visual modality, it learns the main visual semantic features that ignore background noise and focus on the target object (such as icons, UI controls). Based on this pre-training foundation, the model leverages its powerful feature extraction and semantic alignment capabilities to analyze in the vector space whether the actual meaning expressed by the output to be verified matches the semantics of the expected verification content. The model no longer simply compares differences in pixel values ​​or perfect string matches, but can perform cognitive judgments similar to those of a human tester. For example, it can determine whether the actual played speech matches the expected text description in terms of tone and content, or whether the icon in the screenshot visually represents the expected warning meaning.

[0038] Based on the analysis results of the model, the system directly obtains the verification results representing consistency. Specifically, the consistency verification results can refer to the structured evaluation data output by the multimodal large language model, which indicates the degree of semantic matching between the output to be verified and the expected content. These results typically include at least one or more quantitative evaluation indicators (such as the matching degree of the broadcast content, the reasonableness score of the listening experience, and the semantic matching degree of the icon) and qualitative matching status (such as whether a match is achieved and the specific reasons for anomaly identification). The system parses the above verification results and generates the final automated test conclusion (such as "pass" or "fail"). In this embodiment, by using probabilistic semantic understanding instead of traditional rigid rule matching, common problems in HMI testing, such as resolution adaptation, rendering subtle differences, or the diversity of speech synthesis, are effectively addressed, significantly reducing the false alarm rate and achieving intelligent automated HMI acceptance.

[0039] Regarding the implementation details of step S130, in some examples of embodiments of this application, when the output to be verified includes audio output information, the audio file generation path used by the target HMI application to generate the audio file corresponding to the audio output information is monitored.

[0040] Here, for test scenarios involving text-to-speech (TTS) feedback in HMI interactions, the system employs a lossless acquisition strategy based on file system monitoring. Specifically, the test system pre-configures a specific directory path (i.e., the audio file generation path) at the operating system level used by the target HMI application for temporary storage or output of synthesized speech streams. For example, the test engine establishes a real-time monitoring service for this path by calling the operating system's file monitoring interface. This service silently performs monitoring in the background, aiming to capture file system write events, thereby perceiving the real-time activity status of the speech synthesis engine without intruding on the HMI application's business logic.

[0041] Then, if a new audio file is detected to be generated under the audio file generation path, the new audio file is identified as the audio data to be verified in the data set to be verified, wherein the audio data to be verified is the raw digital audio signal generated by the target HMI application.

[0042] More specifically, once the monitoring service detects the generation of a new audio file under the aforementioned path, the system immediately locks the file and marks it as the response data for the current interaction event. Furthermore, to prevent read conflicts caused by concurrent writes or interference from old data, the system may combine timestamp verification or filename matching rules to accurately filter out the latest target file and copy or extract it to the data set to be verified (or a dedicated temporary data cache), establishing it as the audio data to be verified. This achieves synchronous acquisition of audio data from the generation end to the testing end, ensuring the timeliness of the test data and the accuracy of the correspondence.

[0043] It is worth noting that the audio data to be verified obtained through the above method is the raw digital audio signal generated by the target HMI application (such as pure PCM data or WAV files). Compared with the traditional method of physical recording via microphone, this embodiment completely eliminates the physical link of "digital-to-analog conversion - air propagation - analog-to-digital conversion", thereby completely avoiding the contamination of test data by environmental noise (such as air conditioner noise, current noise), speaker distortion, and frequency response differences of recording equipment. By using high-fidelity digital signal acquisition, the purest input source is provided for the subsequent acoustic feature analysis of multimodal large models, significantly improving the accuracy of identifying detailed features such as the pronunciation of polyphonic characters and subtle pauses in speech.

[0044] Figure 2This document illustrates an example of obtaining consistency verification results using a multimodal large language model according to an embodiment of this application. In this embodiment, when the output to be verified includes interface image information, the expected verification content includes expected speech text corresponding to the audio data to be verified.

[0045] like Figure 2 In the illustrated embodiment, the output to be verified includes audio data to be verified, and the expected verification content includes expected speech text. In step S210, a first verification input data pair is constructed based on the audio data to be verified and the expected speech text, and the first verification input data pair is input into the multimodal large language model.

[0046] Here, the system performs data preprocessing and modal fusion operations. Specifically, the system structurally encapsulates the directly acquired raw digital audio data and the expected speech text defined in the test cases, constructing the first verification input data that the model can recognize. During this process, the system may convert the audio data into tensor formats or embedding vectors supported by the model's audio encoder, and convert the expected text into corresponding token sequences. Together, these constitute the cross-modal context input. This establishes a clear "standard-test" alignment context, enabling the multimodal large language model to understand that the subsequent task is not simply speech recognition, but rather verifying the correctness of the audio data based on a given text standard.

[0047] In step S220, an acoustic feature extraction process is performed on the audio data to be verified using a multimodal large language model to obtain audio features used to characterize the broadcast content.

[0048] Specifically, the multimodal large language model can activate its internal audio encoding module to perform deep acoustic feature extraction on the input audio data to be verified. For example, the audio features include at least one of the following: phoneme-related features, stress-related features, and pause rhythm-related features.

[0049] Unlike traditional ASR methods that focus solely on transcribing sound into text, the feature extraction process in this embodiment aims to capture richer, finer-grained information from the audio signal, including the accuracy of the phoneme sequence (for pronunciation identification), the distribution of pitch stress (for identifying emphasis), and the rhythmic patterns of speech rate and pauses (for identifying sentence breaks). This processing mechanism enables the model to deeply analyze the emotional attributes and prosodic features in the audio signal, effectively overcoming the technical shortcomings of traditional methods that, due to a lack of acoustic details, cannot identify abnormal intonation or inappropriate emotional expression in speech broadcasts even when the text content is correct.

[0050] In step S230, a multimodal large language model is used to perform cross-modal semantic alignment of audio features with the expected speech text to output a verification result containing audio verification sub-results.

[0051] Here, the audio verification results include the matching degree of the broadcast content and the reasonableness score of the listening experience. The matching degree of the broadcast content is used to indicate the accuracy of the audio data to be verified in terms of text semantic content and pronunciation of polyphonic characters, while the reasonableness score of the listening experience is used to indicate the naturalness of the audio data to be verified in terms of sentence segmentation logic and tone of voice.

[0052] In this embodiment, the large language model utilizes a cross-modal attention mechanism to align the extracted acoustic features with the semantic features of the expected speech text. The model combines the contextual semantics of the text to verify whether the phoneme features at corresponding positions in the audio are consistent with the semantics of the expected text (e.g., whether they completely cover the word sequence of the expected text, whether they are in the same order as the word sequence in the expected text, etc.), and whether they conform to the correct pronunciation rules for polyphonic characters. Simultaneously, the model analyzes whether the pause rhythm features in the audio match the punctuation marks or logical sentence breaks in the text. Furthermore, it can analyze whether intonation features such as stress distribution and pitch changes conform to natural expression habits in a specific context. Consequently, the system output verification results are quantified into two core indicators: the broadcast content matching degree primarily reflects whether there are misreadings, omissions, or pronunciation errors of polyphonic characters, ensuring accurate information transmission; the auditory reasonableness score primarily reflects whether the broadcast is fluent and natural, and whether the punctuation and intonation conform to human language habits, thereby achieving a comprehensive evaluation of the HMI voice interaction quality. In other embodiments, the verification result may directly include a qualitative judgment indicating whether the current test case ultimately passes, or it may include a qualitative judgment of whether it passes and quantitative scores for the two dimensions mentioned above. In specific implementations, this conclusion can be generated by the system performing logical processing on the two scores (e.g., determining whether both scores fall within a preset acceptable range, or performing a weighted summation of the two scores and comparing it with a preset acceptable threshold); in other alternative embodiments, the multimodal large language model can also directly output the final conclusion of "pass" or "fail" after analyzing the above features.

[0053] Figure 3 The diagram illustrates an architecture diagram for TTS verification based on a specific multimodal large model according to an embodiment of this application. The diagram shows the data flow process when the multimodal large model adopts the Qwen3-Omni model, in which audio files and prompt text are input into the Qwen3-Omni model and structured verification results are output.

[0054] Regarding the aforementioned TTS verification process, a multimodal large language model can be selected that supports full-modal interaction (e.g., Figure 3The Qwen3-Omni model shown.

[0055] In this implementation scenario, the system directly feeds the model with the "expected text" (the prompt text shown in the figure) from the test cases and the "actual audio" (the WAV audio file shown in the figure) acquired in real time. Leveraging its end-to-end audio understanding capabilities, the model directly aligns acoustic features (covering phoneme sequences, stress distribution, and phrasing rhythm) with text semantics within the model itself, without requiring external ASR (Automatic Speech Recognition) conversion.

[0056] like Figure 3 As shown in the output section, the model can ultimately output structured data (such as JSON format) containing detailed verification conclusions. In this example, the model accurately identified the inconsistency between the audio playback content and the expected text (the presence of the extra word "vehicle" and a different word order), and output a verification result containing "is_passed: false" and the specific "fail_reason". Simultaneously, the model can also be configured to output quantified evaluation results for "playback content matching degree" and "sound perception reasonableness". Based on multiple experimental verification data, the inventors found that the overall accuracy of TTS automated verification using this specific model architecture can reach or exceed 99%, effectively solving the misjudgment problem caused by ASR recognition errors in traditional methods and achieving high-precision verification of voice interaction quality.

[0057] It should be noted that the embodiments of this application also provide a variety of alternative technical implementations to adapt to different testing environments and needs: In the selection of multimodal large language models, in addition to the aforementioned preferred models, other general-purpose large models with full-modal understanding capabilities can also be used (such as GPT-4V, multimodal derivatives of CLIP, etc.). In this case, the system only needs to adjust the encapsulation format of the input data according to the interface specification of the target model (for example, adjusting the text-image pairs to model-specific tensor inputs) to achieve similar semantic verification functions.

[0058] In other embodiments, as an alternative to the file system monitoring strategy, the system can also call the operating system's system-level sound capture interface (such as the MediaRecorder API or AudioRecord interface on the Android platform) to directly record the synthesized audio stream from the mixer output of the audio system. This method does not depend on a specific file generation path and is suitable for streaming playback scenarios where the audio stream does not need to be written to disk.

[0059] Regarding the implementation details of step S130, in some examples of embodiments of this application, when the output to be verified includes interface image information, the control layout structure of the current interface of the target HMI application is parsed, and the coordinate position and size parameters of the target element to be verified in the current interface are determined according to the control layout structure.

[0060] Here, for the visual verification scenario of HMI interface elements, the system abandons the traditional mechanical screenshot method based on fixed coordinates and instead adopts a dynamic positioning strategy based on the UI control tree. Specifically, the testing system parses the DOM structure or View control tree of the target HMI application's current interface in real time through the underlying interface of the UI automation testing framework. Based on the unique identifier of the target element preset in the test case, the system retrieves and extracts the layout attributes of the element in the control tree, thereby accurately calculating the absolute coordinate starting point (x, y) and rectangle size parameters (width, height) of the target control at the current screen resolution. This decouples test execution from screen resolution; even if the HMI system runs on a different sized vehicle screen or undergoes minor UI layout adjustments, the system can still adaptively lock the accurate position of the target element, greatly improving the compatibility and reusability of the test scripts.

[0061] Then, the screen capture command is invoked to capture a full-screen image of the current interface, and the region of interest containing the target element to be verified is cropped from the full-screen image based on the coordinate position and size parameters.

[0062] Specifically, the system calls the operating system's low-level screen capture instructions to obtain the complete display buffer data of the current frame and generate a high-fidelity full-screen screenshot. Next, using image processing algorithms (such as OpenCV), based on the coordinates and size parameters obtained in the previous steps, the system constructs a cropping box on the full-screen image, extracting the pixel region containing the target element to be verified and generating an independent region of interest (ROI) image. Thus, through physical cropping, dynamic backgrounds (such as map navigation base maps), adjacent distracting text, or status bar information surrounding the target element are forcibly removed, ensuring that the captured image data contains only the main content of interest for testing.

[0063] Furthermore, the region of interest image is identified as the image data to be verified in the dataset to be verified.

[0064] Specifically, the system formats and encodes the image of the region of interest (e.g., converting it to PNG or JPG format) and establishes it as the image data to be verified in the dataset. By simplifying the complex full-screen visual scene into a single target entity, the system ensures that the data input to the subsequent multimodal large model has extremely high semantic relevance, enabling the model to focus on specific objects when performing visual understanding. This avoids situations where the model's attention is scattered or misled by irrelevant similar elements in the background due to the input of an entire screenshot.

[0065] In this embodiment, a visual focusing mechanism is constructed during the data acquisition phase. Specifically, before data is input into the model, deterministic control layout information is used to filter out non-deterministic background noise. This not only significantly reduces computational redundancy when performing visual reasoning on large multimodal models, but also fundamentally solves the problem of frequent false detections caused by environmental interference in traditional visual testing under complex dynamic backgrounds (such as dynamic maps or video playback backgrounds), ensuring high confidence in automated test conclusions.

[0066] Figure 4 A flowchart illustrating another example of obtaining consistency verification results using a multimodal large language model according to an embodiment of this application is shown. In this embodiment, when the output to be verified includes interface image information, the expected verification content includes natural language prompts describing the visual features of the target element to be verified, and / or a reference image of the expected response of the target HMI application.

[0067] In step S410, based on the image data to be verified, a second verification input data pair is constructed with natural language prompts and / or reference images, and the second verification input data pair is input into the multimodal large language model.

[0068] Here, the system performs a structured assembly task of cross-modal data. Specifically, the system fuses the image data to be verified obtained from the previous steps (such as an ROI image containing only the target control) with the natural language prompts and / or expected reference images defined in the test cases to construct a second verification input data pair that conforms to the multimodal model input specification. In this process, the prompts can include not only simple labels but also detailed descriptions of the visual features of the target element (such as "a red circular warning icon with jagged edges"); while the reference images provide the standard visual form of the expected response of the target HMI application (such as a standard design draft or a screenshot of the correct version). For example, the system encodes the image to be verified and the expected reference image as visual token sequences, encodes the prompts as text token sequences, and embeds the corresponding visual token sequences and text token sequences into preset visual question answering (VQA), image-text matching, or multi-image comparison matching prompt templates. Thus, a clear "visual verification context" (covering the verification of single images and text, or the joint comparison verification of two images and text) is constructed for the large model, transforming the abstract automated testing requirements into image-text consistency judgment tasks that the model can understand.

[0069] In step S420, a multimodal large language model is used to perform visual semantic feature extraction processing on the image data to be verified in order to obtain the main semantic features used to characterize the target element to be verified, and cross-modal semantic alignment of the main semantic features with natural language prompts and / or reference images is performed to output a verification result containing the image verification sub-result.

[0070] Here, the image verification result includes icon semantic matching degree, which is used to indicate the consistency between the target element to be verified and at least one of the pattern elements, color elements, and detail elements and the natural language prompt words and / or reference images.

[0071] In some implementations, the multimodal large language model activates its visual encoder (such as the Vision Transformer module) to perform deep visual semantic feature extraction on the image data to be verified. Instead of focusing on the differences in the underlying RGB pixel matrix of the image, the model extracts the main semantic features representing the essential attributes of the target element through convolution or self-attention mechanisms. These features encompass the topological structure of the pattern (e.g., shape, outline), the distribution patterns of the color space (e.g., dominant hue, gradient colors), and key detail textures (e.g., internal symbols, line thickness). For the expected reference image, the model can also use its visual encoder to extract the corresponding reference visual features. Subsequently, the model uses a cross-modal alignment layer to calculate the matching distance or similarity probability between the aforementioned visual features and the text semantic vector of the natural language prompt and / or the reference visual feature vector. During this process, the model can flexibly perform alignment of pure visual features (i.e., comparing the actual icon with the expected reference icon) or perform multi-source alignment of visual features and image-text joint features (i.e., simultaneously comparing the actual screenshot with the expected icon and the expected text description). Ultimately, the output verification result includes a quantified icon semantic matching degree. This metric objectively reflects whether the currently displayed icon faithfully reproduces the design intent in terms of visual semantics, unaffected by screen resolution scaling or differences in rendering engine anti-aliasing algorithms. Furthermore, similar to the aforementioned voice verification scenario, the system can determine the final test result based on a comparison of this matching degree with a preset threshold. In other alternative embodiments, the multimodal large language model can also be configured to directly output a qualitative judgment conclusion on whether the current icon meets expectations after completing internal feature alignment (e.g., directly outputting a Boolean result of "test passed" or "mismatch").

[0072] This application's embodiments employ a combination of visual semantic feature extraction and cross-modal semantic alignment to achieve a paradigm shift in HMI icon testing, moving from "physical pixel comparison" to "cognitive semantic verification." Leveraging the high-dimensional representation capabilities of large models for visual objects, a flexible verification mechanism with tolerance for resolution and rendering details is established. This allows the testing system to accurately identify genuine business defects such as missing icons, color errors, or inconsistent display content, even when ignoring non-functional differences like screen size variations, UI scaling adjustments, or subtle pixel shifts. This significantly reduces the test case maintenance costs associated with multi-terminal adaptation.

[0073] Based on actual test and verification data from the inventors' research, compared to traditional pixel-level comparison schemes, the technical solution of this application reduces the maintenance cost of test cases by approximately 30%, significantly reducing the workload of updating the standard image library due to frequent UI iterations. Simultaneously, for icons with incomplete pixel matching but identical semantics (such as warning icons with different theme design styles), the recognition success rate is improved by over 50%, achieving a dual breakthrough in HMI automated testing efficiency and robustness while ensuring a 99% verification accuracy rate.

[0074] Figure 5 A schematic diagram of the architecture for icon verification based on a specific multimodal large model according to an embodiment of this application is shown.

[0075] like Figure 5 As shown, as a specific preferred implementation example, for the above icon verification process, the multimodal large language model can specifically choose a model that supports visual-language full-modal interaction (such as the Qwen3-VL-Plus model).

[0076] In this implementation scenario, the system employs a multi-image input validation mode. Specifically, the system will validate the "expected UI icon" (such as...) in the test cases. Figure 5 The standard steering wheel icon image shown) and the real-time captured "HMI actual pop-up icon" (such as...) Figure 5 The actual screenshot area shown is simultaneously fed into the model as visual input. In addition, the system guides the model to ignore background differences (such as the difference between the dark background in the actual screenshot and the standard image) through prompt text (such as "Please check if the two icons are the same, ignore the text and background color outside the icons").

[0077] This model leverages its high-dimensional image semantic understanding capabilities to directly perform the comparison and alignment of multi-source visual features within the model itself. For example... Figure 5 As shown in the output section, the model ultimately outputs structured data (such as JSON format) containing the judgment conclusion. In this example, despite the actual screenshot containing background noise, the model still accurately determines that the two are semantically consistent, outputting `is_passed: true`. Based on experimental verification data, the inventors found that the overall accuracy of automated icon verification using this specific model architecture can reach or exceed 99%, effectively overcoming the shortcomings of traditional pixel-level comparison methods that are sensitive to resolution and require manual background masking, achieving highly robust icon correctness verification.

[0078] It should be noted that, regarding the positioning method for the icon area, as an alternative to the UI control tree-based positioning strategy, the system can also integrate a lightweight object detection model (such as YOLOv8 or SSD model). By performing visual analysis on the full-screen screenshot, the bounding box coordinates of the pop-up or icon are automatically identified and regressed, thus enabling accurate cropping of the target area even when the UI control tree is unavailable or the hierarchy is obfuscated (such as in game engine rendering).

[0079] Regarding the implementation details of step S410 above, in some preferred examples of embodiments of this application, the natural language prompts include region focus instructions and / or non-target element ignore instructions. First, based on the coordinate position and size parameters of the target element to be verified in the current interface, relative position description information of the target element to be verified relative to the current interface is generated.

[0080] Here, to bridge the gap between the absolute coordinates of machine vision and the semantic cognition of large language models, the system first performs spatial semantic mapping processing. Specifically, based on the target element coordinates and size parameters obtained in the previous steps, the system establishes a spatial grid in conjunction with the overall screen resolution, transforming the dry mathematical coordinates (such as pixel x, y) into relative positional descriptions that conform to human natural language habits (such as "located in the upper left corner of the screen", "located on the right side of the navigation bar", or "centered"). Through spatial geometric calculations, precise physical positioning is abstracted into a vague but semantically clear directional description, thus providing an intuitive spatial index for multimodal large models, enabling them to quickly establish the correspondence between spatial coordinate systems and visual areas in complex interface screenshots.

[0081] Furthermore, relative position description information is embedded in natural language prompts to instruct the multimodal large language model to combine with region attention instructions to prioritize the main body region of the target element that matches the relative position description information in the reference image in the image data to be verified and / or the expected verification content through a visual attention mechanism.

[0082] Specifically, the system utilizes cue word engineering technology to dynamically fuse the aforementioned relative positional descriptions with region attention instructions, constructing a composite cue word with spatial guidance. When this cue word is input into a multimodal large language model, the model's internal cross-modal attention mechanism is activated. Based on the directional guidance in the cue word, the system dynamically adjusts the processing weight distribution of the visual encoder on the image data to be verified and / or the expected reference image. This allows the model to simulate human visual focusing behavior during the feature extraction stage, prioritizing computational resources and attention weights to the target element's main body region that matches the relative positional description. This ensures that the extracted visual semantic features are highly focused on the object being tested. Even when the expected verification content is a complex full-screen reference screenshot, the model can accurately lock onto and extract standard main features, rather than aimlessly scanning the entire image or the entire reference screen.

[0083] As an optional or alternative implementation, natural language prompts can also instruct the multimodal large language model to combine non-target element ignoring instructions to perform background feature suppression processing on other regions outside the main region of the target element in the image data to be verified and / or the reference image in the expected verification content, so as to reduce the interference of non-target elements on the image verification results.

[0084] Specifically, for scenarios where the background of the HMI test interface or the expected reference image is complex (such as containing dynamic map base maps, adjacent buttons, or irrelevant text labels), the natural language prompts further integrate instructions to ignore non-target elements. This instruction drives the model to perform a reverse feature suppression operation, that is, during semantic alignment, actively reducing or masking the feature weights of image regions located outside the main area of ​​the target element in the image to be verified or the reference image. By employing background feature suppression processing, the system effectively constructs a soft shielding layer for visual information, enabling the model to automatically filter out interference from surrounding non-target elements (such as adjacent text descriptions, status bar icons, or background color blocks), significantly improving the robustness and accuracy of icon verification in complex rendering environments and scenarios with impure reference image sources.

[0085] This application presents a visual attention guidance mechanism driven by natural language instructions. By converting physical coordinates into semantic relative position descriptions and combining bidirectional prompts with positive attention and / or negative inhibition, the system successfully and precisely constrains the generalized visual understanding capabilities of a multimodal large model to a specific test target. This not only effectively solves the "attention drift" or "background misreading" problems that easily occur when general large models process high-density information HMI interfaces, but also achieves high-precision semantic verification of minute UI elements in complex dynamic scenes at low cost through prompt optimization without modifying model parameters.

[0086] Figure 6 A flowchart illustrating an example of constructing a validation input data pair in a method according to an embodiment of this application is shown.

[0087] like Figure 6 As shown, in step S610, the event type of the target HMI interaction event is parsed, and a structured verification prompt template matching the event type is called from the preset library.

[0088] Here, the system first performs the initialization and parsing of the task context. Specifically, the system reads the metadata of the current test case, identifies the type of the target HMI interaction event (such as TTS voice broadcast or UI icon display), and accordingly retrieves and loads a matching structured validation prompt template from the pre-set prompt word template library.

[0089] The structured validation prompt template includes a validation instruction segment to instruct the multimodal large language model to perform validation tasks, an expected constraint segment to define the validation judgment criteria, and a data description segment to carry the actual data to be validated. Specifically, the template can be written using a structured markup language such as JSON or XML, and the input space is divided into three logically independent areas by reserving slots: the validation instruction segment defines the model's role as an HMI test expert and its task objectives; the expected constraint segment is reserved for filling in specific test pass criteria; and the data description segment is reserved for filling in the actual collected multimodal evidence. Thus, the template mechanism decouples the test logic from the test data, ensuring that the multimodal large language model always follows a unified reasoning framework when processing different test cases, avoiding output instability caused by differences in instruction expression.

[0090] In step S620, the expected verification content is converted into a text description that conforms to the model input specification and filled into the expected constraint segment to define the expected verification criteria for the target HMI interaction event.

[0091] Here, the system transforms human-readable test expectations into executable constraint logic for the model. Specifically, the system extracts the expected verification content from the test cases (such as the text string "Please slow down" or the description of the icon "red circular prohibition symbol") and dynamically populates it into the expected constraint section of the template. During this process, the system can also apply natural language processing techniques to refine the original expectations, transforming them into explicit judgment condition descriptions. For example, "expected to be A" is transformed into "Please strictly check whether the input data contains information with semantic value A." This establishes a clear truth benchmark for the large model, enabling it to have a specific comparison standard during subsequent inference, rather than engaging in aimless open-ended descriptions.

[0092] In step S630, the corresponding data to be verified is extracted from the data set to be verified, and the data to be verified is encapsulated in the modal format required by the data description segment to generate actual data description information.

[0093] Here, the system processes unstructured multimodal data to adapt it to the input interface of a large text model. Specifically, the system extracts the original file (such as a screenshot or audio recording) corresponding to the current event from the dataset to be verified, and serializes and encapsulates it according to the modal format required by the data description section in the template.

[0094] For example, for image data, the system might convert it to a Base64 encoded string or upload it to a temporary storage service and generate an access URL; for audio data, it might transcode it to a specific sampling rate format supported by the model. Subsequently, the system can use specific special tokens (such as...) ... or <audio>... <audio>< / audio> These encapsulated data are then packaged to generate actual data description information. This ensures that heterogeneous binary media data can be correctly parsed into visual or auditory vectors by the multimodal large model.

[0095] In step S640, the verification instruction segment, the filled expected constraint segment, and the actual data description information are sequentially concatenated to generate a combined verification prompt sequence, and the combined verification prompt sequence is determined as the verification input data pair.

[0096] Specifically, following the logical order of "role definition - task constraints - factual evidence," the system linearly concatenates the static verification instruction segment, the dynamically filled expected constraint segment, and the encapsulated actual data description information to generate a complete combined verification prompt sequence containing contextual information. This constitutes the final verification input data pair fed into the model. Thus, a complete inference closed-loop context is constructed, enabling the large model to possess all the necessary information—"what to do (instructions)," "what standard to follow (constraints)," and "what object (data)"—the moment it receives the sequence, allowing it to immediately initiate the inference generation process.

[0097] This application's embodiments employ a structured template-based prompt word engineering construction technology to address the issue of unstable instruction compliance in industrial-grade testing scenarios for general-purpose large language models. By reconstructing open natural language interaction into a standardized three-part input stream of "instructions + constraints + data," the illusion of large models is effectively suppressed, forcing the model to focus on specific verification logic. Furthermore, when adapting to new testing scenarios, only the template library needs updating without reconstructing the underlying code, enabling agile iteration of testing business logic and significantly improving the scalability of the automated testing system.

[0098] Figure 7 A flowchart illustrating an example of an automated test conclusion generated based on the verification result in the method according to an embodiment of this application is shown.

[0099] like Figure 7 As shown, in step S710, the verification results output by the multimodal large language model are parsed, and quantitative evaluation indicators for characterizing the degree of consistency are extracted.

[0100] Here, the system performs structured post-processing on the output response of the multimodal large language model. Specifically, since the raw output of the large model is usually a piece of natural language text (or thought chain) containing the analysis process, the testing system uses a regular expression extractor or a JSON parser to accurately locate and extract predefined quantitative evaluation indicators from the unstructured response stream. The quantitative evaluation indicators include at least one of the following: content matching degree, auditory reasonableness score, and icon semantic matching degree. These indicators are numerical representations of the model's HMI interaction quality, including "content matching degree" for measuring the accuracy of TTS content, "audio reasonableness score" for measuring the naturalness of speech, and "icon semantic matching degree" for measuring the correctness of UI display. This process transforms fuzzy semantic analysis into computer-calcifiable numerical values ​​(e.g., scores of 0-100 or confidence levels of 0.0-1.0).

[0101] In step S720, the quantitative evaluation index is compared with the preset pass threshold.

[0102] Here, the system executes the decision logic, comparing each extracted quantitative evaluation indicator with a preset pass threshold. These thresholds are pre-configured based on business quality standards (e.g., setting the content matching threshold to 0.95 and the listening experience reasonableness score threshold to 0.8), representing the minimum quality level allowed for HMI product release. Thus, by mapping the probabilistic output of the large model to deterministic decision logic, the uncertainty of subjective human judgment is eliminated, ensuring the consistency of testing standards.

[0103] In step S731, if all indicators meet the passing threshold, a test pass conclusion is generated.

[0104] Specifically, if all indicators are higher than or equal to the corresponding pass threshold, the system determines that the current interaction event meets expectations, generates a test pass conclusion, and updates the test case status.

[0105] In step S733, if at least one indicator fails to meet the pass threshold, a test failure conclusion is generated, and the failure attribution type is determined based on the type of indicator that was not met.

[0106] Specifically, if any indicator falls below a preset threshold, the system not only generates a test failure conclusion, but more importantly, it can automatically infer the failure attribution type based on which specific indicator failed. The failure attribution type includes at least one of the following: TTS broadcast content error, TTS auditory abnormality, and icon semantic mismatch. For example, if the "audio reasonableness score" is too low but the "content matching degree" is normal, the system automatically classifies the defect as "TTS auditory abnormality (tone / sentence segmentation error)"; if the "icon semantic matching degree" is too low, it is classified as "icon semantic mismatch". This automatic attribution mechanism effectively reduces the workload of manually reviewing logs, achieving an automated testing leap from "discovering problems" to "locating problems".

[0107] In step S740, an HMI test defect record corresponding to the test failure conclusion is generated.

[0108] Here, the system solidifies the above analysis results into a traceable HMI test defect record. Specifically, the HMI test defect record includes: the failure attribution type, the corresponding evaluation index value, a description of the specific defect content generated by the multimodal large language model based on cross-modal semantic comparison, and the data identification information to be verified corresponding to the evaluation index value. This record not only includes the automatically determined failure attribution type (e.g., "inaccurate TTS broadcast" or "icon semantic mismatch") and the specific measured values ​​of the evaluation index (as quantitative evidence), but also includes detailed comparisons between the expected content and the actual output (e.g., comparison of the literal differences between the expected speech text and the actual broadcast speech), as well as detailed error details (e.g., extra words, missing words, different word order, or mismatched colors). In specific implementation scenarios, the multimodal large language model will ultimately generate and output a test report or prompt information conforming to a preset fixed template (e.g., standard JSON data structure), which includes formatted core fields such as matching status (e.g., is_passed), defect type (e.g., fail_type), and specific defect reason (e.g., fail_reason). Furthermore, the defect record is automatically associated with the identification information of the data to be verified collected during this test (such as the storage path of the abnormal audio file or the snapshot ID of the abnormal interface). This achieves "full-element retention" of test defects. When developers view defect reports, they can directly click to play the abnormal audio or view screenshots, and by combining the specific defect comparison reasons given by the model, they can quickly reproduce and fix the problem, greatly improving the efficiency of defect repair workflow.

[0109] In this embodiment, the problem of standardizing the output results of large models in automated testing applications is solved by using a multi-dimensional index quantification extraction and automatic attribution judgment mechanism and a structured output template. A closed-loop decision logic from "semantic reasoning" to "numerical judgment" to "defect diagnosis" is constructed. This not only ensures the objectivity and repeatability of HMI automated testing conclusions, but also improves the traditional black-box testing, which can only provide "pass / fail" feedback, to provide diagnostic analysis that can provide feedback on "why it failed (e.g., the pronunciation is correct but the tone is wrong, or there are extra or missing words)" through automatic fault classification technology based on index type and detailed error details generated by the large model. This significantly shortens the defect location and repair cycle in HMI R&D iteration.

[0110] As an alternative or supplementary embodiment to the aforementioned quantitative judgment method, the multimodal large language model can also be configured to directly output judgment conclusions, achieving end-to-end direct judgment based on the model. Specifically, when constructing the verification prompt sequence, the system can also explicitly inject binary judgment rules into the instruction segment (e.g., "Please judge whether the audio content is completely correct, directly return True or False"). In this case, after receiving multimodal data, the model uses its internal knowledge base and logical reasoning capabilities to directly perform a comprehensive judgment on consistency, instead of outputting intermediate state score indicators. Thus, the process of generating automated test conclusions is further simplified, and the system directly parses the structured fields output by the model, directly mapping their Boolean values ​​to the conclusion of test pass or failure.

[0111] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0112] In some embodiments, this application also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform any of the above-described HMI application testing methods.

[0113] In some embodiments, this application also provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute a test method for an HMI application.

[0114] The apparatus described in the embodiments of this application can be used to execute the testing method for the HMI application of the embodiments of this application, and accordingly achieve the technical effects achieved by the testing method for the HMI application of the embodiments of this application, which will not be elaborated further here. In the embodiments of this application, the relevant functional modules can be implemented by a hardware processor.

[0115] Figure 8 This is a schematic diagram of the hardware structure of an electronic device for executing a test method for an HMI application, as provided in another embodiment of this application. Figure 8 As shown, the device includes: One or more processors 810 and memory 820, Figure 8 Take the 810 processor as an example.

[0116] The device for executing test methods for HMI applications may also include an input device 830 and an output device 840.

[0117] The processor 810, memory 820, input device 830, and output device 840 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0118] The memory 820, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the HMI application testing method in the embodiments of this application. The processor 810 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 820, thereby implementing the HMI application testing method of the above-described method embodiments.

[0119] The memory 820 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the device. Furthermore, the memory 820 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 820 may optionally include memory remotely located relative to the processor 810, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0120] Input device 830 can receive input digital or character information and generate signals related to user settings and function control of the device. Output device 840 may include display devices such as a display screen.

[0121] The one or more modules are stored in the memory 820, and when executed by the one or more processors 810, they execute the test method of the HMI application in any of the above method embodiments.

[0122] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0123] The electronic devices in this application embodiments exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0124] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0125] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players (such as iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0126] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0127] (5) Other electronic devices with data interaction functions.

[0128] In some embodiments, this application also provides a mobile platform on which the computer device described in any embodiment of this application is installed. The mobile platform includes, but is not limited to, mobile devices such as vehicles, tracked robots, bipedal robots, quadrupedal robots, smart boats, and smart aircraft, wherein the vehicle can be a passenger car, pickup truck, van, or truck. It should be noted that the above are merely examples, and this application does not limit the specific form of the mobile platform.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.< / audio>

Claims

1. A testing method for an HMI application, comprising: Obtain a set of test cases for the target HMI application. Each test case in the set includes at least: triggering information for triggering the target HMI interaction event, and expected verification content corresponding to the target HMI interaction event. Based on the triggering information, an automated triggering operation is performed on the target HMI application to cause the target HMI application to output a verification output corresponding to the target HMI interaction event; wherein, the verification output includes output information of at least one modality; Collect the output to be verified to obtain the set of data to be verified corresponding to the output to be verified; At least one verification input data pair is constructed based on the expected verification content and the set of data to be verified. The verification input data pair includes: expected verification data for the same target HMI interaction event and the data to be verified corresponding to the expected verification data. The at least one verification input data pair is input into a multimodal large language model to obtain a verification result that characterizes the consistency between the output to be verified and the expected verification content, and an automated test conclusion for the corresponding test case is generated based on the verification result.

2. The method of claim 1, wherein, When the output to be verified includes audio output information, the step of acquiring the output to be verified to obtain a set of data to be verified corresponding to the output to be verified includes: Monitor the audio file generation path of the target HMI application used to generate the audio file corresponding to the audio output information; If a new audio file is detected to be generated under the audio file generation path, the new audio file will be identified as the audio data to be verified in the data set to be verified. The audio data to be verified is the raw digital audio signal generated by the target HMI application.

3. The method of claim 2, wherein, The expected verification content includes expected speech text corresponding to the audio data to be verified; the step of inputting the at least one verification input data pair into a multimodal large language model to obtain a verification result characterizing the consistency between the output to be verified and the expected verification content includes: Based on the audio data to be verified and the expected speech text, a first verification input data pair is constructed, and the first verification input data pair is input into the multimodal large language model; The multimodal large language model is used to perform acoustic feature extraction processing on the audio data to be verified to obtain audio features for characterizing the broadcast content. The audio features include at least one of the following: phoneme-related features, stress-related features, and pause rhythm-related features. The multimodal large language model is used to perform cross-modal semantic alignment between the audio features and the expected speech text to output a verification result containing audio verification sub-results. The audio verification sub-results include the broadcast content matching degree and the listening reasonableness score. The broadcast content matching degree is used to indicate the accuracy of the audio data to be verified in terms of text semantic content and pronunciation of polyphonic characters. The listening reasonableness score is used to indicate the naturalness of the audio data to be verified in terms of sentence segmentation logic and speech intonation.

4. The method of claim 1, wherein, When the output to be verified includes interface image information, the step of collecting the output to be verified to obtain a set of data to be verified corresponding to the output to be verified includes: The control layout structure of the current interface of the target HMI application is analyzed, and the coordinate position and size parameters of the target element to be verified in the current interface are determined according to the control layout structure. The screen capture command is invoked to capture a full-screen image of the current interface, and the region of interest containing the target element to be verified is cropped from the full-screen image based on the coordinate position and size parameters. The region of interest image is determined as the image data to be verified in the dataset to be verified.

5. The method of claim 4, wherein, The expected verification content includes natural language prompts describing the visual features of the target element to be verified, and / or a reference image of the expected response of the target HMI application; The step of inputting the at least one verification input data pair into the multimodal large language model includes: Based on the image data to be verified, a second verification input data pair is constructed with the natural language prompt words and / or the reference image, and the second verification input data pair is input into the multimodal large language model; The multimodal large language model is used to perform visual semantic feature extraction processing on the image data to be verified to obtain the main semantic features used to characterize the target element to be verified, and the main semantic features are aligned with the natural language prompt words and / or the reference image to output a verification result containing the image verification sub-result. The image verification result includes icon semantic matching degree, which is used to indicate the consistency between the target element to be verified and the natural language prompt word and / or the reference image in at least one of the pattern elements, color elements and detail elements.

6. The method of claim 5, wherein, The natural language prompts include region attention instructions and / or non-target element ignore instructions; The step of inputting the second verification input data pair into the multimodal large language model includes: Based on the coordinate position and size parameters of the target element to be verified in the current interface, generate relative position description information of the target element to be verified relative to the current interface; The relative position description information is embedded in the natural language prompt word to instruct the multimodal large language model, in conjunction with the region attention instruction, to preferentially focus on the main region of the target element that matches the relative position description information in the image data to be verified through a visual attention mechanism, and / or The multimodal large language model is instructed to combine the non-target element ignoring instruction to perform background feature suppression processing on other regions located outside the main region of the target element in the image data to be verified and / or the expected verification content, so as to reduce the interference of non-target elements on the image verification sub-results.

7. The method of claim 1, wherein, The construction of at least one verification input data pair based on the expected verification content and the data set to be verified includes: The event type of the target HMI interaction event is parsed, and a structured verification prompt template matching the event type is called from the pre-built library; the structured verification prompt template includes a verification instruction segment for instructing the multimodal large language model to perform verification tasks, an expected constraint segment for limiting the verification judgment criteria, and a data description segment for carrying the actual data to be verified; The expected verification content is converted into a text description that conforms to the model input specification and filled into the expected constraint segment to define the expected verification standard of the target HMI interaction event; Extract the corresponding data to be verified from the set of data to be verified, and encapsulate the data to be verified according to the modal format required by the data description segment to generate actual data description information; The verification instruction segment, the filled expected constraint segment, and the actual data description information are sequentially concatenated to generate a combined verification prompt sequence, and the combined verification prompt sequence is determined as the verification input data pair.

8. The method of claim 1, wherein, The automated test conclusions generated based on the verification results for the corresponding test cases include: The verification results output by the multimodal large language model are analyzed, and quantitative evaluation indicators for characterizing the degree of consistency are extracted; the quantitative evaluation indicators include at least one of the following: broadcast content matching degree, listening perception rationality score, and icon semantic matching degree; The quantitative evaluation indicators are compared with preset pass thresholds; if all indicators meet the pass thresholds, a test pass conclusion is generated. If at least one indicator fails to meet the pass threshold, a test failure conclusion is generated, and the failure attribution type is determined based on the type of indicator that was not met; the failure attribution type includes at least one of the following: TTS broadcast content error, TTS listening abnormality, and icon semantic mismatch; Generate an HMI test defect record corresponding to the test failure conclusion. The HMI test defect record includes: the failure attribution type, the evaluation index value corresponding to the failure attribution type, and the data identification information to be verified corresponding to the evaluation index value.

9. A storage medium, wherein, The storage medium stores one or more programs including execution instructions that can be read and executed by an electronic device to perform the steps of the method according to any one of claims 1-8.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein, The processor executes the computer program to implement the steps of the method according to any one of claims 1-8.

Images