A vehicle-mounted assistant test method, system, device and storage medium

Abstract

Embodiments of the present application provide a kind of vehicle-mounted assistant test method, system, equipment and storage medium, it is related to intelligent assistant field, method includes: obtaining including dialogue instruction and target result to be tested data, target result includes target refusal state, target autonomous agent mode, target call tool and target reply content;Dialogue instruction is sent to vehicle-mounted assistant, vehicle-mounted assistant calls preset simulation tool and executes dialogue instruction and returns reply content, generates test record;Each dialogue instruction corresponds to the dialogue interaction event with event identifier;According to event identifier, evaluation data is parsed from test record, evaluation data includes actual refusal state, actual autonomous agent mode, actual call tool and reply content;Evaluation data is checked based on target result with rule and semantics, and evaluation determination result is obtained.Such as event identifier through test record and data analysis, with the check of rule and semantics, evaluation data is judged, and the test closed loop of data connection is realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent assistant technology, and more specifically, to a method, system, device, and storage medium for testing in-vehicle assistants. Background Technology

[0002] SystemAgent is an AI system capable of receiving user commands, autonomously planning, and invoking tools to complete specified tasks. Taking an in-vehicle scenario as an example, after a user says "Navigate to Shanghai's tallest building," SystemAgent automatically invokes a map navigation tool to obtain the destination's location and route information, and then provides the results to the user via voice. During SystemAgent's development, each model version iteration undergoes extensive testing to evaluate its tool invocation recall and success rate, the rationality of task orchestration, and the completeness of the end-to-end link, thus providing data for model optimization.

[0003] Currently, SystemAgent testing is predominantly done manually. Testers manually input commands one by one and monitor the execution process, then verify the tool's calls and responses against expectations based on their subjective experience. As SystemAgent's functionality expands, the number of test cases grows from hundreds to thousands or even tens of thousands, highlighting the increasing efficiency bottleneck of manual testing: the execution and verification of a single test is time-consuming, and the cycle of full regression testing cannot keep up with the rapid iteration of the model. Furthermore, manual verification lacks unified quantitative standards; different testers may have different judgments on the same test result, resulting in a lack of objectivity and comparability, and failing to provide reliable data support for model optimization.

[0004] To address these issues, the industry has begun to introduce partial automation methods, such as using automated scripts to initiate test requests in batches or collecting runtime logs through log collection tools. However, these partial automation methods are independent of each other and lack coordination, making it difficult to form a data-driven connection between the various stages of test case generation, test execution, data collection and analysis, and result evaluation and output. In other words, they cannot form a closed-loop test system where information flows seamlessly through all stages. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a vehicle assistant testing method, system, device and storage medium to solve the problem that the local automation methods are independent of each other and lack connection, making it difficult to form a data-driven connection between the various links of test case generation, test execution, data collection and analysis and result evaluation output, that is, it is impossible to form a test closed loop with information flow through each link.

[0006] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a vehicle assistant testing method, the method comprising: acquiring test data, the test data including at least one dialogue instruction and a target result; the target result including a target rejection state, a target autonomous agent mode, a target invoked tool, and a target response content; sending the dialogue instruction to the vehicle assistant, so that the vehicle assistant executes the dialogue instruction and returns the corresponding response content, generating a test record; wherein, each dialogue instruction corresponds to a dialogue interaction event with an event identifier, and when the vehicle assistant executes the dialogue instruction, it invokes a preset simulation tool and returns the corresponding response content; parsing evaluation data for each dialogue instruction from the test record according to the event identifier; the evaluation data including the actual rejection state, the actual autonomous agent mode, the actual invoked tool, and the response content; and performing rule verification and semantic verification on the evaluation data based on the target result to obtain an evaluation judgment result.

[0007] In this embodiment, test data consisting of dialogue commands and target results is acquired. After the dialogue command is sent to the in-vehicle assistant, the assistant executes the command and returns a corresponding response based on a preset simulation tool. This serves as a dialogue interaction event with a corresponding event identifier. Multiple dialogue interaction events constitute a test record. Evaluation data for each event is then extracted from the test record based on the event identifier. This facilitates rule-based and semantic verification of the evaluation data based on the target result, yielding test results from both multi-dimensional and semantic verification perspectives. Thus, by pre-pairing dialogue commands and target results throughout test execution and verification, connecting test records and data parsing through event identifiers, and using rule-based and semantic verification to complementarily evaluate the same evaluation data, a data-driven connection is achieved between dialogue command input, test record output, data parsing, and result evaluation, forming a closed-loop test system where information flow permeates all stages.

[0008] In some embodiments, the preset simulation tool is configured by: obtaining the input parameter data structure and output parameter data structure of each tool in a preset database; constructing the preset simulation tool according to the input parameter data structure and the output parameter data structure, wherein the preset simulation tool is used to generate simulation return results according to the field format specified by the output parameter data structure.

[0009] This setup allows for the construction of simulation tools based on the input and output parameter data structures of each tool. The simulation tools can generate simulated return results that are consistent with the return result format of the real tools, according to the field format specified in the output parameter data structure. This ensures that the response received by the vehicle assistant when calling tools during the processing of dialogue commands is no different from that when calling real tools, thus completing the entire test execution process without relying on real tools.

[0010] In some embodiments, parsing the evaluation data of each dialogue instruction from the test record based on the event identifier includes: extracting the operation log and terminal log of the vehicle assistant from the test record; associating the operation log and the terminal log according to the event identifier to obtain the operation log and the terminal log of each dialogue instruction; and extracting the fields corresponding to the target result from the operation log and the terminal log according to a preset parsing rule to obtain the evaluation data of each dialogue instruction.

[0011] This setup aligns the runtime logs and client-side logs through event identifiers, ensuring that the data from the server-side processing and terminal reception processes of each dialogue interaction event accurately correspond. Then, according to preset parsing rules, the actual fields corresponding to each dimension of the target result are extracted, thereby transforming the scattered unstructured logs into structured evaluation data, providing directly comparable data for subsequent rule verification and semantic verification.

[0012] In some embodiments, the step of performing rule verification and semantic verification on the evaluation data based on the target result to obtain an evaluation judgment result includes: comparing the actual rejection state, actual autonomous agent mode, actual tool invocation, and response content in the evaluation data with the target rejection state, target autonomous agent mode, target tool invocation, and target response content in the target result to obtain multi-dimensional verification results; performing semantic verification on the evaluation data and the target result to obtain a judgment result on whether the evaluation data passes the test and the corresponding reason; and obtaining the evaluation judgment result based on the multi-dimensional verification results, the judgment result on whether the evaluation data passes the test, and the corresponding reason.

[0013] This setup, by comparing each actual field in the evaluation data with the corresponding target field in the target result one by one, achieves deterministic determination of four dimensions: rejection status, autonomous agent mode, tool call, and response content. It covers all objective dimensions that can be accurately compared, making the determination result certain and unambiguous, without relying on subjective judgment based on human experience.

[0014] In some embodiments, performing semantic verification on the evaluation data and the target result to obtain a determination result of whether the evaluation data passes the test and the corresponding reason includes: submitting the actual response content, the actual calling tool, and the target response content in the evaluation data to a preset evaluation model to obtain a semantic verification score and the corresponding reason; wherein, if the semantic verification score is greater than or equal to a preset score, it is determined that the evaluation data passes the test; if the semantic verification score is less than a preset score, it is determined that the evaluation data fails the test.

[0015] This setup allows the evaluation model to quantitatively assess whether the actual response content meets the requirements of the target response content at the semantic level and to provide explanations. This compensates for the shortcomings of rule validation, which may produce misjudgments by relying solely on keyword matching in complex semantic scenarios. Rule validation and semantic validation complement each other, covering all dimensions of evaluation needs from precise comparison to semantic understanding.

[0016] In some embodiments, the step of submitting the actual response content, the actual invoked tool, and the target response content in the evaluation data to a preset evaluation model to obtain a semantic verification score and the corresponding reason includes: obtaining targeted inspection content; adding the targeted inspection content to the target response content to obtain targeted response content; and submitting the actual response content, the actual invoked tool, and the targeted response content in the evaluation data to a preset evaluation model to obtain a semantic verification score and the corresponding reason.

[0017] This setup, by adding targeted checks to the target response content, allows the evaluation model to prioritize checks on specific contextual issues during the evaluation process. It enables more precise targeted evaluation for specific test cases with known error tendencies, avoiding omissions by general decision rules when dealing with special boundary cases.

[0018] In some embodiments, sending the dialogue command to the vehicle assistant includes: synthesizing the dialogue command into audio; sending the audio as a file to the vehicle assistant; or playing the audio to the terminal where the vehicle assistant is located.

[0019] This setup enables driver testing on the cloud path by synthesizing audio files and driver testing on the terminal path by using real audio recordings, allowing test execution to be flexibly adapted to both cloud-based pure software environments and terminal physical device environments.

[0020] Secondly, embodiments of the present invention provide an in-vehicle assistant testing system, the system comprising: an acquisition module, configured to acquire test data, the test data including at least one dialogue instruction and a target result; the target result including a target rejection state, a target autonomous agent mode, a target invoked tool, and a target reply content; a parsing module, configured to send the dialogue instruction to the in-vehicle assistant, so that the in-vehicle assistant executes the dialogue instruction and returns the corresponding reply content, generating test records; wherein, each dialogue instruction corresponds to a dialogue interaction event with an event identifier, and when the in-vehicle assistant executes the dialogue instruction, it invokes a preset simulation tool and returns the corresponding reply content; evaluation data for each dialogue instruction is parsed from the test records according to the event identifier; the evaluation data includes the actual rejection state, the actual autonomous agent mode, the actual invoked tool, and the actual reply content; and a testing module, configured to perform rule verification and semantic verification on the evaluation data based on the target, and obtain an evaluation judgment result.

[0021] Thirdly, embodiments of the present invention provide an electronic device, including a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement the vehicle assistant testing method as described in the first aspect.

[0022] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the vehicle assistant testing method as described in the first aspect.

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart of a vehicle assistant testing method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the functional modules of the vehicle assistant testing system provided in an embodiment of the present invention; Figure 3 A block diagram of an electronic device provided in an embodiment of the present invention.

[0026] Icons: 1000 - Vehicle Assistant Test System; 1100 - Acquisition Module; 1200 - Parsing Module; 1300 - Test Module; 2000 - Electronic Device; 2100 - Processor; 2200 - Memory; 2300 - Bus; 2400 - Communication Interface. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0028] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0029] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0030] As described in the background section, current testing of SystemAgent is predominantly done manually. Testers manually input commands one by one and monitor the execution process, then verify the tool's calls and responses against expectations based on subjective experience. With the expansion of SystemAgent's functionality, the scale of test cases has grown from hundreds to thousands or even tens of thousands, highlighting the increasing efficiency bottleneck of manual testing: the execution and verification of a single test is time-consuming, and the cycle of full regression testing cannot keep up with the rapid iteration of the model. Furthermore, manual verification lacks unified quantitative standards; different testers may have different judgments on the same test result, resulting in a lack of objectivity and comparability, and failing to provide reliable data support for model optimization.

[0031] To address these issues, the industry has begun to introduce partial automation methods, such as using automated scripts to initiate test requests in batches or collecting runtime logs through log collection tools. However, these partial automation methods are independent of each other and lack coordination, making it difficult to form a data-driven connection between the various stages of test case generation, test execution, data collection and analysis, and result evaluation and output. In other words, they cannot form a closed-loop test system where information flows seamlessly through all stages.

[0032] To address this, embodiments of the present invention provide a vehicle assistant testing method. This method pre-pairs dialogue commands with target results and integrates this pairing throughout test execution and verification; uses event identifiers to connect test records and data parsing; and employs a dual-dimensional approach of rule verification and semantic verification to complementarily evaluate the same test data. This achieves data-driven connectivity between dialogue command input, test record output, data parsing, and result evaluation, thus forming a closed-loop test system where information flow permeates all stages. (See also...) Figure 1 , Figure 1 This is a flowchart of a vehicle assistant testing method provided by an embodiment of the present invention. The vehicle assistant testing method includes steps S100 to S500: S100. Obtain the test data, which includes at least one dialogue command and the target result; the target result includes the target rejection state, the target autonomous agent mode, the target calling the tool, and the target reply content.

[0033] In this embodiment, the test data is a set of standardized test cases pre-generated before test execution. Each piece of test data consists of a dialogue command and a target result corresponding to that dialogue command. Each test scenario has corresponding test data, such as navigation, weather, music, ordering food, purchasing tickets, and booking hotels. The dialogue command is the voice interaction input that the in-vehicle assistant needs to process, such as "Navigate to the tallest building in Shanghai." The target result is a pre-defined standard answer for the dialogue command. Specifically, the target rejection status indicates whether the dialogue command is rejected by the in-vehicle assistant; the target autonomous agent mode indicates whether the in-vehicle assistant enters autonomous agent mode to process the dialogue command; the target invoked tool indicates the name of the tool invoked by the in-vehicle assistant for the dialogue command; and the target response content indicates the semantic requirements that the in-vehicle assistant's response should meet. These four dimensions constitute the standard answer for the subsequent verification phase of this test. Each dialogue command has its own independent target result; different dialogue commands correspond to different target results. In practice, more than four dimensions can be preset, or one or more of the above four dimensions can be selected, depending on the application scenario.

[0034] For example, the test data is generated as follows: a batch of gold standard test cases are pre-defined based on product definition and scenario combination, and then the test case processing agent expands the test scenarios of the gold standard test cases and converts them into a standardized table format that the test framework can directly read, which is the test data; the test case processing agent also supports receiving scenario descriptions in natural language and automatically converting them into test data containing dialogue instructions and target results.

[0035] S200: Send the dialogue command to the vehicle assistant so that the vehicle assistant can execute the dialogue command and return the corresponding response content, and generate a test record; wherein, each dialogue command corresponds to a dialogue interaction event with an event identifier. When the vehicle assistant executes the dialogue command, it calls the preset simulation tool and returns the corresponding response content.

[0036] In this embodiment, the dialogue command is sent to the vehicle assistant in audio form. After receiving the audio, the vehicle assistant performs speech recognition to obtain the command text. After analyzing the intent of the dialogue command, it autonomously determines which tool needs to be invoked. When the vehicle assistant issues a tool invoke request, a preset simulated tool intercepts the invoke request and generates a simulated return result based on the output parameter data structure of the tool, and returns it to the vehicle assistant. After receiving the simulated return result, the vehicle assistant organizes a natural language reply and broadcasts the corresponding reply content, thus completing one dialogue interaction event.

[0037] Furthermore, during the processing of dialogue commands, the in-vehicle assistant originally needed to invoke real tool services to complete the command. For example, a navigation command would invoke a map navigation tool, and a weather query command would invoke a weather query tool. The simulated tool receives the call request at the same interface location where these real tools are invoked, but it does not execute the real tool operations. Instead, it generates a simulated return result according to the field format specified by the pre-configured output parameter data structure. For example, for the map navigation tool, the simulated return result is JSON data containing the destination name and distance, which is then returned to the in-vehicle assistant. The simulated return result received by the in-vehicle assistant is completely consistent with the result returned by the real tool in terms of field format. The in-vehicle assistant cannot distinguish whether the invoked object is a real tool or a simulated tool, so it continues to organize the natural language response content based on the simulated return result. It should be noted that each dialogue command may invoke more than one preset simulated tool. For example, if a dialogue command mentions navigation, asks about the weather, and inquires about nearby restaurants, it will invoke the simulated tools corresponding to navigation, weather, and nearby restaurants.

[0038] Furthermore, the event identifier is a unique identifier assigned to each dialogue interaction event. Multiple rounds of dialogue within the same test each have their own independent event identifier. For example, if a user first says, "Navigate to the tallest building in Shanghai," this is one round of dialogue interaction event, assigned one event identifier. After this dialogue interaction ends, the user says, "What are some good restaurants nearby?" This is a second round of dialogue interaction event, assigned another event identifier. Event identifiers are used throughout all stages of test record analysis and result evaluation, and are crucial for data alignment between different stages. If, during a single dialogue interaction, the user asks about the location, the weather, and nearby restaurants, this corresponds to two dialogue interaction events, with each dialogue command corresponding to a dialogue interaction event with an event identifier.

[0039] Furthermore, the test record is a collection of process data accumulated from multiple dialogue interaction events. Each time a dialogue interaction event is triggered, the corresponding dialogue command content, the parameters passed when calling the tool, the simulation result returned by the simulation tool, the execution status of the vehicle assistant, the final reply text of the vehicle assistant, and the processing time are recorded synchronously. This information is then associated with the event identifier of this event and recorded in the test record.

[0040] S300: Parse the evaluation data of each dialogue instruction from the test record based on the event identifier; the evaluation data includes the actual rejection state, the actual autonomous agent mode, the actual tool call and the response content.

[0041] In this embodiment, the parsing process from test records to evaluation data consists of three steps. First, the system retrieves the in-vehicle assistant server's operation logs from the ELK log collection framework according to the time range and the vehicle's unique identifier. Simultaneously, it obtains the terminal-side Android logs from the terminal device. The operation logs record all key information during the in-vehicle assistant server's processing of dialogue commands, while the terminal-side logs record information about the terminal device's audio reception and response processes. Second, the retrieved operation logs and terminal-side logs are associated with each other according to event identifiers. Operation log fragments and terminal-side log fragments with the same event identifier constitute the complete data for that dialogue interaction event. Third, the system extracts the actual fields corresponding to the four dimensions of the target result from the associated logs according to preset parsing rules. For example, the actual rejection status is extracted from the `trace_z.final_reject` field of the operation log; the existence of the `trace_z.agenticAndTool` field determines the actual autonomous agent mode; the name of the actual called tool is parsed from `trace_z.agenticAndTool`; and the actual response content is extracted from `trace_z.final_result.output`. The extracted fields are then aligned according to event identifiers to obtain the final evaluation data.

[0042] It can be understood that the evaluation data is a mirror image of the target result; the four dimensions in the target result (target rejection state, target autonomous proxy mode, target tool invocation, and target response content) correspond to the actual rejection state, actual autonomous proxy mode, actual tool invocation, and actual response content in the evaluation data.

[0043] In some embodiments, due to a delay of approximately one and a half minutes between log generation and their retrieval by the ELK system, the collection process waits for the logs to be written to the database after the test execution before fetching them. If logs corresponding to certain event identifiers are missing, the time range is automatically expanded and the fetching is retried to ensure the integrity of the data collection. Log parsing can automatically optimize parsing rules according to changes in the test scenario to improve parsing efficiency and accuracy.

[0044] S400. Based on the target results, the evaluation data is subjected to rule verification and semantic verification to obtain the evaluation judgment results.

[0045] In this embodiment, the evaluation result represents whether the dialogue instruction passes the test and the corresponding reason. Rule verification is used to compare the actual rejection state, actual autonomous agent mode, actual tool invocation, and actual response content in the evaluation data with the target rejection state, target autonomous agent mode, target tool invocation, and target response content in the target result, respectively. Semantic verification is used to perform semantic-level evaluation of the actual response content based on the target response content.

[0046] S500 generates an evaluation report based on the evaluation results and synchronizes the evaluation report to the data platform for maintenance.

[0047] In this embodiment, the testing platform generates an evaluation report based on the evaluation results of each dialogue interaction event. The evaluation report may include test case identifiers, dialogue instructions, event identifiers, actual rejection status, actual autonomous agent mode, actual tool invocation, actual response content, verification conclusions for each rule verification dimension, semantic verification score, overall evaluation results, and reasons for failure.

[0048] The testing platform can summarize the evaluation results according to the test task, test scenario, vehicle assistant version or test time, and statistically analyze the pass rate, rejection status consistency rate, autonomous agent mode consistency rate, tool call matching rate and semantic verification pass rate, and generate evaluation reports in MD, HTML, JSON or Excel format.

[0049] After the evaluation report is generated, the evaluation report and its corresponding structured evaluation data are synchronized to the data platform for maintenance. The data platform archives and associates the evaluation reports according to the test batch, vehicle assistant version, and test scenario to support the comparison of test results from different batches or versions, the analysis of indicator trends, and the tracing and location of failed test cases.

[0050] For example, the evaluation report generated from the evaluation results obtained from rule validation and semantic validation clearly presents the following core evaluation indicators and their changing trends: The positive example rejection rate refers to the proportion of dialogue commands that are both targeted for rejection and actually rejected out of the total number of dialogue commands that are targeted for rejection. Rejection means that the in-vehicle assistant discards voice content that is not suitable for it to respond to. Target rejection and actual rejection means that a dialogue command marked as targeted for rejection in the target results is also rejected after the test execution. For example, if out of one hundred dialogue commands that are targeted for rejection, ninety-five are actually rejected by the in-vehicle assistant, then the positive example rejection rate is ninety-five percent.

[0051] Negative example recall refers to the proportion of dialogue commands that are not rejected by the target and are actually not rejected, out of the total number of dialogue commands that are not rejected by the target. "Not rejected by the target and actually not rejected" means that a dialogue command marked as not rejected by the target in the target results is actually not rejected after test execution. For example, if out of one hundred dialogue commands that are not rejected by the target, and ninety-two are not actually rejected by the in-vehicle assistant, then the negative example recall is ninety-two percent.

[0052] Tool recall refers to the percentage of dialogue commands that actually invoked a specific tool, out of the total number of dialogue commands that invoked that tool. "Invoked a specific tool" means that, among a set of dialogue commands marked as invoking that tool in the target results, the actual tool invocation record in the evaluation data includes that tool. For example, if out of fifty dialogue commands that invoked a map navigation tool, forty-eight actually invoked the map navigation tool, then the tool recall for that tool is 96%. The evaluation report calculates the tool recall separately for each tool.

[0053] The success rate of switching to autonomous agent mode refers to the proportion of dialogue commands that actually enter autonomous agent mode out of the total number of dialogue commands that actually enter autonomous agent mode. Autonomous agent mode refers to the operating mode where the vehicle assistant switches from the main processing flow to a sub-processing flow more suitable for multi-tool orchestration when multiple tools need to be invoked simultaneously to complete a complex task. "Target enters autonomous agent mode and actually enters autonomous agent mode" means that in a set of dialogue commands where the target autonomous agent mode is marked as "yes" for the target result, the actual autonomous agent mode of the evaluation data is also "yes". For simple commands that only require invoking a single tool, the target does not enter autonomous agent mode; for complex commands that require multiple tools to work together, the target enters autonomous agent mode.

[0054] The end-to-end success rate refers to the percentage of tests conducted without any abnormalities occurring during the entire process from the input of dialogue commands in audio form to the output of audio played by the in-vehicle assistant.

[0055] After being normalized, the aforementioned core evaluation metrics support comparative analysis of test results from different batches and versions under the same dimension, and the evaluation reports are synchronized to the data platform for maintenance.

[0056] In some embodiments, for step S100, one possible implementation of the present invention is that step S100 includes sub-steps S110~S120: S110. Combine dialogue commands into audio.

[0057] In this embodiment, the dialogue commands are user input in text form, which needs to be synthesized into audio before being sent to the in-vehicle assistant. Audio synthesis supports multiple mainstream audio formats such as WAV, Speex, and Opus, and can also directly use pre-recorded audio files as input to meet the needs of different voice-related testing scenarios.

[0058] S120. Send the audio as a file to the vehicle assistant; or, play the audio to the terminal where the vehicle assistant is located.

[0059] In this embodiment, the path of directly sending audio to the in-vehicle assistant via file corresponds to the cloud execution method. That is, the synthesized audio file is sent to the in-vehicle assistant server via the WebSocket protocol. After receiving the audio, the in-vehicle assistant server performs speech recognition and processing. This path runs in a pure software environment without the involvement of physical devices. The path of playing audio to the terminal where the in-vehicle assistant is located corresponds to the terminal execution method. That is, the synthesized audio is input to the physical device (such as a car head unit or test bench) where the in-vehicle assistant is located through a speaker or a recording method. The device's microphone receives the real audio and then hands it over to the in-vehicle assistant for processing. This path is used to verify the performance of the in-vehicle assistant in a real hardware audio recording environment. However, the tool call part still uses a simulation tool to return simulated results to ensure that the test results' ability to attribute the decision quality of the in-vehicle assistant is not affected by the stability of the real tool.

[0060] In some embodiments, for a preset simulation tool, one possible implementation of the present invention is that the preset simulation tool is configured in the following manner, and step S200 includes sub-steps S210~S220: S210. Obtain the input parameter data structure and output parameter data structure of each tool in the preset database.

[0061] In this embodiment, a pre-defined database stores the input parameter data structures and output parameter data structures for all tools that the vehicle assistant can call. The input parameter data structure specifies the parameter names, parameter types, and whether they are required when calling the tool. Taking the "Add Waypoint" tool as an example, its input parameter data structure specifies that the required parameter is "via_poi," and this parameter is a string. The output parameter data structure specifies the field names and field types that the tool's returned results should include. The input parameter data structure is used in the subsequent result evaluation stage to verify whether the parameter format passed by the vehicle assistant when calling the tool conforms to the tool's data structure definition. The output parameter data structure is used to guide the simulation tool in generating the field format of the simulated return results.

[0062] S220. Construct a preset simulation tool based on the input parameter data structure and the output parameter data structure. The preset simulation tool is used to generate simulation return results according to the field format specified in the output parameter data structure.

[0063] In this embodiment, the preset simulation tool is constructed based on the input parameter data structure and output parameter data structure of each tool. After construction, the simulation tool has the ability to generate simulated return results according to the field format specified in the output parameter data structure. The sources of the simulated return results include at least one of the following: historical tool call return results extracted from online data backflow, simulated return results synthesized in real time based on the output parameter data structure, and abnormal return results generated according to preset random rules (such as abnormal scenarios such as navigation failure and network timeout used to examine the fault tolerance capability of the vehicle assistant).

[0064] For example, when the input and output parameter data structures of the connection protocol definition or tool change during the R&D process, the change detection agent in the testing framework automatically obtains the changed items and compares them with the current state to determine whether the change belongs to the interaction protocol level or the tool definition level. If it belongs to the interaction protocol level, the change detection agent submits the changed content and the code of the corresponding module to the model. After the model identifies the module involved, it returns the code of the corresponding module to the change detection agent. The change detection agent then submits the code of the corresponding module to the model, which returns the modified code snippet to replace the original module implementation code. Since the interaction protocol is implemented in the form of an abstract class, the interaction base capability remains unchanged, and only the modular message code is replaced, thus completing the self-closing update of the interaction protocol. If it belongs to the tool definition level, the change detection agent dynamically modifies the input and output parameter data structure definitions of the corresponding tool in the preset database, so that the simulation tool automatically updates its simulation capabilities after reading the database change.

[0065] In some embodiments, for step S300, one possible implementation of the present invention is that step S300 includes sub-steps S310~S330: S310. Extract the vehicle assistant's operation log and terminal log from the test records.

[0066] In this embodiment, the test log is equivalent to storing the full process data of multiple dialogue interaction events. The operation log generated by the in-vehicle assistant server and the terminal device's edge log are extracted from the test log. The operation log records all the details of the in-vehicle assistant server's handling of the event, while the edge log records the time and status information of the terminal device's audio reception and response.

[0067] S320. Associate the runtime log with the client-side log according to the event identifier to obtain the runtime log and client-side log for each dialogue command.

[0068] In this embodiment, the runtime log and the client-side log exist as independent log streams, and the fields and formats recorded for the same dialogue interaction event differ between the two logs. The runtime log and the client-side log are associated by event identifier, so that the server-side data and terminal data for the same event form a complete event data, providing a complete data source for subsequent parsing.

[0069] S330. Extract the fields corresponding to the target result from the operation log and the terminal log according to the preset parsing rules to obtain the evaluation data of each dialogue command.

[0070] In this embodiment, preset parsing rules define which fields in the log to extract which data. For example, the actual rejection status is extracted from the `trace_z.final_reject` field of the runtime log; the existence and content of the `trace_z.agenticAndTool` field determine the actual autonomous agent mode and parse out the name and input parameters of the actual calling tool; the actual response content is extracted from `trace_z.final_result.output`; and the execution time is extracted from the `trace_z.agenticStartVadEnd` field. After extraction, each field is aligned and arranged into a structured data table according to the event identifier to obtain the evaluation data corresponding to that event.

[0071] In some embodiments, the multi-dimensional verification result includes passing the evaluation data rule verification. For step S400, one possible implementation of this embodiment includes sub-steps S410~S430: S410. Compare the actual rejection status, actual autonomous agent mode, actual tool invocation, and response content in the evaluation data with the target rejection status, target autonomous agent mode, target tool invocation, and target response content in the target results to obtain multi-dimensional verification results.

[0072] In this embodiment, the four comparison dimensions of rule verification correspond to the four target contents of the target result. The first dimension compares the actual rejection state of the evaluation data with the target rejection state in the target result. If the value of the actual rejection state matches the label of the target rejection state, this dimension is considered passed. The second dimension compares the actual autonomous agent mode of the evaluation data with the target autonomous agent mode in the target result. If the actual entry into autonomous agent mode matches the target label, this dimension is considered passed. The third dimension compares the name of the tool actually called in the evaluation data with the target called tool in the target result. If the list of tools actually called contains the name of the target called tool, this dimension is considered passed. This comparison is an inclusion judgment rather than an exact equality judgment, because the in-vehicle assistant may call multiple tools in a single dialogue. The fourth dimension matches the actual response content of the evaluation data with the target response content in the target result. This matching comparison is a keyword or pattern matching at the string level. If the actual response content contains the specified keywords in the target response content or matches the specified matching pattern, this dimension is considered passed. The verification results for each of the four dimensions are generated independently, meaning that each dimension gives a pass or fail judgment. The multi-dimensional verification results composed of the four dimensions fully reflect the degree of consistency between the evaluation data and the target results in the four objective dimensions.

[0073] For example, the target response content in the target result is labeled as a natural language description, such as "the response should include confirmation of navigation to the target location". The rule validation searches for keywords such as "navigation" and "destination" in the actual response content text for matching.

[0074] S420. Perform semantic verification on the evaluation data and the target results to obtain the judgment result of whether the evaluation data passes the test and the corresponding reason.

[0075] In this embodiment, semantic verification is used to compensate for the shortcomings of rule verification at the semantic understanding level. The fourth dimension of rule verification is keyword matching at the string level. That is, when the keywords of the actual response content and the target response content are inconsistent but semantically equivalent, rule verification may misjudge as failing; conversely, when the actual response happens to contain keywords but the overall meaning deviates from the target requirements, rule verification may misjudge as passing. Semantic verification avoids such inaccurate identification through model evaluation. For example, the actual response content in the evaluation data, the actual tool called, and the target response content in the target result are submitted together to a preset evaluation model. The evaluation model judges whether the actual response meets the requirements of the target response content from a semantic level and outputs a semantic verification score and a reason explanation. The semantic verification score represents the degree to which the actual response meets the requirements semantically, and the reason explanation is the textual explanation given by the model for the score. If the semantic verification score is greater than or equal to the preset score, the evaluation data is determined to have passed the test; if the semantic verification score is less than the preset score, the evaluation data is determined to have failed the test.

[0076] For example, the evaluation model is trained on labeled test samples and is used to semantically determine whether the submitted actual response content meets the requirements of the target response content. Semantic verification and rule verification run in parallel. The former is responsible for deterministic comparison of four objective dimensions, while the latter is responsible for in-depth semantic evaluation. The two are complementary but not mutually exclusive.

[0077] For example, the explanation of the semantic validation reason and the semantic validation score are generated synchronously by the evaluation model. After reading the actual response content, actually calling the tool, and the target response content, the evaluation model outputs the score and the corresponding reason; the score reflects the quantitative value of the degree of semantic satisfaction, and the reason explains the basis for the score. Regardless of whether it ultimately passes or fails, the reason is the evaluation model's synchronous explanation of the score.

[0078] S430. Based on the multi-dimensional verification results, the evaluation data, the judgment results of whether the test is passed, and the corresponding reasons, the evaluation judgment result is obtained.

[0079] In this embodiment, the multi-dimensional verification results and the semantic verification results are summarized to obtain the evaluation result. The evaluation result includes at least the conclusion of whether the overall result passes, the verification conclusion of each rule verification dimension, the semantic verification score, and the corresponding reasons.

[0080] Specifically, the evaluation result is determined to be passed when all four dimensions—actual rejection status, actual autonomous proxy mode, actual tool invocation, and actual response content—are compared with the corresponding target content, and the semantic verification judgment evaluation data passes the test. If any rule verification dimension fails, or the semantic verification judgment evaluation data fails, the evaluation result is determined to be failed.

[0081] For cases where the evaluation result is unsuccessful, the evaluation result records the failed rule verification dimensions and the corresponding differences. When semantic verification fails, the reason for the output of the preset evaluation model is also recorded. Thus, rule verification reflects the consistency of objective fields such as rejection status, autonomous agent mode, tool calls, and response content, while semantic verification reflects whether the actual response content meets the requirements of the target response content at the semantic level. Together, they constitute the final evaluation result, facilitating subsequent problem localization, test result statistics, and test report generation.

[0082] In some embodiments, for step S420, one possible implementation of the present invention is that step S420 includes sub-step S421: S421. Submit the actual response content, actual tool usage, and target response content from the evaluation data to the preset evaluation model to obtain the semantic verification score and the corresponding reason; wherein, if the semantic verification score is greater than or equal to the preset score, the evaluation data is determined to have passed the test; if the semantic verification score is less than the preset score, the evaluation data is determined to have failed the test.

[0083] In this embodiment, the actual response text parsed from the evaluation data, the list of actual tool names parsed from the evaluation data, and the target response content from the target result are submitted to the evaluation model. The target response content is a semantic requirement written in natural language, such as "the response should include confirmation of navigation to the target location and must call a navigation tool." The evaluation model judges whether the actual response content and the actual tool called meet the requirements at the semantic level based on this requirement.

[0084] Specifically, the semantic verification score output by the evaluation model is a quantified score that represents the evaluation model's confidence that "the actual response and tool call semantically meet the requirements of the target response content." The higher the score, the higher the degree of satisfaction. The corresponding reason is the natural language explanation given by the model for the score, such as "the response includes confirmation of navigation to the Shanghai Tower and actually calls the map navigation tool, meeting all checkpoint requirements."

[0085] Furthermore, the preset score is a pre-set passing threshold. A semantic verification score greater than or equal to this threshold is considered a pass, while a score less than this threshold is considered a fail. The preset score can be adjusted according to the rigor of the test; a higher preset score indicates stricter test requirements, while a lower preset score indicates more lenient requirements. In this embodiment, it can be 90 points.

[0086] In some embodiments, for step S421, one possible implementation of the present invention is that step S421 includes sub-steps S4211~S4213: S4211. Obtain the content of the targeted inspection.

[0087] In this embodiment, targeted inspection content refers to supplementary inspection requirements added for specific test cases, written in natural language. Targeted inspection content is introduced in two situations: one is for a specific test case where the in-vehicle assistant repeatedly exhibits the same specific error in historical versions, and the general target response is insufficient to accurately detect this specific error. For example, when processing "navigate to the tallest building in Shanghai," the in-vehicle assistant consistently misclassifies the destination as the Oriental Pearl Tower instead of the Shanghai Tower; the general target response only requires "the response should include confirmation of navigation to the target location," which is insufficient to distinguish whether the destination is correct. The other situation is when testers discover common omissions in a certain type of scenario based on iterative feedback, requiring the addition of supplementary requirements to a batch of test cases. Targeted inspection content is not a feature present in every test case; it is only added when the general target response cannot meet specific evaluation requirements.

[0088] S4212. Add the targeted inspection content to the target response content to obtain the targeted response content.

[0089] In this embodiment, the targeted response content is obtained by appending targeted check content to the original target response content. For example, if the original target response content is "the response should include confirmation of navigation to the target location", after adding the targeted check content "and the destination confirmed in the response should be the Shanghai Tower, not the Oriental Pearl Tower", the targeted response content becomes "the response should include confirmation of navigation to the target location, and the destination confirmed in the response should be the Shanghai Tower, not the Oriental Pearl Tower".

[0090] S4213. Submit the actual response content, actual tool usage, and targeted response content from the evaluation data to the preset evaluation model to obtain the semantic verification score and the corresponding reason.

[0091] In this embodiment, the target response content is replaced by enhanced targeted response content in the content submitted to the evaluation model. After receiving the enhanced evaluation criteria, the evaluation model prioritizes the targeted inspection content for evaluation. The evaluation model focuses on the targeted inspection content as a priority point when making judgments, while still making an overall evaluation based on the common parts of the targeted response content.

[0092] Based on the above method, embodiments of the present invention also provide a system corresponding to the above method, such as... Figure 2 As shown, Figure 2This is a schematic diagram of the functional modules of the vehicle assistant testing system provided in this embodiment of the invention. It should be noted that the basic principle and technical effects of the vehicle assistant testing system 1000 provided in this embodiment are the same as those in the above method embodiments. For the sake of brevity, parts not mentioned in this embodiment can be referred to the corresponding content in the method embodiments.

[0093] In this embodiment, the vehicle assistant testing system 1000 includes an acquisition module 1100, a parsing module 1200, and a testing module 1300.

[0094] The acquisition module 1100 is used to acquire test data, which includes at least one dialogue instruction and a target result. The target result includes the target's rejection state, the target's autonomous agent mode, the target's tool invocation, and the target's response content. It can be understood that the acquisition module 1100 is used to execute the above step S100.

[0095] The parsing module 1200 is used to send dialogue commands to the vehicle assistant, enabling the vehicle assistant to execute the dialogue commands and return corresponding response content, generating test records. Each dialogue command corresponds to a dialogue interaction event with an event identifier. When the vehicle assistant executes the dialogue command, it calls a preset simulation tool and returns the corresponding response content. Evaluation data for each dialogue command is parsed from the test records based on the event identifier. The evaluation data includes the actual rejection state, actual autonomous agent mode, actual tool invocation, and actual response content corresponding to the target result. It can be understood that the parsing module 1200 is used to perform the above steps S200~S300.

[0096] The testing module 1300 is used to perform rule validation and semantic validation on the evaluation data based on the target, and obtain the evaluation judgment result. It can be understood that the testing module 1300 is used to execute the above step S400.

[0097] In some embodiments, the acquisition module 1100 is used to synthesize the dialogue instructions into audio; send the audio as a file to the vehicle assistant; or play the audio to the terminal where the vehicle assistant is located. It can be understood that the acquisition module 1100 is used to perform the above steps S110~S120.

[0098] In some embodiments, the parsing module 1200 obtains the input parameter data structure and output parameter data structure of each tool in a preset database; constructs a preset simulation tool based on the input parameter data structure and output parameter data structure, and the preset simulation tool is used to generate simulation return results according to the field format specified in the output parameter data structure. It can be understood that the parsing module 1200 is used to perform the above steps S210~S220.

[0099] In some embodiments, the parsing module 1200 is used to extract the vehicle assistant's operation log and terminal log from the test records; associate the operation log and terminal log with event identifiers to obtain the operation log and terminal log for each dialogue instruction; and extract the fields corresponding to the target result from the operation log and terminal log according to preset parsing rules to obtain the evaluation data for each dialogue instruction. It can be understood that the parsing module 1200 is used to perform the above steps S310~S330.

[0100] In some embodiments, the testing module 1300 is used to compare the actual rejection state, actual autonomous agent mode, actual invoked tool, and response content in the evaluation data with the target rejection state, target autonomous agent mode, target invoked tool, and target response content in the target result to obtain multi-dimensional verification results; to perform semantic verification on the evaluation data and the target result to obtain a judgment result on whether the evaluation data passes the test and the corresponding reason; and to obtain an evaluation judgment result based on the multi-dimensional verification results, the judgment result on whether the evaluation data passes the test, and the corresponding reason. It can be understood that the testing module 1300 is used to execute the above steps S410~S430.

[0101] In some embodiments, the testing module 1300 is used to submit the actual response content, the actual invoked tool, and the target response content from the evaluation data to a preset evaluation model to obtain a semantic verification score and the corresponding reason; wherein, if the semantic verification score is greater than or equal to a preset score, the evaluation data is determined to have passed the test; if the semantic verification score is less than a preset score, the evaluation data is determined to have failed the test. It can be understood that the testing module 1300 is used to perform the above step S421.

[0102] In some embodiments, the testing module 1300 is used to obtain targeted inspection content; add the targeted inspection content to the target response content to obtain targeted response content; and submit the actual response content, the actual invoked tool, and the targeted response content in the evaluation data to a preset evaluation model to obtain a semantic verification score and the corresponding reason. It can be understood that the testing module 1300 is used to perform the above steps S4211~S4213.

[0103] Based on the same inventive concept disclosed above, the present invention also provides a block diagram of an electronic device 2000 performing the above method. Please refer to... Figure 3 , Figure 3 This is a block diagram of an electronic device provided in an embodiment of the present invention. The electronic device 2000 includes a processor 2100, a memory 2200, a bus 2300, and a communication interface 2400. The processor 2100 and the memory 2200 are connected via the bus 2300, and the processor 2100 communicates with external devices via the communication interface 2400.

[0104] Processor 2100 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed through integrated logic circuits in the hardware of processor 2100 or through software instructions. The processor 2100 may be a general-purpose processor 2100, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0105] The memory 2200 is used to store computer programs. For example, the vehicle assistant testing system 1000 in this embodiment of the invention includes at least one software function module that can be stored in the memory 2200 in the form of software or firmware. After receiving the execution instruction, the processor 2100 executes the program to implement the vehicle assistant testing method in this embodiment of the invention.

[0106] The memory 2200 may include high-speed random access memory (RAM) or non-volatile memory. Optionally, the memory 2200 may be a storage device built into the processor 2100 or a storage device independent of the processor 2100.

[0107] Bus 2300 can be ISA bus 2300, PCI bus 2300 or EISA bus 2300, etc. Figure 3 It is indicated by only one double-headed arrow, but does not mean that there is only one bus 2300 or one type of bus 2300.

[0108] Electronic devices 2000 can be mobile phones, tablets, laptops, desktop computers, and other computer devices.

[0109] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program thereon, which, when executed by processor 2100, implements the vehicle assistant testing method described above. The computer-readable storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0110] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for testing an in-vehicle assistant, characterized in that, The method includes: Acquire test data, which includes at least one dialogue command and a target result; the target result includes target rejection state, target autonomous agent mode, target tool invocation, and target response content. The dialogue command is sent to the vehicle assistant so that the vehicle assistant executes the dialogue command and returns the corresponding response content, generating a test record; wherein, each dialogue command corresponds to a dialogue interaction event with an event identifier, and when the vehicle assistant executes the dialogue command, it calls a preset simulation tool and returns the corresponding response content; Evaluation data for each dialogue instruction is parsed from the test record based on the event identifier; the evaluation data includes the actual rejection state, the actual autonomous agent mode, the actual tool invoked, and the response content; Based on the target result, the evaluation data is subjected to rule verification and semantic verification to obtain the evaluation judgment result.

2. The method according to claim 1, characterized in that, The preset simulation tool is configured in the following ways: Obtain the input and output parameter data structures of each tool in the preset database; The preset simulation tool is constructed based on the input parameter data structure and the output parameter data structure. The preset simulation tool is used to generate simulation return results according to the field format specified by the output parameter data structure.

3. The method according to claim 1, characterized in that, The step of parsing the evaluation data for each dialogue instruction from the test record based on the event identifier includes: Extract the vehicle assistant's operation log and terminal log from the test records; The runtime log and the client-side log are associated according to the event identifier to obtain the runtime log and the client-side log for each dialogue instruction; From the operation log and the terminal log, fields corresponding to the target result are extracted according to preset parsing rules to obtain the evaluation data of each dialogue instruction.

4. The method according to claim 1, characterized in that, The step of performing rule validation and semantic validation on the evaluation data based on the target result to obtain an evaluation judgment result includes: The actual rejection status, actual autonomous agent mode, actual tool invocation, and response content in the evaluation data are compared with the target rejection status, target autonomous agent mode, target tool invocation, and target response content in the target result to obtain multi-dimensional verification results. Semantic verification is performed on the evaluation data and the target result to obtain the judgment result of whether the evaluation data passes the test and the corresponding reason; The evaluation result is obtained based on the multi-dimensional verification results, the determination results of whether the evaluation data passes the test, and the corresponding reasons.

5. The method according to claim 4, characterized in that, The step of performing semantic verification on the evaluation data and the target result to obtain a judgment result on whether the evaluation data passes the test and the corresponding reason includes: The actual response content, the actual tool used, and the target response content in the evaluation data are submitted to a preset evaluation model to obtain a semantic verification score and the corresponding reason. If the semantic verification score is greater than or equal to a preset score, the evaluation data is determined to have passed the test. If the semantic verification score is less than the preset score, the evaluation data is determined to have failed the test.

6. The method according to claim 5, characterized in that, The step of submitting the actual response content, the actual tool used, and the target response content from the evaluation data to a preset evaluation model to obtain a semantic verification score and the corresponding reason includes: Obtain the content of the targeted inspection; The targeted inspection content is added to the target response content to obtain the targeted response content; The actual response content, the actual tool used, and the targeted response content in the evaluation data are submitted to a preset evaluation model to obtain a semantic verification score and the corresponding reason.

7. The method according to claim 1, characterized in that, Sending the dialogue command to the vehicle assistant includes: The dialogue commands are then synthesized into audio. Send the audio file to the in-vehicle assistant; or... The audio is played to the terminal where the in-vehicle assistant is located.

8. A vehicle-mounted assistant testing system, characterized in that, The system includes: The acquisition module is used to acquire test data, which includes at least one dialogue command and a target result; the target result includes the target rejection state, the target autonomous agent mode, the target calling a tool, and the target reply content. The parsing module is used to send the dialogue instructions to the in-vehicle assistant, so that the in-vehicle assistant executes the dialogue instructions and returns the corresponding response content, generating test records; wherein, each dialogue instruction corresponds to a dialogue interaction event with an event identifier, and when the in-vehicle assistant executes the dialogue instruction, it calls a preset simulation tool and returns the corresponding response content; the evaluation data of each dialogue instruction is parsed from the test records according to the event identifier; the evaluation data includes the actual rejection state, the actual autonomous agent mode, the actual tool call, and the actual response content; The testing module is used to perform rule verification and semantic verification on the evaluation data based on the target, and obtain the evaluation judgment result.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program that can be executed by the processor to implement the vehicle assistant testing method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the vehicle assistant testing method as described in any one of claims 1-7.

Images