Method and apparatus for multi-round semantic testing of vehicle voice interaction systems

Abstract

This application relates to a method and apparatus for multi-round semantic testing of a vehicle voice interaction system. The method includes: acquiring the domain and actions from the previous round of voice test data; generating the previous round of test corpus and corresponding expected results using a configuration library based on the domain and actions from the previous round of voice test data; generating the next round of test corpus and corresponding expected results using the configuration library based on the domain or actions from the next round of voice test data; sequentially inputting the previous round of test corpus and the next round of test corpus into the vehicle voice interaction system to obtain corresponding feedback results, and comparing the feedback results with the corresponding expected results to obtain the test results. This application solves the problem of low testing efficiency caused by the long testing time of manual testing of vehicle interaction systems in the prior art.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to a method and apparatus for performing multi-round semantic testing on a vehicle voice interaction system. Background Technology

[0002] During testing of vehicle voice interaction systems, a common issue arises where a command, once spoken, doesn't elicit the expected response, requiring second and third rounds of interaction that omit key information. This multi-round testing presents challenges, particularly with variations in the rear wheel area and the unpredictable nature of actions. As voice functionality becomes increasingly sophisticated, multi-round voice interaction scenarios are also on the rise. Currently, testing is conducted manually, resulting in lengthy testing times and low efficiency. Summary of the Invention

[0003] This application provides a method and apparatus for multi-round semantic testing of a vehicle voice interaction system, in order to solve the problem that the testing efficiency is very low due to the long testing time of the existing technology based on manual testing of the vehicle interaction system.

[0004] In a first aspect, this application provides a method for conducting multi-round semantic testing on a vehicle voice interaction system, comprising: acquiring the domain and actions from the previous round of voice test data, wherein the domain refers to a functional device in the vehicle, and the action refers to an operation that the functional device can perform; based on the domain and actions from the previous round of voice test data, generating the previous round of test corpus and corresponding expected results using a configuration library, wherein the previous round of test corpus includes multiple combinations, each combination including the domain and an action associated with the action; based on the domain or actions from the next round of voice test data, generating the next round of test corpus and corresponding expected results using the configuration library, wherein the next round of test corpus includes multiple combinations; the multiple combinations have the same domain but different actions, or the multiple combinations have the same actions but different domains; the expected results are used to characterize the expected semantic structure corresponding to each domain under each action; sequentially inputting the previous round of test corpus and the next round of test corpus into the vehicle voice interaction system to obtain corresponding feedback results, and comparing the feedback results with the corresponding expected results to obtain test results.

[0005] Optionally, if the domain of the previous round of voice test data is the same as the domain of the next round of voice test data, based on the domain or actions in the next round of voice test data, the configuration library is used to generate the test corpus and corresponding expected results for the next round, including: obtaining all actions associated with the domain in the next round of voice test data from the configuration library; removing actions from the previous round of voice test data from all actions, and generating corresponding test corpus based on the remaining actions using the configuration library; generating initial expected results for the test corpus corresponding to each of the remaining actions using the configuration library; and merging the initial expected results with the expected results corresponding to the test corpus in the previous round to obtain the expected results corresponding to each of the remaining actions.

[0006] Optionally, if the domain of the previous round of voice test data is different from the domain of the next round of voice test data, based on the domain or action in the next round of voice test data, the configuration library is used to generate the test corpus and corresponding expected results for the next round, including: removing the domain from the previous round of voice test data from the domain in the next round of voice test data; using the configuration library to obtain all actions associated with the remaining domains; determining the actions that match the actions in the previous round of voice test data from all the actions associated with the remaining domains; generating the test corpus corresponding to the next round of voice test data for the domain corresponding to the matched action using the configuration library; generating an initial expected result for the test corpus corresponding to the next round of voice test data using the configuration library; and merging the initial expected result with the expected result corresponding to the test corpus in the previous round to obtain the expected result corresponding to the test corpus in the next round of voice test data.

[0007] Optionally, merging the initial expected result with the expected result corresponding to the previous round of test corpus includes: determining the union of fields in the initial expected result and the expected result corresponding to the previous round of test corpus; for fields in the union that exist in the initial expected result, filling the value of the corresponding field in the initial expected result into the union; for fields in the union that do not exist in the initial expected result, inheriting the value of the corresponding field in the expected result corresponding to the previous round of test corpus.

[0008] Optionally, comparing the feedback result with the corresponding expected result to obtain the test result includes: if the fields in the feedback result include the fields in the expected result and the values ​​in the fields are consistent, determining the test result as a test passed; if the fields in the feedback result include the fields in the expected result, but at least one value in the fields is inconsistent, or if the fields in the feedback result do not include the fields in the expected result, determining the test result as a test failed.

[0009] Optionally, before comparing the feedback results with the corresponding expected results to obtain the test results, the method further includes: selecting a preset number of feedback results from the feedback results for the next round of test corpus.

[0010] Optionally, the method further includes: constructing a domain action library, a domain name library, an action corpus, and an expected result library, wherein the domain action library is used to store all actions supported by each domain; the domain name library is used to store at least one natural language alias corresponding to each domain; the action corpus is used to store at least one natural language sentence template corresponding to each action; the expected result library includes multiple expected results, and the expected results are used to store the expected speech intent corresponding to each domain under each action; and the configuration library is constructed based on the domain action library, the domain name library, the action corpus, and the expected result library.

[0011] Secondly, this application provides an apparatus for performing multi-round semantic testing on a vehicle voice interaction system, comprising: an acquisition module, configured to acquire the domain and actions from the previous round of voice test data, wherein the domain refers to a functional device in the vehicle, and the action refers to an operation that the functional device can perform; a first processing module, configured to generate the previous round of test corpus and corresponding expected results based on the domain and actions from the previous round of voice test data using a configuration library, wherein the previous round of test corpus includes multiple combinations, each combination including the domain and an action associated with the action; and a second processing module, configured to... Based on the domains or actions in the next round of voice test data, the configuration library is used to generate the next round of test corpus and the corresponding expected results. The next round of test corpus includes multiple combinations; the multiple combinations have the same domain but different actions, or the multiple combinations have the same actions but different domains. The expected results are used to characterize the expected semantic structure of each domain under each action. The third processing module is used to sequentially input the previous round of test corpus and the next round of test corpus into the vehicle voice interaction system, obtain the corresponding feedback results, and compare the feedback results with the corresponding expected results to obtain the test results.

[0012] Thirdly, this application provides a vehicle, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to execute the method for performing multi-round semantic testing of a vehicle voice interaction system as described in the first aspect of this application.

[0013] Fourthly, this application also provides a computer storage medium storing computer-executable instructions for executing the method for performing multi-round semantic testing of a vehicle voice interaction system as described in the first aspect of this application.

[0014] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application first obtains the domain and actions in the previous round of voice test data, then generates the previous round of test corpus and corresponding expected results based on the domain and actions in the previous round of voice test data using a configuration library. Next, based on the domain or actions in the next round of voice test data, it generates the next round of test corpus and corresponding expected results using a configuration library. Finally, it sequentially inputs the previous round of test corpus and the next round of test corpus into the vehicle voice interaction system to obtain the corresponding feedback results, and compares the feedback results with the corresponding expected results to obtain the test results. Therefore, in this application, multi-round contextual semantic testing can be performed, and the testing can distinguish between multi-round tests within the same domain and cross-domain tests. Manual testing is unnecessary, greatly improving testing efficiency and making the testing more comprehensive. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0018] Figure 1 A flowchart illustrating a method for performing multi-round semantic testing on a vehicle voice interaction system, provided in an embodiment of this application; Figure 2 A schematic diagram of a device structure for performing multi-round semantic testing on a vehicle voice interaction system, provided in an embodiment of this application; Figure 3 This is a schematic diagram of the vehicle structure provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0021] To address the problem of low testing efficiency due to the long testing time of existing manual testing methods for vehicle interaction systems, this application provides a method for multi-round semantic testing of vehicle voice interaction systems, such as... Figure 1 As shown, the steps of this method include: Step 101: Obtain the domain and actions from the previous round of voice test data. The domain refers to the functional devices in the vehicle, and the actions refer to the operations that the functional devices can perform. In the embodiments of this application, the field can specifically refer to functional devices in a vehicle such as windows, doors, air conditioners, and audio systems; the action refers to the action associated with the field. For example, the action of a window can include opening and closing; the action of a door can also be opening and closing; the action of an air conditioner includes turning on, turning off, raising the temperature, lowering the temperature, etc. That is, different functional devices have their own corresponding actions, which may be the same as the actions of other functional devices or different from the actions of other functional devices.

[0022] Furthermore, in the embodiments of this application, the previous round of voice test data and the next round of voice test data typically refer to two consecutive voice data sentences during the test. For example, if the previous round of voice test data is "open the car window," and a while later the user adds "open it a little more," it is clear that both rounds involve voice test data related to the car window, thus constituting a multi-round voice test within the same domain. If the previous round of voice test data is "open the car window," and a while later the user adds "and the sunroof," it is clear that the two rounds involve voice test data from two different domains.

[0023] Step 102: Based on the domain and actions in the previous round of voice test data, use the configuration library to generate the test corpus of the previous round and the corresponding expected results. The test corpus of the previous round includes multiple combinations, and each combination includes the domain and the action associated with the action. In this context, if the previous round of voice test data is "open the car window," then the domain = car window, and the action = open. Then, all actions associated with the car window are retrieved from the configuration library. For example, if the actions for the car window in the configuration library include "open" and "open briefly," then the corresponding voice test data generated from the configuration library in the previous round of voice test data would be "open the car window" and "open the car window briefly." The expected result refers to the expected voice structure corresponding to each action in the corresponding domain. For example, in the expected result for "open the car window," the domain = car window, and the action = open. That is, the fields in the voice test data are consistent with the fields in the expected result.

[0024] Step 103: Based on the domains or actions in the next round of speech test data, generate the test corpus and corresponding expected results for the next round using the configuration library. The test corpus for the next round includes multiple combinations; the domains in these multiple combinations are the same but the actions are different, or the actions in these multiple combinations are the same but the domains are different. The expected results are used to characterize the expected speech structure corresponding to each domain under each action. In this embodiment of the application, the test data for the next round needs to be determined separately for those in the same domain and those in different domains. If the next round of voice test data is in the same domain as the previous round of voice test data, then the domains of the combinations in the next round of test data are the same but the actions are different. If the next round of voice test data is in a different domain than the previous round of voice test data, then the actions of the combinations in the next round of test data are the same but the domains are different. Taking the previous round of voice test data as "open the car window" and the next round of voice test data as "open the car window briefly" as an example, based on the domain = car window in the previous round of voice test data, all actions corresponding to this domain are obtained from the configuration library, such as "open", "close", "open to which position", etc. After removing duplicate actions from the previous round of voice test data, "close" and "open to which position" remain. Then, test corpus related to the car window is generated based on these two actions. In order to better fit the actual scenario, corresponding slot information can be added to the generated test corpus, such as "driver's side", "passenger's side", "left rear", and "right rear". The generated test corpus can include: "open the driver's side window", "open the passenger's side window", "open the left rear window", "open the right rear window", "open the driver's side window", "open the passenger's side window", "open the left rear window", and "open the right rear window".

[0025] Step 104: Input the previous round of test data and the next round of test data into the vehicle voice interaction system in sequence to obtain the corresponding feedback results, and compare the feedback results with the corresponding expected results to obtain the test results.

[0026] As described in steps 101 to 104 above, the domain and actions from the previous round of voice test data are first obtained. Then, based on the domain and actions from the previous round of voice test data, the configuration library is used to generate the test corpus and corresponding expected results for the previous round. Next, based on the domain or actions from the next round of voice test data, the configuration library is used to generate the test corpus and corresponding expected results for the next round. Finally, the test corpus from the previous round and the test corpus from the next round are sequentially input into the vehicle voice interaction system to obtain the corresponding feedback results. The feedback results are then compared with the corresponding expected results to obtain the test results. Therefore, this application allows for multi-round contextual semantic testing, distinguishing between multi-round tests within the same domain and those across domains, eliminating the need for manual testing, greatly improving testing efficiency, and providing a more comprehensive test.

[0027] In an optional embodiment of this application, a configuration library is built before testing. The specific process includes the following steps: Step 11: Construct a domain action library, a domain name library, an action corpus, and an expected result library. The domain action library stores all actions supported by each domain; the domain name library stores at least one natural language alias corresponding to each domain; the action corpus stores at least one natural language sentence template corresponding to each action; and the expected result library includes multiple expected results, which are used to store the expected speech intent corresponding to each domain under each action. Step 12: Construct a configuration library based on the domain action library, domain name library, action corpus, and expected result library.

[0028] In this specific example, the domain action library records all actions supported by each domain. For example: window: open, close, open position; sunroof: open, close; air conditioner: open, close, temperature adjustment.

[0029] Domain name database, used to record natural language aliases for each domain. For example: car window: ["car window"]. sunroof: ["sunroof"]. air conditioner: ["air conditioner"].

[0030] An action corpus records natural language sentence templates for each action. These templates can include placeholders. For example: Open: ["Open DD", "Open DD briefly"] (DD is a domain placeholder). Open Location: ["Open [Location] DD", "Open [Location] DD"] ([Location] is a slot placeholder). Close: ["Close DD"]. Adjust Temperature: ["Adjust DD to [Temperature] degrees"].

[0031] Slot value library: Records the possible values ​​of each slot placeholder. For example: [Position]: ["Driver", "Passenger", "Rear"]. [Temperature]: ["22", "24", "26"].

[0032] The expected results library includes multiple expected results, which are used to record the expected voice intent returned for each action in the current domain. For example, for "open the driver's window", the expected result would be: {"domain": "window", "action": "open", "position": "driver"}.

[0033] In an optional embodiment of this application, when the domain of the previous round of voice test data is the same as the domain of the next round of voice test data, the method of generating the next round of test corpus and the corresponding expected results based on the domain or action in the next round of voice test data, as mentioned in step 103 above, using the configuration library, may further include: Step 21: Retrieve all actions related to the domain in the next round of voice test data from the configuration library; Step 22: Remove the actions from the previous round of voice test data from all actions, and generate the corresponding test corpus based on the remaining actions using the configuration library; Step 23: For the test corpus corresponding to each of the remaining actions, generate the initial expected results using the configuration library; Step 24: Combine the initial expected result with the expected result corresponding to the test corpus in the previous round to obtain the expected result corresponding to each of the remaining actions.

[0034] In this specific example, taking the previous round (e.g., the first round) of test data as follows: Domain: window (car window); Action: open, then all actions associated with the window domain are retrieved from the configuration library. The current list of actions for window in the configuration library is: ["open", "close", "open_position"]. After removing the action "open" that was executed in the previous round, the remaining actions are: ["close", "open_position"]. Test corpora and initial expectations are then generated for each of these remaining actions. For the corpus template "Close DD" → generated corpus "Close car window", the corresponding initial expectation is: {"domain": "window", "action": "close"}.

[0035] For `open_position`: corpus template: "Open [position]DD", "Open [position]DD"; slot values: ["Driver's seat", "Passenger's seat", "Rear left", "Rear right"]; generated corpus: "Open driver's seat window", "Open passenger's seat window", etc.; initial expectation: {"domain": "window", "action": "open", "position": "10"} (assuming "Driver's seat" is mapped to "10") Then, the initial expectation is merged with the expectation of the previous round. The expectation of the previous round (actually returned after the first round of execution) is: {"domain": "window", "action": "open", "window_state": "opening"}.

[0036] For close: the final expected value after merging is {"domain": "window", "action": "close", "window_state": "opening"}.

[0037] For open_position (taking "open driver's window" as an example): the final expected value after merging is {"domain":"window", "action": "open", "position": "10", "window_state": "opening"}.

[0038] As can be seen, by removing actions already executed in the previous round, it ensures that the same semantics are not repeatedly tested within the same domain. It can also automatically test all unexecuted operations in the domain, verifying the system's complete understanding of the domain context. Finally, by merging expectations, it verifies whether the system correctly retains the state information from the previous round (such as window_state), ensuring that subsequent rounds of dialogue can continue the context.

[0039] In an optional embodiment of this application, when the domain of the previous round of voice test data is different from the domain of the next round of voice test data, the method of generating the next round of test corpus and the corresponding expected results based on the domain or action in the next round of voice test data, as mentioned in step 103 above, using the configuration library, may further include: Step 31: After removing the domains from the previous round of voice test data from the domains in the next round of voice test data, use the configuration library to obtain all actions associated with the remaining domains; Step 32: Identify the actions that match the actions in the previous round of voice test data from all the actions associated with the remaining domain; Step 33: For the domain corresponding to the matched action, use the configuration library to generate the test corpus corresponding to the next round of voice test data; Step 34: For the test corpus corresponding to the next round of voice test data, generate the initial expected results using the configuration library; Step 35: Merge the initial expected result with the expected result corresponding to the test corpus of the previous round to obtain the expected result corresponding to the test corpus of the next round of speech test data.

[0040] In a specific example, for instance, if the test data from the previous round (e.g., the first round) was: Domain: window (car window); Action: open, then the domain set for the next round would be ["sunroof", "seat_heat", "window"]. After removing the previous domain "window", the remaining domains would be ["sunroof", "seat_heat"]. Based on this, actions matching the previous action "open" are filtered from the remaining domains, taking into account action extension information. The actions for "sunroof" in the configuration library: ["open_page", "close"], match "open_page". The actions for "seat_heat" in the configuration library: ["close", "set_gear"], do not match any actions related to the extension of the action "open". Therefore, there are no expected results associated with "seat_heat" in the subsequent steps. However, "sunroof" matches the action "open_page" based on the extension of the action "open", thus generating test corpus for the matched domain (sunroof). For sunroof: domain alias ["skyroof"], the cross-domain corpus templates are "DD settings page is also" and "and DD settings page". Based on this, the corpus "skyroof settings page is also" and "and skyroof settings page" are generated.

[0041] Therefore, the initial expectation can be generated: initial expectation of the open action of sunroof: {"domain": "sunroof", "action": "open_page"}.

[0042] Merge the initial expectation with the previous expectation: Previous expectation: {"domain": "window", "action":"open", "window_state": "opening"}.

[0043] For the sunroof corpus "skyroof settings page is also": the final expected value after merging is {"domain": "sunroof", "action": "open_page", "window_state": "opening"} Through steps 31 to 35 above, only cross-domain scenarios related to the semantics of the previous action are tested to avoid meaningless cross-domain combinations. Then, by inheriting fields from the previous round (such as window_state), it is verified whether the system can correctly retain or migrate context information when switching domains. Finally, a specific cross-domain corpus template (such as "DD settings page is also") is used to simulate the cross-domain expression habits of real users.

[0044] In optional embodiments of this application, the method of merging the initial expected result with the expected result corresponding to the previous round of test corpus involved in step 24 or step 35 may further include: Step 41: Determine the union of the fields in the initial expected result and the expected result corresponding to the test corpus in the previous round; Step 42: For fields in the union set where the initial expected result exists, fill the union set with the values ​​of the corresponding fields from the initial expected result; for fields in the union set where the initial expected result does not exist, inherit the values ​​of the corresponding fields from the expected result of the previous test corpus.

[0045] In this specific example, the expectation of the previous round is { "domain": "window", "action": "open", "position": "10", "window_state": "opening"}. The initial expectation of the current round (generated based on the instructions of the next round) is: { "domain": "sunroof", "action": "open_page"}.

[0046] Merging process: Field union: ["domain", "action", "position", "window_state"]; Field-by-field filling: domain: if initially expected to exist, take "sunroof"; action: if initially expected to exist, take "open_page"; position: if initially expected not to exist, inherit "10" from the previous round; window_state: if initially expected not to exist, inherit "opening" from the previous round. Final expectation: {"domain": "sunroof", "action": "open_page", "position": "10", "window_state": "opening"}.

[0047] As can be seen, in this embodiment of the application, there is no need to manually write expectations for each subsequent scenario. The fields are automatically completed through rules. This also verifies whether the system has correctly retained the key state information of the previous round, thereby avoiding context loss.

[0048] In an optional embodiment of this application, the method of comparing the feedback result with the corresponding expected result to obtain the test result in step 105 above may further include: Step 51: If the fields in the feedback result include the fields in the expected result, and the values ​​in the fields are consistent, then the test result is determined to be a successful test. Step 52: If the fields in the feedback result include the fields in the expected result, but at least one of the values ​​in the fields is inconsistent, or if the fields in the feedback result do not include the fields in the expected result, the test result is determined to be a test failure.

[0049] In this specific example, if the current expected result is: { "domain": "sunroof", "action": open", "window_state": "opening"}.

[0050] If the feedback result is: { "domain": "sunroof", "action": "open_page", "window_state": "opening", "confidence": 0.95}, then the test passed, as the feedback result contains all fields and the values ​​match (additional fields are allowed).

[0051] If the feedback result is: { "domain": "sunroof", "action": "open_page "}, it can be seen that window_state is missing in the feedback result, so the test is confirmed to have failed.

[0052] If the feedback result is: { "domain": "sunroof", "action": "open_page", "window_state": closed"}, then the feedback result does not match the expected window_state value, and the test fails.

[0053] As can be seen, in this embodiment of the application, the automated testing rules are clear, avoiding subjective judgment, and can distinguish between missing fields and incorrect values, making it easier for developers to quickly locate defects. It also allows the system to return additional fields without affecting the test pass determination and avoids false alarms.

[0054] In this embodiment of the application, before comparing the feedback result with the corresponding expected result to obtain the test result, the method of this embodiment of the application may further include: Step 61: Select a preset number of feedback results for the next round of test data from the feedback results.

[0055] In a specific example, 100 test cases for subsequent actions are generated using a configuration library (e.g., 8 cases for the `open_position` action, 1 case for `close`, and several cases for cross-domain actions). By setting a preset quantity limit (e.g., 30 cases), only these 30 cases are tested, while the rest are not. This approach avoids test case explosion, ensuring controllable test duration, especially in continuous integration environments, and by using sampling, it covers as many scenarios as possible within a limited time.

[0056] The present application will now be explained in detail with reference to specific embodiments of the present application, which provide a general testing method for multi-round semantic testing.

[0057] Before conducting the test, data preparation mainly includes the following parts: a) First-round domain action section: represented using a dictionary, such as: { "window": [action1, action2, …], "sunroof": [action1, action2, ...], … (This section contains data on other domains and actions) } b) Rear wheel area motion: represented using a dictionary, such as: { "window": [action1, action2, …], "sunroof": [action1, action2, ...], … (This section contains data on other domains and actions) } c) Single-intent expectation part, which assembles the expected data according to the same domain protocol, such as: When action = open window_expect = { "domain": "window", "action": "open", …(Other fields described in the agreement) In the expectation, some areas and actions may involve variables. These variables need to be obtained from the "Extended Information" section, such as: When Action = open_position, that is, when opening a car window at a certain position. window_expect = { "domain": "window", "action": "open", "position": functionUtils().get_position_expect(input_text), ----- (The position information here needs to be obtained from the extended information section based on the input test corpus) …(Other fields described in the agreement) } d) Domain Name section: This section sets all possible names for the domain, as shown below: { "window": ["car window"], “trunk”: [“tailgate”, “trunk”] … (This refers to data from other fields) } e) First round of action corpus { "open": ["Open DD", "Open DD"], "open_position": ["Open [position]DD", "Open [position]DD"], … (This section represents corpus data for other actions) } Here, DD represents the domain name. In actual testing, DD will be replaced with the corresponding domain name. For example, when the domain is "trunk" and the action is "open", according to steps c) and d), the corpus used is: open the tailgate, open the tailgate briefly, open the trunk, open the trunk briefly.

Position

[0058] h) Extended Information Section: This section describes the input and output components for the slot, primarily in dictionary form, such as: Gear = { "Low": "1", "Medium": "2", "High": "3", } Among them, other slot information is prepared similarly to the gear position slot, and it can also be in other forms, but the one-to-one correspondence between input and output must be ensured. The field values of the slots will be used to replace the corresponding slots in the corpus and assemble into a complete corpus; for example: replace the corpus "Set the windshield wiper to the [gear position] gear" with: Set the windshield wiper to the low gear, Set the windshield wiper to the medium gear, Set the windshield wiper to the high gear. The values of the slots will be used in the expectation to obtain the corresponding expected values according to the corpus. For example: the expected value corresponding to setting the windshield wiper to the low gear is 1, the expected value corresponding to setting the windshield wiper to the medium gear is 2, and the expected value corresponding to setting the windshield wiper to the high gear is 3.

[0059] i) Action extension information part, this part classifies all actions in the above steps. The classification standard is in the way of action + effect. Taking the action of "open" as an example, the fields it involves include: windows, windshield wipers, etc.; the effects include: directly open, open (to) a certain position, open a page, etc. Then the action extension information can be used to expand and divide actions in combination with the above steps. The example is as follows: action_extend_obj = { "open": ["open", "open_position",... (actions related to directly opening)], "open_value": ["open_value", "open_percentage", "open_max",... (actions related to opening to a certain position, that is, having a certain step)], "open_page": ["open_page",... (actions related to opening a certain page)], "close": ["close", "close_position",... (actions related to directly closing)], } j) Cross-domain rear-wheel corpus, this part defines the corpus used later in the cross-domain context. The configuration is as follows: rear_extend_text_list = ["DD is also", "There is also DD", … (other later corpus)] action_extend_rear_text_obj = { "open_page": ["DD set page", "DD interface"], "open_position": ["

Position

Position

[0060] In the embodiments of the present application, for multi-round context semantic tests, only the selection of intra-domain context or cross-domain context tests needs to be made: a) Intra-domain context test: If the first round is in domain A, then all subsequent rounds must be in the same domain A. b) Cross-domain context test: If the first round is in domain A, then all subsequent rounds must not be in domain A.

[0061] a) For intra-domain context tests: First, based on the domain action data of the first round, obtain the domain action dictionary to be executed in the first round: domain_action_obj. For example: {'window': ['open','open_position', 'close']}; then use domain_action_obj combined with the domain action data of the subsequent rounds, and according to the key domain of domain_action_obj, obtain the current domain action dictionary to be executed in the subsequent rounds: rear_domain_action_obj, such as: {'window': ['open','open_position', 'close']}.

[0062] Immediately afterwards, traverse the domain_action_obj obtained from the configuration library, and then traverse each action in each domain to achieve full-scale testing.

[0063] For each domain domain and each action action during traversal, obtain the domain actions of the subsequent rounds. For example, when the domain is window and the action is open: i) Based on the domain window and action open in the first round, we get: domain_name_list(["window"]) and slot_text_list(["open DD", "open DD"]).

[0064] Specifically, the process iterates through the slot text list `slot_text_list`. For each `slot_text` item, it performs slot replacement based on the slot text value. Then, it combines the replaced text value list `text_list` with the `domain_name_list`, replacing `DD` with the specific domain name. It also adds necessary information: domain, domain_name, and action. The output is the final text list `first_text_obj_list`. For example: `{'domain':'window', 'domain_name': 'car window', 'action': 'open', 'text': 'open car window'}`.

[0065] ii) Based on the obtained `rear_domain_action_obj` and the domain and action executed in the first round, after filtering out the executed actions, the final actions to be executed in the subsequent round, `remain_actions`, are obtained as: ['open_position', 'close']. The list is then traversed again, processed, and written into a new list, ultimately yielding the corpus list to be executed in the subsequent round, `rear_text_obj_list`. For example: [{'domain': 'window', 'domain_name': 'car window', 'action': 'open_position', 'text': 'Open for the first row'}, {'domain': 'window', 'domain_name': 'car window', 'action': 'open_position', 'text': 'Open for the left rear'}, {'domain': 'window', 'domain_name': 'car window', 'action': 'open_position', 'text': 'Open for the passenger seat'}].

[0066] iii) Iterate through the obtained first_text_obj_list, and for each first_text_obj, obtain the expected data first_text_expect for the first round of the current test using a single intent expectation. Then, execute the test on the first round of the corpus, recording the unique representation and round number for each round of testing. For example: {'domain': window, 'action': 'open', ... other returned fields}.

[0067] iv) After the first round of testing, iterate through the obtained rear_text_obj_list and execute all subsequent rounds of testing. For each rear_text_obj item, combine the single intent expectation to obtain the first round of expected data in the current test. For example: {'domain': window, 'action': 'open', 'position': ['10'], …other return fields}.

[0068] v) The field names of the expected values ​​in the previous and current rounds are combined into a set all_union_keys (which can be a list or a set. The specific implementation is not limited). For example, we can get: ['domain', 'action', 'position'].

[0069] vi) If the action in the current round is different from the action in the previous round, obtain a new expected value `rear_now_expect_obj` based on `all_union_keys` and the expected value of the current round. Specifically, iterate through `all_union_keys`. If an action in `all_union_keys` is present in the current round, use the action value from the current round; otherwise, use the action value from the previous round. Example follows: Iterate through ['domain', 'action', 'position']. If 'domain' is the expected key for the current round, then the expected domain for the current round uses the corresponding value from the current round. That is, 'domain = window'. If 'position' is not the expected key for the current round, then the expected domain for the current round uses the value from the previous round. That is, 'position = ["10"].

[0070] vii) Use `rear_now_expect_obj` to call the subsequent API, incrementing the round number by 1. After receiving the API response, perform an assertion check: if they are equal, the test passes; otherwise, the test fails, and the test result is output.

[0071] b) For cross-domain context testing: First, based on the initial domain action data, obtain the domain action dictionary to be executed: `domain_action_obj`. For example: `{'window': ['open', 'open_position', 'close']}`. Then, iterate through the obtained `domain_action_obj`, and then iterate through the actions of each domain within it. This completes the full testing.

[0072] For each domain and action during traversal, retrieve the subsequent domain action. For example, if the domain is `window` and the action is `open`: i) Based on the domain window and action open in the first round, we get: domain_name_list(["window"]) and slot_text_list(["open DD", "open DD"]).

[0073] Specifically, the process iterates through the slot text list `slot_text_list`. For each `slot_text` item, it performs slot replacement based on the slot text value. Then, it combines the replaced text value list `text_list` with the `domain_name_list`, replacing `DD` with the specific domain name. Necessary information is added: domain and action. The final output is the final text list `first_text_obj_list`. For example: `{'domain': 'window', 'domain_name': 'car window', 'action': 'open', 'text': 'open car window'}`.

[0074] ii) Obtain the data to be executed in the subsequent rounds (hereinafter) based on the domain and action used in the first round: 1. Based on the initial action (action), and combined with the action's extended information, obtain the key and value of the extended information. For example, if the initial action is "open", then the found `rear_action_key = "open"`. rear_action_values ​​= ["open", "open_position",…] If not found, then: rear_action_key = ""; rear_action_values ​​= [] 2. Determine the value of rear_action_key: If no value is found, return [] to obtain empty data for the next round of execution. If a value exists, iterate through the domain action data of the next round. If the domain and the domain executed in the first round are not equal, combine their corresponding values ​​into the `rear_data_list` of data to be executed in the next round. When combining, the actions from the previous round must be added. For example: `[{'domain': 'seat_heat', 'actions': ['open_position'], 'prev_action: 'open'}, {'domain': 'sunroof', 'actions': ['open'], 'prev_action': 'open'}]` iii) Iterate through first_text_obj_list to perform test execution: 1. For each item `first_text_obj`, combine it with the resulting `rear_data_list` data and the `rear_domains_obj_list` list of all domains to be executed for the rear wheels. For example: `[[{"text": "Front seat heating also", "expect": {expected data}}, ... (other text and their corresponding expected data)]`, `[{"text": "Sunroof also needed", "expect": {expected data}}, ... (other text and their corresponding expected data)]]`. The specific steps are as follows: 1.1) Iterate through `rear_data_list`. For each domain data (domain and corresponding actions) to be executed, if the action from the first round is in `actions`, then use that action as the action for the current round; otherwise, randomly select an action from `actions` as the action for the current round. It should be noted that this action selection does not change the action type, only the effect of the action.

[0075] 1.2) Based on the cross-domain rear wheel corpus data, corpus names, and corpus slot values, the final list of corpora to be executed, all_text_list, is obtained. For example: ['Driver's seat heating also needs', 'Left front seat heating also needs']: Iterate through all_text_list and combine it with the expected value of each single intent. Combine this to form all_text_obj_list, as shown below: { "text": text, "expect": expect } 1.3) Obtain the list of data to be executed for all domains: rear_domains_obj_list.

[0076] 2. If rear_domains_obj_list has a value, perform the test; otherwise, skip the current loop.

[0077] 3. Based on the domain, action, and corpus of the first round, obtain the expectation of the first round and execute the first round test. At this time, it is necessary to record the unique expression and round number of the multi-round test.

[0078] 4. Process rear_domains_obj_list, combining each data point in the corpus into a new single-level list rear_obj_list. For example: [{"text": "Front seat heating", "expect": {expected data}}, ... (other texts and their corresponding expected data)].

[0079] 5. Iterate through rear_obj_list to perform subsequent tests and output the test results.

[0080] As can be seen, the above methods can be used to quickly test the multi-intent capabilities of any domain or any / specified domain; moreover, during testing, different test results can be obtained based on testing within the same domain or across domains, which can greatly improve the efficiency of development and testing.

[0081] Corresponding to the above Figure 1 This application provides an apparatus for performing multi-round semantic testing on a vehicle voice interaction system, such as... Figure 2 As shown, the device includes: The acquisition module 202 is used to acquire the domain and action from the previous round of voice test data. The domain refers to the functional devices in the vehicle, and the action refers to the operation that the functional devices can perform. The first processing module 204 is used to generate the previous round of test corpus and corresponding expected results based on the domain and actions in the previous round of voice test data using a configuration library. The previous round of test corpus includes multiple combinations, and each combination includes a domain and an action associated with the action. The second processing module 206 is used to generate the next round of test corpus and corresponding expected results based on the domains or actions in the next round of voice test data using a configuration library. The next round of test corpus includes multiple combinations; the domains in the multiple combinations are the same but the actions are different, or the actions in the multiple combinations are the same but the domains are different. The expected results are used to characterize the expected semantic structure of each domain under each action. The third processing module 208 is used to input the previous round of test data and the next round of test data into the vehicle voice interaction system in sequence, obtain the corresponding feedback results, and compare the feedback results with the corresponding expected results to obtain the test results.

[0082] In an optional embodiment of this application, when the domain of the previous round of voice test data is the same as the domain of the next round of voice test data, the second processing module includes: The acquisition unit is used to retrieve all actions associated with the domain in the next round of voice test data from the configuration library; The first processing unit is used to remove actions from the previous round of voice test data from all actions, and generate corresponding test corpus based on the remaining actions using the configuration library; The second processing unit is used to generate the initial expected result for the test corpus corresponding to each of the remaining actions using the configuration library. The third processing unit is used to merge the initial expected result with the expected result corresponding to the test corpus in the previous round to obtain the expected result corresponding to each of the remaining actions.

[0083] In an optional embodiment of this application, when the domain of the previous round of voice test data is different from the domain of the next round of voice test data, the second processing module includes: The fourth processing unit is used to remove the domains from the previous round of voice test data from the domains in the next round of voice test data, and then use the configuration library to obtain all actions associated with the remaining domains. The first determining unit is used to determine the action that matches the action in the previous round of voice test data from all actions associated with the remaining domain; The fifth processing unit is used to generate test corpus corresponding to the next round of voice test data for the domain corresponding to the matched action using the configuration library; The sixth processing unit is used to generate the initial expected results for the test corpus corresponding to the next round of voice test data using the configuration library; The seventh processing unit is used to merge the initial expected result with the expected result corresponding to the test corpus of the previous round to obtain the expected result corresponding to the test corpus of the next round of speech test data.

[0084] In an optional embodiment of this application, the third processing unit or the seventh processing unit is used to perform the following steps: Determine the union of the fields in the initial expected result and the expected result corresponding to the previous round of test corpus; For fields in the union set that have an initial expected result, fill the union set with the values ​​of the corresponding fields from the initial expected result; for fields in the union set that do not have an initial expected result, inherit the values ​​of the corresponding fields from the expected result of the previous test corpus.

[0085] In an optional embodiment of this application, the third processing module includes: The second determining unit is used to determine the test result as passed if the fields in the feedback result include the fields in the expected result and the values ​​in the fields are consistent. The third determining unit is used to determine the test result as "test failed" when the fields in the feedback result include the fields in the expected result, but at least one of the values ​​in the fields is inconsistent, or when the fields in the feedback result do not include the fields in the expected result.

[0086] In optional embodiments of this application, the apparatus further includes: The fourth processing module is used to select a preset number of feedback results for the next round of test data from the feedback results before comparing the feedback results with the corresponding expected results to obtain the test results.

[0087] In optional embodiments of this application, the apparatus further includes: The first building module is used to build a domain action library, a domain name library, an action corpus, and an expected result library. The domain action library stores all actions supported by each domain; the domain name library stores at least one natural language alias corresponding to each domain; the action corpus stores at least one natural language sentence template corresponding to each action; and the expected result library includes multiple expected results, which are used to store the expected speech intent corresponding to each domain under each action. The second building module is used to build a configuration library based on the domain action library, domain name library, action corpus, and expected result library.

[0088] like Figure 3 As shown in the figure, this application embodiment provides a vehicle, including a processor 311, a communication interface 312, a memory 313, and a communication bus 314, wherein the processor 311, the communication interface 312, and the memory 313 communicate with each other through the communication bus 314. Memory 313 is used to store computer programs; In one embodiment of this application, when the processor 311 executes the program stored in the memory 313, it implements the method for performing multi-round semantic testing on the vehicle voice interaction system provided in any of the foregoing method embodiments. Its function is similar and will not be described again here.

[0089] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method for performing multi-round semantic testing on a vehicle voice interaction system as provided in any of the foregoing method embodiments.

[0090] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0091] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0092] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0093] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for conducting multi-round semantic testing on a vehicle voice interaction system, characterized in that, include: Obtain the domain and actions from the previous round of voice test data, wherein the domain refers to the functional devices in the vehicle, and the actions refer to the operations that the functional devices can perform; Based on the domain and actions in the previous round of voice test data, the previous round of test corpus and corresponding expected results are generated using the configuration library. The previous round of test corpus includes multiple combinations, and each combination includes the domain and the action associated with the action. Based on the domains or actions in the next round of voice test data, the configuration library is used to generate the test corpus for the next round and the corresponding expected results. The test corpus for the next round includes multiple combinations; the domains in the multiple combinations are the same but the actions are different, or the actions in the multiple combinations are the same but the domains are different. The expected results are used to characterize the expected semantic structure of each domain under each action. The previous round of test data and the next round of test data are sequentially input into the vehicle voice interaction system to obtain the corresponding feedback results. The feedback results are then compared with the corresponding expected results to obtain the test results.

2. The method according to claim 1, characterized in that, If the domain of the previous round of voice test data is the same as that of the next round of voice test data, based on the domain or actions in the next round of voice test data, the configuration library is used to generate the next round of test corpus and the corresponding expected results, including: Retrieve all actions associated with the domain in the next round of voice test data from the configuration library; Remove the actions from the previous round of voice test data from all the actions, and generate the corresponding test corpus based on the remaining actions using the configuration library; For the test corpus corresponding to each of the remaining actions, the initial expected results are generated using the configuration library; The initial expected result is merged with the expected result corresponding to the test corpus in the previous round to obtain the expected result corresponding to each of the remaining actions.

3. The method according to claim 1, characterized in that, If the domain of the previous round of voice test data differs from that of the next round of voice test data, based on the domain or actions in the next round of voice test data, the configuration library is used to generate the next round of test corpus and the corresponding expected results, including: After removing the domains from the previous round of voice test data from the domains in the next round of voice test data, the configuration library is used to obtain all actions associated with the remaining domains; Identify the actions that match the actions in the previous round of voice test data from all actions associated with the remaining domain; For the domain corresponding to the matched action, the configuration library is used to generate the test corpus corresponding to the next round of voice test data; For the test corpus corresponding to the next round of voice test data, the initial expected results are generated using the configuration library; The initial expected result is merged with the expected result corresponding to the test corpus of the previous round to obtain the expected result corresponding to the test corpus of the next round of speech test data.

4. The method according to claim 2 or 3, characterized in that, The initial expected result is merged with the expected result corresponding to the test corpus in the previous round, including: Determine the union of the fields in the initial expected result and the expected result corresponding to the previous round of test corpus; For fields in the union set where the initial expected result exists, the values ​​of the corresponding fields in the initial expected result are filled into the union set; for fields in the union set where the initial expected result does not exist, the values ​​of the corresponding fields in the expected result of the previous test corpus are inherited.

5. The method according to claim 1, characterized in that, The feedback results are compared with the corresponding expected results to obtain the test results, including: If the fields in the feedback result include the fields in the expected result, and the values ​​in the fields are consistent, the test result is determined to be a successful test. If the fields in the feedback result include the fields in the expected result, but at least one of the values ​​in the fields is inconsistent, or if the fields in the feedback result do not include the fields in the expected result, the test result is determined to be a test failure.

6. The method according to claim 1, characterized in that, Before comparing the feedback result with the corresponding expected result to obtain the test result, the method further includes: Select a preset number of feedback results from the feedback results for the next round of test data.

7. The method according to claim 1, characterized in that, The method further includes: Construct a domain action library, a domain name library, an action corpus, and an expected result library. The domain action library stores all actions supported by each domain; the domain name library stores at least one natural language alias corresponding to each domain; the action corpus stores at least one natural language sentence template corresponding to each action; and the expected result library includes multiple expected results, which store the expected speech intent corresponding to each domain under each action. The configuration library is constructed based on the domain action library, the domain name library, the action corpus, and the expected result library.

8. A device for performing multi-round semantic testing on a vehicle voice interaction system, characterized in that, include: The acquisition module is used to acquire the domain and action from the previous round of voice test data, wherein the domain refers to the functional device in the vehicle, and the action refers to the operation that the functional device can perform; The first processing module is used to generate the previous round of test corpus and the corresponding expected results based on the domain and actions in the previous round of voice test data using a configuration library. The previous round of test corpus includes multiple combinations, and each combination includes the domain and the action associated with the action. The second processing module is used to generate the next round of test corpus and corresponding expected results based on the domains or actions in the next round of voice test data using the configuration library. The next round of test corpus includes multiple combinations; the domains in the multiple combinations are the same but the actions are different, or the actions in the multiple combinations are the same but the domains are different. The expected results are used to characterize the expected semantic structure of each domain under each action. The third processing module is used to input the previous round of test data and the next round of test data into the vehicle voice interaction system in sequence, obtain the corresponding feedback results, and compare the feedback results with the corresponding expected results to obtain the test results.

9. A vehicle, characterized in that, include: The processor, communication interface, memory, and communication bus are connected, with the processor, communication interface, and memory communicating with each other via the communication bus. The memory is used to store computer programs; the processor is used to execute the computer programs to implement the method for performing multi-round semantic testing on a vehicle voice interaction system as described in any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method for performing multi-round semantic testing of a vehicle voice interaction system as described in any one of claims 1-7.

Images