Interactive assertion method, related device, equipment, system and storage medium

Abstract

The invention discloses an interactive assertion method, a related device, equipment, a system and a storage medium. The method comprises the following steps: in response to an interaction instruction of to-be-tested equipment, generating an assertion task for checking whether the to-be-tested equipment successfully executes the interaction instruction; based on the assertion task, performing decomposition to obtain sub-tasks of a plurality of detection types; wherein the plurality of detection types correspond to the hardware types of the to-be-detected device for executing the interaction instruction response; acquiring hardware response data of the to-be-tested equipment about each subtask in the process of executing the interaction instruction by the to-be-tested equipment; and analyzing the hardware response data of each sub-task based on the first large language model to obtain an assertion result of executing the interaction instruction by the to-be-tested device. According to the scheme, the interaction function of the to-be-detected equipment can be automatically detected, and the assertion accuracy is improved.

Description

Technical Field

[0001] This application relates to the technical field of interaction detection, and particularly to an interaction assertion method and related devices, equipment, systems, and storage media. Background Art

[0002] With the development of artificial intelligence technology, interaction systems such as intelligent vehicle cockpits and smart homes can recognize user interaction intentions and control the corresponding hardware functions to respond. Therefore, the detection of the performance of intelligent interaction systems has received increasing attention.

[0003] In the prior art, the assertion scheme for the interaction results of intelligent interaction systems usually relies on manual testing. The interaction results of each round of the interaction system need to be judged by testers based on experience, resulting in low testing efficiency, inability to meet the product iteration requirements, and the testing results are subject to the subjective influence of testers, and the accuracy of the generated assertion results is relatively low. In view of this, how to achieve automatic detection of the interaction functions of the device under test and improve the accuracy of assertion has become an urgent problem to be solved. Summary of the Invention

[0004] The main technical problem to be solved by this application is to provide an interaction assertion method and related devices, equipment, systems, and storage media, which can achieve automatic detection of the interaction functions of the device under test and improve the accuracy of assertion.

[0005] To solve the above technical problem, in the first aspect of this application, an interaction assertion method is provided, including generating an assertion task for verifying whether the device under test successfully executes the interaction instruction in response to the interaction instruction of the device under test; decomposing the assertion task to obtain subtasks of several detection types; where the several detection types correspond to the hardware types of the device under test in response to the execution of the interaction instruction; obtaining the hardware response data of the device under test for each subtask during the process of the device under test executing the interaction instruction; and analyzing the hardware response data of each subtask based on the first large language model to obtain the assertion result of the device under test executing the interaction instruction.

[0006] To solve the above technical problem, in the second aspect of this application, an interaction assertion device is provided, including a generation module, a decomposition module, an acquisition module, and an analysis module. The generation module is used to generate an assertion task for verifying whether the device under test successfully executes the interaction instruction in response to the interaction instruction of the device under test; the decomposition module is used to decompose the assertion task to obtain subtasks of several detection types; where the several detection types correspond to the hardware types of the device under test in response to the execution of the interaction instruction; the acquisition module is used to obtain the hardware response data of the device under test for each subtask during the process of the device under test executing the interaction instruction; and the analysis module is used to analyze the hardware response data of each subtask based on the first large language model to obtain the assertion result of the device under test executing the interaction instruction.

[0007] To solve the above technical problems, a third aspect of the present application provides an electronic device, including a memory and a processor coupled to each other. Program instructions are stored in the memory, and the processor is configured to execute the program instructions to implement the interaction assertion method in the first aspect above.

[0008] To solve the above technical problems, a fourth aspect of the present application provides an interaction assertion system, including a device under test and an assertion device. The assertion device at least includes a data acquisition device, an instruction output device, and a data processing device. The instruction output device is configured to output interaction instructions for interacting with the device under test. The data acquisition device is configured to acquire hardware response data of the device under test, and the data processing device is the electronic device in the third aspect above.

[0009] To solve the above technical problems, a fifth aspect of the present application provides a computer-readable storage medium storing program instructions that can be run by a processor, and the program instructions are used to implement the interaction assertion method in the first aspect above.

[0010] In the above solution, in response to the interaction instructions of the device under test, an assertion task for verifying whether the device under test successfully executes the interaction instructions is generated. Based on the assertion task, several subtasks of detection types are decomposed, and several detection types correspond to the hardware types of the responses of the device under test when executing the interaction instructions. Hardware response data of the device under test for each subtask during the process of the device under test executing the interaction instructions is obtained. Based on the analysis of the hardware response data of each subtask by the first large language model, an assertion result of the device under test executing the interaction instructions is obtained. Therefore, based on the collected hardware response data of each subtask, the automatic detection of the interaction function of the device under test can be realized. Moreover, the several decomposed subtasks can analyze the execution situation of the interaction instructions from multiple angles according to the hardware types of the responses of the device under test when executing the interaction instructions, improving the accuracy of assertion task parsing. Further, the first large language model can analyze the hardware response data of multiple data types and is applicable to the generation of assertion results in multiple scenarios. Therefore, the automatic detection of the interaction function of the device under test can be realized, and the accuracy of assertion can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a schematic flowchart of an embodiment of the interaction assertion method of the present application; Figure 2 is a schematic framework diagram of an embodiment of the interaction assertion device of the present application; Figure 3 is a schematic framework diagram of an embodiment of the electronic device of the present application; Figure 4 is a schematic framework diagram of an embodiment of the interaction assertion system of the present application; Figure 5 is a schematic framework diagram of an embodiment of the computer-readable storage medium of the present application. Detailed implementation manners

[0012] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the scope of protection of the present application.

[0013] The terms "system" and "network" are often used interchangeably herein. The term "and / or" herein is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "" herein generally indicates that the associated objects before and after are in an "or" relationship. In addition, "plurality" herein means two or more than two.

[0014] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of an embodiment of the interactive assertion method of the present application. Specifically, the following steps may be included: Step S10: In response to an interaction instruction of a device under test, generate an assertion task for verifying whether the device under test successfully executes the interaction instruction.

[0015] In the embodiments of the present disclosure, the device under test is a device capable of performing human-computer interaction, and the device under test runs at least a processing control system, which can execute interaction instructions and control the corresponding hardware functions to respond. For example, the device under test is an intelligent vehicle cockpit, and the in-vehicle computer in the intelligent vehicle cockpit runs a processing control system. The in-vehicle computer can receive interaction instructions from users, control the hardware response of the vehicle cockpit according to the interaction instructions. For example, when the interaction instruction is "open the co-pilot window", the in-vehicle computer issues a control instruction to control the window to rise and fall, and issues a control instruction to control the speaker to output the voice "the co-pilot window has been opened" after the instruction is executed. When the interaction instruction is "play song A", the in-vehicle computer issues a control instruction to control the display screen to display the song information of song A, and issues a control instruction to control the speaker to output the audio content of song A. For another example, the device under test is a smart home, and the voice assistant device in the smart home runs a processing control system. The voice assistant device can receive interaction instructions from users, control the hardware response of the smart home according to the interaction instructions. For example, when the interaction instruction is "close the master bedroom curtain", the voice assistant device issues a control instruction to control the curtain to move along the curtain track.

[0016] It should be noted that the above embodiments are only possible implementation scenarios, and the specific form of the device under test is not limited in the present application.

[0017] In an implementation scenario, the interaction instruction is based on a natural language description. It should be noted that the language system used for the interaction instruction is not limited in this application. For example, it can be Chinese, English, French, etc., or a combination of multiple language systems. For the sake of brevity, they will not be elaborated one by one here. And the data form of the interaction instruction is not limited in this application. For example, it can be audio, text, etc. Based on the interaction intention represented by the interaction instruction, the natural language input is converted into an executable task description. For example, taking the device under test as an intelligent vehicle cockpit, the interaction instruction is "Open the music player", and the generated assertion task is "Whether the music player is opened".

[0018] In an implementation scenario, the assertion task is generated based on a large language model. The large language model can understand the human multi-modal natural language input, and obtain the keyword information about the input content and the logical relationship between various elements according to the input content, and generate semantically relevant outputs. The specific principle can refer to more technical details of the large language model, which will not be elaborated here. It should be noted that the large language model can include but is not limited to: open source large models such as LLAMA and Bloom. The network architecture of the large language model is not limited in this application. The large language model in this application can be an open source large model or a finished large model after parameter adjustment, which is not specifically limited in this application.

[0019] In a specific implementation scenario, at least based on the interaction instruction, a second prompt instruction is constructed to instruct the large language model to understand the interaction instruction and generate an assertion task about the interaction instruction. The second prompt instruction is input into the second large language model to obtain the output content of the second large language model as the assertion task. The above solution, based on the natural language understanding ability and multi-modal data processing ability of the large language model, can be applied to the generation of assertion tasks in multiple scenarios and improve the accuracy of assertion task generation.

[0020] In a specific implementation scenario, the second large language model can be fine-tuned based on synthetic instructions and historical interaction instructions, and adversarial sample testing is introduced to further verify the error bound of the second large language model in intention parsing, continuously optimize the second large language model, and improve the intention understanding ability and robustness of the second large language model.

[0021] In a specific implementation scenario, when it is difficult to understand the meaning of an interaction instruction. For example, the interaction instruction is "Favorite this song", that is, the specific song referred to by "this song" is unknown. Before constructing at least a second prompt instruction based on the interaction instruction at least, obtain the output data of at least one target hardware regarding the interaction instruction before the device under test executes the interaction instruction, and obtain at least one historical interaction instruction that the device under test has executed. Still taking the foregoing embodiment as an example, according to the historical interaction instruction that the device under test has executed, the historical interaction instruction "Play song B" executed before "Favorite this song" is obtained. The output data of at least one target hardware regarding the interaction instruction before the device under test executes the interaction instruction is the audio content of song B output by the speaker. Analyze the output data and the historical interaction instruction to obtain reference data for instructing the large language model to understand the interaction instruction. Based on the reference data and the interaction instruction, construct a second prompt instruction. In the above solution, based on the output data of at least one target hardware regarding the interaction instruction before the device under test executes the interaction instruction and at least one historical interaction instruction that the device under test has executed, reference data for assisting the second large language model to understand the interaction instruction is obtained, which can further improve the accuracy of assertion task generation.

[0022] In an implementation scenario, before generating an assertion task for verifying whether the device under test has successfully executed an interaction instruction in response to the interaction instruction of the device under test, based on the semantic relationship between a number of historical interaction instructions executed by the device under test and the interaction instruction, obtain the historical assertion result of the historical interaction instruction that has an instruction dependency relationship with the interaction instruction. For example, the interaction instruction "Play the first song in the favorite list" requires the previous historical interaction instruction "Favorite song C" to be successfully executed. When the historical assertion result indicates that the device under test has successfully executed the historical interaction instruction, perform the steps of generating an assertion task for verifying whether the device under test has successfully executed the interaction instruction in response to the interaction instruction of the device under test and subsequent steps. In the above solution, by verifying the front-back dependency relationship of the assertion path regarding the interaction instruction, and verifying whether the implementation of the interaction instruction depends on the execution result of the previous instruction based on logical dependency analysis, the executability of the assertion task can be guaranteed, the logical consistency of the front and back tasks can be verified, and the accuracy and reliability of assertion task parsing and execution are improved.

[0023] It should be noted that the method for obtaining the instruction dependency relationship is not limited in this application. For example, a large language model, a network model based on the Encoder-Decoder architecture, etc. can be used.

[0024] Step S20: Decompose the assertion task to obtain sub-tasks of several detection types.

[0025] In an implementation scenario, several detection types correspond to the hardware types for the device under test to execute interaction instruction responses. For example, when the device under test executes "Play Song D", it controls the display screen to display the song information of Song D and controls the speaker to output the audio content of Song D. That is, the hardware response types are the display screen and the speaker. Another example is that when the device under test executes "Open the front window", it controls the driver's window 1 and the passenger's window 2 to open. That is, the hardware response types are window 1 and window 2. Taking the device under test executing "Play Song D" in the foregoing embodiment as an example, the generated assertion task is "Whether to play Song D". According to the response hardware types, it is decomposed into subtask 1 "Whether the display screen displays the song content of Song D and the status icon indicates playing" and subtask 2 "Whether the audio output by the speaker is Song D". The above solution decomposes the assertion task into subtasks of several detection types, which can improve the accuracy of generating assertion results.

[0026] In a specific implementation scenario, different devices under test may have different execution results for the same interaction instruction. For example, when device under test A executes "Play Song D", it controls the display screen to display the song information of Song D and controls the speaker to output the audio content of Song D. When device under test B executes "Play Song D", it only controls the speaker to output the audio content of Song D. Therefore, before decomposing the assertion task, log data of the device under test can be obtained, etc., to obtain the hardware types that respond when the device under test executes different types of interaction instructions.

[0027] Step S30: Obtain the hardware response data of the device under test for each subtask during the execution of the interaction instruction.

[0028] In an implementation scenario, the hardware response data for each subtask can be collected based on the hardware type. For example, for obtaining the hardware response data of subtask 1 "Whether the display screen displays the song content of Song D and the status icon indicates playing", it can be obtained based on an image acquisition device, such as a captured image of the display screen. For obtaining the hardware response data of subtask 2 "Whether the audio output by the speaker is Song D", it can be obtained based on an audio acquisition device, such as a recorded audio of the speaker.

[0029] It should be noted that specific parameters such as the acquisition duration and acquisition time of different hardware response data are not limited in this application. For example, the process of the device under test executing the interaction instruction corresponding to the subtask "whether the window is closed" is: from the window starting to rise until the window is completely closed and the instruction completion prompt sound is output. Therefore, the hardware response data collected corresponding to the subtask "whether the window is closed" is the completion prompt sound output during the process of the device under test executing the interaction instruction and the captured image of the window when the interaction instruction is executed. Again, for example, the process of the device under test executing the interaction instruction corresponding to the subtask "whether the audio output by the speaker is song D" is: continuously playing song D. Therefore, the hardware response data collected corresponding to the subtask "whether the audio output by the speaker is song D" is at least part of the audio information continuously output after the device under test executes the interaction instruction.

[0030] In addition, in order to ensure the generation of subsequent assertion results as much as possible, the hardware response data corresponding to each subtask can be obtained sequentially. Of course, considering that under the premise of relatively abundant network environment, hardware resources, etc., data acquisition can usually be triggered and completed more quickly. In order to accelerate the generation of subsequent assertion results as much as possible, the hardware response data can also be obtained synchronously. The above examples are only several typical examples in the actual application process, and do not limit the acquisition timing of the hardware response data accordingly.

[0031] In an implementation scenario, after obtaining the hardware response data of the device under test regarding each subtask during the process of the device under test executing the interaction instruction, preprocess the hardware response data. For example, perform adaptive noise reduction, voice enhancement, etc. on audio data, perform super-resolution reconstruction under low-light conditions on image data, perform frame rate optimization on video data, perform formatting denoising and semantic enhancement on log files, etc., to improve the integrity and usability of the hardware response data.

[0032] Step S40: Analyze the hardware response data of each subtask based on the first large language model to obtain the assertion result of the device under test executing the interaction instruction.

[0033] In an implementation scenario, based on the collected hardware response data regarding each subtask, the automated detection of the interaction function of the device under test can be realized, and the decomposed several subtasks can analyze the execution situation of the interaction instruction from multiple angles according to the hardware type to which the device under test responds when executing the interaction instruction, improving the accuracy of assertion task parsing. Further, the first large language model can analyze the hardware response data of multiple data types and is applicable to the generation of assertion results in multiple scenarios. Therefore, it can realize the automated detection of the interaction function of the device under test and improve the accuracy of the assertion.

[0034] In one implementation scenario, the first large language model has the ability to process multimodal data. Therefore, the data type of the hardware response data is not limited in this application, such as images, audio, etc.

[0035] In another implementation scenario, the target text that matches the subtask is parsed from the hardware response data of the subtask. For example, based on the captured image of the display screen, the target text that matches the subtask "whether the display screen shows the song content of song D and the status icon indicates that it is playing" is parsed as "the display screen shows the lyrics information of song D, and the play button indicates that the song is playing". Based on the target text of each subtask, a first prompt instruction for instructing the first large language model to respond to the assertion task is constructed, and the first prompt instruction is input into the first large language model to obtain the output content of the first large language model as the assertion result. In the above solution, the hardware response data is converted into the target text related to the subtask, and the target text is used as the input data of the first large language model, which can reduce the data processing pressure of the first large language model and improve the accuracy of generating the assertion result.

[0036] In a specific implementation scenario, after the target text that matches the subtask is parsed from the hardware response data of the subtask, since the data representation forms of the target texts are the same, the target texts are fused to obtain a fused text. The specific fusion process is not limited in this application, such as text splicing, duplicate text filtering, etc. Constructing the first prompt instruction based on the fused text can reduce the data processing pressure of the first large language model and improve the efficiency of generating the assertion result.

[0037] It should be noted that the first large language model and the second large language model can be the same large language model or different large language models, and the specific network parameters of the first large language model and the second large language model can be the same or different, which are not limited in this application.

[0038] In a specific implementation scenario, based on the detection type of the subtask and the data type of the hardware response data, a target parsing model is selected. Specifically, the target parsing model includes a target detection model, an image classification model, a text recognition model, etc., which can be used to implement different parsing tasks. The hardware response data is processed based on the target parsing model to obtain the target text. It should be noted that the specific structure of the target parsing model is not limited in this application.

[0039] In a specific implementation scenario, based on the task intention of the subtask, the same hardware response data needs to be processed by multiple target parsing models. It can be understood that the number of selected target parsing models and the specific processing process are not limited in this application.

[0040] In a specific implementation scenario, the target parsing model is trained based on the basic model for the functional data of the device under test. When the physical information such as the models and implemented functions of several devices under test is different, the trained target parsing model is labeled with tags corresponding to the devices under test and stored in a classified manner to establish an index database that can be quickly searched. It is applicable to the generation of assertion results in multiple scenarios, so it can realize the automatic detection of the interaction functions of the devices under test and improve the accuracy of assertions.

[0041] In a specific implementation scenario, before processing the hardware response data based on the target parsing data, it is necessary to verify whether the target parsing model has been loaded and is running properly. When the verification result indicates that the target parsing model has been loaded and is running properly, the step of generating the target text is executed.

[0042] In an implementation scenario, when the device under test is interrupted during the execution of the previous interaction instruction and the current interaction instruction is executed, the generation progress of the assertion result for the previous interaction instruction is saved, and the assertion result for the device under test to execute the current interaction instruction is generated. In response to the device under test resuming the execution of the previous interaction instruction, the assertion result for the device under test to execute the previous interaction instruction is continuously generated based on the generation progress.

[0043] In a specific implementation scenario, the priorities of several types of interaction instructions executed by the device under test are obtained in advance. For example, in a smart car cockpit, the priority of answering a call is higher than that of playing music. Using the priorities of the interaction instructions as reference information for generating assertion results can further improve the accuracy of generating assertion results.

[0044] In a specific implementation scenario, the assertion result includes the content used to answer the assertion task. For example, the assertion result is "yes", "successfully executed", "no", "not successfully executed", etc. It can also include the relevant parsing content during the assertion process, such as "the previous interaction instruction before this interaction failed to be successfully executed, resulting in the failure of this interaction" and "the target parsing model was not successfully called", etc., further improving the reliability of the assertion result.

[0045] In a specific implementation scenario, the assertion result of each interaction is recorded in the system log, and a multi-dimensional statistical report is generated using a visualization analysis tool to comprehensively evaluate the performance of generating assertion results.

[0046] In the above solution, in response to the interaction instruction of the device under test, an assertion task is generated to verify whether the device under test successfully executes the interaction instruction. Based on the assertion task, several subtasks of different detection types are decomposed, and the several detection types correspond to the hardware types of the responses of the device under test when executing the interaction instruction. The hardware response data of the device under test for each subtask during the execution of the interaction instruction is obtained, and the hardware response data of each subtask is analyzed based on the first large language model to obtain the assertion result of the device under test when executing the interaction instruction. Therefore, based on the collected hardware response data for each subtask, the automatic detection of the interaction function of the device under test can be realized. Moreover, the several decomposed subtasks can analyze the execution situation of the interaction instruction from multiple angles according to the hardware types of the responses of the device under test when executing the interaction instruction, improving the accuracy of assertion task parsing. Further, the first large language model can analyze the hardware response data of multiple data types and is applicable to the generation of assertion results in multiple scenarios. Therefore, the automatic detection of the interaction function of the device under test can be realized and the accuracy of the assertion can be improved.

[0047] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of an embodiment of the interaction assertion device 20 of the present application. As Figure 2 shown, the interaction assertion device 20 includes a generation module 21, a decomposition module 22, an acquisition module 23, and an analysis module 24. The generation module 21 is configured to generate an assertion task for verifying whether the device under test successfully executes the interaction instruction in response to the interaction instruction of the device under test; the decomposition module 22 is configured to decompose, based on the assertion task, several subtasks of different detection types; wherein, the several detection types correspond to the hardware types of the responses of the device under test when executing the interaction instruction; the acquisition module 23 is configured to acquire the hardware response data of the device under test for each subtask during the execution of the interaction instruction; and the analysis module 24 is configured to analyze the hardware response data of each subtask based on the first large language model to obtain the assertion result of the device under test when executing the interaction instruction.

[0048] Therefore, in response to the interaction instruction of the device under test, the interaction assertion device 20 generates an assertion task for verifying whether the device under test successfully executes the interaction instruction, decomposes the assertion task into subtasks of several detection types, and several detection types correspond to the hardware types in response to the execution of the interaction instruction by the device under test. The hardware response data of the device under test for each subtask during the execution of the interaction instruction is obtained, and the hardware response data of each subtask is analyzed based on the first large language model to obtain the assertion result of the device under test for executing the interaction instruction. Therefore, based on the collected hardware response data for each subtask, the automatic detection of the interaction function of the device under test can be realized, and several decomposed subtasks can analyze the execution situation of the interaction instruction from multiple angles according to the hardware type in response when the device under test executes the interaction instruction, improving the accuracy of assertion task parsing. Further, the first large language model can analyze the hardware response data of multiple data types and is applicable to the generation of assertion results in multiple scenarios. Therefore, the automatic detection of the interaction function of the device under test can be realized, and the accuracy of the assertion can be improved.

[0049] In some disclosed embodiments, the analysis module 24 further includes a text parsing module (not shown) for parsing out the target text matching the subtask from the hardware response data of the subtask; the analysis module 24 further includes a first construction module (not shown) for constructing a first prompt instruction based on the target text of each subtask; wherein, the first prompt instruction is used to instruct the first large language model to respond to the assertion task; the analysis module 24 further includes a first output module (not shown) for inputting the first prompt instruction to the first large language model to obtain the output content of the first large language model as the assertion result.

[0050] In some disclosed embodiments, the text parsing module (not shown) further includes a model selection module (not shown) for selecting a target parsing model based on the detection type of the subtask and the data type of the hardware response data; the text parsing module (not shown) further includes a model processing module (not shown) for processing the hardware response data based on the target parsing model to obtain the target text.

[0051] In some disclosed embodiments, the generation module 21 further includes a second construction module (not shown) for constructing a second prompt instruction based at least on the interaction instruction; wherein, the second prompt instruction is used to instruct the second large language model to understand the interaction instruction and generate an assertion task for the interaction instruction; the generation module 21 further includes a second output module (not shown) for inputting the second prompt instruction to the second large language model to obtain the output content of the second large language model as the assertion task.

[0052] In some disclosed embodiments, before constructing the second prompt instruction at least based on the interaction instruction, the interaction assertion device 20 further includes a data acquisition module (not shown) for acquiring output data of at least one target hardware regarding the interaction instruction before the device under test executes the interaction instruction, and acquiring at least one historical interaction instruction that the device under test has executed; the interaction assertion device 20 further includes a reference data module (not shown) for parsing the output data and the historical interaction instruction to obtain reference data for instructing the large language model to understand the interaction instruction; the second construction module (not shown) further includes a construction sub-module (not shown) for constructing the second prompt instruction based on the reference data and the interaction instruction.

[0053] In some disclosed embodiments, before generating an assertion task for verifying whether the device under test has successfully executed the interaction instruction in response to the interaction instruction of the device under test, the interaction assertion device 20 further includes a historical detection module (not shown) for obtaining a historical assertion result of a historical interaction instruction having an instruction dependency relationship with the interaction instruction based on the semantic relationship between a number of historical interaction instructions executed by the device under test and the interaction instruction; the interaction assertion device 20 further includes an execution judgment module (not shown) for, in response to the historical assertion result indicating that the device under test has successfully executed the historical interaction instruction, performing the step of generating an assertion task for verifying whether the device under test has successfully executed the interaction instruction in response to the interaction instruction of the device under test and subsequent steps.

[0054] In some disclosed embodiments, in the case where the device under test is interrupted during the execution of the previous interaction instruction and executes the current interaction instruction, the interaction assertion device 20 further includes a progress saving module (not shown) for saving the generation progress of the assertion result regarding the previous interaction instruction and generating an assertion result for the device under test to execute the current interaction instruction; the interaction assertion device 20 further includes a progress recovery module (not shown) for, in response to the device under test resuming the execution of the previous interaction instruction, continuing to generate the assertion result for the device under test to execute the previous interaction instruction based on the generation progress.

[0055] Please refer to Figure 3 , Figure 3 which is a schematic framework diagram of an embodiment of the electronic device 30 of the present application. As Figure 3As shown, the electronic device 30 includes a memory 31 and a processor 32 that are coupled to each other. Program instructions are stored in the memory 31, and the processor 32 is configured to execute the program instructions to implement the steps in any of the above-described interactive assertion method embodiments. Specifically, the electronic device 30 may include, but is not limited to, a server, a desktop computer, a laptop computer, a tablet computer, a smart phone, etc., which are not limited herein. Specifically, the processor 32 is configured to control itself and the memory 31 to implement the steps in any of the above-described interactive assertion method embodiments. The processor 32 may also be referred to as a CPU (Central Processing Unit). The processor 32 may be an integrated circuit chip with signal processing capabilities. The processor 32 may also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Additionally, the processor 32 may be implemented jointly by integrated circuit chips.

[0056] Therefore, in response to an interaction instruction of a device under test, the electronic device 30 generates an assertion task for verifying whether the device under test has successfully executed the interaction instruction, decomposes the assertion task into subtasks of several detection types, and the several detection types correspond to the hardware types in response to the execution of the interaction instruction by the device under test. The electronic device 30 obtains the hardware response data of the device under test for each subtask during the execution of the interaction instruction, analyzes the hardware response data of each subtask based on a first large language model, and obtains an assertion result of the device under test for executing the interaction instruction. Therefore, based on the collected hardware response data for each subtask, the automatic detection of the interaction function of the device under test can be realized, and the several decomposed subtasks can analyze the execution situation of the interaction instruction from multiple angles according to the hardware types in response when the device under test executes the interaction instruction, improving the accuracy of assertion task parsing. Further, the first large language model can analyze the hardware response data of multiple data types and is applicable to the generation of assertion results in multiple scenarios. Therefore, the automatic detection of the interaction function of the device under test can be realized, and the accuracy of the assertion can be improved.

[0057] Please refer to Figure 4 , Figure 4 which is a schematic framework diagram of an embodiment of the interactive assertion system 40 of the present application. As Figure 4As shown, the interaction assertion system 40 includes a device under test 41 and an assertion device 42. The assertion device 42 at least includes a data acquisition device 421, an instruction output device 422, and a data processing device 423. The instruction output device 422 is used to output interaction instructions for interacting with the device under test 41. The data acquisition device 421 is used to acquire the hardware response data of the device under test 41, and the data processing device 423 is the electronic device 30 in the foregoing embodiment.

[0058] In one implementation scenario, the source of the interaction instructions output by the instruction output device 422 is not limited in this application. For example, intelligent synthesized sound, historically acquired audio, etc.

[0059] In one implementation scenario, the data acquisition device 421 includes, but is not limited to, an image acquisition device, an audio acquisition device, a temperature acquisition device, etc. Taking the device under test 41 as an intelligent vehicle cockpit as an example, the data acquisition device 421 includes an image acquisition device A for acquiring the image of the display screen, an image acquisition device B for acquiring the image of the window, an image acquisition device C for acquiring the image of the seat, and the data acquisition device 421 also includes an audio acquisition device D for acquiring the audio of speaker 1, an audio acquisition device E for acquiring the audio of speaker 2, etc.

[0060] In a specific implementation scenario, the data acquisition device 421 is communicatively connected to the data processing device 423. The data acquisition device 421 can obtain the data acquisition instruction sent by the data processing device 423, acquire the data collected by the corresponding acquisition device, and transmit it back to the data processing device 423. For details, reference can be made to the foregoing embodiment. For the sake of brevity, it will not be elaborated here.

[0061] In one implementation scenario, the data processing device 423 runs an interaction assertion model and a data statistics model. The interaction assertion model is used to generate an assertion result for the device under test 41 to execute the interaction instruction. The data statistics model is used to count the assertion results of the interaction instructions and generate feedback data for optimizing the interaction assertion model based on the statistical results. For details, reference can be made to the foregoing steps. For the sake of brevity, it will not be elaborated here.

[0062] In the above solution, the assertion device 42 in the interaction assertion system 40 generates an assertion task for verifying whether the device under test 41 has successfully executed the interaction instruction in response to the interaction instruction of the device under test 41. A number of subtasks of several detection types are obtained based on the decomposition of the assertion task, and the several detection types correspond to the hardware types in response to the execution of the interaction instruction by the device under test 41. The hardware response data of the device under test 41 for each subtask during the execution of the interaction instruction is obtained, and the hardware response data of each subtask is analyzed based on the first large language model to obtain the assertion result of the device under test 41 for executing the interaction instruction. Therefore, based on the collected hardware response data for each subtask, the automatic detection of the interaction function of the device under test 41 can be realized, and the several subtasks obtained by decomposition can analyze the execution situation of the interaction instruction from multiple angles according to the hardware type in response when the device under test 41 executes the interaction instruction, improving the accuracy of assertion task parsing. Further, the first large language model can analyze the hardware response data of multiple data types and is applicable to the generation of assertion results in multiple scenarios. Therefore, the automatic detection of the interaction function of the device under test 41 can be realized, and the accuracy of the assertion can be improved.

[0063] Please refer to Figure 5 , Figure 5 which is a schematic framework diagram of an embodiment of the computer-readable storage medium 50 of the present application. The computer-readable storage medium 50 includes program instructions 51 that can be run by a processor, and the program instructions 51 are used to implement the steps in any of the above-described interaction assertion method embodiments.

[0064] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0065] The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to each other. For the sake of brevity, they will not be repeated here.

[0066] In several embodiments provided by the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation manners described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0067] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed over multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0068] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0069] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0070] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. An interactive assertion method, characterized in that, Including: Generating an assertion task for verifying whether the device under test has successfully executed the interaction instruction in response to the interaction instruction of the device under test; Decomposing the assertion task into subtasks of several detection types; wherein, the several detection types correspond to the hardware types of the responses of the device under test to the execution of the interaction instruction; Obtaining the hardware response data of the device under test for each of the subtasks during the execution of the interaction instruction by the device under test; Analyzing the hardware response data of each of the subtasks based on a first large language model to obtain an assertion result of the device under test for the execution of the interaction instruction.

2. The method according to claim 1, characterized in that, The analyzing the hardware response data of each of the subtasks based on a first large language model to obtain an assertion result of the device under test for the execution of the interaction instruction includes: Parsing a target text matching the subtask from the hardware response data of the subtask; Constructing a first prompt instruction based on the target texts of each of the subtasks; wherein, the first prompt instruction is used to instruct the first large language model to respond to the assertion task; Inputting the first prompt instruction into the first large language model to obtain the output content of the first large language model as the assertion result.

3. The method according to claim 2, wherein The parsing a target text matching the subtask from the hardware response data of the subtask includes: Selecting a target parsing model based on the detection type of the subtask and the data type of the hardware response data; Processing the hardware response data based on the target parsing model to obtain the target text.

4. The method according to claim 1, wherein The generating an assertion task for verifying whether the device under test has successfully executed the interaction instruction includes: Constructing a second prompt instruction based at least on the interaction instruction; wherein, the second prompt instruction is used to instruct a second large language model to understand the interaction instruction and generate an assertion task for the interaction instruction; Inputting the second prompt instruction into the second large language model to obtain the output content of the second large language model as the assertion task.

5. The method according to claim 4, characterized in that, Before the constructing a second prompt instruction based at least on the interaction instruction, the method further includes: Obtaining the output data of at least one target hardware for the interaction instruction before the device under test executes the interaction instruction, and obtaining at least one historical interaction instruction executed by the device under test; Parsing the output data and the historical interaction instruction to obtain reference data for instructing the large language model to understand the interaction instruction; The constructing a second prompt instruction based at least on the interaction instruction includes: Constructing the second prompt instruction based on the reference data and the interaction instruction.

6. The method according to any one of claims 1 to 5, characterized in that Before the generating an assertion task for verifying whether the device under test has successfully executed the interaction instruction in response to the interaction instruction of the device under test, the method further includes: Obtaining the historical assertion results of the historical interaction instructions having an instruction dependency relationship with the interaction instruction based on the semantic relationship between several historical interaction instructions executed by the device under test and the interaction instruction; In response to the historical assertion result indicating that the device under test successfully executed the historical interaction instruction, perform the step of generating an assertion task for verifying whether the device under test successfully executed the interaction instruction in response to the interaction instruction of the device under test and subsequent steps.

7. The method according to claim 1, wherein In the case where the execution of the previous interaction instruction by the device under test is interrupted and the current interaction instruction is executed, the method further includes: Saving the generation progress of the assertion result for the previous interaction instruction and generating the assertion result for the device under test to execute the current interaction instruction; In response to the device under test resuming the execution of the previous interaction instruction, continuing to generate the assertion result for the device under test to execute the previous interaction instruction based on the generation progress.

8. An interactive assertion device, characterized in that, It includes: A generation module, configured to generate an assertion task for verifying whether the device under test successfully executes the interaction instruction in response to the interaction instruction of the device under test; A decomposition module, configured to decompose, based on the assertion task, to obtain subtasks of several detection types; wherein, the several detection types correspond to the hardware types corresponding to the execution of the interaction instruction by the device under test; An acquisition module, configured to acquire the hardware response data of the device under test for each of the subtasks during the execution of the interaction instruction by the device under test; An analysis module, configured to analyze the hardware response data of each of the subtasks based on a first large language model to obtain the assertion result for the device under test to execute the interaction instruction.

9. An electronic device, characterized in that, It includes a mutually coupled memory and a processor, and program instructions are stored in the memory, and the processor is configured to execute the program instructions to implement the interaction assertion method according to any one of claims 1 to 7.

10. An interactive assertion system, characterized in that, It includes a device under test and an assertion device, and the assertion device at least includes a data acquisition device, an instruction output device, and a data processing device. The instruction output device is configured to output an interaction instruction for interacting with the device under test, the data acquisition device is configured to acquire the hardware response data of the device under test, and the data processing device is the electronic device according to claim 9.

11. The interactive assertion system according to claim 10, characterized in that, The data processing device runs an interaction assertion model and a data statistics model. The interaction assertion model is configured to generate the assertion result for the device under test to execute the interaction instruction, and the data statistics model is configured to count the assertion results of the interaction instruction and generate feedback data for optimizing the interaction assertion model based on the statistical results.

12. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, the interaction assertion method according to any one of claims 1 to 7 is implemented.

Images