Intelligent agent testing system and method, electronic equipment, storage medium and computer program product

Abstract

The invention discloses an agent testing system and method, electronic equipment, a storage medium and a computer program product. The intelligent agent testing system comprises a first subsystem and a second subsystem, the first subsystem comprises a first module and a second module; the first module is used for acquiring a test script corresponding to a to-be-tested agent, executing the test script and controlling the second module to execute one or more test operations corresponding to the to-be-tested agent in the process of executing the test script; and the second module is used for executing the test operation based on the control of the first module.

Description

Technical Field

[0001] The present application relates to the field of testing, and in particular to an intelligent agent testing system, method, electronic device, storage medium and computer program product. Background Art

[0002] At present, in the ever-changing wave of artificial intelligence, the rapid rise of large model technology, especially multimodal large model technology, has deeply penetrated into all walks of life and led a new round of scientific and technological revolution. This trend has spawned a series of intelligent products (i.e., intelligent agents) based on large models, including but not limited to terminal applications, smart devices, digital humans, robots, etc., which are gradually changing people's lifestyles and work patterns. Therefore, how to automatically initiate test operations for intelligent agents has become an urgent problem to be solved. Summary of the invention

[0003] To solve related technical problems, the embodiments of the present application provide an intelligent agent testing system, method, electronic device, storage medium and computer program product.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The embodiment of the present application provides an intelligent agent testing system, including: a first subsystem; the first subsystem includes: a first module and a second module; wherein,

[0006] The first module is used to obtain a test script corresponding to the intelligent agent to be tested, execute the test script, and in the process of executing the test script, control the second module to execute one or more test operations corresponding to the intelligent agent to be tested;

[0007] The second module is used to perform the test operation based on the control of the first module.

[0008] In the above scheme, the test operation includes one or more of the following:

[0009] Controlling one or more first devices to start audio recording;

[0010] controlling one or more second devices to start video recording;

[0011] Controlling the mechanical test tool to perform one or more test actions;

[0012] Play one or more voice commands;

[0013] controlling one or more first devices to stop audio recording;

[0014] Control one or more second devices to stop video recording; wherein,

[0015] Different first devices / second devices correspond to different recording objects, and the recording objects include one or more marked points / marked areas on the intelligent entity to be tested and / or the mechanical testing tool.

[0016] In the above scheme, the agent testing system further includes: a second subsystem;

[0017] The first module is further used to control the second subsystem to collect multimodal data of the intelligent agent to be tested during the execution of the test script, wherein the multimodal data is generated by the second module executing the test operation;

[0018] The second subsystem is used to collect the multimodal data based on the control of the first module.

[0019] In the above scheme, the first subsystem further includes: a third module; the intelligent agent testing system further includes: a third subsystem;

[0020] The third module is used to obtain the multimodal data collected by the second subsystem and send the multimodal data to the third subsystem;

[0021] The third subsystem is used to receive the multimodal data sent by the third module and present the multimodal data and / or analysis data corresponding to the multimodal data; wherein,

[0022] The multimodal data includes one or more of the following:

[0023] audio data of one or more recorded objects;

[0024] video data of one or more recorded objects;

[0025] Optical motion data of one or more recording objects, wherein the optical motion data includes position information and time information corresponding to the position information.

[0026] In the above scheme, the third subsystem is specifically used to present the multimodal data and / or the analysis data corresponding to the multimodal data through multiple display areas, and different display areas correspond to different data features, and the data features include one or more of the data type, test indicators, and recording objects.

[0027] In the above scheme, the agent testing system further includes: a fourth subsystem;

[0028] The fourth subsystem is used to determine the clock deviation between the multiple display areas;

[0029] The third subsystem is also used to perform clock synchronization between the multiple display areas based on the clock deviation between the multiple display areas determined by the fourth subsystem, during the process of presenting the multimodal data and / or analysis data corresponding to the multimodal data through the multiple display areas.

[0030] In the above solution, the fourth subsystem is further used for:

[0031] Based on the multimodal data and / or analysis data corresponding to the multimodal data, determine an event list of the intelligent agent to be tested, wherein the event list includes multiple event information and time information corresponding to each event information, the event information includes one or more of a recording object, an event type, and event description information, and the event includes one or more of a behavior event, a voice event, and a video event;

[0032] Determine the correlation between the multiple event information, and based on the correlation between the multiple event information and one or more preset test indicators, perform performance analysis on the intelligent agent to be tested to obtain analysis results, wherein the one or more test indicators are associated with the performance of the intelligent agent to be tested.

[0033] In the above solution, the first subsystem, the second subsystem, the third subsystem and the fourth subsystem are arranged on one electronic device, or the first subsystem, the second subsystem, the third subsystem and the fourth subsystem are arranged on at least two electronic devices.

[0034] The present application also provides an agent testing method, including:

[0035] The first module of the first subsystem of the agent testing system obtains a test script corresponding to the agent to be tested, executes the test script, and in the process of executing the test script, controls the second module of the first subsystem to execute one or more test operations corresponding to the agent to be tested;

[0036] The second module performs the test operation based on the control of the first module.

[0037] In the above scheme, the test operation includes one or more of the following:

[0038] Controlling one or more first devices to start audio recording;

[0039] controlling one or more second devices to start video recording;

[0040] Controlling the mechanical test tool to perform one or more test actions;

[0041] Play one or more voice commands;

[0042] controlling one or more first devices to stop audio recording;

[0043] Control one or more second devices to stop video recording; wherein,

[0044] Different first devices / second devices correspond to different recording objects, and the recording objects include one or more marked points / marked areas on the intelligent entity to be tested and / or the mechanical testing tool.

[0045] In the above scheme, the method further includes:

[0046] The first module controls the second subsystem of the agent testing system to collect multimodal data of the agent to be tested during the execution of the test script, wherein the multimodal data is generated by the second module executing the test operation;

[0047] The second subsystem collects the multimodal data based on the control of the first module.

[0048] In the above scheme, the method further includes:

[0049] The third module of the first subsystem acquires the multimodal data collected by the second subsystem, and sends the multimodal data to the third subsystem of the intelligent agent testing system;

[0050] The third subsystem receives the multimodal data sent by the third module, and presents the multimodal data and / or analysis data corresponding to the multimodal data; wherein,

[0051] The multimodal data includes one or more of the following:

[0052] audio data of one or more recorded objects;

[0053] video data of one or more recorded objects;

[0054] Optical motion data of one or more recording objects, wherein the optical motion data includes position information and time information corresponding to the position information.

[0055] In the above solution, presenting the multimodal data and / or the analysis data corresponding to the multimodal data includes:

[0056] The third subsystem presents the multimodal data and / or analysis data corresponding to the multimodal data through multiple display areas, and different display areas correspond to different data features, and the data features include one or more of data types, test indicators, and recording objects.

[0057] In the above scheme, the method further includes:

[0058] A fourth subsystem of the agent testing system determines clock skew between the plurality of display regions;

[0059] Based on the clock deviation between the multiple display areas determined by the fourth subsystem, the third subsystem performs clock synchronization between the multiple display areas during the process of presenting the multimodal data and / or analysis data corresponding to the multimodal data through the multiple display areas.

[0060] In the above scheme, the method further comprises:

[0061] The fourth subsystem determines an event list of the intelligent agent to be tested based on the multimodal data and / or analysis data corresponding to the multimodal data, wherein the event list includes multiple event information and time information corresponding to each event information, the event information includes one or more of a recording object, an event type, and event description information, and the event includes one or more of a behavior event, a voice event, and a video event;

[0062] The fourth subsystem determines the correlation between the multiple event information, and based on the correlation between the multiple event information and one or more preset test indicators, performs performance analysis on the intelligent entity to be tested to obtain analysis results, and the one or more test indicators are associated with the performance of the intelligent entity to be tested.

[0063] The present application also provides an electronic device, including: a processor and a memory for storing a computer program that can be run on the processor.

[0064] Wherein, the processor is used to execute the steps of any of the above methods when running the computer program.

[0065] An embodiment of the present application further provides a storage medium having a computer program stored thereon, wherein the computer program implements the steps of any of the above methods when executed by a processor.

[0066] An embodiment of the present application also provides a computer program product, including a computer program, which implements the steps of any of the above methods when executed by a processor.

[0067] The intelligent agent testing system, method, electronic device, storage medium and computer program product provided by the embodiment of the present application, the intelligent agent testing system includes: a first subsystem; the first subsystem includes: a first module and a second module; wherein the first module is used to obtain the test script corresponding to the intelligent agent to be tested, execute the test script, and control the second module to execute one or more test operations corresponding to the intelligent agent to be tested during the execution of the test script; the second module is used to execute the test operation based on the control of the first module. The solution provided by the embodiment of the present application, through the intelligent agent testing system, can automatically obtain the test script corresponding to the intelligent agent to be tested, automatically execute the test script, and can automatically initiate the test operation for the intelligent agent according to the test requirements (i.e. according to the test script) during the execution of the test script, such as starting and / or stopping audio recording, starting and / or stopping video recording, playing a specific voice, and a specific action of a mechanical test tool (such as a mechanical arm, etc.) One or more of the test operation behaviors, thereby significantly reducing the manual operation in the intelligent agent testing process, reducing the test deviation caused by human factors, and improving the efficiency and accuracy of the intelligent agent testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a schematic diagram of the structure of an intelligent agent testing system according to an embodiment of the present application;

[0069] Figure 2 This is a schematic diagram of another intelligent agent testing system structure according to an embodiment of the present application;

[0070] Figure 3 This is a flow chart of the intelligent agent testing method according to an embodiment of the present application;

[0071] Figure 4 This is a hardware deployment diagram of the multi-source information acquisition subsystem for this application example;

[0072] Figure 5 This is a schematic diagram of the structure of the test automation subsystem for this application example;

[0073] Figure 6 A visualization diagram of the partitions of the information aggregation subsystem for the application example of this application;

[0074] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0075] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0076] At present, how to scientifically and professionally evaluate the performance indicators of key functions of intelligent products (i.e., intelligent entities) such as terminal applications, smart devices, digital humans, robots, etc. to ensure that they can provide users with excellent service experience has become an important issue that operators urgently need to solve in the process of digital transformation.

[0077] In the related technologies, operators can adopt the following three solutions in the field of intelligent testing such as smart homes and robots:

[0078] Solution 1, manual evaluation solution; this solution focuses on verifying the basic functions and business logic of the intelligent agent, such as input / output, user interaction, data processing, etc. Taking the smart speaker as an example, manual evaluation can comprehensively test whether it can accurately recognize voice commands, play music smoothly, and accurately query the weather. Through manual operation and evaluation, this solution can ensure that the product's performance indicators (such as response time, stability, etc.) meet the preset standards.

[0079] Solution 2, semi-automated testing solution; this solution uses advanced testing tools, such as programmable robotic arms and intelligent terminal automatic control tools, to automate some operations, so that action instructions can be issued to the intelligent agent efficiently and accurately. At the same time, the behavior of the intelligent agent can be recorded through full video recording, and then professionals can perform video analysis to calculate key performance indicators such as latency and stability. This solution can improve test efficiency to a certain extent while ensuring test accuracy.

[0080] Solution 3, intelligent testing solution: This solution targets specific behavior scenarios of intelligent entities and combines the latest artificial intelligence recognition technologies, such as speech recognition, optical character recognition (OCR) and target detection algorithms, to achieve automated evaluation of key business indicators. This solution not only improves the intelligence level of testing, but also ensures the accuracy and reliability of evaluation results, providing strong technical support for operators.

[0081] However, the above solutions have at least the following three problems:

[0082] Question 1: Although the above solution 1 (i.e., manual evaluation mode) is intuitive and direct, it relies on a large amount of human resources, and the test results are often affected by the subjective factors of the testers, which can easily lead to deviations in the results. The limitations of this mode are particularly prominent in tests involving key indicators such as long-term performance and stability; for example, when evaluating the timeliness of a robot's response to human commands, the tester needs to accurately correlate the time difference between the voice command and the robot's action, which undoubtedly increases the difficulty and complexity of the analysis.

[0083] Question 2: Although the above solution 2 (i.e., semi-automatic test mode) realizes the automation of the test process to a certain extent, it still relies on manual analysis to determine the test results, and cannot meet the needs of large-scale, high-efficiency testing. Taking the magnetic performance evaluation of smart doors and windows as an example, although the opening and closing operations of doors and windows can be realized through simulation tools, manual intervention is still required to analyze the time delay between the operation of the robotic arm and the reception of messages on the mobile phone, which undoubtedly limits the efficiency and accuracy of the test.

[0084] Question 3: Although the above solution 3 (i.e., intelligent test mode) uses artificial intelligence recognition technology, it relies on strict rules and preset conditions and is unable to handle abnormal situations or unexpected behaviors. Especially in multimodal test scenarios, each new test scenario requires the rules to be re-formulated and adapted, which not only increases the consumption of human resources, but also increases the complexity and uncertainty of the test.

[0085] In summary, with the rapid development of large model technology, multimodal agents have become a new trend in the development of the industry. However, the testing tools of related technologies still have obvious shortcomings in terms of automated evaluation, cross-modal, and cross-platform testing. These problems not only restrict the innovation and development of smart products, but also pose a severe challenge to the digital transformation of operators. Therefore, the development of a new test tool that can efficiently, accurately, and intelligently evaluate the performance of multimodal agents has become an urgent problem to be solved in the industry.

[0086] Based on this, in various embodiments of the present application, through the intelligent agent testing system, it is possible to automatically obtain the test script corresponding to the intelligent agent to be tested, automatically execute the test script, and in the process of executing the test script, automatically initiate test operations for the intelligent agent according to the test requirements (i.e., according to the test script), such as starting and / or stopping audio recording, starting and / or stopping video recording, playing specific voices, specific movements of mechanical testing tools (such as robotic arms, etc.), and other test operation behaviors (i.e., one or more of the test operation behaviors), thereby significantly reducing manual operations in the intelligent agent testing process, reducing test deviations caused by human factors, and improving the efficiency and accuracy of the intelligent agent testing, thereby at least solving the above-mentioned problems 1 to 3.

[0087] Specifically, the present application embodiment provides an intelligent agent testing system, such as Figure 1 As shown, the system includes: a first subsystem 101; the first subsystem 101 includes: a first module 1011 and a second module 1012; wherein,

[0088] The first module 1011 is used to obtain a test script corresponding to the intelligent agent to be tested, execute the test script, and in the process of executing the test script, control the second module 1012 to execute one or more test operations corresponding to the intelligent agent to be tested;

[0089] The second module 1012 is used to perform the test operation based on the control of the first module 1011.

[0090] In actual application, the specific type of the intelligent agent to be tested can be set according to the test requirements, and the embodiment of the present application does not limit this. Exemplarily, the intelligent agent to be tested may include intelligent products such as terminal applications, intelligent devices, digital humans, robots, and intelligent business systems; the intelligent devices may include smart home devices, such as smart speakers, smart doors and windows, etc.; the intelligent business system may include a smart large-screen set-top box system, etc. The smart large-screen set-top box system may specifically include a set-top box remote control, a smart set-top box, and a display.

[0091] In actual application, the first module 1011 can read (i.e., obtain) the test script corresponding to the intelligent agent to be tested locally, or obtain the test script from the cloud, or obtain the test script from a specific storage space inside or outside the intelligent agent testing system. The embodiment of the present application does not limit the specific method for the first module 1011 to obtain the test script.

[0092] In actual application, the first module 1011 controls the second module 1012 to execute one or more test operations corresponding to the intelligent entity to be tested during the execution of the test script, which means that the test script can contain relevant content (such as control instructions and related codes, etc.) for controlling the second module 1012 to execute one or more test operations corresponding to the intelligent entity to be tested, or it can be understood that the first module 1011 can control the second module 1012 to execute one or more test operations corresponding to the intelligent entity to be tested according to the test script.

[0093] Among them, the test script can be pre-written according to the test requirements or pre-generated in a specific way, and can specifically include some specific test case codes, etc. The embodiment of the present application does not limit the specific type and generation method of the test script, as long as its function is realized. It can be understood that the test script can reflect the test requirements, and the first module 1011 controls the second module 1012 to execute one or more test operations corresponding to the intelligent agent to be tested according to the test script. It can be understood that the first module 1011 controls the second module 1012 to execute one or more test operations corresponding to the intelligent agent to be tested according to the test requirements. Here, the one or more test operations refer to at least one test operation, and the multiple test operations refer to at least two test operations. In addition, when the first module 1011 controls the second module 1012 to execute one or more test operations corresponding to the intelligent agent to be tested, a specific control instruction can be sent to the second module 1012; the second module 1012 executes the test operation based on the control of the first module 1011, which means that the second module 1012 can execute the test operation based on the control instruction of the first module 1011, or it can be understood that the second module 1012 can execute the test operation by responding to the control instruction of the first module 1011.

[0094] In actual application, the test operation may include one or more of the following (that is, the test operation may include at least one of the following):

[0095] Controlling at least one (i.e., one or more first devices) to start audio recording;

[0096] Controlling at least one (i.e., one or more second devices) to start video recording;

[0097] Controlling the mechanical testing tool to perform at least one testing action (i.e., one or more testing actions);

[0098] Playing at least one (i.e. one or more) voice commands;

[0099] controlling at least one (i.e., one or more first devices) to stop audio recording;

[0100] Control at least one (ie, one or more second devices) to stop video recording.

[0101] Among them, different first devices can correspond to different recording objects, and different second devices can also correspond to different recording objects; the recording object may include at least one marking point / marking area (that is, one or more marking points / marking areas) on the intelligent entity to be tested and / or the mechanical testing tool.

[0102] In actual application, the annotation points may include physical annotation points and / or virtual annotation points. The physical annotation points may include rigid annotation points (which may also be understood as optical annotation points or specific rigid ball devices), etc., and the virtual annotation points may be implemented through data and / or code. The annotation area may include a physical annotation area and / or a virtual annotation area. The physical annotation area may be understood as an area delineated in a physical manner, and the virtual annotation area may be implemented through data and / or code. Here, the annotation points / annotation areas may be pre-set according to test requirements, and / or the annotation points / annotation areas may be configured in the video picture recorded by the second device by manual or machine selection.

[0103] In actual application, it can be understood that the first device has at least an audio recording function. Specifically, the first device may include a recorder, a camera with an audio recording function, etc. The specific name and function of the first device are not limited in the embodiment of the present application. In addition, the first device may communicate with the second module 1012 by wired communication or wireless communication. The wired communication may include communication based on a universal serial bus (USB) interface, etc., and the wireless communication may include communication based on Wi-Fi, communication based on Bluetooth, etc. When the second module 1012 controls the first device to start / stop audio recording, a specific control instruction may be sent to the first device by wired communication or wireless communication. The first device may start / stop audio recording based on the control instruction of the second module 1012 (i.e., respond to the control instruction of the second module 1012).

[0104] In actual application, it can be understood that the second device has at least a video recording function, and the second device can be the same as or different from the first device. Specifically, the second device may include a camera, etc., and the specific name and function of the second device are not limited in the embodiment of the present application. In addition, the second device can communicate with the second module 1012 by wired communication or wireless communication, and the wired communication method may include communication based on a USB interface, etc., and the wireless communication method may include communication based on Wi-Fi, communication based on Bluetooth, etc. When the second module 1012 controls the second device to start / stop video recording, a specific control instruction can be sent to the second device by wired communication or wireless communication, and the second device can start / stop video recording based on the control instruction of the second module 1012 (i.e., responding to the control instruction of the second module 1012).

[0105] In actual application, the mechanical testing tool may include a mechanical arm, etc., and the embodiment of the present application does not limit the specific type of the mechanical testing tool. In addition, the mechanical testing tool may communicate with the second module 1012 by wired communication or wireless communication. The wired communication may include communication based on a USB interface, etc., and the wireless communication may include communication based on Wi-Fi, communication based on Bluetooth, etc. When the second module 1012 controls the mechanical testing tool to perform a test action, a specific control instruction may be sent to the mechanical testing tool by wired communication or wireless communication. The mechanical testing tool may perform the corresponding specific test action based on the control instruction of the second module 1012 (i.e., responding to the control instruction of the second module 1012).

[0106] In actual application, the second module 1012 plays one or more voice instructions, which may include controlling a voice playback component to play the one or more voice instructions, and the voice playback component may be arranged inside or outside the second module 1012. When the voice playback component is arranged outside the second module 1012, the voice playback component may communicate with the second module 1012 via a wired communication method or a wireless communication method; specifically, the second module 1012 may send a specific control instruction to the voice playback component using a wired communication method or a wireless communication method, and the voice playback component may play the corresponding specific voice instruction based on the control instruction of the second module 1012 (i.e., responding to the control instruction of the second module 1012).

[0107] In actual application, it can be seen from the above description that the intelligent agent testing system can also include the first device, the second device, and mechanical testing tools and other hardware devices for performing the testing operations. Based on these hardware devices, the intelligent agent testing system can collect multimodal data of the intelligent agent to be tested.

[0108] Based on this, in one embodiment, the intelligent agent testing system may further include: a second subsystem;

[0109] The first module 1011 may also be used to control the second subsystem to collect multimodal data of the agent to be tested during the execution of the test script, wherein the multimodal data is generated by the second module 1012 executing the test operation;

[0110] Correspondingly, the second subsystem is used to collect the multimodal data based on the control of the first module 1011.

[0111] Here, it can be understood that the second subsystem can include the first device, the second device, and hardware devices such as mechanical testing tools, and the second subsystem can collect the multimodal data through these hardware devices. In addition, when the first module 1011 controls the second subsystem to collect the multimodal data, a specific control instruction can be sent to the second subsystem; the second subsystem collects the multimodal data based on the control of the first module 1011, which means that the second subsystem can collect the multimodal data based on the control instruction of the first module 1011 (i.e., respond to the control instruction of the first module 1011).

[0112] In actual application, the second subsystem can be intelligently constructed based on an optical motion capture system and video acquisition, audio acquisition and other technologies. Accordingly, the multimodal data can include one or more of the following (that is, the multimodal data can include at least one of the following):

[0113] audio data of at least one recording object (i.e., one or more recording objects);

[0114] video data of one or more recorded objects;

[0115] Optical motion data of one or more recorded objects, wherein the optical motion data includes position information and time information corresponding to the position information (such as a timestamp, etc.).

[0116] In actual application, after collecting the multimodal data, the intelligent agent testing system can present the multimodal data and / or analysis data corresponding to the multimodal data.

[0117] Based on this, in one embodiment, the first subsystem 101 may further include: a third module; the intelligent agent testing system may further include: a third subsystem;

[0118] The third module is used to obtain the multimodal data collected by the second subsystem and send the multimodal data to the third subsystem;

[0119] The third subsystem is used to receive the multimodal data sent by the third module, and present the multimodal data and / or analysis data corresponding to the multimodal data.

[0120] In actual application, the third module may have wired or wireless communication capabilities such as Gigabit Ethernet, Wi-Fi, and Bluetooth. Based on these communication capabilities, the third module may synchronize the multimodal data such as the video stream (i.e., the video data) and the audio stream (i.e., the audio data) obtained from the second subsystem to the third subsystem in real time through different protocols, so that the third subsystem can display (i.e., present) the multimodal data and / or the analysis data corresponding to the multimodal data in real time and synchronously.

[0121] In actual application, the specific type of analysis data corresponding to the multimodal data can be set according to the test requirements, and the embodiments of the present application do not limit this. For example, the analysis data corresponding to the audio data may include image data such as spectrum and waveform; the analysis data corresponding to the audio data and video data of the same recording object may include image data related to audio and video synchronization.

[0122] In actual application, taking into account that the multimodal data and / or the analysis data corresponding to the multimodal data may contain different data characteristics such as different data types, different test indicators, and different recording objects, therefore, in order to further improve the efficiency and accuracy of the intelligent agent test, the third subsystem can present the multimodal data and / or the analysis data corresponding to the multimodal data in partitions.

[0123] Based on this, in one embodiment, the third subsystem can be specifically used to present the multimodal data and / or the analysis data corresponding to the multimodal data through multiple display areas, and different display areas correspond to different data features, and the data features include one or more of a data type, a test indicator, and a recording object (that is, the data features may include at least one of a data type, a test indicator, and a recording object).

[0124] Wherein, in actual application, the specific number of the display areas and the specific content presented in each display area can be set according to the test requirements, and the embodiment of the present application does not limit this. Exemplarily, when the intelligent body to be tested includes the above-mentioned intelligent large-screen set-top box system (the intelligent large-screen set-top box system can specifically include a set-top box remote control, an intelligent set-top box, and a display and other devices), the third subsystem may include four display areas: the first display area can be used to play (i.e., present) the action of the mechanical arm (i.e., the mechanical test tool) captured by camera 1 (i.e., the second device) in real time, the second display area can be used to play the picture of the device under test (i.e., the display) captured by camera 2 (i.e., another second device) and the audio and video synchronization benchmark program display picture (i.e., audio and video synchronization related image data) in real time, the third display area can be used to present optical action data in real time and synchronously, that is, present the position information and timestamp information (i.e., time information) of each annotation point, and the fourth display area can be used to display (i.e., present) the spectrum, waveform and other image data of the recorded audio data in real time.

[0125] In actual application, taking into account that there may be clock deviations between the multimodal data collected by different devices, correspondingly, there may also be clock deviations between the multiple display areas. Therefore, in order to further improve the efficiency and accuracy of the intelligent agent test, the intelligent agent test system can perform clock synchronization between the multiple display areas.

[0126] Based on this, in one embodiment, the intelligent agent testing system may further include: a fourth subsystem;

[0127] The fourth subsystem is used to determine the clock deviation between the multiple display areas;

[0128] The third subsystem is also used to perform clock synchronization between the multiple display areas based on the clock deviation between the multiple display areas determined by the fourth subsystem, during the process of presenting the multimodal data and / or analysis data corresponding to the multimodal data through the multiple display areas.

[0129] Among them, in actual application, the specific method by which the fourth subsystem determines the clock deviation between the multiple display areas can be set according to the test requirements, and the embodiment of the present application does not limit this. Exemplarily, the fourth subsystem can use a preset specific formula to calculate the clock deviation between the multiple display areas; or, before the intelligent agent test system starts the intelligent agent test, the clock deviation between the multiple display areas can be calculated by manual pre-testing, and the calculation results can be stored in the fourth subsystem, that is, the fourth subsystem can determine the clock deviation between the multiple display areas based on the locally stored clock deviation calculation results.

[0130] In addition, when the fourth subsystem determines the clock deviation between the multiple display areas, it can also determine the clock deviation between the multimodal data and the corresponding data acquisition device, such as the clock deviation between the optical motion data and the corresponding camera (i.e., the second device mentioned above), and perform clock synchronization, thereby further improving the efficiency and accuracy of the intelligent agent test.

[0131] In actual application, the intelligent agent testing system can also perform performance analysis on the intelligent agent to be tested based on the multimodal data and / or analysis data corresponding to the multimodal data.

[0132] Based on this, in one embodiment, the fourth subsystem may also be used for:

[0133] Based on the multimodal data and / or the analysis data corresponding to the multimodal data, determine an event list of the intelligent agent to be tested, wherein the event list includes multiple event information and time information corresponding to each event information (such as a start timestamp, an end timestamp, etc.), the event information includes one or more of a recording object, an event type, and event description information (that is, the event information may include at least one of a recording object, an event type, and event description information), and the event includes one or more of a behavior event, a voice event, and a video event (that is, the event may include at least one of a behavior event, a voice event, and a video event);

[0134] Determine the correlation between the multiple event information, and based on the correlation between the multiple event information and at least one preset test indicator (i.e., one or more test indicators), perform performance analysis on the intelligent agent to be tested to obtain analysis results, wherein the one or more test indicators are associated with the performance of the intelligent agent to be tested.

[0135] In actual application, the fourth subsystem can calculate the value of one or more test indicators based on the correlation between the multiple event information and one or more preset test indicators during the performance analysis of the intelligent agent to be tested based on the correlation between the multiple event information; in other words, the analysis result can include the value of one or more test indicators.

[0136] In actual application, the specific type of the test indicator can be set according to the test requirements, and the embodiment of the present application does not limit this. For example, when the intelligent body to be tested includes the above-mentioned intelligent large-screen set-top box system (the intelligent large-screen set-top box system can specifically include a set-top box remote control, an intelligent set-top box, and a display and other devices), the test indicator can be associated with the voice control performance of the intelligent large-screen set-top box system. For example, the test indicator can include key performance indicators such as voice remote control retrieval feedback delay and set-top box remote control screen switching delay.

[0137] In actual application, the specific content of the event list and the specific expression form of the event information of the behavioral event, voice event, and video event can be set according to the test requirements, and the embodiments of the present application do not limit this. For example, when the intelligent body to be tested includes the above-mentioned intelligent large-screen set-top box system (the intelligent large-screen set-top box system can specifically include a set-top box remote control, an intelligent set-top box, and a display and other devices), the event list can include multiple groups of behavioral events of a specific rigid body annotation point, one or more voice command events, one or more voice response events, a video rapid change event of a specific annotation area, and related information of the video preservation unchanged event (i.e., event information and time information). For example, the relevant information of a video rapid change event can be expressed as "annotated area 1, video rapid change, start timestamp, end timestamp".

[0138] In practical application, it can be seen from the above description that if Figure 2 As shown, the intelligent agent test system may specifically include: a second subsystem 201, a first subsystem 101, a third subsystem 203 and a fourth subsystem 204; the first subsystem 101 may specifically include a first module 1011, a second module 1012 and a third module 202. The second subsystem 201 may also be referred to as a multi-source information acquisition subsystem, etc.; the first subsystem 101 may also be referred to as a test automation subsystem, etc., the first module 1011 may also be referred to as a test engine, etc., the second module 1012 may also be referred to as a test interaction module, etc., the third module 202 may also be referred to as an information interaction module, etc.; the third subsystem 203 may also be referred to as an information convergence subsystem, etc.; the fourth subsystem 204 may also be referred to as an intelligent analysis subsystem, etc. The embodiments of the present application do not limit the specific names of these subsystems and modules, as long as the corresponding functions are realized.

[0139] In actual application, the first subsystem 101, the second subsystem 201, the third subsystem 203 and the fourth subsystem 204 can be set on an electronic device (such as a server, etc.), that is, they are set on the same electronic device, that is, the intelligent agent test system can be set on an electronic device; or, the first subsystem 101, the second subsystem 201, the third subsystem 203 and the fourth subsystem 204 can be set on at least two electronic devices, that is, the intelligent agent test system can be set on at least two electronic devices. Among them, the first subsystem 101, the second subsystem 201, the third subsystem 203 and the fourth subsystem 204 can be specifically implemented by the processor in the corresponding electronic device in combination with the communication interface.

[0140] It should be noted that: when the intelligent agent testing system provided in the above embodiment performs intelligent agent testing, it only uses the division of the above-mentioned program modules (i.e., the first subsystem 101, the second subsystem 201, the third subsystem 203 and the fourth subsystem 204, as well as the first module 1011, the second module 1012 and the third module 202) as an example. In actual application, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the intelligent agent testing system can be divided into different program modules to complete all or part of the processing described above.

[0141] Accordingly, the present application also provides an intelligent agent testing method, such as Figure 3 As shown, the method includes:

[0142] Step 301: The first module of the first subsystem of the agent testing system obtains a test script corresponding to the agent to be tested, executes the test script, and in the process of executing the test script, controls the second module of the first subsystem to execute one or more test operations corresponding to the agent to be tested;

[0143] Step 302: The second module performs the test operation based on the control of the first module.

[0144] The test operation may include one or more of the following:

[0145] Controlling one or more first devices to start audio recording;

[0146] controlling one or more second devices to start video recording;

[0147] Controlling the mechanical test tool to perform one or more test actions;

[0148] Play one or more voice commands;

[0149] controlling one or more first devices to stop audio recording;

[0150] Controlling one or more second devices to stop video recording.

[0151] Among them, different first devices / second devices correspond to different recording objects, and the recording objects include one or more marked points / marked areas on the intelligent entity to be tested and / or the mechanical testing tool.

[0152] In one embodiment, the method may further include:

[0153] The first module controls the second subsystem of the agent testing system to collect multimodal data of the agent to be tested during the execution of the test script, wherein the multimodal data is generated by the second module executing the test operation;

[0154] The second subsystem collects the multimodal data based on the control of the first module.

[0155] In one embodiment, the method may further include:

[0156] The third module of the first subsystem acquires the multimodal data collected by the second subsystem, and sends the multimodal data to the third subsystem of the intelligent agent testing system;

[0157] The third subsystem receives the multimodal data sent by the third module, and presents the multimodal data and / or analysis data corresponding to the multimodal data.

[0158] The multimodal data may include one or more of the following:

[0159] audio data of one or more recorded objects;

[0160] video data of one or more recorded objects;

[0161] Optical motion data of one or more recording objects, wherein the optical motion data includes position information and time information corresponding to the position information.

[0162] In one embodiment, presenting the multimodal data and / or analysis data corresponding to the multimodal data may include:

[0163] The third subsystem presents the multimodal data and / or analysis data corresponding to the multimodal data through multiple display areas, and different display areas correspond to different data features, and the data features include one or more of data types, test indicators, and recording objects.

[0164] In one embodiment, the method may further include:

[0165] A fourth subsystem of the agent testing system determines clock skew between the plurality of display regions;

[0166] Based on the clock deviation between the multiple display areas determined by the fourth subsystem, the third subsystem performs clock synchronization between the multiple display areas during the process of presenting the multimodal data and / or analysis data corresponding to the multimodal data through the multiple display areas.

[0167] In one embodiment, the method may further include:

[0168] The fourth subsystem determines an event list of the intelligent agent to be tested based on the multimodal data and / or analysis data corresponding to the multimodal data, wherein the event list includes multiple event information and time information corresponding to each event information, the event information includes one or more of a recording object, an event type, and event description information, and the event includes one or more of a behavior event, a voice event, and a video event;

[0169] The fourth subsystem determines the correlation between the multiple event information, and based on the correlation between the multiple event information and one or more preset test indicators, performs performance analysis on the intelligent entity to be tested to obtain analysis results, and the one or more test indicators are associated with the performance of the intelligent entity to be tested.

[0170] Here, it should be noted that the intelligent agent testing method provided in the above embodiment and the intelligent agent testing system embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0171] The intelligent agent testing system and method provided by the embodiment of the present application, the intelligent agent testing system includes: a first subsystem; the first subsystem includes: a first module and a second module; wherein the first module is used to obtain the test script corresponding to the intelligent agent to be tested, execute the test script, and control the second module to execute one or more test operations corresponding to the intelligent agent to be tested during the execution of the test script; the second module is used to execute the test operation based on the control of the first module. The solution provided by the embodiment of the present application, through the intelligent agent testing system, can automatically obtain the test script corresponding to the intelligent agent to be tested, automatically execute the test script, and can automatically initiate the test operation for the intelligent agent according to the test requirements (i.e., according to the test script) during the execution of the test script, such as starting and / or stopping audio recording, starting and / or stopping video recording, playing a specific voice, and one or more of the test operation behaviors of a mechanical test tool (such as a mechanical arm, etc.), thereby significantly reducing the manual operation in the intelligent agent testing process, reducing the test deviation caused by human factors, and improving the efficiency and accuracy of the intelligent agent testing, thereby at least being able to solve the above-mentioned problems 1 to 3.

[0172] The present application is described in further detail below in conjunction with application examples.

[0173] In this application example, the first subsystem is called a test automation subsystem, the second subsystem is called a multi-source information acquisition subsystem, the third subsystem is called an information convergence subsystem, the fourth subsystem is called an intelligent analysis subsystem, the first module is called a test engine, the second module is called a test interaction module, and the third module is called an information interaction module.

[0174] In response to the current difficulty in automating the interaction process and the analysis of test results in multimodal testing scenarios, this application example proposes a testing solution for multimodal agents to achieve automated testing of agents based on multimodal information association analysis. Furthermore, based on multimodal information processing in the test scenario, a set of visual agent testing devices (i.e., the above-mentioned agent testing system) is provided to users to achieve automated test initiation and performance indicator analysis capabilities.

[0175] The intelligent agent test system includes a multi-source information acquisition subsystem, a test automation subsystem, an information convergence subsystem and an intelligent analysis subsystem. The following takes an intelligent large-screen set-top box system as an example to describe the functions of the intelligent agent test system in detail. The intelligent large-screen system under test (i.e., the intelligent agent to be tested) includes a set-top box remote control, an intelligent set-top box, and a display. A typical test indicator of the system is to test the voice control performance of the system, such as the voice remote control retrieval feedback delay, the set-top box remote control screen switching delay and other key performance indicators.

[0176] First, if Figure 4 As shown, the multi-source information acquisition subsystem is mainly constructed by optical motion capture system, high-precision camera (i.e. the above-mentioned second device), voice collector (i.e. the above-mentioned first device) and other technologies. Among them, the optical motion capture system can be applied to the test tool (i.e. the above-mentioned mechanical test tool, such as a mechanical arm, etc.) and the device under test (such as a set-top box remote control, etc.), and is mainly used to accurately obtain the change of the object position information (i.e. the above-mentioned optical motion data) of the hardware device. The optical motion capture system is mainly based on optical principles and computer vision technology. The specific understanding can refer to the relevant technology. This application example does not elaborate on the principle of the optical motion capture system. The optical motion capture system can capture the mark points set on the target object from different angles at the same time through multiple high-speed cameras (usually called optical motion capture lenses, i.e. the above-mentioned second device) above the test field (which can be understood as a specific test area) (usually a mark point made of special reflective materials, which can be understood as a rigid body mark point). The mark point (i.e. the above-mentioned mark point) can be placed on the test tool to record the position information of the test initiator, or it can be placed on the device under test to monitor its motion behavior. The data accuracy obtained can usually reach the millimeter level. The optical motion capture system can perform joint analysis and calculation on the information fed back by each camera (i.e., the second device mentioned above) in real time, output the location information and time information of each marker point, and output the result data to the intelligent analysis subsystem in the test field. In addition to the optical motion capture system, the multi-source information acquisition subsystem can also include information acquisition modules such as video acquisition and audio acquisition, which are controlled and used by the test automation subsystem.

[0177] In this application example, the test automation subsystem is mainly responsible for automatically initiating some specified test operations according to the test requirements. For the typical information input / output sources of intelligent agents based on large models, the test automation subsystem can be implemented through a programmable industrial computer / portable development board and other devices. Figure 5 As shown in the figure, the test automation subsystem mainly involves the following three modules:

[0178] The test engine module is responsible for interpreting (i.e. executing) pre-written test case codes;

[0179] The information interaction module has the connection capabilities of Gigabit Ethernet, Wi-Fi and Bluetooth, and can synchronize video streams, audio streams and other information (i.e. the above-mentioned multimodal data) to the information convergence subsystem in real time through different protocols;

[0180] The test interaction module is responsible for interacting with the multi-source information acquisition sources (i.e., the multi-source information acquisition subsystem) connected to the local machine. These information acquisition sources may include but are not limited to: one or more USB cameras (i.e., the above-mentioned second device), a voice output module (i.e., the above-mentioned voice playback component), a voice input module (i.e., the above-mentioned first device) and a robotic arm.

[0181] Among them, the test interaction module may include a camera control module for controlling a USB camera so that the USB camera can be responsible for real-time video recording of robotic arm movements, tested intelligent body movements and other behaviors; the test interaction module may also include a voice interaction module for controlling a voice output module and a voice input module so that the voice output module can be responsible for sending voice commands to the intelligent body; the test interaction module may also include a robotic arm control module for controlling the robotic arm to repeatedly complete specified test actions according to a pre-compiled program.

[0182] In this application example, the test process of the test automation subsystem may include the following steps:

[0183] Step 1: The tester prepares for the test by placing two cameras (i.e., the second device mentioned above), a set-top box, and a large-screen display (i.e., the intelligent agent to be tested mentioned above) in a specific test area (i.e., the test field), and prepares a robotic arm fixture by placing no less than three optically marked point rigid body sphere devices on the robotic arm fixture;

[0184] Here, the audio-visual synchronization benchmark program can be played through a personal computer (PC) or other devices such as speakers with screens, and the screen is placed next to the large screen under test (the content presented on the screen can be used as part of the multimodal data of the intelligent agent under test, and can also be used as analysis data corresponding to the above multimodal data);

[0185] Step 2: The test engine reads the test script and starts executing the test program;

[0186] Step 3: The test engine controls the two cameras by controlling the test interaction module to start recording audio and video respectively. The camera that captures the large screen data needs to capture the image of the audio and video synchronization benchmark program simultaneously, and the other camera is responsible for capturing the entire movement of the robotic arm.

[0187] Step 4: The test engine controls the test interaction module to control the robot arm to press the voice input button of the set-top box remote control and maintain the pressing state;

[0188] Step 5: The test engine controls the test interaction module to make the test interaction module play a pre-recorded sound file through the voice output module, such as the voice of "Please play the first episode of the first season of Joy of Life";

[0189] Step 6: After the voice command is played, the test engine controls the test interaction module to control the robotic arm to release the button of the set-top box remote control (i.e., control the robotic arm to leave the pressing state);

[0190] Step 7: The test engine controls the test interaction module to reset each device, stop recording audio and video, and end this round of testing.

[0191] In this application example, all relevant audio and video information in the above test process (i.e., the above multimodal data and / or the analysis data corresponding to the multimodal data) can be synchronized to the information convergence subsystem in real time for synchronous display. The information convergence subsystem is mainly responsible for visually analyzing and displaying the collected multi-source information (i.e., the above multimodal data and / or the analysis data corresponding to the multimodal data) according to the test requirements. Specifically, Figure 6 As shown, the information convergence subsystem can present the following four areas:

[0192] Area 1, the test tool recording area, is used to play the robot arm movements captured by camera 1 in real time;

[0193] Area 2, the device under test recording area, is used to play the device under test image captured by camera 2 and the audio and video synchronization benchmark program display image in real time;

[0194] Area 3, the rigid body annotation point location area, is used to synchronize the location information and timestamp information of each annotation point collected by the optical motion capture system in real time;

[0195] Area 4, the audio analysis area, is used to analyze the recorded audio information in real time and display the spectrum, waveform and other images in real time.

[0196] In this application example, the intelligent analysis subsystem has the information convergence subsystem test configuration function, which is used to select one or more intelligent analysis frames (i.e., the above-mentioned marked areas) in the test tool recording area (i.e., the above-mentioned area 1) and the device under test recording area (i.e., the above-mentioned area 2) after displaying the images in each area before starting the test. The selection process can be performed manually (for example, the tester uses the editing tools provided by the intelligent analysis subsystem to drag and drop in the corresponding video screen to form the corresponding focus frame) or automatically by the machine. Afterwards, intelligent recognition of the test can be carried out for the intelligent analysis frame, for example, Figure 6The smart box (i.e., smart analysis box) in the recording area of ​​the test tool can be used to analyze the physical position and motion information of the robot arm in real time through video. Before the formal test, the clock deviation between the rigid body position information and the test camera can be calculated through the image motion changes of the box. For example, the smart box in the recording area of ​​the device under test can be used to provide the intelligent recognition system with a focus area related to key event recognition. In the future, correlation analysis can be performed only on the video changes in this area.

[0197] In this application example, it can be seen from the above description that the intelligent analysis subsystem also has a multi-source information clock synchronization function. Specifically, for data such as audio, video, and position information transmitted by multiple subsystems (i.e., the above-mentioned multimodal data), considering that the transmission passes through different devices and channels, and there are certain differences in the amount of data encoding and decoding calculations of each system, before carrying out the evaluation work, the rigid body position information collected by the optical capture system, the sound information decoded by the audio system (i.e., audio information), and the video information transmitted by the camera can be synchronized to calculate the clock compensation parameters of the multi-source system data.

[0198] Among them, for the test tool recording area and the device under test recording area, since they are video streams shot by two cameras on the same main control device, the parameters of various software and hardware, transmission protocols and codecs are consistent, so there is no clock deviation between the two, that is, the deviation coefficient is 0. For the rigid body annotation point position area and the test tool recording area, before the test, the test robot can be used to perform n actions (n ​​is an integer greater than 2) according to the specified fixed action (such as from top to bottom), and each action calculates the picture from video recognition to the rigid body in the smart frame reaching the lowest point (k1 second), and the picture of the lowest position of the rigid body output in the rigid body annotation point position area (k2 second), and calculates the deviation O1 = Σ (k2-k1) / n. For the device under test recording area, the image analysis mechanism can be used to view the pictures of various standard sounds of the benchmark audio and video synchronization program (y1 second), and the picture of the pulse information appearing in the audio analysis area (y2 second), and the test is performed m times (m is an integer greater than 2), and the deviation O2 = Σ (y2-y1) / m. After obtaining the deviation O1 and the deviation O2, the clock of the multi-source information acquisition data of the entire system can be synchronized.

[0199] In this application example, the intelligent analysis subsystem also has a key event identification and analysis function. Specifically, the intelligent analysis subsystem can perform correlation analysis on key events of multi-source information and automatically filter out a variety of key event lists with timestamps from the multi-source system. The list may include but is not limited to the following:

[0200] Multiple groups of behavior events for rigid body annotation points; among them, multiple focus points (i.e., annotation points) can be pre-set, and each focus point outputs events separately, that is, a group of behavior events can contain multiple behavior event information of a annotation point, such as "annotation point 1, spatial motion type, start timestamp, end timestamp";

[0201] Voice command events, such as "voice command content 1, start timestamp, end timestamp", "audio-visual synchronization reference sound 1, start timestamp, end timestamp", "voice response content 1, start timestamp, end timestamp", etc.;

[0202] Video events of the device under test, such as "marking area 1 (i.e., the smart box in the recording area of ​​the device under test), rapid changes in the area video, start timestamp, end timestamp", "marking area 1 (i.e., the smart box in the recording area of ​​the device under test), the area video is saved unchanged, start timestamp, end timestamp", etc.

[0203] In this application example, after obtaining the list of key events, the tester can use the visual arrangement function provided by the intelligent analysis subsystem or a small amount of code, through information such as event type and timestamp, to assist in calculating the clock deviation parameters between the subsystems before the performance test, and can calculate indicators such as the delay and success rate of each key indicator. For example, the large screen switching delay can be calculated based on the start timestamp of the uplink (i.e., upward movement) key event of a specific rigid body standard point (i.e., the marked point), the start timestamp of the regional video rapid change event of the marked area 1 (i.e., the smart box in the recording area of ​​the device under test), and the clock deviation O1; for another example, the set-top box voice control retrieval delay can be calculated based on the end timestamp of the voice command content 1, the start timestamp of the regional video rapid change event of the marked area 1 (i.e., the smart box in the recording area of ​​the device under test), and the clock deviation O2.

[0204] The solution provided by this application example optimizes the multi-source information acquisition and synchronization technology, that is, the multi-source information acquisition subsystem can accurately collect data from multiple sensors and devices, and achieve high-precision data synchronization through technologies such as optical motion capture tools and high-precision cameras; and it realizes automated test execution and control, that is, the test automation subsystem can automatically initiate test operations through the test engine and information interaction module, significantly improve test efficiency, reduce manual intervention, and improve the consistency and repeatability of intelligent agent testing; in addition, it realizes intelligent analysis and performance indicator evaluation, that is, the intelligent analysis subsystem can conduct in-depth analysis of the collected multi-source information, identify key events, and combine clock deviation parameters to calculate performance indicators, such as latency and success rate, so as to realize automated analysis and evaluation of test results.

[0205] From the above description, it can be seen that the solution provided by this application example has the following advantages:

[0206] 1) Significantly reduced manual operations, reduced test deviations caused by human factors, and improved test efficiency and accuracy;

[0207] 2) It can handle more complex test scenarios, meet cross-modal and cross-platform test requirements, and enhance test coverage and adaptability. Especially in the testing of multi-modal intelligent agents, it can be widely used by operators to evaluate the quality of smart devices and services.

[0208] 3) Through the intelligent analysis subsystem, the test results can be automatically analyzed and key performance indicators can be calculated without the need for tedious manual data analysis, which greatly improves the efficiency and accuracy of post-test result processing.

[0209] Based on the hardware implementation of each program module of the above-mentioned intelligent agent testing system, and in order to implement the intelligent agent testing method of the embodiment of the present application, the embodiment of the present application also provides an electronic device, such as Figure 7 As shown, the electronic device 700 includes:

[0210] Communication interface 701, capable of exchanging information with other electronic devices;

[0211] A processor 702, connected to the communication interface 701, to implement information interaction with other electronic devices, and used to execute the method provided by one or more of the above technical solutions when running a computer program;

[0212] The memory 703 stores computer programs that can be executed on the processor 702 .

[0213] Specifically, the processor 702 is used to obtain a test script corresponding to the intelligent agent to be tested, execute the test script, and in the process of executing the test script, perform one or more test operations corresponding to the intelligent agent to be tested.

[0214] In one embodiment, the processor 702 is further used to collect multimodal data of the intelligent agent to be tested during the execution of the test script, and the multimodal data is generated by executing the test operation.

[0215] In one embodiment, the processor 702 is further configured to:

[0216] Acquiring the collected multimodal data;

[0217] Presenting the multimodal data and / or analysis data corresponding to the multimodal data; wherein,

[0218] The multimodal data includes one or more of the following:

[0219] audio data of one or more recorded objects;

[0220] video data of one or more recorded objects;

[0221] Optical motion data of one or more recording objects, wherein the optical motion data includes position information and time information corresponding to the position information.

[0222] In one embodiment, the processor 702 is further used to present the multimodal data and / or analysis data corresponding to the multimodal data through multiple display areas, and different display areas correspond to different data features, and the data features include one or more of a data type, a test indicator, and a recording object.

[0223] In one embodiment, the processor 702 is further configured to:

[0224] determining a clock offset between the plurality of display regions;

[0225] Based on the clock deviation between the multiple display areas, clock synchronization is performed between the multiple display areas during the process of presenting the multimodal data and / or analysis data corresponding to the multimodal data through the multiple display areas.

[0226] In one embodiment, the processor 702 is further configured to:

[0227] Based on the multimodal data and / or analysis data corresponding to the multimodal data, determine an event list of the intelligent agent to be tested, wherein the event list includes multiple event information and time information corresponding to each event information, the event information includes one or more of a recording object, an event type, and event description information, and the event includes one or more of a behavior event, a voice event, and a video event;

[0228] Determine the correlation between the multiple event information, and based on the correlation between the multiple event information and one or more preset test indicators, perform performance analysis on the intelligent agent to be tested to obtain analysis results, wherein the one or more test indicators are associated with the performance of the intelligent agent to be tested.

[0229] It should be noted that the specific processing process of the processor 702 can be understood by referring to the above method, which will not be repeated here.

[0230] Of course, in actual application, the various components in the electronic device 700 are coupled together through the bus system 704. It can be understood that the bus system 704 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 7 Various buses are labeled as bus system 704 .

[0231] The memory 703 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 700. Examples of such data include: any computer program used to operate on the electronic device 700.

[0232] The method disclosed in the above embodiment of the present application can be applied to the processor 702, or implemented by the processor 702. The processor 702 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 702. The above processor 702 may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 702 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 703, and the processor 702 reads the information in the memory 703 and completes the steps of the above method in combination with its hardware.

[0233] In an exemplary embodiment, the electronic device 700 can be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to execute the aforementioned method.

[0234] It can be understood that the memory 703 of the embodiment of the present application can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory described in the embodiments of the present application is intended to include, but is not limited to, these and any other suitable types of memory.

[0235] In an exemplary embodiment, the present application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, a memory 703 storing a computer program, and the computer program can be executed by a processor 702 of an electronic device 700 to complete the steps of any of the aforementioned methods. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0236] In an exemplary embodiment, the embodiment of the present application further provides a computer program product, including a computer program, and the computer program can be executed by the processor 702 of the electronic device 700 to complete the steps of any of the aforementioned methods.

[0237] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0238] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0239] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. An intelligent agent testing system, characterized in that: include: First subsystem; The first subsystem includes: a first module and a second module; wherein, The first module is used to obtain a test script corresponding to the intelligent agent to be tested, execute the test script, and in the process of executing the test script, control the second module to execute one or more test operations corresponding to the intelligent agent to be tested; The second module is used to perform the test operation based on the control of the first module.

2. The system according to claim 1, characterized in that The test operation includes one or more of the following: Controlling one or more first devices to start audio recording; controlling one or more second devices to start video recording; Controlling the mechanical test tool to perform one or more test actions; Play one or more voice commands; controlling one or more first devices to stop audio recording; Control one or more second devices to stop video recording; wherein, Different first devices / second devices correspond to different recording objects, and the recording objects include one or more marked points / marked areas on the intelligent entity to be tested and / or the mechanical testing tool.

3. The system according to claim 2, characterized in that The agent testing system further includes: a second subsystem; The first module is further used to control the second subsystem to collect multimodal data of the intelligent agent to be tested during the execution of the test script, wherein the multimodal data is generated by the second module executing the test operation; The second subsystem is used to collect the multimodal data based on the control of the first module.

4. The system according to claim 3, characterized in that The first subsystem further includes: a third module; the intelligent agent testing system further includes: a third subsystem; The third module is used to obtain the multimodal data collected by the second subsystem and send the multimodal data to the third subsystem; The third subsystem is used to receive the multimodal data sent by the third module and present the multimodal data and / or analysis data corresponding to the multimodal data; wherein, The multimodal data includes one or more of the following: audio data of one or more recorded objects; video data of one or more recorded objects; Optical motion data of one or more recording objects, wherein the optical motion data includes position information and time information corresponding to the position information.

5. The system according to claim 4, characterized in that The third subsystem is specifically used to present the multimodal data and / or analysis data corresponding to the multimodal data through multiple display areas, different display areas correspond to different data features, and the data features include one or more of the data type, test indicators, and recording objects.

6. The system according to claim 5, characterized in that The intelligent agent testing system further includes: a fourth subsystem; The fourth subsystem is used to determine the clock deviation between the multiple display areas; The third subsystem is also used to perform clock synchronization between the multiple display areas based on the clock deviation between the multiple display areas determined by the fourth subsystem, during the process of presenting the multimodal data and / or analysis data corresponding to the multimodal data through the multiple display areas.

7. The system according to claim 6, characterized in that The fourth subsystem is further used for: Based on the multimodal data and / or analysis data corresponding to the multimodal data, determine an event list of the intelligent agent to be tested, wherein the event list includes multiple event information and time information corresponding to each event information, the event information includes one or more of a recording object, an event type, and event description information, and the event includes one or more of a behavior event, a voice event, and a video event; Determine the correlation between the multiple event information, and based on the correlation between the multiple event information and one or more preset test indicators, perform performance analysis on the intelligent agent to be tested to obtain analysis results, wherein the one or more test indicators are associated with the performance of the intelligent agent to be tested.

8. The system according to claim 6, characterized in that The first subsystem, the second subsystem, the third subsystem and the fourth subsystem are arranged on one electronic device, or the first subsystem, the second subsystem, the third subsystem and the fourth subsystem are arranged on at least two electronic devices.

9. An agent testing method, characterized in that: include: The first module of the first subsystem of the agent testing system obtains a test script corresponding to the agent to be tested, executes the test script, and in the process of executing the test script, controls the second module of the first subsystem to execute one or more test operations corresponding to the agent to be tested; The second module performs the test operation based on the control of the first module.

10. The method according to claim 9, characterized in that The test operation includes one or more of the following: Controlling one or more first devices to start audio recording; controlling one or more second devices to start video recording; Controlling the mechanical test tool to perform one or more test actions; Play one or more voice commands; controlling one or more first devices to stop audio recording; Control one or more second devices to stop video recording; wherein, Different first devices / second devices correspond to different recording objects, and the recording objects include one or more marked points / marked areas on the intelligent entity to be tested and / or the mechanical testing tool.

11. The method according to claim 10, characterized in that The method further comprises: The first module controls the second subsystem of the agent testing system to collect multimodal data of the agent to be tested during the execution of the test script, wherein the multimodal data is generated by the second module executing the test operation; The second subsystem collects the multimodal data based on the control of the first module.

12. The method according to claim 11, characterized in that The method further comprises: The third module of the first subsystem acquires the multimodal data collected by the second subsystem, and sends the multimodal data to the third subsystem of the intelligent agent testing system; The third subsystem receives the multimodal data sent by the third module, and presents the multimodal data and / or analysis data corresponding to the multimodal data; wherein, The multimodal data includes one or more of the following: audio data of one or more recorded objects; video data of one or more recorded objects; Optical motion data of one or more recording objects, wherein the optical motion data includes position information and time information corresponding to the position information.

13. The method according to claim 12, characterized in that The presenting of the multimodal data and / or the analysis data corresponding to the multimodal data includes: The third subsystem presents the multimodal data and / or analysis data corresponding to the multimodal data through multiple display areas, and different display areas correspond to different data features, and the data features include one or more of data types, test indicators, and recording objects.

14. The method according to claim 13, characterized in that The method further comprises: A fourth subsystem of the agent testing system determines clock skew between the plurality of display regions; Based on the clock deviation between the multiple display areas determined by the fourth subsystem, the third subsystem performs clock synchronization between the multiple display areas during the process of presenting the multimodal data and / or analysis data corresponding to the multimodal data through the multiple display areas.

15. The method according to claim 14, characterized in that The method further comprises: The fourth subsystem determines an event list of the intelligent agent to be tested based on the multimodal data and / or analysis data corresponding to the multimodal data, wherein the event list includes multiple event information and time information corresponding to each event information, the event information includes one or more of a recording object, an event type, and event description information, and the event includes one or more of a behavior event, a voice event, and a video event; The fourth subsystem determines the correlation between the multiple event information, and based on the correlation between the multiple event information and one or more preset test indicators, performs performance analysis on the intelligent entity to be tested to obtain analysis results, and the one or more test indicators are associated with the performance of the intelligent entity to be tested.

16. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the steps of the method described in any one of claims 9 to 15.

17. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 9 to 15 are implemented.

18. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 9 to 15 are implemented.

Images