Automatic test method and system for intelligent cockpit large model

Abstract

The embodiment of the invention discloses an automatic test method and system for an intelligent cockpit large model, and the method comprises the steps: uploading a to-be-tested case document, and generating a test input signal according to the to-be-tested case document; receiving the test input signal, executing a test task for the test input signal, and generating a test response result; collecting a test response result, wherein the test response result comprises a voice recognition result, a human-computer interaction interface screenshot and a system operation log; performing semantic comparison and scoring on the test response result through a judgment large model, and outputting a related index corresponding to the test response result; and generating a test result evaluation report according to the related indexes corresponding to the test response result, and outputting a typical error case list according to the test cases which do not reach the standard in the test result evaluation report. According to the method, the accuracy, the efficiency and the reproducibility of the intelligent cabin model test can be improved.

Description

Technical Field

[0001] The present application relates to the field of smart cockpit technology, and is related to, but not limited to, an automated testing method and system for a large smart cockpit model. Background Art

[0002] With the continuous development of smart cockpit technology, Large Language Models (LLMs) have been widely used in the field of smart cockpit technology due to their core advantages of large scale, strong versatility, and support for multimodal expansion. For example, in smart cockpit functions such as voice assistants, navigation command understanding, and artificial intelligence-generated content (AIGC). The intelligence level and user interaction experience of smart cockpit technology using LLMs are increasing with the continuous deepening of related research.

[0003] In existing technologies, researchers evaluate real smart cockpits through LLM testing in smart cockpit scenarios. However, because existing tests generally rely on manual methods, the test results have certain deficiencies in accuracy and efficiency. Specifically, the manual text-to-speech (TTS) process for speech synthesis and evaluation is time-consuming, resulting in low test efficiency. Currently, there is a lack of unified testing metrics and methods for smart cockpit scenarios, resulting in insufficient standardization of the testing process and affecting test results. The poor consistency of manual scoring during the testing process leads to excessive subjectivity in multi-round dialogue generation tasks and insufficient test accuracy. Existing testing methods struggle to accurately simulate the diverse factors present in real smart cockpit scenarios, such as vehicle noise, speech speed, and accent. This makes it impossible to replicate real scenarios end-to-end and reflect the evaluation results of real smart cockpits through test results. Existing testing methods lack structured storage, visualization, and management of typical bad cases, leading to vulnerabilities in data management mechanisms, an inability to complete a closed-loop testing process, and incomplete test results.

[0004] Therefore, a more systematic and automated intelligent cockpit model testing method is urgently needed to make up for the shortcomings of the existing testing methods in the testing process and results, and ultimately achieve improvements in the testing efficiency, accuracy and reproducibility of the intelligent cockpit model. Summary of the Invention

[0005] The embodiments of the present application provide an automated testing method and system for a large model of an intelligent cockpit.

[0006] The technical solution of the embodiment of the present application is implemented as follows: In a first aspect, an embodiment of the present application provides an automated testing method for a smart cockpit large model, the method comprising: uploading a case document to be tested, and generating a test input signal based on the case document to be tested; receiving the test input signal, and executing a test task for the test input signal to generate a test response result, wherein the test task includes a semantic understanding task, a dialogue generation task, and an interface update task; collecting test response results, wherein the test response results include speech recognition results, screenshots of the human-computer interaction interface, and system operation logs; performing semantic comparison and scoring on the test response results through a referee large model, and outputting relevant indicators corresponding to the test response results; generating a test result evaluation report based on the relevant indicators corresponding to the test response results, and outputting a list of typical error cases based on the test cases that do not meet the standards in the test result evaluation report.

[0007] The technical solution provided by the present application, by uploading the test case document to be tested, generating a test input signal according to the test case document to be tested, and introducing a preset noise voice signal, improves the adaptability in the highly complex real smart cockpit scenario, and by executing the test task for the received test input signal, generates a test response result, and assists in the analysis of problems in the test process by collecting the test response results including voice recognition results, screenshots of the human-computer interaction interface and system operation logs, and performs semantic comparison and scoring of the test response results through the referee large model to reduce subjective scoring bias, thereby improving the objectivity and consistency of the test response result analysis and scoring, and testing the response results in multiple rounds of test scenarios. The accuracy and rationality of the results are significantly improved, and by outputting the relevant indicators corresponding to the test response results, a systematic test indicator system is formed to comprehensively reflect the actual performance of the smart cockpit model in different application scenarios. Finally, a test result evaluation report is generated according to the relevant indicators corresponding to the test response results, so that the project team can evaluate the performance of the smart cockpit model, and output a list of typical error cases based on the test cases that do not meet the standards in the test result evaluation report to support the retrospective analysis of typical error cases, thereby forming an automated closed loop of testing-feedback-optimization for the smart cockpit model, providing structured data support for the subsequent adjustment and optimization of the smart cockpit model.

[0008] Optionally, uploading the test case document and generating a test input signal based on the test case document includes: parsing the test case document and extracting the text to be tested, wherein the text to be tested includes voice input text, expected response results and test target domain; converting the text to be tested into a voice file through a text-to-speech component to generate a test input signal.

[0009] The technical solution provided by this application generates a test input signal through a test case document. The test case document can be a plain text file or a binary file, supports multiple file formats and covers complex functions such as multiple worksheets, formulas and charts. In addition, the test cases can cover multiple aspects such as chatting, encyclopedia, generative artificial intelligence (AIGC), navigation and itinerary planning, thereby making the test dimension comprehensive. Specifically, the test case document is uploaded and parsed, and the test text is extracted based on the key information content in the test case document. The test text includes the voice input text corresponding to the test case, the expected response result and the test target domain. The voice input text corresponding to the test case is the original text content input by the user through voice, the expected response result corresponding to the test case is the correct reply or correct operation to be performed for the test case, and the test target domain corresponding to the test case is the functional category to which the test case belongs (such as chatting, navigation, music control, etc.). After the test text is extracted from the test case document, the test text is converted into a voice file through the TTS component to generate the test input signal.

[0010] Optionally, the test input signal also includes a synthesized voice signal and a preset noise voice signal, wherein the synthesized voice signal is a voice file played to the terminal under test through a voice playback device; the preset noise voice signal is a voice file played to the terminal under test through a background noise playback device, and the preset noise voice signal includes street noise, wind noise corresponding to different vehicle speeds, tire noise and engine sound; the playback speed, delay time and signal strength of the synthesized voice signal and the preset noise voice signal are dynamically adjusted according to the driving scenario.

[0011] The technical solution provided by the present application, in addition to being able to convert the text to be tested into a voice file through the TTS component to generate a test input signal, the test input signal also includes a synthesized voice signal and a preset noise voice signal, wherein the synthesized voice signal is a voice file played to the terminal under test through a voice playback device, for example, a voice file played to the terminal under test through a voice playback device such as an "artificial mouth", and the preset noise voice signal is a voice file played to the terminal under test through a background noise playback device with extremely high fidelity. The preset noise voice signal can be street noise, wind noise corresponding to different vehicle speeds, tire noise, engine sound, etc., and in different driving scenarios, the playback speed, delay time, signal type and signal strength of the synthesized voice signal and the preset noise voice signal can be dynamically adjusted according to different driving scenarios; for example, the playback speed, delay time, signal type and signal strength of the synthesized voice signal and the preset noise voice signal in different actual driving scenarios can be dynamically adjusted through the delay control component. The embodiment of the present application combines the voice file converted by the TTS component, the voice file played to the terminal under test by the voice playback device, and the voice file played to the terminal under test by the background noise playback device to generate a test input signal, which can simulate the voice input environment in the vehicle driving state in a real smart cockpit environment, improve the adaptability of the test method provided by the present application in highly complex real smart cockpit scenarios, and greatly enhance the reproducibility and integrity of the test method provided by the present application.

[0012] Optionally, the relevant indicators corresponding to the test response result include speech recognition accuracy, natural language understanding (NLU) intent recognition accuracy, semantic generation accuracy, response delay time, generation speed and rejection ability.

[0013] The technical solution provided by this application, through the introduction of a referee model, can perform semantic analysis (e.g., semantic comparison) and scoring of the test response results corresponding to the test case documents, thereby outputting relevant indicators corresponding to the test response results. Specifically, the relevant indicators corresponding to the test response results include speech recognition accuracy, NLU intent recognition accuracy, semantic generation accuracy, response latency, generation speed, and rejection capability. Speech recognition accuracy measures the match between the test input signal and the speech recognition result generated after the test input signal executes the test task; NLU intent recognition accuracy verifies the correctness of the intent classification corresponding to the test case; semantic generation accuracy evaluates the semantic match between the test response result and the reference result; response latency measures the time difference between the input of the test input signal and the generation of the first minimum semantic unit response, thereby evaluating the user experience; generation speed quantifies the real-time generation efficiency of the test response result; and rejection capability reflects the rationality of the recognition and rejection of unsupported requests. By outputting relevant indicators corresponding to the test response results, the technical solution provided by this application forms a systematic test indicator system to comprehensively reflect the actual performance of the smart cockpit model in different application scenarios.

[0014] Optionally, a test result evaluation report is generated based on the relevant indicators corresponding to the test response results, and a list of typical error cases is output based on the test cases that do not meet the standards in the test result evaluation report, including: displaying the relevant indicators corresponding to the test response results in a graphical form to generate the test result evaluation report; identifying the test cases that do not meet the standards in the test result evaluation report, recording the error type, occurrence timestamp and functional domain corresponding to the test cases that do not meet the standards, outputting the list of typical error cases, and labeling, classifying and storing the list of typical error cases.

[0015] The technical solution provided in this application displays the relevant indicators corresponding to the test response results of each round for the test cases in the form of charts (such as bar charts, line charts, and radar charts, etc.) or tables to generate a summary test result evaluation report. The test result evaluation report supports PDF format, Word format, CSV format, etc., and can intuitively display the evaluation results of the test results. Testers can export detailed test result evaluation reports for the project team to evaluate the performance of the smart cockpit model.

[0016] The technical solution provided in this application automatically identifies test cases that do not meet the standards in the test result evaluation report, and records the error type, occurrence timestamp and functional domain corresponding to the test cases that do not meet the standards, and finally outputs a list of typical error cases, analyzes the test cases that do not meet the standards in the typical error case list, and annotates, classifies and stores the typical error case list to support retrospective analysis of typical error cases, thereby forming an automated closed loop of testing-feedback-optimization for the smart cockpit model, and providing structured data support for subsequent adjustment and optimization of the smart cockpit model and enhancement of the training of the smart cockpit model.

[0017] Optionally, the automated testing method for a large smart cockpit model includes: a fully automatic testing method applied to a dedicated test room for the smart cockpit model; and a semi-automatic testing method applied to a mobile test terminal device.

[0018] The technical solution provided in this application includes fully automated and semi-automated testing methods. The semi-automatic method is applied to mobile test terminals (such as tablets and portable PCs) to meet on-site debugging or remote testing needs. The fully automated method is applied to a dedicated test room for intelligent cockpit models, which contains a complete, fully automated testing environment suitable for formal acceptance and stability testing. By flexibly selecting between semi-automatic and fully automated testing methods based on different test resources and site conditions, a highly flexible test deployment method can be achieved, facilitating the expansion of test methods.

[0019] In a second aspect, an embodiment of the present application provides an automated testing system for a large model of an intelligent cockpit, comprising: a test input signal generation module, a test response result generation module, a test response result acquisition module, a test response result processing module, and a test result evaluation report generation module, wherein: A test input signal generating module is used to upload a test case document and generate a test input signal according to the test case document; A test response result generation module is used to receive the test input signal, execute a test task according to the test input signal, and generate a test response result, wherein the test task includes a semantic understanding task, a dialogue generation task, and an interface update task; A test response result collection module is used to collect test response results, including speech recognition results, screenshots of the human-computer interaction interface, and system operation logs; A test response result processing module is used to perform semantic comparison and scoring on the test response results through the referee model, and output relevant indicators corresponding to the test response results; The test result evaluation report generation module is used to generate a test result evaluation report based on the relevant indicators corresponding to the test response results, and output a list of typical error cases based on the test cases that do not meet the standards in the test result evaluation report.

[0020] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the above-mentioned automated testing method for a large smart cockpit model are implemented.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-mentioned method for automated testing of a large intelligent cockpit model.

[0022] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least: The present application provides an automated testing method and system for a large model of an intelligent cockpit. By uploading a document of a case to be tested, a test input signal is generated according to the document of the case to be tested, and a preset noise voice signal is introduced to improve the adaptability in a highly complex real intelligent cockpit scenario. In addition, a test task is executed for the received test input signal to generate a test response result. By collecting the test response results including voice recognition results, screenshots of the human-computer interaction interface and system operation logs, the problem analysis in the test process is assisted. The test response results are semantically compared and scored by a large referee model to reduce subjective scoring bias, thereby improving the objectivity and consistency of the analysis and scoring of the test response results, and ensuring the accuracy of multiple rounds of testing. The accuracy and rationality of the test response results in the test scenarios are significantly improved. In addition, by outputting the relevant indicators corresponding to the test response results, a systematic test indicator system is formed to comprehensively reflect the actual performance of the smart cockpit model in different application scenarios. Finally, a test result evaluation report is generated according to the relevant indicators corresponding to the test response results, so that the project team can evaluate the performance of the smart cockpit model. A list of typical error cases is output based on the test cases that do not meet the standards in the test result evaluation report to support the retrospective analysis of typical error cases, thereby forming an automated closed loop of testing, feedback and optimization for the smart cockpit model, and providing structured data support for the subsequent adjustment and optimization of the smart cockpit model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. Those skilled in the art can also derive other drawings based on these drawings without inventive work, among which: Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application; Figure 2 A flowchart of an automated testing method for a large intelligent cockpit model provided in an embodiment of the present application; Figure 3 A schematic diagram of a test result evaluation report provided in an embodiment of the present application; Figure 4 A schematic diagram of a typical error case list provided in an embodiment of the present application; Figure 5 A schematic diagram of the testing process of an automated testing method for a large intelligent cockpit model provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of an automated testing system for a large intelligent cockpit model provided in an embodiment of the present application; Figure 7 A hardware entity diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. The following examples are used to illustrate the present application, but are not intended to limit the scope of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0025] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0026] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0027] Those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as generally understood by those skilled in the art in the art to which the embodiments of the present application belong. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art, and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0028] The embodiments of the present application will be further described below with reference to the accompanying drawings.

[0029] In view of the current problems in the testing and research of smart cockpit models in the field of smart cockpits, the embodiments of the present application provide an automated testing method and system for a large smart cockpit model.

[0030] The technical solution of this application is introduced below, and the application scenario of this application is first introduced.

[0031] Please refer to Figure 1 , which shows a schematic diagram of an application scenario provided by an embodiment of the present application, wherein the application scenario provides an intelligent driving system in an intelligent cockpit, the intelligent driving system including at least one information collection device, an intelligent driving server 101, and at least one terminal device, Figure 1 The intelligent driving system includes an information collection device 102 , an information collection device 103 and a terminal device 104 as an example for explanation.

[0032] The intelligent driving server 101 is a single server, a server cluster consisting of multiple servers, or a cloud computing service center. The information collection device is an electronic device capable of collecting intelligent driving environment information, such as an in-vehicle environment information collection device (e.g., an in-vehicle camera, a microphone array), a driver / passenger status collection device (e.g., a steering wheel / seat built-in sensor, a seat pressure sensor, an eye-tracking camera), etc. For example, information collection device 102 and information collection device 103 are an in-vehicle camera and a steering wheel built-in sensor, respectively, or information collection device 102 and information collection device 103 are a microphone array and a seat pressure sensor, respectively. Figure 1 The information collection device 102 and the information collection device 103 are respectively an in-car camera and a microphone array for illustration.

[0033] The terminal device is an electronic device with an intelligent driving function. For example, an intelligent driving client is running in the terminal device, and the intelligent driving client has the intelligent driving function. For example, the terminal device is a smart phone, a tablet computer, a smart wearable device (such as a smart watch), a laptop computer, a desktop computer, an Internet of Things (IoT) device, etc. The terminal device 104 and the terminal device 105 can be the same type of electronic device, for example, the terminal device 104 and the terminal device 105 are both smart phones. Alternatively, the terminal device 104 and the terminal device 105 can be different types of electronic devices, for example, the terminal device 104 is a smart phone and the terminal device 105 is a tablet computer. Figure 1 It is assumed that the terminal device 104 is a tablet computer and the terminal device 105 is a desktop computer.

[0034] The information collection device 102, the information collection device 103, the terminal device 104, and the terminal device 105 are respectively connected to the intelligent driving server 101 through a wired network or a wireless network. Each information collection device (for example, the information collection device 102 and the information collection device 103) can send collection information to the intelligent driving server 101, and the intelligent driving server 101 sends the message to at least one other terminal device (for example, the terminal device 104 and the terminal device 105). Thus, by using the intelligent driving server 101 as an information processing transfer station, the information collected by the information collection device 102 and the information collection device 103 is processed and presented in the terminal device 104 and the terminal device 105.

[0035] Figure 1 The intelligent driving system shown is only used as an example and is not intended to limit the technical solution of this application. In the implementation process, the intelligent driving system may include Figure 1 As shown, more or fewer information collection devices and terminal devices may be included in the intelligent driving system. The number of information collection devices and terminal devices and the devices included in the intelligent driving system can be configured as needed. This is not limited in the embodiments of the present application.

[0036] The above is an introduction to the application scenarios of this application. The following introduces the method embodiments of this application.

[0037] Please refer to Figure 2 , which shows a flow chart of an automated testing method for a large model of an intelligent cockpit provided in an embodiment of the present application, such as Figure 2 As shown, the method at least includes the following steps S210 to S250.

[0038] Step S210: uploading a test case document, and generating a test input signal according to the test case document.

[0039] In an embodiment of the present application, a test input signal is generated by a test case document. The test case document can be a plain text file that supports the Comma-Separated Values ​​(.CSV) format, or the test case document can be a binary file that supports the Excel Open Extensible Markup Language Spreadsheet (.XLSX) format, covering complex functions such as multiple worksheets, formulas, and charts. In addition, the test case can cover multiple aspects such as chatting, encyclopedia, AIGC, navigation, and itinerary planning, thereby making the test dimension comprehensive. Specifically, the test case document is uploaded and parsed, and the test text is extracted based on the key information content in the test case document. The test text includes the voice input text corresponding to the test case, the expected response result, and the test target domain. The voice input text corresponding to the test case is the original text content input by the user through voice, the expected response result corresponding to the test case is the correct reply or correctly performed operation that should be given for the test case, and the test target domain corresponding to the test case is the functional category to which the test case belongs (such as chatting, navigation, music control, etc.). For example, the test case is a navigation request initiated by the user to the smart cockpit. During the test process, the test case document corresponding to the test case "navigation request" is first uploaded and parsed, and the test text corresponding to the test case "navigation request" is extracted according to the key information content in the test case document corresponding to the test case "navigation request". The test text corresponding to the test case "navigation request" includes the voice input text corresponding to the test case "navigation request" "Please navigate to the nearest high-speed rail station", the expected response result corresponding to the test case "navigation request" "The optimal route to the nearest high-speed rail station is being planned for you, and the estimated driving time is 6 minutes" and the test target domain "navigation" corresponding to the test case "navigation request".

[0040] In an embodiment of the present application, after extracting the test text from the test case document, the test text is converted into a voice file through the TTS component to generate a test input signal. In an optional embodiment, in addition to converting the test text into a voice file through the TTS component to generate the test input signal, the test input signal also includes a synthesized voice signal and a preset noise voice signal, wherein the synthesized voice signal is a voice file played to the terminal under test through a voice playback device, for example, a voice file played to the terminal under test through a voice playback device such as an "artificial mouth", and the preset noise voice signal is a voice file played to the terminal under test through a background noise playback device with extremely high fidelity. The preset noise voice signal can be street noise, wind noise corresponding to different vehicle speeds, tire noise, engine sound, etc., and the playback speed, delay time, signal type, and signal strength of the synthesized voice signal and the preset noise voice signal can be dynamically adjusted according to different driving scenarios. For example, the playback speed, delay time, signal type, and signal strength of the synthesized voice signal and the preset noise voice signal in different actual driving scenarios can be dynamically adjusted through the delay control component. The embodiment of the present application combines the voice files converted by the TTS component, the voice files played to the terminal under test by the voice playback device, and the voice files played to the terminal under test by the background noise playback device to generate a test input signal, which can simulate the voice input environment in the vehicle driving state under a variety of different real smart cockpit environments, improve the adaptability of the test method provided by the embodiment of the present application in highly complex real smart cockpit scenarios, and greatly enhance the reproducibility and integrity of the test method provided by the embodiment of the present application.

[0041] Step S220: receiving the test input signal, executing a test task for the test input signal, and generating a test response result. The test task includes a semantic understanding task, a dialogue generation task, and an interface update task.

[0042] In an embodiment of the present application, the test input signal generated by the test case document is first received, and then the test tasks including semantic understanding tasks, dialogue generation tasks and interface update tasks are executed for the test input signal, and finally a test response result is generated. Specifically, the semantic understanding task executed for the test input signal is to perform intent recognition, target recognition and multi-language and multi-accent adaptation on the test input signal corresponding to the test case; the dialogue generation task executed for the test input signal is to make a contextual reply according to the test input signal corresponding to the test case, when the test case is a normal user request, a normal contextual reply is made according to the test input signal corresponding to the test case, and when the test case is a dangerous user request, a rejection reply is made according to the test input signal corresponding to the test case; the interface update task executed for the test input signal is to update the execution result of the test input signal corresponding to the test case in the human-computer interaction interface, and, at the same time as the test input signal corresponding to the test case triggers the execution of the interface update task, the operation event triggered by the test input signal corresponding to the test case is recorded through the system log.

[0043] For example, the test case is a navigation request to the nearest high-speed railway station initiated by the user for the smart cockpit. The semantic understanding task performed on the test input signal corresponding to the test case "Navigate to the nearest high-speed railway station" is to perform intent recognition, target recognition and multi-language and multi-accent adaptation on the test input signal corresponding to the test case "Navigate to the nearest high-speed railway station". The intent corresponding to the test case "Navigate to the nearest high-speed railway station" is "navigation", the target corresponding to the test case "Navigate to the nearest high-speed railway station" is "nearest high-speed railway station", and the multi-language and multi-accent adaptation corresponding to the test case "Navigate to the nearest high-speed railway station" is "reporting the navigation route in Mandarin". For example, when the test case is a normal user request, for example, the test case is a weather query request for Beijing within a week initiated by the user for the smart cockpit, for the test case "Weather query in Beijing within a week" The dialogue generation task executed by the corresponding test input signal is to make a normal context reply based on the test input signal corresponding to the test case "Query the weather in Beijing within a week", and the reply result is "The weather in Beijing will be sunny within a week". When the test case is a dangerous user request, for example, the test case is a user's inquiry request for how to speeding in the smart cockpit, for the test case "How to speeding" The dialogue generation task executed by the corresponding test input signal is to give a rejection reply according to the test input signal corresponding to the test case to be tested "How to overspeed"; for example, the test case to be tested is a temperature adjustment request initiated by the user to the smart cockpit, "Please adjust the temperature in the cockpit to the temperature that is best for human comfort", and the interface update task executed for the test input signal corresponding to the test case to be tested "Please adjust the temperature in the cockpit to the temperature that is best for human comfort" is to update the execution result of the test input signal corresponding to the test case to be tested "Please adjust the temperature in the cockpit to the temperature that is best for human comfort" "The air conditioning temperature is adjusted to 23 degrees Celsius" in the human-computer interaction interface, and, while the test input signal corresponding to the test case to be tested "Please adjust the temperature in the cockpit to the temperature that is best for human comfort" triggers the execution of the interface update task, the operation event "Air conditioning temperature set successfully" triggered by the test input signal corresponding to the test case to be tested "Please adjust the temperature in the cockpit to the temperature that is best for human comfort" is recorded through the system log.

[0044] Step S230 , collecting test response results, which include speech recognition results, screenshots of the human-computer interaction interface, and system operation logs.

[0045] In an embodiment of the present application, the test response results corresponding to the test case document are collected after the test task is executed. The test response results include the speech recognition results corresponding to the test case document, screenshots of the human-computer interaction interface, and system operation logs. Specifically, the speech recognition results include the text response content and voice response content corresponding to the test case document output by the smart cockpit model captured in real time. The text response content corresponding to the test case document can be captured in real time through the application programming interface (API) interface in the smart cockpit model. For example, the normal context reply generated after the dialogue generation task is executed according to the test input signal corresponding to the test case. The collected text response content is used to determine the match between the expected text response content and the expected text response content; the voice response content corresponding to the test case document can be captured in real time through the audio output interface (such as Bluetooth, speaker, etc.) in the smart cockpit model. For example, the voice response result generated after the intent recognition task is executed according to the test input signal corresponding to the test case. The collected voice response content is used to evaluate the voice response result quality according to voice quality indicators (such as voice clarity, voice naturalness, and voice error, etc.); the human-computer interaction interface screenshot corresponding to the test case document can be captured by the test task execution completion instruction (such as interface update). The system operation log corresponding to the test case document can be obtained by automatically pulling the system operation content in the real-time log stream monitoring. For example, after executing the dialogue generation task according to the test input signal corresponding to the test case, the system operation content corresponding to the dialogue generation task executed according to the test input signal corresponding to the test case is pulled from the real-time log stream monitoring. The collected system operation log is used to assist in problem analysis during the test process of the test solution provided in the embodiment of the present application.

[0046] Step S240: semantically compare and score the test response results through the referee model, and output relevant indicators corresponding to the test response results.

[0047] In an embodiment of the present application, by introducing a large referee model, semantic-level analysis (such as semantic comparison, etc.) and scoring of the test response results corresponding to the test case document can be achieved, and then relevant indicators corresponding to the test response results can be output. Scoring the test response results corresponding to the test case document by the large referee model instead of manual scoring can reduce subjective scoring bias, thereby improving the objectivity and consistency of the analysis and scoring of the test response results, and significantly improving the accuracy and rationality of the test response results in multi-round testing scenarios. Specifically, the relevant indicators corresponding to the test response results include speech recognition accuracy, NLU intent recognition accuracy, semantic generation accuracy, response delay time, generation speed and rejection ability, etc., among which, the speech recognition accuracy is used to measure the matching degree between the test input signal and the speech recognition result generated after the test input signal executes the test task; the NLU intent recognition accuracy is used to verify whether the intent classification corresponding to the test case is correct; the semantic generation accuracy is used to evaluate the matching degree between the test response result and the reference result at the semantic level; the response delay time is used to measure the time difference between the input of the test input signal and the generation of the response of the first minimum semantic unit, so as to evaluate the user experience; the generation speed is used to quantify the real-time generation efficiency of the test response result; the rejection ability is used to reflect the rationality of the recognition and rejection of unsupported requests. The technical solution provided in the embodiment of the present application forms a set of systematic test indicator system by outputting the relevant indicators corresponding to the test response results, so as to fully reflect the real performance of the smart cockpit model in different application scenarios.

[0048] For example, for the test case "Navigate to the nearest high-speed rail station", the real intention is "navigation". After executing the test task for the test input signal corresponding to the test case "Navigate to the nearest high-speed rail station", a speech recognition result is generated. If the generated speech recognition result is "A route to the nearest high-speed rail station has been planned, and it is expected to arrive in 30 minutes", it means that the speech recognition accuracy is high. If the generated speech recognition result is "A route to the nearest airport has been planned, and it is expected to arrive in 40 minutes", it means that the speech recognition accuracy is low. For the test input signal corresponding to the test case "Navigate to the nearest high-speed rail station", a semantic understanding task is performed. If the corresponding test intention is "navigation", it means that the intention recognition is correct. If the corresponding test intention is "vehicle control", it means that the intention recognition is incorrect.

[0049] Step S250: generating a test result evaluation report based on relevant indicators corresponding to the test response results, and outputting a list of typical error cases based on the test cases that do not meet the standards in the test result evaluation report.

[0050] In the embodiment of the present application, the relevant indicators corresponding to the test response results of each round for the test case are displayed in the form of a chart (such as a bar chart, a line chart, a radar chart, etc.) or a table to generate a summary test result evaluation report. The test result evaluation report supports the portable document format (PDF), Word format and comma-separated values ​​(CSV) format, etc., which can intuitively display the evaluation results of the test results. The tester can export a detailed test result evaluation report for the project team to evaluate the performance of the smart cockpit model. For an example, please refer to Figure 3 , which shows a schematic diagram of a test result evaluation report provided by an embodiment of the present application, the test result evaluation report includes a speech recognition accuracy line graph, an NLU intent recognition accuracy line graph, a semantic generation accuracy line graph, a response delay time bar graph, and a generation speed bar graph. The speech recognition accuracy line graph shows the speech recognition accuracy corresponding to tests 1 to 3, that is, the speech recognition accuracy in the test response results corresponding to the test case to be tested during the first to third rounds of testing; the NLU intent recognition accuracy line graph shows the NLU intent recognition accuracy corresponding to tests 1 to 3, that is, the test case to be tested during the first to third rounds of testing. The NLU intent recognition accuracy in the corresponding test response results; the semantic generation accuracy line chart shows the semantic generation accuracy corresponding to tests 1 to 3, that is, the semantic generation accuracy in the test response results corresponding to the test cases from the first to the third round of testing; the response delay time bar chart shows the response delay time corresponding to tests 1 to 3, that is, the response delay time in the test response results corresponding to the test cases from the first to the third round of testing; the generation speed bar chart shows the generation speed corresponding to tests 1 to 3, that is, the generation speed in the test response results corresponding to the test cases from the first to the third round of testing.

[0051] In an embodiment of the present application, by automatically identifying test cases that do not meet the standards in the test result evaluation report, and recording the error type, occurrence timestamp and functional domain corresponding to the test cases that do not meet the standards, a list of typical error cases is finally output, the test cases that do not meet the standards in the typical error case list are analyzed, and the typical error case list is annotated, classified and stored to support retrospective analysis of typical error cases (such as manual review, etc.), thereby forming an automated closed loop of test-feedback-optimization for the smart cockpit model, and providing structured data support for the subsequent adjustment and optimization of the smart cockpit model and enhancement of the training of the smart cockpit model. For examples, please refer to Figure 4 , which shows a typical error case list provided by the embodiment of the present application. The typical error case list includes test cases X that do not meet the standards.i , test cases that did not meet the standards X i Corresponding error type A i , test cases that did not meet the standards X i The corresponding occurrence timestamp B i and failed test case X i Corresponding functional domain M i , i is a positive integer between 1 and n.

[0052] In an optional embodiment, the test method includes a fully automatic test method and a semi-automatic test method. The semi-automatic test method is applied to mobile test terminal devices (such as iPad, portable PC, etc.), which can meet the needs of on-site debugging or remote testing; the fully automatic test method is applied to a dedicated test room for the smart cockpit model. A complete fully automated test environment is built in the dedicated test room, which is suitable for the formal acceptance and stability testing stages. By flexibly selecting semi-automatic and fully automatic test methods under different test resources and site conditions, a highly flexible test deployment method can be achieved, which is easy to expand the test method. Moreover, from test input signal generation, test task execution, test response result collection, test response result processing to the final test result evaluation report generation, the overall test process of the test method provided in the embodiment of the present application does not require human intervention, which greatly improves the test efficiency.

[0053] The automated testing method for the above-mentioned large intelligent cockpit model is described below with reference to a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustrating the present application and does not constitute an improper limitation on the present application.

[0054] For specific examples, please refer to Figure 5, which shows a test process diagram of an automated testing method for a large model of a smart cockpit provided by an embodiment of the present application. During the test process, the test case document to be tested is first uploaded, and the test case document to be tested is converted into a test input signal through the TTS component, and a synthetic voice signal played to the terminal under test through a voice playback device and a preset noise voice signal played to the terminal under test through a background noise playback device (including street noise, wind noise corresponding to different vehicle speeds, tire noise and engine sound), the test input signal is received, the test task is started to be issued through a test task start instruction, the type of test task (such as semantic understanding task and dialogue generation task) is determined, and the test task is executed for the test input signal to generate a corresponding test response result, and it is determined whether the test response is successful. If the test response is not successfully generated, exception handling is performed (initializing the test software and determining the current test task position, restarting from the test task issuance). Execute the test task) and store the exception handling; if the test response is successfully generated, the test response results including voice recognition results, human-computer interaction interface screenshots and system operation logs are collected, and the test response results are detected and scored according to the referee big model. If the detection result corresponding to the test response result is an error, the smart cockpit model is restarted and the test task is restarted; if the detection result corresponding to the test response result is not an error, the relevant indicators corresponding to the test response result (such as semantic generation accuracy, response delay time, generation speed) are output. When all test cases are detected and scored according to the referee big model, a report generation instruction is issued, and a test result evaluation report is generated according to the relevant indicators corresponding to the test response result. A list of typical error cases is output according to the test cases that do not meet the standards in the test result evaluation report, and the test result evaluation report and the typical error case list are stored. A test end instruction is issued, and the test process is completed.

[0055] In summary, the embodiment of the present application provides an automated testing method for a large model of an intelligent cockpit. By uploading a document of a case to be tested, a test input signal is generated according to the document of the case to be tested, and a preset noise voice signal is introduced to improve the adaptability in a highly complex real intelligent cockpit scenario. In addition, by executing a test task for the received test input signal, a test response result is generated. By collecting test response results including voice recognition results, screenshots of the human-computer interaction interface, and system operation logs, problem analysis in the test process is assisted. The test response results are semantically compared and scored by a large referee model to reduce subjective scoring bias, thereby improving the objectivity and consistency of the analysis and scoring of the test response results. The accuracy and rationality of the test response results in multiple rounds of test scenarios have been significantly improved. In addition, by outputting the relevant indicators corresponding to the test response results, a systematic test indicator system has been formed to comprehensively reflect the actual performance of the smart cockpit model in different application scenarios. Finally, a test result evaluation report is generated according to the relevant indicators corresponding to the test response results for the project team to evaluate the performance of the smart cockpit model, and a list of typical error cases is output based on the test cases that do not meet the standards in the test result evaluation report to support the retrospective analysis of typical error cases, thereby forming an automated closed loop of testing, feedback and optimization for the smart cockpit model, providing structured data support for the subsequent adjustment and optimization of the smart cockpit model.

[0056] The above is an introduction to the method embodiment of the present application. Based on the aforementioned embodiment, the system embodiment of the present application is introduced below. The system of the present application is used to execute the method of the present application. For details not disclosed in the system embodiment, please refer to the method embodiment. The embodiment of the present application further provides an automated testing system for a large model of an intelligent cockpit. The system includes the modules included and the units included in each module, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0057] Please refer to Figure 6 , which shows a schematic diagram of an automated testing system 600 for a large model of a smart cockpit provided by an embodiment of the present application. The automated testing system 600 for the smart cockpit model can be implemented as follows Figure 2 The automated testing method of the smart cockpit model is shown in FIG. Figure 6As shown, an automated testing system 600 for a large model of an intelligent cockpit includes: a test input signal generating module 610, a test response result generating module 620, a test response result collecting module 630, a test response result processing module 640 and a test result evaluation report generating module 650. Specifically, The test input signal generating module 610 is used to upload a test case document and generate a test input signal according to the test case document; A test response result generating module 620 is configured to receive the test input signal, execute a test task based on the test input signal, and generate a test response result. The test task includes a semantic understanding task, a dialogue generation task, and an interface update task. A test response result collection module 630 is used to collect test response results, including speech recognition results, screenshots of the human-computer interaction interface, and system operation logs; The test response result processing module 640 is used to perform semantic comparison and scoring on the test response results through the referee model, and output relevant indicators corresponding to the test response results; The test result evaluation report generating module 650 is used to generate a test result evaluation report according to the relevant indicators corresponding to the test response results, and output a list of typical error cases according to the test cases that do not meet the standards in the test result evaluation report.

[0058] In summary, the embodiment of the present application provides an automated testing system for a large model of an intelligent cockpit. By uploading a document of a case to be tested, a test input signal is generated according to the document of the case to be tested, and a preset noise voice signal is introduced to improve the adaptability in a highly complex real intelligent cockpit scenario. In addition, by executing a test task for the received test input signal, a test response result is generated. By collecting test response results including voice recognition results, screenshots of the human-computer interaction interface, and system operation logs, problem analysis in the test process is assisted. The test response results are semantically compared and scored by a large referee model to reduce subjective scoring bias, thereby improving the objectivity and consistency of the analysis and scoring of the test response results. The accuracy and rationality of the test response results in multiple rounds of test scenarios have been significantly improved. In addition, by outputting the relevant indicators corresponding to the test response results, a systematic test indicator system has been formed to comprehensively reflect the actual performance of the smart cockpit model in different application scenarios. Finally, a test result evaluation report is generated according to the relevant indicators corresponding to the test response results for the project team to evaluate the performance of the smart cockpit model, and a list of typical error cases is output based on the test cases that do not meet the standards in the test result evaluation report to support the retrospective analysis of typical error cases, thereby forming an automated closed loop of testing, feedback and optimization for the smart cockpit model, providing structured data support for the subsequent adjustment and optimization of the smart cockpit model.

[0059] It should be noted that in the embodiments of the present application, if the automated testing system for the large model of the intelligent cockpit described above is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0060] Correspondingly, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the steps of the automated testing system for a large intelligent cockpit model described in any of the above embodiments. Correspondingly, an embodiment of the present application also provides a computer program product. When executed by a processor of an electronic device, the computer program product is used to implement the steps of the automated testing system for a large intelligent cockpit model described in any of the above embodiments.

[0061] Based on the same technical concept, an embodiment of the present application provides an electronic device for implementing an automated testing system for a large model of a smart cockpit as described in the above method embodiment. Figure 7 A hardware entity diagram of an electronic device provided in an embodiment of the present application is shown as follows: Figure 7 As shown, the electronic device 700 includes a memory 710 and a processor 720. The memory 710 stores a computer program that can be run on the processor 720. When the processor 720 executes the program, the steps in the automated testing system of the smart cockpit large model described in any embodiment of the present application are implemented.

[0062] The memory 710 is configured to store instructions and applications executable by the processor 720, and can also cache data to be processed or processed by the processor 720 and various modules in the electronic device (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (RAM).

[0063] When the processor 720 executes the program, the steps of any one of the above-mentioned methods for automatically testing a large intelligent cockpit model are implemented. The processor 720 generally controls the overall operation of the electronic device 700.

[0064] The processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understood that the electronic device that implements the functions of the processor may also be other electronic devices, and the embodiments of the present application are not specifically limited thereto.

[0065] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface storage device, an optical disc, or a compact disc read-only memory (CD-ROM); it can also be various electronic devices that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0066] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0067] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0068] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0069] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0070] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0071] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0072] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling the automatic test line of the device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0073] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0074] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0075] The above is merely an embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An automated testing method for a large intelligent cockpit model, characterized in that: The method comprises: Upload the test case document to be tested, and generate a test input signal according to the test case document to be tested; Receive the test input signal, and execute a test task according to the test input signal to generate a test response result, wherein the test task includes a semantic understanding task, a dialogue generation task, and an interface update task; Collect test response results, including speech recognition results, screenshots of the human-computer interaction interface, and system operation logs; Perform semantic comparison and scoring on the test response results through the referee big model, and output relevant indicators corresponding to the test response results; A test result evaluation report is generated based on relevant indicators corresponding to the test response results, and a list of typical error cases is output based on the test cases that do not meet the standards in the test result evaluation report.

2. The method according to claim 1, characterized in that The uploading of the test case document and generating a test input signal according to the test case document includes: Parsing the test case document to extract the test text, wherein the test text includes the voice input text, the expected response result and the test target domain; The text to be tested is converted into a voice file by a text-to-speech component to generate a test input signal.

3. The method according to claim 2, characterized in that The test input signal also includes a synthetic speech signal and a preset noise speech signal, wherein, The synthesized voice signal is a voice file played to the terminal under test through a voice playback device; The preset noise voice signal is a voice file played to the terminal under test by a background noise playback device. The preset noise voice signal includes street noise, wind noise corresponding to different vehicle speeds, tire noise, and engine sound; The playback speed, delay time and signal strength of the synthesized voice signal and the preset noise voice signal are dynamically adjusted according to the driving scenario.

4. The method according to claim 1, wherein The relevant indicators corresponding to the test response results include speech recognition accuracy, natural language understanding intent recognition accuracy, semantic generation accuracy, response delay time, generation speed and rejection ability.

5. The method according to claim 1, characterized in that Generating a test result evaluation report based on relevant indicators corresponding to the test response results, and outputting a list of typical error cases based on test cases that do not meet the standards in the test result evaluation report, including: Display relevant indicators corresponding to the test response results in the form of charts and generate the test result evaluation report; Identify the test cases that do not meet the standards in the test result evaluation report, record the error type, occurrence timestamp and functional domain corresponding to the test cases that do not meet the standards, output the list of typical error cases, and mark, classify and store the list of typical error cases.

6. The method according to any one of claims 1 to 5, characterized in that The automated testing method for a large intelligent cockpit model comprises: Fully automated testing methods, applied to dedicated test chambers for intelligent cockpit models; Semi-automatic testing method, applied to mobile test terminal equipment.

7. An automated testing system for a large intelligent cockpit model, characterized in that: The system comprises: A test input signal generating module is used to upload a test case document and generate a test input signal according to the test case document; A test response result generation module is used to receive the test input signal, execute a test task according to the test input signal, and generate a test response result, wherein the test task includes a semantic understanding task, a dialogue generation task, and an interface update task; A test response result collection module is used to collect test response results, including speech recognition results, screenshots of the human-computer interaction interface, and system operation logs; A test response result processing module is used to perform semantic comparison and scoring on the test response results through the referee model, and output relevant indicators corresponding to the test response results; The test result evaluation report generation module is used to generate a test result evaluation report based on the relevant indicators corresponding to the test response results, and output a list of typical error cases based on the test cases that do not meet the standards in the test result evaluation report.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the program, the steps in the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images