An offline voice control intelligent speaker test system and method

Abstract

The embodiment of the application provides a kind of offline voice control's intelligent speaker test system and method, it is applied to intelligent voice interaction technical field, by constructing multiple scene noise and multiple user voice control data form offline situation voice sample set, and based on the voice sample set generates the test task sequence covering diversified voice features and complex environmental noise, control intelligent speaker executes test task sequence and records recognition result and instruction execution result;Calculate identification accuracy parameter, execution matching rate parameter and execution response delay parameter generate performance evaluation value;Mark abnormal task and generate variation test task based on abnormal task and join test task sequence, the performance evaluation value of all test tasks is counted to form performance evaluation set and output test result report for representing the comprehensive processing capacity and user instruction response effect of voice control system in offline voice interaction scene, to realize the overall real evaluation of offline intelligent speaker.

Description

Technical Field

[0001] This application relates to the field of intelligent voice interaction technology, and in particular to an offline voice-controlled intelligent speaker testing system and method. Background Technology

[0002] Offline voice-controlled smart speaker testing methods typically refer to testing the speaker's recognition and response using pre-stored or played voice signals when the speaker is not connected to the network, in order to evaluate its basic voice recognition function and command execution capability.

[0003] In existing technologies, offline voice-controlled smart speaker testing methods typically involve inputting pre-recorded or manually generated voice commands into the smart speaker. The speaker's internal voice recognition module converts the voice signal into a corresponding command signal, and the execution module then triggers the corresponding action (such as playing audio, adjusting volume, or switching modes). During this process, the testing system collects the recognition results and execution status, and statistically analyzes the voice recognition accuracy and command execution based on preset indicators to evaluate the speaker's basic voice recognition and response functions in an offline environment.

[0004] The above-mentioned solution still has some problems in practical application. Since the offline voice control smart speaker test method mainly relies on a single scene or static noise environment for recognition and response evaluation, and the test tasks usually cannot fully cover the diverse user voice expressions and background noise in complex environments, the test results cannot truly reflect the speaker's recognition stability, command execution consistency and response performance in offline state. Summary of the Invention

[0005] This application provides an offline voice-controlled smart speaker testing system and method. By constructing an offline voice sample set with multi-scene noise and multi-user voice commands, and generating a test task sequence covering diverse voice features and complex environmental noise, a comprehensive, realistic, and quantitative evaluation of the smart speaker's voice recognition stability, command execution consistency, and response performance in offline mode is achieved. This overcomes the problem of insufficient test coverage in single-scene or static noise environments in existing methods.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] This application provides a method for testing an offline voice-controlled smart speaker, wherein the smart speaker is in an offline state. The method includes: constructing a voice sample set for offline scenarios based on collected multi-scene noise data and multi-user voice control data; generating a test task sequence based on the voice sample set, wherein the test task sequence includes at least one test task, and the test task includes voice command features and a corresponding scene type; controlling the smart speaker to execute each test task in the test task sequence, and recording the voice recognition result and command execution result of each test task; comparing the voice recognition result with the voice sample set to calculate a recognition accuracy parameter; and calculating an execution matching rate based on the command execution result. The system generates performance evaluation values ​​for each test task based on the recognition accuracy parameter, execution matching rate parameter, and execution response latency parameter. Test tasks with performance evaluation values ​​below a preset performance threshold are marked as anomalous tasks. Mutant test tasks are generated based on the voice command characteristics and corresponding scenario types of the anomalous tasks, and these mutant test tasks are added to the test task sequence. After all test tasks in the test task sequence have been executed, the performance evaluation values ​​of each test task are statistically analyzed to form a performance evaluation set. Based on the performance evaluation set, a test result report is output, which characterizes the comprehensive processing capability and user command response effect of the voice control system in offline voice interaction scenarios.

[0008] In some possible implementations, generating a mutation test task based on the voice command features and corresponding scene type of the abnormal task includes: performing correlation analysis between the voice command features and corresponding scene type of the abnormal task and the recognition accuracy parameter, execution matching rate parameter, and execution response latency parameter of the abnormal task to determine the cause of the low performance evaluation value; setting an adjustment coefficient level according to the cause of the abnormal task and the corresponding performance evaluation value, wherein the adjustment coefficient level is used to control the adjustment range of scene noise and voice commands of the abnormal task; and performing targeted feature mutation on the abnormal task based on the adjustment coefficient level to generate a mutation test task.

[0009] In some possible implementations, setting the adjustment coefficient level based on the cause of the abnormal task and the corresponding performance evaluation value includes: calculating the difference between the performance evaluation value of the abnormal task and a preset performance threshold as the performance deviation; and setting the corresponding adjustment coefficient level based on the cause of the abnormal task and the performance deviation through a preset level mapping strategy.

[0010] In some possible implementations, the voice command features include speech rate, accent, and voiceprint. The step of performing targeted feature mutation on the anomalous task based on the adjustment coefficient level to generate a mutation test task includes: adjusting the speech rate, accent, and voiceprint of the voice command features of the anomalous task according to the adjustment coefficient level to obtain a mutated voice command; adjusting the noise energy distribution of the scene noise of the anomalous task according to the adjustment coefficient level to obtain mutated scene noise; combining the mutated voice command with the mutated scene noise to obtain a set of mutated test tasks; and determining a unique mutated test task based on the similarity between the test task sequence and the mutated test task set. The mutated test task is used to enhance the coverage and relevance of the test task sequence.

[0011] In some possible implementations, determining a unique variant test task based on the similarity between the test task sequence and the variant test task set includes: calculating the speech feature offset and scene noise energy distribution offset for each test task; mapping the speech feature offset and scene noise energy distribution offset to a feature difference vector, which is used to characterize the comprehensive difference between the variant test task and existing test tasks in the speech and noise dimensions; based on the feature difference vector, calculating the comprehensive difference between each variant test task in the variant test task set and existing test tasks in the test task sequence, and statistically obtaining a comprehensive difference sequence; selecting the variant test task with the largest comprehensive difference in the comprehensive difference sequence as the unique variant test task, which is used to improve the differentiated coverage of the test task sequence in complex speech environments.

[0012] In some possible implementations, constructing an offline context speech sample set based on collected multi-scene noise data and multi-user voice control data includes: collecting multi-scene noise data and multi-user voice control data; classifying the multi-scene noise data to obtain various scene noises; extracting features from the multi-user voice control data to obtain speech rate variation parameters, accent shift parameters, and voiceprint feature parameters; setting initial standard voice commands, and adjusting the speech rate, accent, and voiceprint of the initial standard voice commands according to the speech rate variation parameters, accent shift parameters, and voiceprint feature parameters to obtain various voice commands; and constructing an offline context speech sample set by combining the scene noises and the voice commands, wherein the offline context speech sample set is used to subsequently generate test task sequences.

[0013] In some possible implementations, the voice sample set includes voice commands and scene noise. Generating a test task sequence based on the voice sample set includes: combining a single voice command with randomly selected single scene noise to form a first type of test task, which is used to evaluate the recognition accuracy and response performance of the smart speaker under a single command and single noise environment; sequentially combining multiple consecutive voice commands with multiple randomly selected segments of scene noise to form a second type of test task, which is used to verify the recognition stability and response consistency of the smart speaker under continuous command input and noise variation conditions in the same recognition session; mixing the first type of test task and the second type of test task according to a preset ratio, and randomly combining and sorting the mixed test tasks to form a test task sequence.

[0014] In some possible implementations, after generating the performance evaluation value for each test task based on the recognition accuracy parameter, execution matching rate parameter, and execution response latency parameter, the method further includes correcting the performance evaluation value. The correction includes: calculating a correction factor based on the voice command features of the test task, the corresponding scene type, and historical performance data; and using the correction factor to correct the performance evaluation value to obtain the final performance evaluation value.

[0015] In some possible implementations, after marking the test task corresponding to the performance evaluation value that is less than a preset performance threshold as an abnormal task, the method further includes executing an abnormal alarm based on the abnormal task. Executing the abnormal alarm includes: statistically analyzing the abnormal tasks to determine whether the number of consecutive occurrences of the abnormal task reaches a preset threshold; when the number of consecutive abnormal tasks reaches the preset threshold, triggering an abnormal alarm and issuing corresponding abnormal alarm information, the abnormal alarm information including the voice command characteristics of the abnormal task, the corresponding scenario type, and the number of consecutive abnormal occurrences, to assist R&D or testing personnel in locating weak points in the system.

[0016] This application provides an offline voice-controlled smart speaker testing system, comprising: a voice sample construction module for constructing an offline voice sample set based on collected multi-scene noise data and multi-user voice control data; a test task generation module for receiving the voice sample set from the voice sample construction module and generating a test task sequence based on the voice sample set, wherein the test task sequence includes at least one test task, and each test task includes voice command features and a corresponding scene type; a test execution module for receiving the test task sequence from the test task generation module and controlling the smart speaker to execute each test task in the test task sequence, recording the voice recognition result and command execution result of each test task; and a performance calculation module for receiving the voice recognition result and command execution result from the test execution module and calculating a performance evaluation value based on the voice recognition result and command execution result, specifically including: based on the voice recognition result and the command execution result... The system compares audio sample sets to calculate recognition accuracy parameters; calculates execution matching rate parameters and execution response delay parameters based on the instruction execution results; generates performance evaluation values ​​for each test task based on the recognition accuracy parameters, execution matching rate parameters, and execution response delay parameters; an abnormal task marking module receives performance evaluation values ​​from the performance calculation module and marks test tasks with performance evaluation values ​​less than a preset performance threshold as abnormal tasks; a variant task generation module generates variant test tasks based on the voice instruction features and corresponding scenario types of the abnormal tasks and adds the variant test tasks to the test task sequence; a performance evaluation module calculates the performance evaluation values ​​of each test task after all test tasks in the test task sequence have been executed to form a performance evaluation set; and a report output module outputs a test result report based on the performance evaluation set, which characterizes the comprehensive processing capability and user instruction response effect of the voice control system in offline voice interaction scenarios.

[0017] As can be seen from the above technical solution, this application has the following beneficial effects:

[0018] 1. This application constructs an offline voice sample set of multi-scene noise and multi-user voice commands and generates a structured test task sequence to achieve a comprehensive quantitative evaluation of the stability of voice recognition, consistency of command execution and response performance of smart speakers in offline state, thereby making the test results more representative, comparable and traceable, and making up for the problem of insufficient coverage of existing single-scene or static noise tests.

[0019] 2. This application generates mutation test tasks based on the correlation analysis of abnormal tasks, adjustment of coefficient levels and targeted feature mutations, and selects unique mutation test tasks to be added to the sequence through feature difference degree. This enables targeted stress testing and efficient coverage of weak links in the system, significantly improving the test's relevance and efficiency, providing R&D personnel with refined quantitative diagnostic basis, and supporting targeted algorithm optimization and system improvement. Attached Figure Description

[0020] Figure 1 This is a flowchart of an offline voice-controlled smart speaker testing method according to this application;

[0021] Figure 2 This is an example diagram illustrating the composition of an offline voice-controlled smart speaker testing system according to this application. Detailed Implementation

[0022] The terms "first," "second," and "third," etc., used in this application specification, claims, and drawings are used to distinguish different objects, not to limit a specific order.

[0023] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0024] Research has revealed that existing offline voice-controlled smart speaker testing methods typically rely solely on single-scene or static noise environments for recognition and response evaluation. The testing tasks are primarily based on pre-recorded voice commands, lacking coverage of diverse voice expression characteristics such as different users' speech rates, accents, and voiceprints. Therefore, when faced with diverse background noise in complex environments, the testing tasks cannot fully simulate actual usage scenarios, resulting in test results that cannot accurately reflect the speaker's recognition stability, command execution consistency, and response performance in offline states. Consequently, it is impossible to accurately evaluate the system's comprehensive processing capabilities and user command response effects under complex voice interaction conditions.

[0025] To address the aforementioned issues, this application provides an offline voice-controlled smart speaker testing method. The smart speaker is in an offline state. The method includes: constructing an offline context voice sample set based on collected multi-scene noise data and multi-user voice control data; generating a test task sequence based on the voice sample set, wherein the test task sequence includes at least one test task, and the test task includes voice command features and a corresponding scene type; controlling the smart speaker to execute each test task in the test task sequence, and recording the voice recognition result and command execution result of each test task; comparing the voice recognition result with the voice sample set to calculate a recognition accuracy parameter; and calculating the execution accuracy parameter based on the command execution result. The system generates performance evaluation values ​​for each test task based on the recognition accuracy parameter, execution matching rate parameter, and execution response latency parameter. Test tasks with performance evaluation values ​​less than a preset performance threshold are marked as abnormal tasks. Mutant test tasks are generated based on the voice command characteristics and corresponding scenario types of the abnormal tasks, and these mutant test tasks are added to the test task sequence. After all test tasks in the test task sequence are executed, the performance evaluation values ​​of each test task are statistically analyzed to form a performance evaluation set. Based on the performance evaluation set, a test result report is output, which characterizes the comprehensive processing capability and user command response effect of the voice control system in offline voice interaction scenarios.

[0026] Example 1

[0027] like Figure 1 As shown, this application provides a testing method for offline voice-controlled smart speakers, applied to evaluate the speech recognition and command execution performance of smart speakers in offline states. The aim is to comprehensively evaluate the overall processing capabilities and user command response effects of smart speakers in complex offline voice interaction environments by comprehensively analyzing multi-scenario noise data and multi-user voice commands, thereby generating test task sequences, analyzing abnormal tasks, and expanding mutated tasks. Specifically, it includes the following steps:

[0028] S1. Based on the collected multi-scene noise data and multi-user voice control data, construct an offline scenario voice sample set.

[0029] Specifically, noise data from various usage scenarios, including homes, offices, and outdoors, is collected, along with voice control data from users of different genders and age groups. The collected scene noise data is categorized to obtain several typical scene noise types. Feature extraction is performed on the multi-user voice control data to obtain speech rate variation parameters, accent shift parameters, and voiceprint feature parameters. Initial standard voice commands are set and adjusted based on the extracted speech rate variation parameters, accent shift parameters, and voiceprint feature parameters to generate multiple voice commands. Finally, these multiple voice commands are combined with various scene noises to form an offline contextual voice sample set.

[0030] It should be noted that scene noise type refers to a representative noise category obtained after classifying and analyzing the collected multi-scene noise data. Examples include television and kitchen operation sounds in a home environment, printer and telephone sounds in an office environment, and traffic and wind sounds in an outdoor environment. Each scene noise type includes typical acoustic characteristics of that environment, including frequency distribution, energy intensity, and temporal variation patterns. These are used to combine with different voice commands when constructing the speech sample set for offline scenarios, ensuring that the test task sequence covers diverse environmental noise conditions and enabling a comprehensive evaluation of the smart speaker's speech recognition and command execution performance in offline mode.

[0031] The beneficial effects of offline speech sample sets include: by combining multi-scene noise and multi-user speech features, unlike existing methods that rely solely on a single scene or static noise for testing, it can cover multi-dimensional speech features such as speech rate, accent, and voiceprint, as well as noise changes in complex environments. This ensures the diversity and relevance of tasks in the subsequently generated test task sequence, significantly improving the test's ability to comprehensively reflect the accuracy of speech recognition, consistency of command execution, and response performance of smart speakers in offline states.

[0032] S2, Based on the speech sample set, generate a test task sequence.

[0033] Specifically, a single voice command is combined with randomly selected single-scene noise to form the first type of test task, which is used to evaluate the recognition accuracy and response performance of the smart speaker under a single command and single noise environment. Multiple consecutive voice commands are combined sequentially with multiple randomly selected scene noise segments to form the second type of test task, which is used to verify the recognition stability and response consistency of the smart speaker under continuous command input and noise variation conditions in the same recognition session. Then, the first type of test task and the second type of test task are mixed according to a preset ratio, and the mixed test tasks are randomly combined and sorted to finally form a complete test task sequence.

[0034] It should be noted that the test task sequence includes at least one test task, which includes voice command features and corresponding scenario types. The voice command features include speech rate, accent, and voiceprint.

[0035] It should be noted that the preset ratio can be set according to the test objectives and scenario coverage requirements. For example, the first type of test task can be set to account for 30% of the test task sequence, and the second type of test task can account for 70% to ensure a reasonable distribution of single instruction and continuous instruction test tasks in the test sequence, thereby comprehensively evaluating the recognition accuracy and response performance of the smart speaker under different noise environments and voice input modes.

[0036] It should be noted that the first type of test task and the second type of test task are mixed in a preset ratio. Specifically, different types of test tasks are arranged in an alternating manner to form a set of structured but randomly ordered test tasks. This ensures that the test sequence includes both evaluation tasks of single instructions under specific noise conditions and evaluation tasks of continuous instructions under varying noise conditions. This improves the coverage and diversity of the test task sequence, so that the stability of speech recognition and the continuity of instruction execution of the smart speaker in offline mode can be fully reflected.

[0037] It should be noted that, in some implementations, the generated first and second type of test tasks can be combined according to a preset ratio to form a test task sequence. Subsequently, the differences between each test task in the mixed test task set and other test tasks in terms of speech command features and scene noise type are analyzed and mapped into a task difference vector, which is used to characterize the comprehensive differences of each task in the speech and noise dimensions. A comprehensive difference degree sequence is calculated based on the task difference vector, and the mixed test task set is sorted according to the magnitude of the comprehensive difference degree, prioritizing tasks with larger differences to ensure that the test task sequence has high coverage and diversity under different speech commands and scene noise conditions. Finally, a test task sequence is generated according to the sorting result, forming a comprehensive and logically ordered offline test task sequence.

[0038] The advantages of using test task sequences include: by combining multiple dimensions and randomizing them, compared with existing methods of testing tasks in a single scenario or fixed order, the coverage and relevance of test tasks can be significantly improved, and the voice recognition capability and command execution performance of smart speakers in offline state can be more comprehensively reflected.

[0039] S3, control the smart speaker to execute each test task in the test task sequence, and record the speech recognition result and instruction execution result of each test task.

[0040] Specifically, the voice commands for the test tasks are input into the smart speaker, enabling its voice recognition system to receive the commands and trigger the corresponding execution actions. During execution, the voice recognition results for each test task are collected via a connected recording device or internal log function, including the recognized command text, recognition confidence level, and recognition time. Simultaneously, the command execution results are collected, including the actual action type, execution status, and response latency. After all recorded information is saved, it can be used to calculate the recognition accuracy, execution matching rate, and response latency for each test task, thereby generating performance evaluation data and providing a basis for subsequent anomaly task identification and variant test task generation.

[0041] S4. Based on the comparison between the speech recognition result and the speech sample set, calculate the recognition accuracy parameter.

[0042] Specifically, first, the text content of the recognition result is matched with the standard instruction text, and the text matching score is recorded; second, the speech features in the recognition result (such as speech rate, voiceprint features and accent shift) are compared with the feature parameters of the corresponding instructions in the speech sample set in multiple dimensions, and the speech feature matching score is calculated; then, the text matching score and the speech feature matching score are weighted and synthesized according to preset weights to obtain the comprehensive recognition accuracy parameter for each test task.

[0043] It should be noted that the standard instruction text refers to the pre-set benchmark text used to evaluate the accuracy of the smart speaker's voice recognition. Each standard instruction text corresponds to a specific functional instruction and is used to compare with the smart speaker's recognition results.

[0044] It should be noted that the preset weights are used to weight and synthesize the text matching score and the speech feature matching score. The weights can be set according to the test objectives and the sensitivity requirements of each dimension of speech recognition. For example, the weight of the text matching score can be set to 0.6 to 0.8 and the weight of the speech feature matching score can be set to 0.2 to 0.4, so as to take into account both the correctness of the instruction and the degree of speech feature matching in the overall recognition accuracy, thereby more realistically reflecting the recognition ability of the smart speaker in the offline environment.

[0045] S5. Based on the execution result of the instruction, calculate the execution matching rate parameter and the execution response delay parameter.

[0046] Specifically, the actual execution results of the smart speaker's commands are compared with the expected actions of the corresponding test tasks. This comparison considers not only whether the action types are consistent but also the completeness and correctness of the action completion, such as whether the volume adjustment reaches the preset value or whether the content of the played audio is completely matched. By quantifying the action deviation of each test task, an execution matching rate parameter is calculated. The execution matching rate parameter ranges from 0 to 1, with a value closer to 1 indicating a higher degree of matching between the actual execution and the expected action.

[0047] During execution, the time interval from voice command input to action completion is recorded. By statistically analyzing the mean, variance, and peak value of the response delay, a delay feature vector is formed. The delay feature vector is then compared with a preset reference delay interval to obtain the execution response delay parameter.

[0048] S6. Generate a performance evaluation value for each test task based on the recognition accuracy parameter, execution matching rate parameter, and execution response latency parameter.

[0049] Specifically, for each test task, the recognition accuracy parameter and the execution matching rate parameter are weighted according to preset weights to obtain a comprehensive accuracy value, where the weights reflect the relative importance of speech recognition and command execution in the overall performance. Second, the execution response latency parameter is standardized, and its value is converted to the range of 0 to 1, so that the larger the value, the more timely the response. Finally, the comprehensive accuracy value and the standardized response latency value are calculated together to obtain the performance evaluation value of each test task. The performance evaluation value is used to quantify the overall performance of the smart speaker under the test task, and the closer the value is to 1, the better the task execution performance.

[0050] Furthermore, after generating performance evaluation values ​​for each test task based on recognition accuracy parameters, execution matching rate parameters, and execution response latency parameters, these performance evaluation values ​​can be corrected to improve evaluation accuracy. Specifically, based on the voice command characteristics, corresponding scene type, and historical performance data of each test task, a correction factor is calculated to reflect the performance deviation of the smart speaker under specific voice characteristics or scene noise conditions. Subsequently, the correction factor is applied to the generated performance evaluation values ​​to adjust the overall accuracy value and the standardized response latency value, resulting in the final performance evaluation value. The final performance evaluation value can more realistically characterize the overall performance of the smart speaker under different voice commands and complex noise environments.

[0051] S7, mark the test task corresponding to the performance evaluation value that is less than the preset performance threshold as an abnormal task.

[0052] Specifically, for each test task, after obtaining its performance evaluation value, it is compared with a preset performance threshold. When the performance evaluation value is lower than the preset performance threshold, the test task is marked as an abnormal task, and its corresponding voice command features, scene type, and performance evaluation value are recorded for subsequent analysis and processing. The preset performance threshold can be set according to the performance requirements of the offline voice control system. It can reflect the system's comprehensive processing capability under normal conditions and accurately identify tasks that perform abnormally under specific voice features or scene noise, thereby providing a basis for generating targeted variation test tasks.

[0053] Furthermore, for test tasks marked as anomalous, the number of consecutive occurrences of each task in the test task sequence is counted, and this consecutive anomaly count is compared with a preset consecutive anomaly threshold. When the number of consecutive anomalous tasks reaches the preset threshold, an anomaly alarm is triggered, and an anomaly alarm message is generated. The anomaly alarm message includes the voice command characteristics of the anomalous task, the corresponding scenario type, and the number of consecutive anomalies. This information is used to assist R&D or testing personnel in analyzing the system's weaknesses under specific voice characteristics or scenarios, thereby providing a basis for optimizing test task design and improving smart speaker performance.

[0054] S8. Generate a mutation test task based on the voice command characteristics and corresponding scene type of the abnormal task, and add the mutation test task to the test task sequence.

[0055] Specifically, firstly, for each anomalous task, its voice command features are correlated with the corresponding scene type, as well as the recognition accuracy, execution matching rate, and execution response latency parameters of that anomalous task, to determine the specific reasons for the low performance evaluation value, including but not limited to abnormal speech rate, accent deviation, voiceprint differences, or background noise interference. This step accurately identifies the performance weaknesses of smart speakers under specific voice features and scene conditions, providing a basis for the targeted generation of variant test tasks.

[0056] Secondly, based on the cause of the abnormal task and its performance evaluation value, the performance deviation is calculated against a preset performance threshold. Then, based on the performance deviation and the cause of the abnormality, a corresponding adjustment coefficient level is set using a preset level mapping strategy. This adjustment coefficient level quantifies the adjustment magnitude of voice command features and scene noise, enabling targeted adjustments to the speech rate, accent, voiceprint, and scene noise of the abnormal task when generating mutation test tasks, thus achieving precise control of directional mutation.

[0057] Furthermore, based on the set adjustment coefficient level, targeted feature mutation is performed on the abnormal task. The targeted feature mutation includes: adjusting the speech rate, accent and voiceprint of the speech command features of the abnormal task to obtain the mutated speech command; adjusting the noise energy distribution of the scene noise of the abnormal task to obtain the mutated scene noise; and combining the mutated speech command with the mutated scene noise to generate a set of mutated test tasks, so as to increase the coverage of the test task sequence under different speech features and noise conditions.

[0058] Finally, the speech feature offset and scene noise energy distribution offset of the existing test tasks and the variant test task set in the test task sequence are calculated, and these are mapped to feature difference vectors to characterize the comprehensive difference between the variant test tasks and the existing test tasks in the speech and noise dimensions. The comprehensive difference degree between each variant test task and the existing test tasks in the variant test task set is calculated, and a comprehensive difference degree sequence is generated. The variant test task with the largest comprehensive difference degree is selected as the unique variant test task. This task is used to improve the differentiated coverage of the test task sequence in complex speech environments, and the unique variant test task is added to the original test task sequence to form an expanded test task sequence.

[0059] It should be noted that in some implementations, the speech feature offset and scene noise energy distribution offset of each mutation test task are first normalized, and then weighted and integrated according to preset weights to obtain the comprehensive difference degree, which is used to reflect the comprehensive difference degree of the mutation test task in speech features and noise conditions.

[0060] It should be noted that the adjustment coefficient level is used to quantify the adjustment range of speech command features and scene noise when generating mutation test tasks for anomalous tasks. In some implementations, the adjustment coefficient level can be divided into multiple preset levels, each level corresponding to a performance deviation range. The greater the performance deviation, the higher the corresponding adjustment coefficient level, thus allowing for greater targeted adjustments to speech rate, accent, voiceprint, and scene noise; conversely, a smaller performance deviation corresponds to a lower adjustment coefficient level, requiring only fine-tuning. The level division and corresponding adjustment range can be determined through statistical analysis of historical test data and speech recognition sensitivity to ensure that the mutation test task covers the features of anomalous tasks without introducing unnatural speech or noise deviations, achieving precise control of targeted mutation.

[0061] It should be noted that, in some implementations, the speech feature offset can be obtained by collecting and analyzing the differences between the speech instructions of the test task and the corresponding reference instructions in terms of multi-dimensional features such as speech rate, accent and voiceprint. By normalizing the deviations of each dimension and weighting and integrating them, the speech feature offset that reflects the overall deviation of the speech instructions is finally obtained. This offset is used to guide the directional feature variation of abnormal tasks and enhance the coverage of test task sequences.

[0062] It should be noted that, in some implementations, the scene noise energy distribution offset can be obtained by collecting and analyzing the energy distribution characteristics of noise samples in the test task, including frequency band energy, overall loudness, and temporal variation patterns. By comparing the characteristic differences between the mutated scene noise and the original test task scene noise, and normalizing and integrating the differences in each dimension, the scene noise energy distribution offset, reflecting the overall deviation of the noise samples in terms of energy distribution and acoustic characteristics, is finally obtained. This offset is used to guide the selection of mutated test tasks, ensuring that the generated mutated tasks have maximum difference in the noise dimension, thereby enhancing the coverage and representativeness of the test task sequence in complex environments.

[0063] Furthermore, identifying a unique mutation test task and adding it to the original test task sequence has the following advantages:

[0064] 1. Existing technologies typically rely solely on static or randomly combined voice commands and noise for testing. In contrast, this application generates unique test tasks through targeted mutation based on anomalous tasks, enabling targeted verification of weak links in the system and revealing performance bottlenecks that are difficult to detect using traditional methods.

[0065] 2. By comprehensively analyzing the speech feature offset and the scene noise energy distribution offset, the unique variant test task selected by this method has the greatest difference in both speech and noise dimensions, thus ensuring the multidimensional coverage effect of the test task sequence in complex environments, which is something that existing technologies cannot achieve.

[0066] 3. The newly added unique variant test task can fill the weak areas in the original sequence, making the test task sequence not only more comprehensive but also more targeted, effectively improving the test results' ability to characterize the offline recognition stability and response consistency of smart speakers.

[0067] 4. Selecting a unique mutation test task from the mutation test task set and adding it to the original test task sequence effectively avoids duplicate testing and redundant task inputs, enabling the test task sequence to maintain comprehensive coverage while possessing higher structural simplicity and execution efficiency. This method generates more representative performance evaluation data, providing developers with quantitative analysis of the performance weaknesses of smart speakers under specific voice features or scenario types, thereby supporting targeted optimization of subsequent algorithm parameters and improvement of system performance. This method significantly outperforms existing technologies in terms of the refinement and differential coverage of test feedback.

[0068] S9. After all test tasks in the test task sequence have been executed, the performance evaluation value of each test task is calculated to form a performance evaluation set.

[0069] Specifically, the performance evaluation values ​​for each test task are organized according to the task execution order, and an index is created based on the task type (including single-instruction tasks and multi-instruction tasks) and its corresponding scenario type to form a structured performance evaluation dataset. This performance evaluation dataset systematically reflects the speech recognition accuracy, instruction execution matching rate, and response latency characteristics of the smart speaker under different combinations of speech features and scene noise conditions, providing a unified data foundation for subsequent performance trend calculations and abnormal task identification.

[0070] Furthermore, the technical advantage of generating the performance evaluation set lies in the following: by systematically archiving and multi-dimensionally correlating the performance data of each test task, quantitative analysis of offline voice control performance can be achieved, making the test data comparable, traceable, and cross-scenario correlated. This performance evaluation set provides high-resolution data support for subsequent comprehensive performance analysis, enabling developers to discover potential deviations in the speech recognition and execution process from a global perspective, thereby supporting algorithm parameter correction and continuous optimization of the speech recognition model. Compared to the existing technology that only performs discrete statistics on the results of a single task, this embodiment achieves a technical shift from single-task results to system-level performance mapping by establishing a performance evaluation set, significantly improving the accuracy and analytical value of offline voice control testing.

[0071] S10, based on the performance evaluation set, output a test result report.

[0072] Specifically, the final performance evaluation values ​​for each test task in the performance evaluation set are statistically grouped according to task type and scenario type. Comprehensive statistical indicators such as average recognition accuracy, average execution matching rate, and average response latency are calculated, and a system-level performance evaluation result is generated through a multi-parameter fusion algorithm. The test result report is presented in the form of charts, indicator ranges, and performance trend curves to characterize the smart speaker's comprehensive processing capabilities, speech recognition stability, and command response effect in offline voice interaction scenarios.

[0073] Furthermore, the technical advantage of generating test result reports lies in the following: through grouped calculations and result visualization based on the performance evaluation set, it can intuitively display the overall distribution and fluctuation trend of system performance under different speech characteristics and noise scenarios, providing R&D and testing personnel with quantifiable and comparable system-level performance feedback. Unlike the existing technology that relies solely on static reports based on single test data, this embodiment achieves dynamic characterization and decision-oriented output of offline voice control system performance through statistical analysis and result modeling of the comprehensive performance evaluation set, significantly enhancing the guiding value and engineering application depth of the test results.

[0074] This application presents an offline voice-controlled smart speaker testing method with the following advantages: Existing technologies typically rely on single or limited test data, covering limited voice features and noise environments, making it difficult to comprehensively quantify the overall performance of the system in complex scenarios. Furthermore, they lack targeted analysis and optimization methods for abnormal tasks. The method in this application constructs offline voice sample sets across multiple scenarios and users, generating structured and comprehensive test task sequences. It then combines abnormal task identification and targeted feature mutation to generate unique variant test tasks, achieving differentiated coverage and refined control of the test task sequences. In addition, it provides systematic and traceable quantitative performance information through performance evaluation sets and visualization reports, effectively supporting developers in accurately identifying system weaknesses, optimizing algorithm parameters, and improving voice recognition and command response capabilities, thereby significantly enhancing the comprehensiveness, relevance, and engineering application value of the testing. This method achieves a refined and systematic evaluation mechanism that existing technologies cannot reach.

[0075] Example 2

[0076] like Figure 2 As shown in the figure, this application provides an offline voice-controlled smart speaker testing system, specifically including:

[0077] The speech sample construction module is used to build an offline contextual speech sample set based on multi-scene noise data and multi-user voice control data. Specifically, the module performs spectral analysis and energy distribution statistics on the collected background noise data to extract scene noise energy parameters; it also performs voiceprint feature extraction, speech rate normalization, and accent shift parameter calculation on the user voice control data to obtain a speech command feature set. The module combines multi-scene noise parameters with multi-user speech feature data to construct an offline speech sample set containing multiple types of speech samples, providing a data foundation for subsequent test task generation.

[0078] The test task generation module receives a speech sample set from the speech sample construction module and generates a test task sequence based on the speech sample set. Specifically, the test task generation module matches the speech samples in the speech sample set according to speech command features and scene types, generating at least one test task based on different combinations of speech features and scene noise conditions. Each test task includes speech command features and a corresponding scene type label. The test task generation module divides the generated test tasks into single-command tasks and multi-command tasks based on task complexity to cover different voice interaction scenarios. Through the above generation method, the test task sequence has high representativeness and completeness in terms of speech feature distribution and scene types, which can improve the diversity and adaptability of offline voice control testing. Finally, the test task generation module generates a complete test task sequence for subsequent test execution and performance evaluation.

[0079] The test execution module receives a sequence of test tasks from the test task generation module, controls the smart speaker to execute each test task sequentially, and records the speech recognition results and command execution results generated during the execution process. Specifically, the module inputs the test task content to the smart speaker through the command scheduling interface. After the speaker outputs a response, the module synchronously records the recognition results and execution response, and inputs the records to the performance calculation module.

[0080] The performance calculation module receives speech recognition results and command execution results from the test execution module and calculates performance evaluation values ​​based on these results. First, the module compares the speech recognition results with the speech sample set to calculate the recognition accuracy parameter; second, it calculates the execution matching rate parameter based on the command execution results; and finally, it calculates the execution response latency parameter based on the command execution response time. Based on the recognition accuracy parameter, execution matching rate parameter, and execution response latency parameter, the performance calculation module generates performance evaluation values ​​for the corresponding test task, thereby achieving multi-dimensional quantification of the offline voice control performance of the smart speaker.

[0081] The abnormal task marking module receives performance evaluation values ​​from the performance calculation module and marks test tasks with performance evaluation values ​​below a preset performance threshold as abnormal tasks. This module performs hierarchical management of performance evaluation values ​​to ensure accurate identification of test tasks with significant deviations in speech recognition performance or command execution performance, providing a basis for the generation of subsequent variant tasks.

[0082] The mutation task generation module generates mutation test tasks based on the voice command features of anomalous tasks and their corresponding scene types. Specifically, the module generates new voice command samples by controlling the perturbation of voice feature parameters (such as voiceprint features, speech rate parameters, or accent shift parameters) of anomalous tasks; and constructs mutation test tasks by combining these with the scene type of the original anomalous tasks. This module automatically adds the generated mutation test tasks to the test task sequence to enhance test coverage and robustness testing capabilities, thereby making subsequent performance evaluation results more comprehensive.

[0083] The performance evaluation module, after all test tasks in the test task sequence have been executed, calculates the performance evaluation value for each test task and forms a performance evaluation set. This module then indexes and organizes the performance evaluation values ​​according to the task execution order and task category, creating a structured performance evaluation dataset. The performance evaluation set comprehensively reflects the multi-dimensional performance characteristics of the smart speaker under different combinations of voice features and noisy environments, including recognition accuracy, execution matching rate, and response latency, providing unified data support for the final report output.

[0084] The report output module receives the performance evaluation set from the performance evaluation module and outputs a test result report based on the set. Specifically, the module performs grouped statistical analysis on the performance data in the performance evaluation set, calculating comprehensive performance parameters such as average recognition accuracy, average execution matching rate, and average response latency; and generates system-level performance indicators based on a multi-dimensional performance parameter fusion method. The test result report is output in the form of visual charts and data summaries, used to characterize the comprehensive processing capabilities and user command response effects of the intelligent speaker voice control system in offline voice interaction scenarios.

[0085] This application discloses an offline voice-controlled intelligent speaker testing system, which offers the following advantages: Through the collaborative work of a voice sample construction module, a test task generation module, a test execution module, a performance calculation module, an abnormal task marking module, a variant task generation module, a performance evaluation module, and a report output module, it achieves fully automated testing from voice sample acquisition, test task generation, performance calculation to result analysis. This system can quantitatively evaluate the voice recognition accuracy, command execution matching, and response performance of intelligent speakers in offline mode, and generate variant test tasks based on the characteristics of abnormal tasks, thereby dynamically expanding the test task sequence and improving test coverage and relevance. Compared with existing technologies that rely on fixed voice samples and manual judgment of test results, this system not only enables systematic testing across multiple scenarios and features, but also provides traceable and comparable performance evaluation data, significantly improving the accuracy, efficiency, and engineering application value of offline voice control testing.

[0086] The foregoing has shown and described the basic principles, main features, and advantages of this application. Those skilled in the art should understand that this application is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this application. Various changes and modifications can be made to this application without departing from the spirit and scope thereof, and all such changes and modifications fall within the scope of this application as claimed. The scope of protection of this application is defined by the appended claims and their equivalents.

Claims

1. A method for testing an off-line voice-controlled smart speaker, the method comprising: The method comprises the following steps: The smart speaker is in an offline state, and the method comprises the following steps: According to the collected multi-scene noise data and multi-user voice control data, a voice sample set of an offline context is constructed; Based on the voice sample set, a test task sequence is generated, which at least includes one test task, and the test task includes voice instruction features and corresponding scene types; Control the smart speaker to execute each test task of the test task sequence, and record the voice recognition result and instruction execution result of each test task; Based on the voice recognition result and the voice sample set, the recognition accuracy parameter is calculated; Based on the instruction execution result, the execution matching rate parameter and the execution response delay parameter are calculated; According to the recognition accuracy parameter, the execution matching rate parameter and the execution response delay parameter, the performance evaluation value of each test task is generated; The performance evaluation value less than the preset performance threshold is marked as an abnormal task; The voice instruction features and corresponding scene types of the abnormal task are associated with the recognition accuracy parameter, the execution matching rate parameter and the execution response delay parameter of the abnormal task to determine the cause of the low performance evaluation value; According to the cause of the abnormal task and the performance evaluation value corresponding to the abnormal task, an adjustment coefficient level is set, which is used to control the adjustment range of the scene noise and voice instruction of the abnormal task; Based on the adjustment coefficient level, directional feature variation is performed on the abnormal task to generate a variation test task, and the variation test task is added to the test task sequence; When all the test tasks of the test task sequence are executed, the performance evaluation values of each test task are formed into a performance evaluation set; 2. The method of claim 1, wherein, Based on the performance evaluation set, a test result report is output, which is used to represent the comprehensive processing capability and user instruction response effect of the voice control system in the offline voice interaction scene. The adjustment coefficient level is set according to the cause of the abnormal task and the performance evaluation value corresponding to the abnormal task, which comprises: The difference between the performance evaluation value of the abnormal task and the preset performance threshold is calculated as a performance deviation; 3. The method of claim 2, wherein, According to the cause of the abnormal task and the performance deviation, a corresponding adjustment coefficient level is set through a preset level mapping strategy. The voice instruction features include speech rate, accent and voiceprint, and the directional feature variation is performed on the abnormal task based on the adjustment coefficient level to generate a variation test task, which comprises: According to the adjustment coefficient level, the speech rate, accent and voiceprint of the voice instruction features of the abnormal task are adjusted to obtain a variation voice instruction; According to the adjustment coefficient level, the noise energy distribution of the scene noise of the abnormal task is adjusted to obtain a variation scene noise; The variation voice instruction and the variation scene noise are combined to obtain a variation test task set; Based on the similarity of the test task sequence and the variation test task set, a unique variation test task is determined, which is used to enhance the coverage and pertinence of the test task sequence.

4. The method of claim 3, wherein, The determining of the unique variant test task based on the similarity between the test task sequence and the variant test task set comprises: calculating the voice feature offset degree and the scene noise energy distribution offset degree of each test task; mapping the voice feature offset degree and the scene noise energy distribution offset degree into a feature difference vector, which is used to represent the comprehensive difference between the variant test task and the existing test task in the voice and noise dimensions; based on the feature difference vector, calculating the comprehensive difference degree between each variant test task in the variant test task set and the existing test task in the test task sequence, and obtaining a comprehensive difference degree sequence by statistics; selecting the variant test task with the maximum comprehensive difference degree in the comprehensive difference degree sequence as the unique variant test task, which is used to improve the difference coverage of the test task sequence in a complex voice environment.

5. The method of claim 1, wherein, The voice sample set of the offline context is constructed according to the collected multi-scene noise data and multi-user voice control data, comprising: collecting multi-scene noise data and multi-user voice control data; classifying the multi-scene noise data to obtain multiple scene noises; extracting features from the multi-user voice control data to obtain a speech speed variation parameter, an accent offset parameter and a voiceprint feature parameter; setting an initial voice standard instruction, adjusting the speech speed, accent and voiceprint of the initial voice standard instruction according to the speech speed variation parameter, accent offset parameter and voiceprint feature parameter to obtain multiple voice instructions; constructing a voice sample set of an offline context by combining the scene noise with the voice instruction, which is used for subsequent generation of a test task sequence.

6. The method of claim 5, wherein, The voice sample set comprises voice instructions and scene noises, and the test task sequence is generated based on the voice sample set, comprising: combining a single voice instruction with a single randomly selected scene noise to form a first type of test task, which is used to evaluate the recognition accuracy and response performance of the intelligent loudspeaker in a single instruction and single noise environment; combining a plurality of consecutive voice instructions with a plurality of randomly selected scene noises to form a second type of test task, which is used to verify the recognition stability and response continuity of the intelligent loudspeaker under the conditions of continuous instruction input and noise change in the same recognition session; mixing the first type of test task and the second type of test task according to a preset proportion, and randomly combining and sorting the mixed test tasks to form a test task sequence.

7. The method of claim 1, wherein, After generating the performance evaluation value of each test task according to the recognition accuracy parameter, the execution matching rate parameter and the execution response delay parameter, the performance evaluation value is further modified, which comprises: calculating a correction factor based on the voice instruction feature of the test task, the corresponding scene type and the historical performance data; correcting the performance evaluation value using the correction factor to obtain the final performance evaluation value.

8. The method of claim 1, wherein, The method further comprises the following steps of: marking the test task corresponding to the performance evaluation value less than the preset performance threshold as an abnormal task; and performing abnormal alarm based on the abnormal task. The method further comprises the following steps of: counting the abnormal tasks; and determining whether the continuous number of abnormal tasks reaches a preset threshold. When the number of continuous abnormal tasks reaches the preset threshold, the abnormal alarm is triggered and corresponding abnormal alarm information is thrown, the abnormal alarm information including voice instruction features of the abnormal task, a corresponding scene type, and a continuous abnormal number, for assisting a research and development or test personnel in locating weak links of the system.

9. An off-line voice controlled smart speaker testing system, characterized in that, The system comprises the following modules: A voice sample construction module, configured to construct a voice sample set of an offline scenario based on collected multi-scene noise data and multi-user voice control data; A test task generation module, configured to receive the voice sample set from the voice sample construction module, and generate a test task sequence based on the voice sample set, the test task sequence including at least one test task, each test task including voice instruction features and a corresponding scene type; A test execution module, configured to receive the test task sequence from the test task generation module, and control the smart speaker to execute each test task of the test task sequence, and record voice recognition results and instruction execution results of each test task; A performance calculation module, configured to receive the voice recognition results and the instruction execution results from the test execution module, and calculate a performance evaluation value based on the voice recognition results and the instruction execution results, specifically including: calculating an identification accuracy parameter based on a comparison between the voice recognition results and the voice sample set; calculating an execution matching rate parameter and an execution response delay parameter based on the instruction execution results; and generating the performance evaluation value of each test task based on the identification accuracy parameter, the execution matching rate parameter, and the execution response delay parameter; An abnormal task marking module, configured to receive the performance evaluation value from the performance calculation module, and mark the test task corresponding to the performance evaluation value less than a preset performance threshold as an abnormal task; A variant task generation module, configured to perform correlation analysis on the voice instruction features and the corresponding scene type of the abnormal task and the identification accuracy parameter, the execution matching rate parameter, and the execution response delay parameter of the abnormal task, to determine a cause of the low performance evaluation value; set an adjustment coefficient level based on the cause of the abnormal task and the performance evaluation value corresponding to the abnormal task, the adjustment coefficient level being used to control an adjustment range of the scene noise and the voice instruction of the abnormal task; perform directional feature variation on the abnormal task based on the adjustment coefficient level, to generate a variant test task, and add the variant test task to the test task sequence; A performance evaluation module, configured to count the performance evaluation values of all test tasks in the test task sequence to form a performance evaluation set. A report output module is configured to output a test result report based on the performance evaluation set, and the test result report is used to represent the comprehensive processing capability and user instruction response effect of the voice control system in the offline voice interaction scenario.

Images