A performance evaluation method for human-computer interaction system based on big data
Through a big data-based method, combined with user feedback and system test data, the interaction quality evaluation index is calculated, and the problem of overly one-sided evaluation method of human-computer interaction system in the existing technology is solved, achieving a more accurate and comprehensive human-computer interaction quality evaluation , improving system performance and user experience.
Patent Information
- Application Number
- CN202410734969.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-07
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-06-07
AI Technical Summary
The existing human-computer interactive system performance evaluation methods are too one-sided, mainly based on task completion as the evaluation standard, and cannot fully reflect the performance and user experience of the system. Especially after technological progress and user needs increase, it is difficult to further optimize.
Using the performance evaluation method of human-computer interaction system based on big data, the user's historical feedback data and system test data of the intelligent voice assistant are collected, and the service quality, wake-up rate and environmental interference values are comprehensively analyzed, and the interaction quality evaluation index is calculated, and the interaction quality evaluation index is quantitatively evaluated. Human-computer interaction service quality provides more accurate and comprehensive human-computer interaction evaluation.
Through the evaluation method combining subjectivity and objectivity, the accuracy and efficiency of the evaluation results are improved, and the performance of intelligent voice assistants in different environments can be more comprehensively understood, help optimize system performance and user experience, and enhance system robustness and User satisfaction.
Smart Images

Figure CN118505063B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a human-computer interaction system performance evaluation method based on big data. Background Art
[0002] With the rapid development of big data technology and the advent of the information and digital era, enterprises are increasingly in urgent need of improving efficiency, saving costs, and enhancing competitiveness. Enterprises and organizations are gradually realizing that the accuracy of decision-making and the automation level of business processes can be effectively improved through the mining and analysis of massive data. In this context, the performance evaluation of human-computer interaction systems, as a key platform for realizing data-driven decision-making and intelligent operations, is becoming increasingly important.
[0003] For example, the invention patent with announcement number: CN112016786B discloses a method for evaluating the performance of a spatial human-computer interaction system, including: (1) analyzing the spatial human-computer interaction task process, constructing an event tree model for the spatial human-computer interaction task with two clues: human system and machine system, and defining the consequence state of the spatial human-computer interaction task; (2) analyzing the human-computer interaction characteristics during the task; (3) sorting out the key influencing factors affecting the success or failure of the task based on the constructed event tree model and the analyzed human-computer interaction characteristics; (4) modifying the CREAM method based on the characteristics of the spatial human-computer interaction task, and modifying the relationship between the scenario environment and the cognitive control mode accordingly; (5) establishing a human-computer interaction system performance evaluation model based on event tree modeling, key influencing factor analysis and the modified CREAM method, and obtaining the task reliability result. This application comprehensively and fully considers the impact of human error, environment, equipment and various coupling factors on performance, and the evaluation results are more scientific and reasonable.
[0004] Based on the above scheme, it is found that there are still some shortcomings in the performance evaluation of human-computer interaction systems. Specifically, when evaluating the performance of human-computer interaction systems, quantitative indicators based on task completion are generally used, based on whether the system can accurately and quickly respond to user instructions, and the ability and efficiency to complete predetermined tasks. This evaluation method can help improve the execution speed and accuracy of tasks in the early stages of technology research and development. However, with the continuous advancement of technology and the improvement of user experience requirements, evaluating the performance of human-computer interaction systems solely based on task completion is too one-sided and is not conducive to further optimization of the system. Summary of the invention
[0005] In view of the deficiencies in the prior art, the present invention provides a human-computer interaction system performance evaluation method based on big data, which can effectively solve the problems involved in the above-mentioned background technology.
[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: The present invention provides a human-computer interaction system performance evaluation method based on big data, including: collecting historical user feedback data of the intelligent voice assistant, analyzing the service quality of the intelligent voice assistant, and obtaining a user feedback service quality evaluation index of the intelligent voice assistant.
[0007] The system test data of the intelligent voice assistant is obtained, and the system test data is grouped according to the system test environment and marked as system test groups. The service wake-up rate of each system test group is obtained by analysis, and the environmental interference data of each system test group is collected. The environmental interference value of each system test group is obtained by comprehensive analysis.
[0008] A comprehensive analysis is conducted on the user feedback service quality evaluation index of the intelligent voice assistant, the service wake-up rate of each system test group, and the environmental interference value to obtain the interaction quality evaluation index of the intelligent voice assistant. Based on the interaction quality evaluation index of the intelligent voice assistant, the human-computer interaction service quality of the intelligent voice assistant is analyzed to obtain the analysis results and provide feedback.
[0009] As a further method, the user historical feedback data of the intelligent voice assistant is collected to analyze the service quality of the intelligent voice assistant. The specific analysis process is: based on the user historical feedback data of the intelligent voice assistant, including the number of new users of the intelligent voice assistant at each monitoring time point and the session volume of each user, the time-dependent function of each user's session volume is obtained through curve fitting, and the time-dependent function of each user's session volume is derived to obtain the time derivative of each user's session volume, and the number of new users at the current time monitoring point is extracted, and a comprehensive analysis is performed to obtain the user feedback service quality evaluation index of the intelligent voice assistant.
[0010] As a further method, the analysis obtains the service wake-up rate of each system test group. The specific analysis process is: based on the system test data of each system test group of the intelligent voice assistant, including the number of service calls and the number of wake-up times of each system test group, the ratio of the number of service calls and the number of wake-up times of each system test group is marked as the service wake-up rate of each system test group.
[0011] As a further method, the environmental interference data of each system test group is collected, and the environmental interference value of each system test group is obtained through comprehensive analysis. The specific analysis process is: based on the environmental interference data of each system test group, including the noise and electromagnetic interference intensity of each system test group during the test period, the critical noise and critical electromagnetic interference intensity are extracted from the intelligent voice assistant database, and the environmental interference value of each system test group is obtained through comprehensive analysis.
[0012] As a further method, a comprehensive analysis is performed on the user feedback service quality evaluation index of the intelligent voice assistant, the service wake-up rate of each system test group, and the environmental interference value. The specific analysis process is: based on the user feedback service quality evaluation index of the intelligent voice assistant, the service wake-up rate of each system test group, and the environmental interference value, a comprehensive analysis is performed to obtain the interaction quality evaluation index of the intelligent voice assistant. The interaction quality evaluation index of the intelligent voice assistant is used to quantitatively evaluate the human-computer interaction service quality of the intelligent voice assistant.
[0013] As a further method, the human-computer interaction service quality of the intelligent voice assistant is analyzed according to the interaction quality evaluation index of the intelligent voice assistant, the analysis results are obtained and feedback is provided, and the specific analysis process is: extracting the interaction quality evaluation index threshold of the intelligent voice assistant from the intelligent voice assistant database, comparing the interaction quality evaluation index of the intelligent voice assistant with the interaction quality evaluation index threshold of the intelligent voice assistant, if the interaction quality evaluation index of the intelligent voice assistant is higher than or equal to the threshold, then the human-computer interaction service quality of the intelligent voice assistant is marked as qualified, if the interaction quality evaluation index of the intelligent voice assistant is lower than the threshold, then the human-computer interaction service quality of the intelligent voice assistant is marked as unqualified.
[0014] The analysis results will be fed back to relevant personnel, who will improve the intelligent voice assistant based on the quality of human-computer interaction services.
[0015] As a further method, the user feedback service quality evaluation index of the intelligent voice assistant is a quantitative evaluation data obtained by comprehensively analyzing the number of users at each time monitoring point and the time derivative of each user's session volume. It is used to quantitatively evaluate the service quality of the intelligent voice assistant and provide a basis for evaluating the human-computer interaction service quality of the intelligent voice assistant.
[0016] As a further method, the environmental interference value of each system test group is a quantitative evaluation data obtained by comprehensively analyzing the noise and electromagnetic interference intensity of each system test group during the test cycle, which is used to quantitatively evaluate the degree of environmental interference in the intelligent voice assistant system test, and provide a basis for evaluating the human-computer interaction service quality of the intelligent voice assistant.
[0017] As a further method, the user feedback service quality evaluation index of the intelligent voice assistant is specifically expressed as:
[0018]
[0019] Where Y represents the user feedback service quality evaluation index of the intelligent voice assistant, α represents the number of new users at the current time monitoring point, s represents the number of each time monitoring point, s = 1, 2, 3, ..., h, h represents the total number of time monitoring points, αs represents the number of new users at the sth time monitoring point, r represents the number of each user, r = 1, 2, 3, ..., u, u represents the total number of users, βr represents the value of the time derivative of the rth user session volume at the current time point, ρ 1 represents the user feedback service quality impact factor corresponding to the set number of new users, ρ 2 Indicates the user feedback service quality impact factor corresponding to the set user session volume.
[0020] As a further method, the interaction quality evaluation index of the intelligent voice assistant is specifically expressed as:
[0021]
[0022] In the formula, represents the interaction quality evaluation index of the intelligent voice assistant, e represents a natural constant, Y represents the user feedback service quality evaluation index of the intelligent voice assistant, n represents the number of each system test group, n=1,2,3,...,m, m represents the total number of system test groups, Hn represents the service wake-up rate of the nth system test group, Fn represents the number of service wake-ups of the nth system test group, Xn represents the number of service calls of the nth system test group, ω represents the interaction quality evaluation influence weight factor corresponding to the set user feedback service quality evaluation index, Zn represents the environmental interference value of the nth system test group, and Z 0 Indicates the reference environmental interference threshold.
[0023] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects:
[0024] (1) The present invention provides a human-computer interaction system performance evaluation method based on big data, comprehensively analyzes the user feedback data of the intelligent voice assistant and the wake-up rate under different environmental interferences, and evaluates the quality of human-computer interaction by combining subjectivity and objectivity, which makes the evaluation results more accurate. It helps to avoid the subjective bias that may occur in traditional survey methods. Through big data analysis, a large amount of user feedback information can be quickly analyzed, greatly improving the efficiency of human-computer interaction quality evaluation.
[0025] (2) The present invention analyzes the user feedback data of the intelligent voice assistant. The user feedback data covers a variety of usage scenarios and functional experiences of the intelligent voice assistant. The user feedback data is also highly representative, which helps to fully understand the performance of the intelligent voice assistant in actual use, including its advantages and potential disadvantages.
[0026] (3) The present invention compares and analyzes the wake-up rates of each system test group of the intelligent voice assistant under different environmental interferences, and obtains the anti-interference ability of the intelligent voice assistant to environmental interference, which helps to improve the robustness of the corresponding service in a specific noise environment. A high wake-up rate means that the intelligent voice assistant can accurately recognize and respond to the user's voice commands in various environments, which helps to improve user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.
[0028] Figure 1 It is a schematic diagram of the method flow of the present invention.
[0029] Figure 2 It is a schematic diagram of the functional relationship between the noise level of each system test group and the environmental interference value of each system test group involved in an embodiment of the present invention. DETAILED DESCRIPTION
[0030] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0031] Reference Figure 1 As shown, the present invention provides a human-computer interaction system performance evaluation method based on big data, including: collecting historical user feedback data of the intelligent voice assistant, analyzing the service quality of the intelligent voice assistant, and obtaining a user feedback service quality evaluation index of the intelligent voice assistant.
[0032] Specifically, the user historical feedback data of the intelligent voice assistant is collected, and the service quality of the intelligent voice assistant is analyzed. The specific analysis process is: based on the user historical feedback data of the intelligent voice assistant, including the number of new users of the intelligent voice assistant at each monitoring time point and the session volume of each user, a time-dependent function of each user's session volume is obtained through curve fitting, and the time-dependent function of each user's session volume is derived to obtain the time derivative of each user's session volume, the most recent time monitoring point is marked as the current time monitoring point, the number of new users at the current time monitoring point is extracted, and a user feedback service quality evaluation index of the intelligent voice assistant is obtained through comprehensive analysis.
[0033] It should be explained that the above-mentioned curve fitting specifically refers to fitting an exponential function curve of the number of new users and the number of sessions of each user by using the maximum likelihood method and linear algebra.
[0034] Specifically, the user feedback service quality evaluation index of the intelligent voice assistant is a quantitative evaluation data obtained by comprehensively analyzing the number of users at each time monitoring point and the time derivative of each user's session volume. It is used to quantitatively evaluate the service quality of the intelligent voice assistant and provide a basis for evaluating the human-computer interaction service quality of the intelligent voice assistant.
[0035] In a specific embodiment, the user feedback service quality evaluation index of the intelligent voice assistant is specifically expressed as:
[0036]
[0037] Where Y represents the user feedback service quality evaluation index of the intelligent voice assistant, α represents the number of new users at the current time monitoring point, s represents the number of each time monitoring point, s = 1, 2, 3, ..., h, h represents the total number of time monitoring points, αs represents the number of new users at the sth time monitoring point, r represents the number of each user, r = 1, 2, 3, ..., u, u represents the total number of users, βr represents the value of the time derivative of the rth user session volume at the current time point, ρ 1 represents the user feedback service quality impact factor corresponding to the set number of new users, ρ 2 Indicates the user feedback service quality impact factor corresponding to the set user session volume.
[0038] Table 1 Example of user feedback service quality evaluation index data for intelligent voice assistants
[0039]
[0040] It should be explained that, in this embodiment, the user feedback service quality impact factor corresponding to the newly added user volume and the user feedback service quality impact factor corresponding to the time derivative of the user session volume have a value range of [0.2, 0.8]. The user feedback service quality impact factor corresponding to the newly added user volume is used to adjust the degree of influence of the newly added user volume on the final service quality evaluation index. A mapping set of the newly added user volume and the user session volume and the impact factor is established based on historical data to determine the corresponding relationship between the newly added user volume and the user session volume and the impact factor. If the user feedback service quality impact factor corresponding to the newly added user volume is large, it means that the impact of the newly added user on the service quality is more significant. The user feedback service quality impact factor corresponding to the time derivative of the user session volume is used to measure the impact weight of the user session volume change rate on the service quality evaluation index. If the user session volume increases or decreases rapidly, and the user feedback service quality impact factor corresponding to the time derivative of the user session volume is large, it means that this change has an important impact on the service quality.
[0041] It should be explained that in this embodiment, the formula provides a comprehensive service quality evaluation method by integrating data from different dimensions, including the number of new users and changes in user activity, which can more accurately reflect the complexity of the intelligent voice assistant service. By considering factors that change over time, such as the rate of change of new users and session volume, the session volume change rate refers to the change in the value of the time derivative of the user session volume, and comprehensively analyzing the rate of change of new users and session volume, the formula can dynamically evaluate the service quality and adapt to changes in user behavior and market conditions. Analyzing the impact of the user feedback service quality influencing factor corresponding to the specific parameter of the number of new users and the user feedback service quality influencing factor corresponding to the time derivative of the user session volume can help decision makers identify which factors most affect the service quality and adjust strategies accordingly, such as optimizing the new user onboarding process or improving the ability of the intelligent voice assistant to handle concurrent requests.
[0042] It should be understood that, as shown in Table 1, the greater the number of new users at the current time monitoring point, the greater the user feedback service quality evaluation index of the intelligent voice assistant. At the same time, the greater the time derivative of each user's session volume, the greater the user feedback service quality evaluation index of the intelligent voice assistant. The two are positively correlated.
[0043] It should be explained that the number of new users of the intelligent voice assistant and the number of sessions of each user at each monitoring time point in this embodiment are based on a certain degree of user growth and activity data provided by a third-party statistical platform (such as Sensor Tower, App Annie) and a data reporting system or business intelligence tool (such as Tableau, Power BI) developed internally by the intelligent voice assistant system. The above data sources are integrated to provide a convenient monitoring and analysis of the visualization report of the changing trend of the number of new users and the number of sessions over time. The time derivative of each user's session volume is a quantitative evaluation data obtained by curve fitting of the user's session volume at each time monitoring point and deriving the function, which is used to quantitatively evaluate the changing trend of each user's session volume over time. The value of the time derivative of the user's session volume at each time monitoring point represents the changing trend of the user's session volume over time. If the time derivative of a user's session volume is closer to 0, it means that the user has used the intelligent voice assistant less.
[0044] It should be explained that in this embodiment, analyzing the user feedback service quality has a significant impact on the evaluation of the quality of human-computer interaction. User feedback can help developers understand the user's satisfaction with the product, so as to optimize and improve the areas where the user is dissatisfied. Through continuous adjustment and optimization, the user's satisfaction in the process of using the product is improved, and then the user's favorability and loyalty to the product are improved. User feedback can provide developers with valuable opinions on human-computer interaction design, such as interface layout, operation logic, function settings, etc. Developers can optimize the human-computer interaction design based on user feedback, so that the product is more in line with the user's usage habits and expectations, and improve the ease of use and convenience of the product. User feedback can help developers discover problems and deficiencies in the actual use of the product, such as missing functions, system bugs, performance bottlenecks, etc. In response to these problems, developers can repair and adjust them in time to ensure the stability and reliability of the product. User feedback can stimulate the developer's sense of innovation and guide them to continuously optimize product functions, performance, user experience, etc. Through the analysis and mining of user feedback, developers can find new technical breakthroughs and promote the continuous advancement of human-computer interaction technology. User feedback helps developers better understand market demand and user needs, so as to improve products in a targeted manner. Products optimized through user feedback are more competitive in terms of user experience, functional improvement, and performance optimization, which helps them stand out in the fierce market competition. Actively paying attention to and responding to user feedback can make users feel the sincerity and care of developers, which helps to establish and maintain good user relationships. Good user relationships can bring more user reputation to products and promote the dissemination and promotion of products.
[0045] The system test data of the intelligent voice assistant is obtained, and the system test data is grouped according to the system test environment and marked as system test groups. The service wake-up rate of each system test group is obtained by analysis, and the environmental interference data of each system test group is collected. The environmental interference value of each system test group is obtained by comprehensive analysis.
[0046] Specifically, the analysis obtains the service wake-up rate of each system test group. The specific analysis process is: based on the system test data of each system test group of the intelligent voice assistant, including the number of service calls and the number of wake-up times of each system test group, the ratio of the number of service calls and the number of wake-up times of each system test group is marked as the service wake-up rate of each system test group.
[0047] It should be explained that the system test data of each system test group of the intelligent voice assistant in this embodiment, including the number of service calls and wake-up times of each system test group, is obtained by simulating user behavior by using an automated testing framework (such as Selenium, Appium or tools specific to speech recognition), writing scripts to automatically execute a series of preset wake-up and service request operations, and then using an integrated testing platform to record and process the test data.
[0048] Furthermore, the environmental interference data of each system test group is collected, and the environmental interference value of each system test group is obtained through comprehensive analysis. The specific analysis process is: based on the environmental interference data of each system test group, including the noise and electromagnetic interference intensity of each system test group during the test period, the critical noise and critical electromagnetic interference intensity are extracted from the intelligent voice assistant database, and the environmental interference value of each system test group is obtained through comprehensive analysis.
[0049] It should be explained that the intelligent voice assistant database mainly refers to various data storage systems used to support the operation and service provision of intelligent voice assistants. These databases contain data in multiple aspects such as speech recognition, natural language processing, user personalization settings, historical interaction records, knowledge graphs, etc. In this embodiment, the intelligent voice assistant database specifically includes critical noise, critical electromagnetic interference intensity, and the threshold of the interaction quality evaluation index of the intelligent voice assistant. The critical noise and critical electromagnetic interference intensity can be simulated in a professional laboratory using acoustic and electromagnetic compatibility test equipment. The noise and electromagnetic interference conditions in different scenarios are measured, and the critical threshold of the equipment under these conditions is measured. These test results can be used as a reference standard for designing the anti-interference ability of voice assistant hardware. It is also possible to collect data for a long time by installing noise and electromagnetic interference monitoring equipment in the actual deployment environment to determine the interference level in a typical environment. These data help optimize the performance of voice assistants in real environments. It is also possible to identify frequent recognition errors or functional abnormalities in specific noise or electromagnetic environments through user feedback and system log analysis, and indirectly infer possible critical thresholds. The threshold of the interaction quality evaluation index of the intelligent voice assistant can be collected through questionnaires, interviews, A / B testing, etc., to collect user feedback on the interaction quality of the voice assistant, set a baseline value of user satisfaction, and use it as the evaluation threshold of the interaction quality. It can also be used to continuously monitor and evaluate the various performance indicators of the voice assistant during product development, continuously optimize the model through machine learning algorithms, and dynamically adjust the evaluation threshold based on a large amount of actual interaction data to achieve a better user experience.
[0050] Specifically, the environmental interference value of each system test group is a quantitative evaluation data obtained by comprehensively analyzing the noise and electromagnetic interference intensity of each system test group during the test cycle, which is used to quantitatively evaluate the degree of environmental interference in the intelligent voice assistant system test and provide a basis for evaluating the human-computer interaction service quality of the intelligent voice assistant.
[0051] In a specific embodiment, the specific numerical expression of the environmental interference value of each system test group is:
[0052]
[0053] Where Zn represents the environmental interference value of the nth system test group, n represents the number of each system test group, n = 1, 2, 3, ..., m, m represents the total number of system test groups, Zh → n represents the noise of the nth system test group in the test cycle, Zj → n represents the electromagnetic interference intensity of the nth system test group in the test cycle, represents the critical noise, represents the critical electromagnetic interference intensity, δ 1Indicates the environmental interference factor corresponding to the set noise, δ 2 Indicates the environmental interference level impact factor corresponding to the set electromagnetic interference intensity.
[0054] It needs to be explained that Figure 2 As shown, curve a represents the relationship between the noise of the system test group and the environmental interference value of each system test group when the electromagnetic interference intensity of the nth system test group in the test cycle is 20V / m, curve b represents the relationship between the noise of the system test group and the environmental interference value of each system test group when the electromagnetic interference intensity of the nth system test group in the test cycle is 40V / m, and curve c represents the relationship between the noise of the system test group and the environmental interference value of each system test group when the electromagnetic interference intensity of the nth system test group in the test cycle is 60V / m, Figure 2 It can be obtained that the greater the electromagnetic interference intensity of the system test group, the greater the environmental interference value of each system test group. At the same time, the greater the noise of the system test group, the greater the environmental interference value of each system test group. The two are positively correlated.
[0055] It should be explained that in this embodiment, the value range of the environmental interference degree influence factor corresponding to the noise and the environmental interference degree influence factor corresponding to the electromagnetic interference intensity is [0.3, 0.5]. A linear regression model of noise, electromagnetic interference intensity and influence factor is established based on the simulation test data. The linear regression model is a multivariate linear regression model established by combining the noise and electromagnetic interference intensity with the environmental interference degree. The training model determines the corresponding relationship between the noise and electromagnetic interference intensity and the influence factor. The environmental interference degree influence factor corresponding to the noise is an adjustment factor used to adjust the weight of the contribution of noise to the final environmental interference value. The larger the value, the more significant the impact on environmental interference at the same noise level. The environmental interference degree influence factor corresponding to the electromagnetic interference intensity determines the proportion of electromagnetic interference in the calculation of the overall interference value. The larger the value, the more significant the impact on environmental interference at the same electromagnetic interference level.
[0056] It should be understood that the main function of this formula is to quantitatively analyze and predict the comprehensive environmental interference effect caused by the combined effect of noise level and electromagnetic interference in the system test environment. The environmental interference value of each system test group obtained by calculation can directly reflect the total interference degree of the system in a specific test environment. This is crucial for evaluating the stability and reliability of the system. The formula reveals the specific impact of noise level and electromagnetic interference intensity on system performance. Designers can adjust the layout of system components, select shielding materials or take other measures to reduce interference based on the calculation results, and optimize the system design to improve its anti-interference ability. When the system behaves abnormally, the formula can help engineers analyze whether noise or electromagnetic interference (or both) is more likely to be the root cause of the problem, thereby guiding fault diagnosis and effective intervention. Before new product development or system upgrade, using this formula for simulation and prediction can help estimate the system performance under different interference conditions, take measures in advance to prevent potential problems, and reduce the cost of later debugging and correction. This formula is not only a tool for accurately measuring and understanding the impact of environmental interference on the system, but also an important basis for system design optimization, test verification, fault analysis and prevention strategy formulation.
[0057] It should be explained that the environmental interference data of each system test group in this embodiment includes the noise and electromagnetic interference intensity of each system test group during the test cycle. The noise of each system test group is obtained by deploying a dedicated noise level meter or sound level meter in the test environment, continuously monitoring and recording the environmental noise level, and the electromagnetic interference intensity of each system test group is obtained by using an automated test system, automatically performing the test according to a preset program, scanning, locating the interference source, and recording the interference source related data during the test cycle, and processing the data. In human-computer interaction, especially in applications such as speech recognition or speech control, noise may interfere with the collection and processing of voice signals, leading to misunderstandings or erroneous operations. Electromagnetic interference is a type of interference caused by electromagnetic waves, which can propagate between electronic devices and affect the transmission and processing of data. In wireless communication and sensor applications, electromagnetic interference may interfere with the transmission of signals, resulting in data loss or errors. Strong electromagnetic interference may cause the performance of electronic equipment to degrade, or even damage the equipment, affecting the normal progress of human-computer interaction. By evaluating the impact of noise and electromagnetic interference on human-computer interaction, corresponding measures can be taken to reduce these interferences, such as using noise isolation or shielding technology to improve the stability and reliability of the system, thereby improving the user experience. Analyzing the impact of environmental interference helps developers identify weaknesses in system design, prompting them to consider noise and electromagnetic interference issues during the design phase, adopt better materials, designs or shielding measures, and improve system robustness. By evaluating environmental interference, corresponding standards and specifications can be formulated for system design and operation to ensure that the system can maintain good performance in different environments. The evaluation of environmental interference can promote innovation in interactive technology, such as developing more advanced noise suppression algorithms and improving wireless communication technology to adapt to complex and changing environments.
[0058] A comprehensive analysis is conducted on the user feedback service quality evaluation index of the intelligent voice assistant, the service wake-up rate of each system test group, and the environmental interference value to obtain the interaction quality evaluation index of the intelligent voice assistant. Based on the interaction quality evaluation index of the intelligent voice assistant, the human-computer interaction service quality of the intelligent voice assistant is analyzed to obtain the analysis results and provide feedback.
[0059] Specifically, a comprehensive analysis is performed on the user feedback service quality evaluation index of the intelligent voice assistant, the service wake-up rate of each system test group, and the environmental interference value. The specific analysis process is: based on the user feedback service quality evaluation index of the intelligent voice assistant, the service wake-up rate of each system test group, and the environmental interference value, a comprehensive analysis is performed to obtain the interaction quality evaluation index of the intelligent voice assistant. The interaction quality evaluation index of the intelligent voice assistant is used to quantitatively evaluate the human-computer interaction service quality of the intelligent voice assistant.
[0060] In a specific embodiment, the interaction quality evaluation index of the intelligent voice assistant is specifically expressed as:
[0061]
[0062] In the formula, represents the interaction quality evaluation index of the intelligent voice assistant, e represents a natural constant, Y represents the user feedback service quality evaluation index of the intelligent voice assistant, n represents the number of each system test group, n=1,2,3,...,m, m represents the total number of system test groups, Hn represents the service wake-up rate of the nth system test group, Fn represents the number of service wake-ups of the nth system test group, Xn represents the number of service calls of the nth system test group, ω represents the interaction quality evaluation influence weight factor corresponding to the set user feedback service quality evaluation index, Zn represents the environmental interference value of the nth system test group, and Z 0 Indicates the reference environmental interference threshold.
[0063] It should be explained that the interaction quality evaluation index of the intelligent voice assistant in this embodiment can not only be calculated through the above formula, but also be obtained by implementing A / B testing, comparing user interaction experiences under different versions or function settings, and collecting data for analysis. It can also be obtained through the evaluation of machine learning models, that is, continuous training and evaluation of the artificial intelligence model behind the voice assistant, including the accuracy and generalization ability of key models such as intent recognition and emotion understanding, and comprehensive analysis.
[0064] It should be explained that the value range of the interaction quality assessment impact weight factor corresponding to the user feedback service quality evaluation index in this embodiment is [0, 1]. This is a weight factor used to adjust the relative importance between the user feedback service quality and the system performance reflected by the service wake-up rate under environmental interference. A mapping set of user feedback service quality, service wake-up rate under environmental interference and impact factors is established based on historical data to determine the corresponding relationship between user feedback service quality, service wake-up rate under environmental interference and impact factors. When the interaction quality assessment impact weight factor corresponding to the user feedback service quality evaluation index is close to 1, it means that more attention is paid to the impact of user direct feedback on the interaction quality evaluation; when the interaction quality evaluation impact weight factor corresponding to the user feedback service quality evaluation index is close to 0, it depends more on the impact of system performance indicators and environmental factors on the interaction quality evaluation.
[0065] It should be understood that this formula comprehensively considers the user feedback service quality, the technical performance of each test group, the environmental adaptability, and the sensitivity to user feedback, that is, the adjustment of the interaction quality evaluation influencing weight factor corresponding to the user feedback service quality evaluation index. The technical performance of each test group is represented by the service wake-up rate, and the environmental adaptability is reflected by the environmental interference ratio, thereby providing a comprehensive interaction quality evaluation index. This comprehensive consideration helps to fully grasp the performance of the voice assistant, rather than being limited to a single dimension. The user feedback service quality evaluation index reflects the user's satisfaction with the service quality of the voice assistant. The higher the user feedback service quality evaluation index, the better the user feedback, which means better service quality, and a positive contribution to the interaction quality evaluation index of the intelligent voice assistant. The service wake-up rate of the system test group indicates the proportion of the number of times the service is successfully awakened to the total number of calls. The higher the service wake-up rate of the system test group, the higher the system response efficiency, the better the user interaction experience, and the greater the contribution to the interaction quality evaluation index of the intelligent voice assistant. Environmental interference ratio Used to measure the degree of interference of the current test environment compared to the standard environment. The smaller the ratio, the less environmental interference, and the smaller the negative impact on speech recognition and interaction quality. By analyzing the calculation results and components of the interaction quality evaluation index of the intelligent voice assistant, developers and researchers can identify the key factors that affect the quality of interaction. For example, if it is found that the environmental interference ratio of a test group has a significant impact on the overall score, the robustness of speech recognition in complex environments can be optimized in a targeted manner; if users feedback that the service quality index is low, it is necessary to focus on improving service quality and enhancing user satisfaction.
[0066] It should be explained that in this embodiment, the anti-interference ability of different system test groups to the environment has a significant impact on the evaluation of the quality of human-computer interaction. By comparing the performance of different test groups under environmental interference, the stability and reliability of the human-computer interaction system in the actual use environment can be more accurately evaluated. This helps to formulate scientific test standards and ensure that the test results are practical and representative. In different test environments, the human-computer interaction system may show different performances. By analyzing the test results, the weaknesses of the system under specific environmental interference can be found, so as to optimize and improve them in a targeted manner. After optimizing and improving the human-computer interaction system, by comparing the data of different test groups, the effectiveness of the improvement measures can be verified to ensure that the system has better anti-interference ability in the actual use environment. Environmental interference is often one of the main reasons for poor user experience. By evaluating the anti-interference ability of different test groups, the impact of environmental interference on user experience can be effectively reduced and user satisfaction can be improved. Analyzing the performance of different test groups under environmental interference can provide a useful reference for system design, so that the system has better environmental adaptability. For example, when designing a speech recognition system, the algorithm can be optimized for different noise levels to improve the system's anti-noise ability. In different application scenarios, the characteristics and degree of environmental interference may be different. By analyzing the anti-interference capabilities of different test groups, targeted suggestions and solutions can be provided for practical applications.
[0067] Furthermore, the human-computer interaction service quality of the intelligent voice assistant is analyzed according to the interaction quality evaluation index of the intelligent voice assistant, the analysis results are obtained and feedback is provided, and the specific analysis process is: extracting the interaction quality evaluation index threshold of the intelligent voice assistant from the intelligent voice assistant database, comparing the interaction quality evaluation index of the intelligent voice assistant with the interaction quality evaluation index threshold of the intelligent voice assistant, if the interaction quality evaluation index of the intelligent voice assistant is higher than or equal to the threshold, then the human-computer interaction service quality of the intelligent voice assistant is marked as qualified, if the interaction quality evaluation index of the intelligent voice assistant is lower than the threshold, then the human-computer interaction service quality of the intelligent voice assistant is marked as unqualified.
[0068] The analysis results will be fed back to relevant personnel, who will improve the intelligent voice assistant based on the quality of human-computer interaction services.
[0069] In a specific embodiment, the analysis results can be fed back to relevant personnel by designing an automated reporting system. When the interaction quality assessment is completed and compared with the threshold, the system automatically generates a detailed assessment report, including the scores of each indicator, the total score, whether it is qualified, etc., and automatically sends it to the responsible team and managers via email, SMS or internal messaging system.
[0070] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.
Claims
1. A performance evaluation method for human-computer interaction system based on big data, It is characterized in that include: Collect historical user feedback data of the intelligent voice assistant, analyze the service quality of the intelligent voice assistant, and obtain the user feedback service quality evaluation index of the intelligent voice assistant; The user historical feedback data of the intelligent voice assistant is collected to analyze the service quality of the intelligent voice assistant. The specific analysis process is as follows: According to the historical user feedback data of the intelligent voice assistant, including the number of new users of the intelligent voice assistant and the number of conversations of each user at each monitoring time point, the time-dependent function of the number of conversations of each user is obtained through curve fitting, and the time-dependent function of the number of conversations of each user is derived to obtain the time derivative of the number of conversations of each user, and the number of new users at the current time monitoring point is extracted, and the user feedback service quality evaluation index of the intelligent voice assistant is obtained through comprehensive analysis; The user feedback service quality evaluation index of the intelligent voice assistant is a quantitative evaluation data obtained by comprehensively analyzing the number of users at each time monitoring point and the time derivative of each user's session volume, which is used to quantitatively evaluate the service quality of the intelligent voice assistant and provide a basis for evaluating the human-computer interaction service quality of the intelligent voice assistant; The user feedback service quality evaluation index of the intelligent voice assistant is specifically expressed as: Where Y represents the user feedback service quality evaluation index of the intelligent voice assistant, α represents the number of new users at the current time monitoring point, s represents the number of each time monitoring point, s = 1, 2, 3, ..., h, h represents the total number of time monitoring points, α s represents the number of new users at the sth time monitoring point, r represents the number of each user, r=1,2,3,...,u, u represents the total number of users, β r represents the value of the time derivative of the rth user session volume at the current time point, ρ 1 represents the user feedback service quality impact factor corresponding to the set number of new users, ρ 2 Indicates the user feedback service quality impact factor corresponding to the set user session volume; Obtain the system test data of the intelligent voice assistant, and group the system test data according to the system test environment, mark them into system test groups, analyze and obtain the service wake-up rate of each system test group, collect the environmental interference data of each system test group, and comprehensively analyze and obtain the environmental interference value of each system test group; A comprehensive analysis is conducted on the user feedback service quality evaluation index of the intelligent voice assistant, the service wake-up rate of each system test group, and the environmental interference value to obtain the interaction quality evaluation index of the intelligent voice assistant. Based on the interaction quality evaluation index of the intelligent voice assistant, the human-computer interaction service quality of the intelligent voice assistant is analyzed to obtain the analysis results and provide feedback.
2. According to the human-computer interaction system performance evaluation method based on big data according to claim 1, Features: The analysis obtains the service wake-up rate of each system test group. The specific analysis process is as follows: According to the system test data of each system test group of the intelligent voice assistant, including the number of service calls and wake-up times of each system test group, the ratio of the number of service calls and the number of wake-up times of each system test group is marked as the service wake-up rate of each system test group.
3. According to the performance evaluation method of a human-computer interaction system based on big data as described in claim 1, Features: The environmental interference data of each system test group is collected and the environmental interference value of each system test group is obtained through comprehensive analysis. The specific analysis process is as follows: According to the environmental interference data of each system test group, including the noise and electromagnetic interference intensity of each system test group during the test period, the critical noise and critical electromagnetic interference intensity are extracted from the intelligent voice assistant database, and the environmental interference value of each system test group is obtained by comprehensive analysis.
4. According to claim 3, a human-computer interaction system performance evaluation method based on big data, Features: The user feedback service quality evaluation index of the intelligent voice assistant, the service wake-up rate of each system test group and the environmental interference value are comprehensively analyzed. The specific analysis process is as follows: Based on the user feedback service quality evaluation index of the intelligent voice assistant, the service wake-up rate and environmental interference value of each system test group, a comprehensive analysis is conducted to obtain the interaction quality evaluation index of the intelligent voice assistant. The interaction quality evaluation index of the intelligent voice assistant is used to quantitatively evaluate the human-computer interaction service quality of the intelligent voice assistant.
5. According to claim 4, a human-computer interaction system performance evaluation method based on big data, Features: The human-computer interaction service quality of the intelligent voice assistant is analyzed according to the interaction quality evaluation index of the intelligent voice assistant, and the analysis results are obtained and feedback is provided. The specific analysis process is as follows: Extracting the interaction quality evaluation index threshold of the intelligent voice assistant from the intelligent voice assistant database, comparing the interaction quality evaluation index of the intelligent voice assistant with the interaction quality evaluation index threshold of the intelligent voice assistant, if the interaction quality evaluation index of the intelligent voice assistant is higher than or equal to the threshold, marking the human-computer interaction service quality of the intelligent voice assistant as qualified, and if the interaction quality evaluation index of the intelligent voice assistant is lower than the threshold, marking the human-computer interaction service quality of the intelligent voice assistant as unqualified; The analysis results will be fed back to relevant personnel, who will improve the intelligent voice assistant based on the quality of human-computer interaction services.
6. According to claim 3, a human-computer interaction system performance evaluation method based on big data, Features: The environmental interference value of each system test group is a quantitative evaluation data obtained by comprehensively analyzing the noise and electromagnetic interference intensity of each system test group during the test cycle, which is used to quantitatively evaluate the degree of environmental interference in the intelligent voice assistant system test and provide a basis for evaluating the human-computer interaction service quality of the intelligent voice assistant.
7. According to claim 4, a human-computer interaction system performance evaluation method based on big data, Features: The interaction quality evaluation index of the intelligent voice assistant is expressed as follows: In the formula, φ represents the interaction quality evaluation index of the intelligent voice assistant, e represents the natural constant, Y represents the user feedback service quality evaluation index of the intelligent voice assistant, n represents the number of each system test group, n = 1, 2, 3, ..., m, m represents the total number of system test groups, H n Indicates the service wake-up rate of the nth system test group, F n Indicates the number of service wake-up times of the nth system test group, X n represents the number of service calls of the nth system test group, ω represents the interaction quality evaluation influence weight factor corresponding to the set user feedback service quality evaluation index, Z n represents the environmental interference value of the nth system test group, Z 0 Indicates the reference environmental interference threshold.
