Application evaluation method and system

Abstract

The invention discloses an application evaluation method and system. The application evaluation method comprises the following steps: determining a model framework; determining data variables according to the model framework; performing data collection according to the determined data variables; constructing a model according to the collected data; and performing evaluation and display according to the constructed model. According to the method, the UTAUT model is expanded, more influence factors are covered, and the application effect of the artificial intelligence technology in actual use can be comprehensively evaluated through the more influence factors.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular, to an application evaluation method and system thereof. Background Art

[0002] With the development of digital technology, power grid enterprises have gradually begun to introduce artificial intelligence technology to improve operational efficiency, customer service level and decision-making quality. Existing technology evaluation models, such as the Unified Theory of Acceptance and Use of Technology (UTAUT), are mainly used to evaluate the acceptance and use behavior of technology at the user level. The core constructs of the UTAUT model include performance expectation (PE), effort expectation (EE), social influence (SI) and facilitating conditions (FC), which directly affect the user's use behavior (UB). However, the traditional UTAUT model does not fully consider the specific impact of artificial intelligence technology in enterprise applications, especially in the complex organizational structure of power grid enterprises. Artificial intelligence technology not only affects the application behavior of enterprise employees, but also has an overall impact on the organizational structure of the enterprise. The existing UTAUT model has the following main defects: 1. Lack of evaluation of organizational impact: The traditional UTAUT model mainly focuses on the use behavior and acceptance at the individual level, and fails to fully evaluate the multiple impacts of the application of artificial intelligence technology in power grid enterprises on the organizational structure. 2. Failure to cover all relevant variables: Although the existing model sets core constructs (latent variables), such as performance expectations, effort expectations, social influence, and facilitating conditions, in actual applications, the specific observation variable settings fail to fully cover all important factors that may affect the use of artificial intelligence technology. For example, performance expectation, as a latent variable, may involve multiple observation variables, such as self-efficacy and trust. Self-efficacy refers to an individual's confidence in his or her ability to successfully perform a certain behavior, which is particularly important in technology applications. If employees lack confidence in their ability to use artificial intelligence technology, they may not actively use it even if the technology itself has high performance and ease of use. In addition, trust is one of the key factors for users to accept new technologies, and employees' trust in artificial intelligence technology will directly affect their willingness to use and use behavior. The observation variables in the existing model fail to fully reflect these key factors, resulting in the evaluation results that may not be comprehensive and accurate. 3. Lack of systematic evaluation tools: The existing technology evaluation methods lack systematic tools and cannot effectively support the scientific evaluation of power grid companies in the process of technology application and decision-making. Current evaluations mostly rely on scattered questionnaires and empirical judgments, lacking unified standards and systematic analysis tools. This approach is not only inefficient, but also prone to subjective bias, affecting the objectivity and reliability of the evaluation.

[0003] Therefore, how to provide an application evaluation method that includes comprehensive data collection, processing and analysis functions to evaluate the actual application effect of artificial intelligence has become an urgent problem to be solved in this field. Summary of the invention

[0004] The present application proposes an application evaluation method, comprising the following steps: determining a model framework; determining data variables based on the model framework; collecting data based on the determined data variables; constructing a model based on the collected data; and evaluating and displaying the constructed model.

[0005] As mentioned above, the specific framework of the model is the UTAUT model framework.

[0006] As above, the data variables include: performance expectancy, effort expectancy, social influence, facilitating conditions, usage behavior, and organizational influence.

[0007] As above, constructing a model based on the collected data includes the following sub-steps: processing the collected data; constructing a model based on the processed data, and calculating the values ​​of each latent variable of the six major data variables.

[0008] As above, among other things, the evaluation and presentation based on the constructed model includes determining the moderating variables and evaluating the impact of the moderating variables on the walking data variables.

[0009] An application evaluation system specifically includes: a model framework determination unit, a data variable determination unit, a data collection unit, a model construction unit and an evaluation and display unit; the model framework determination unit is used to determine the model framework; the data variable determination unit is used to determine the data variables according to the model framework; the data collection unit is used to collect data according to the determined data variables; the model construction unit is used to construct a model according to the collected data; the evaluation and display unit is used to evaluate and display according to the constructed model.

[0010] As mentioned above, in the model framework determination unit, the specific framework of the model is the UTAUT model framework.

[0011] As above, the data variables in the data variable determination unit include: performance expectations, effort expectations, social influence, facilitating conditions, usage behavior, and organizational influence.

[0012] As mentioned above, the model building unit builds a model based on the collected data, including the following sub-steps: processing the collected data; building a model based on the processed data, and calculating the values ​​of each latent variable of the six major data variables.

[0013] As above, the evaluation and display unit performs evaluation and display according to the constructed model, including determining the adjustment variable and evaluating the influence of the adjustment variable on the walking data variable.

[0014] This application has the following beneficial effects:

[0015] This application can comprehensively evaluate the application effect of artificial intelligence technology in actual use, including not only the use behavior of employees, but also the impact on organizational structure. At the same time, this application has expanded the UTAUT model to cover more influencing factors, such as self-efficacy and trust, providing a more accurate and comprehensive evaluation. In addition, this application can also continuously optimize the system through the application effect obtained, thereby improving the decision-making quality and application effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0017] Figure 1 is a flow chart of an application evaluation method provided according to an embodiment of the present application;

[0018] Figure 2 It is a schematic diagram of a model provided according to an embodiment of the present application;

[0019] Figure 3 It is a schematic diagram of the internal structure of an application evaluation system provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0021] The present application provides an artificial intelligence technology application evaluation system based on the extended unified theory UTAUT (Unified Theory of Acceptance and Use of Technology, user acceptance and use technology model), which can comprehensively evaluate the application effect of artificial intelligence technology in actual use (which can be "power grid enterprises").

[0022] Embodiment 1

[0023] like Figure 1 As shown, this embodiment provides an application evaluation method, which specifically includes the following steps:

[0024] Step S110: Determine the model framework.

[0025] The specific framework of the model is the UTAUT model framework.

[0026] Step S120: Determine data variables according to the model framework.

[0027] In this embodiment, the UTAUT model is expanded, six data variables are introduced, and the application effect of artificial intelligence technology (AI) in power grid enterprises is evaluated based on the six variables.

[0028] The six major data variables include:

[0029] 1. Performance expectations (PE): Employees’ expectations of AI technology to improve work efficiency. The observed variables include work efficiency, work performance, self-efficacy in using AI, trust, job satisfaction, performance indicator data, etc.

[0030] 2. Effort Expectancy (EE): Employees’ expectations on the ease of learning and using AI technology. The observed variables are ease of learning, ease of use, work pressure, and technology adaptation.

[0031] 3. Social influence (SI): The degree of support from colleagues and leaders for employees to use AI technology. The observed variables are colleague influence and superior influence.

[0032] 4. Facilitating conditions (FC): resources and support for employees in using AI technology. The observed variables are technical support, time availability, training and resources, organizational support, training needs, etc.

[0033] 5. Usage behavior (UB): employees’ specific use of AI technology. The observed variables are the frequency of AI use and the diversity of AI applications.

[0034] 6. Organizational impact (OI): The ability of an organization to adapt and change after the introduction of AI technology. The observed variables are organizational structure, organizational flexibility, interdepartmental cooperation, decision-making process, and organizational efficiency.

[0035] The observed variables configuration in the above variables include but are not limited to work efficiency, work performance, self-efficacy, trust, ease of learning, ease of use, work pressure, colleague influence, superior influence, technical support, time availability, training and resources, organizational support, frequency of use, diversity, organizational structure, organizational flexibility, interdepartmental cooperation, and decision-making process.

[0036] Step S130: Collect data according to the determined data observation variables.

[0037] Among them, relevant data on employees' use of artificial intelligence technology are collected based on six major variables. Data collection can be carried out through various methods such as online questionnaires, system log records and on-site interviews to ensure the comprehensiveness and accuracy of the data.

[0038] Step S140: construct a model based on the collected data.

[0039] The construction of the model based on the collected data includes the following sub-steps:

[0040] Step S1401: Process the collected data.

[0041] The collected observation variables of the six major variables are standardized to ensure the comparability between indicators of different units. The standardization formula is as follows:

[0042]

[0043] Among them, μ is the mean and σ is the standard deviation.

[0044] Step S1402: Perform regression analysis based on the processed data to obtain the regression coefficient of each latent variable, thereby building a complete model.

[0045] The regression analysis includes:

[0046] Step S14021: Perform the first regression analysis.

[0047] Step S14022: Perform a second regression analysis.

[0048] Step S14021 performs the first regression analysis, including evaluating the impact of the independent variables (performance expectations, effort expectations, social influence, and facilitating conditions) on the dependent variable (use behavior). Specifically, the following steps are included:

[0049] Step T1: Conduct factor analysis.

[0050] The observed indicators of the independent variables and dependent variables were subjected to factor analysis (principal component analysis).

[0051] To identify the latent factors, we used the factor analysis function in SPSS, selected “principal component analysis” as the extraction method, and chose the Varimax rotation method to obtain factors that are easy to interpret.

[0052] Step T2: Evaluate the number of factors.

[0053] Kaiser-Meyer-Olkin (KMO) test and Bartlett's sphericity test were used to assess whether the data were suitable for factor analysis. The KMO value should be greater than 0.6 and the Bartlett's test significance level should be less than 0.05, indicating that it is suitable for factor analysis. The number of factors was selected by scree plot and the total variance percentage explained. The present embodiment selected factors whose cumulative explained variance exceeded 70%.

[0054] Step T3: Interpret the factor loadings.

[0055] The larger the factor loading value, the stronger the explanatory power of the observed indicator for the factor. Generally speaking, an observed indicator with a loading greater than 0.4 can be considered to have a significant contribution to the factor. Based on the results of factor analysis, a factor score is generated for each factor.

[0056] Step T4: Perform regression analysis.

[0057] The scores of the independent variable factors are used as the predictor variables (X), and the scores of the dependent variable factors are used as the result variables (Y) for regression analysis. This embodiment uses the linear regression function in SPSS, takes the factor scores as input variables, and checks the influence of the independent variable factors on the dependent variable factors.

[0058] Step T5: Analyze the regression coefficients.

[0059] The standardized regression coefficient (Beta value) compares the relative impact of each independent variable on the dependent variable. Generally speaking, the larger the absolute value of the Beta coefficient, the greater the impact of the variable. Significance test (p-value): Determine whether the regression coefficient is significant. A p-value less than 0.05 indicates that the variable has a statistically significant impact on the dependent variable.

[0060] Step T6: Determine coefficient weights:

[0061] The standardized regression coefficients with a significance level less than 0.05 are selected as effective weights, and these coefficients are used to construct the complete expression of the UTAUT model:

[0062] Z 使用行为 =β 1 Performance Expectation Score + β 2 Expected Effort Score + β 3 Social impact score + β 4

[0063] ·Promote condition score

[0064] Step S14022 performs a second regression analysis, including evaluating the direct impact of the independent variable (usage behavior) on the dependent variable (organizational impact). Specifically, the following steps are included:

[0065] Step Q1: Data preparation.

[0066] Calculate the standardized data of various observed indicators of the dependent variable (Y) organizational influence, organizational structure, organizational flexibility, interdepartmental cooperation, decision-making process, and organizational efficiency.

[0067] The independent variable (X) is the comprehensive score of the usage behavior, which has been previously calculated by the formula Z 使用行为 Calculated.

[0068] Step Q2: Perform regression analysis.

[0069] Use regression analysis to evaluate the impact of usage behavior on organizational impact. Specifically, use the comprehensive score of usage behavior as the independent variable (X) and the various observed indicators of organizational impact as the dependent variable (Y). The steps are as follows: Select "Linear Regression Analysis" in SPSS, set the independent variable as the comprehensive score of usage behavior, and set the dependent variable as the standardized data of each observed indicator of organizational impact. Run the regression analysis and record the corresponding regression coefficient (β 5 ).

[0070] The final regression model will contain two levels:

[0071] 1. The impact of independent variables (such as performance expectations, effort expectations, etc.) on usage behavior.

[0072] 2. The direct impact of usage behavior on organizational impact.

[0073] The mathematical representation of the complete model is:

[0074] Organizational impact = β 5 ·Z 员工AI使用行为 +∈

[0075] Among them, β 5 is the regression coefficient of the impact of usage behavior on organization, and ∈ is the error term.

[0076] This embodiment conducts two regression analyses to analyze the effects of performance expectations, effort expectations, social influence, and facilitating conditions on usage behavior and organizational influence. This not only helps companies understand how usage behavior affects various aspects of the organization, but also provides important decision-making support for organizational strategies in the application of AI technology, such as supporting companies to propose effective solutions to change employees' performance expectations and effort expectations to influence employees' use of AI technology, thereby further positively affecting the company's organizational efficiency.

[0077] like Figure 2As shown, the model constructed for this application is an extension of the UTAUT (Unified Theory of Technology Acceptance and Use) model, in which variables are added in this embodiment, and the UTAUT model is expanded to cover more influencing factors, such as self-efficacy and trust, to provide a more accurate and comprehensive evaluation. This extended model aims to study how employees' attitudes towards AI technology affect their actual usage behavior, and further analyze the impact of this usage behavior on the organizational structure of the enterprise. The model includes four main independent variables: performance expectancy (PE), effort expectancy (EE), social influence (SI), and facilitating conditions (FC). These independent variables affect employees' AI technology usage behavior (UB), and usage behavior has an effect on organizational impact (OI).

[0078] In the construction of the model, first, the impact of independent variables on employees' AI usage behavior is evaluated through regression analysis. This analysis reveals how employees' attitudes affect their actual usage behavior and measures the extent of these effects. Next, the impact of employees' AI usage behavior on organizational structure is analyzed. Through this step, how usage behavior directly affects organizational structure is explored, and the indirect effect of independent variables on organizational structure through usage behavior is evaluated. This process provides insights into how the actual use of AI technology drives organizational change and improves adaptability. Combining these analyses helps to understand how employees' attitudes towards AI technology affect organizational structure through actual usage behavior, and provides a scientific basis for enterprises to optimize their organizational structure when implementing AI technology.

[0079] Specifically, the newly added variables in this embodiment include organizational impact (OI), which further refines the observed variables of the existing core constructs (such as self-efficacy and trust).

[0080] like Figure 2 As shown, the relationship between each core construct and use behavior (UB) is established, especially the direct impact of use behavior on organizational impact.

[0081] Step S150: Evaluate and display based on the constructed model.

[0082] Wherein step S150 specifically includes the following sub-steps:

[0083] Step S1501: Evaluate based on the constructed model.

[0084] The evaluation based on the constructed model is to evaluate the direct impact of each latent variable in the six major variables on usage behavior, as well as the direct impact of usage behavior on the organization, so as to obtain specific feedback on the use of artificial intelligence technology (AI) by employees.

[0085] The evaluation results include employees' specific feedback on AI. For example, in performance expectations, the evaluation result is that AI technology has significantly improved production efficiency and fault response time, but the learning pressure of employees needs to be further reduced. In effort expectations, the evaluation result is that there are differences in employees' adaptability to AI technology, and more training and support are needed. In social impact, the evaluation result is that inter-departmental collaboration has improved, but the timeliness of information sharing needs to be improved. In promoting conditions, the evaluation result is that employees reflect that training and technical support are insufficient and need to be further strengthened. In job satisfaction, the evaluation result is that after the introduction of AI, the job challenges have increased and satisfaction has improved. In organizational flexibility, the evaluation result is that the frequency of departmental collaboration and the timeliness of information sharing need to be increased to enhance organizational flexibility.

[0086] Step S1502: Display the evaluation results.

[0087] The display of evaluation results includes automatic summary of data analysis results and generation of charts.

[0088] The evaluation report includes text description, data tables and visual charts to fully display the evaluation results. The report generation module adopts a template design, supports custom report content and format, and facilitates customization according to different user needs.

[0089] Furthermore, this embodiment can also collect user feedback and continuously optimize according to user feedback and actual operation conditions.

[0090] Specifically, the latent variables can be dynamically adjusted based on user feedback and actual operating conditions for continuous optimization.

[0091] For example, based on employee feedback, the impact on the organizational model (organizational impact) can be optimized as follows: Organizational efficiency: Continue to optimize the application of AI technology to reduce employee learning pressure. Department collaboration: Improve the timeliness of information sharing and enhance inter-departmental collaboration. Job satisfaction: Maintain job challenges and further improve satisfaction. Technology adaptation: Provide more AI technology training for different age groups. Training needs: Increase AI technology training and technical support.

[0092] Continuous optimization can include technical adaptability and providing organizational support. In technical adaptability, customized training and technical support can be provided to help employees adapt to AI technology more quickly. In organizational support, the organization can increase its support and resource investment in the application of AI technology to ensure that employees have sufficient resources to use AI technology.

[0093] You can also show employees the results of feedback processing through internal communication platforms and invite them to continue to provide opinions and suggestions in order to continuously optimize the organizational model.

[0094] Embodiment 2

[0095] like Figure 3 As shown, this embodiment provides an application evaluation system, which specifically includes: a model framework determination unit 301, a data variable determination unit 302, a data collection unit 303, a model construction unit 304 and an evaluation display unit 305.

[0096] The model framework determination unit 301 is used to determine the model framework.

[0097] The specific framework of the model is the UTAUT model framework.

[0098] The data variable determination unit 302 is used to determine the data variables according to the model framework.

[0099] In this embodiment, the UTAUT model is expanded, six data variables are introduced, and the application effect of artificial intelligence technology (AI) in power grid enterprises is evaluated based on the six variables.

[0100] The six major data variables include:

[0101] 1. Performance Expectation (PE): Employees’ expectations of AI technology to improve work efficiency. The observed variables include work efficiency, work performance, self-efficacy in using AI, trust, job satisfaction, performance indicator data, etc.

[0102] 2. Effort Expectancy (EE): Employees’ expectations on the ease of learning and using AI technology. The observed variables are ease of learning, ease of use, work pressure, and technology adaptation.

[0103] 3. Social influence (SI): The degree of support from colleagues and leaders for employees to use AI technology. The observed variables are colleague influence and superior influence.

[0104] 4. Facilitating conditions (FC): resources and support for employees in using AI technology. The observed variables are technical support, time availability, training and resources, organizational support, training needs, etc.

[0105] 5. Usage behavior (UB): employees’ specific use of AI technology. The observed variables are the frequency of AI use and the diversity of AI applications.

[0106] 6. Organizational impact (OI): The ability of an organization to adapt and change after the introduction of AI technology. The observed variables are organizational structure, organizational flexibility, interdepartmental cooperation, decision-making process, and organizational efficiency.

[0107] The observed variables configuration in the above variables include but are not limited to work efficiency, work performance, self-efficacy, trust, ease of learning, ease of use, work pressure, colleague influence, superior influence, technical support, time availability, training and resources, organizational support, frequency of use, diversity, organizational structure, organizational flexibility, interdepartmental cooperation, and decision-making process.

[0108] The data collection unit 303 is used to collect data according to the determined data variables.

[0109] Among them, relevant data on employees' use of artificial intelligence technology are collected based on six major variables. Data collection can be carried out through various methods such as online questionnaires, system log records and on-site interviews to ensure the comprehensiveness and accuracy of the data.

[0110] In the data collection unit 303, sensors and IoT devices can be used to monitor in real time various operating data related to the AI ​​technology of the power grid company, such as the frequency of use of AI-related equipment in smart distribution rooms, substations and smart business halls and the types of AI technologies involved.

[0111] The data collection unit 303 also includes a questionnaire system for collecting indirect related data, such as employee behavior and psychology related data.

[0112] The data collection unit 303 also includes a data upload interface for transmitting the collected data to the data server cluster.

[0113] The model building unit 304 is used to build a model according to the collected data.

[0114] The model building unit 304 performs the following sub-steps:

[0115] Step Y1: Process the collected data.

[0116] The collected observation variables of the six major variables are standardized to ensure the comparability between indicators of different units. The standardization formula is as follows:

[0117]

[0118] Among them, μ is the mean and σ is the standard deviation.

[0119] Step Y2: Perform regression analysis based on the processed data to obtain the regression coefficients of each latent variable, thereby building a complete model.

[0120] The regression analysis includes:

[0121] Step S14021: Perform the first regression analysis.

[0122] Step S14022: Perform a second regression analysis.

[0123] Step S14021 performs the first regression analysis, including evaluating the impact of the independent variables (performance expectations, effort expectations, social influence, and facilitating conditions) on the dependent variable (use behavior). Specifically, the following steps are included:

[0124] Step T1: Conduct factor analysis.

[0125] The observed indicators of the independent variables and dependent variables were subjected to factor analysis (principal component analysis).

[0126] To identify the latent factors, we used the factor analysis function in SPSS, selected “principal component analysis” as the extraction method, and chose the Varimax rotation method to obtain factors that are easy to interpret.

[0127] Step T2: Evaluate the number of factors.

[0128] Kaiser-Meyer-Olkin (KMO) test and Bartlett's sphericity test were used to assess whether the data were suitable for factor analysis. The KMO value should be greater than 0.6 and the Bartlett's test significance level should be less than 0.05, indicating that it is suitable for factor analysis. The number of factors was selected by scree plot and the total variance percentage explained. The present embodiment selected factors whose cumulative explained variance exceeded 70%.

[0129] Step T3: Interpret the factor loadings.

[0130] The larger the factor loading value, the stronger the explanatory power of the observed indicator for the factor. Generally speaking, an observed indicator with a loading greater than 0.4 can be considered to have a significant contribution to the factor. Based on the results of factor analysis, a factor score is generated for each factor.

[0131] Step T4: Perform regression analysis.

[0132] The scores of the independent variable factors are used as the predictor variables (X), and the scores of the dependent variable factors are used as the result variables (Y) for regression analysis. This embodiment uses the linear regression function in SPSS, takes the factor scores as input variables, and checks the influence of the independent variable factors on the dependent variable factors.

[0133] Step T5: Analyze the regression coefficients.

[0134] The standardized regression coefficient (Beta value) compares the relative impact of each independent variable on the dependent variable. Generally speaking, the larger the absolute value of the Beta coefficient, the greater the impact of the variable. Significance test (p-value): Determine whether the regression coefficient is significant. A p-value less than 0.05 indicates that the variable has a statistically significant impact on the dependent variable.

[0135] Step T6: Determine coefficient weights:

[0136] The standardized regression coefficients with a significance level less than 0.05 are selected as effective weights, and these coefficients are used to construct the complete expression of the UTAUT model:

[0137] Z 使用行为 =β 1 Performance Expectation Score + β 2 Expected Effort Score + β 3 Social impact score + β 4

[0138] ·Promote condition score

[0139] Step S14022 performs a second regression analysis, including evaluating the direct impact of the independent variable (usage behavior) on the dependent variable (organizational impact). Specifically, the following steps are included:

[0140] Step Q1: Data preparation.

[0141] Calculate the standardized data of various observed indicators of the dependent variable (Y) organizational influence, organizational structure, organizational flexibility, interdepartmental cooperation, decision-making process, and organizational efficiency.

[0142] The independent variable (X) is the comprehensive score of the usage behavior, which has been previously calculated by the formula Z 使用行为 Calculated.

[0143] Step Q2: Perform regression analysis.

[0144] Use regression analysis to evaluate the impact of usage behavior on organizational impact. Specifically, use the comprehensive score of usage behavior as the independent variable (X) and the various observed indicators of organizational impact as the dependent variable (Y). The steps are as follows: Select "Linear Regression Analysis" in SPSS, set the independent variable as the comprehensive score of usage behavior, and set the dependent variable as the standardized data of each observed indicator of organizational impact. Run the regression analysis and record the corresponding regression coefficient (β 5 ).

[0145] The final regression model will contain two levels:

[0146] 3. The impact of independent variables (such as performance expectations, effort expectations, etc.) on usage behavior.

[0147] 4. The direct impact of usage behavior on organizational impact.

[0148] The mathematical representation of the complete model is:

[0149] Organizational impact = β 5 ·Z 员工AI使用行为 +∈

[0150] Among them, β 5 is the regression coefficient of the impact of usage behavior on organization, and ∈ is the error term.

[0151] This embodiment conducts two regression analyses to analyze the effects of performance expectations, effort expectations, social influence, and facilitating conditions on usage behavior and organizational influence. This not only helps companies understand how usage behavior affects various aspects of the organization, but also provides important decision-making support for organizational strategies in the application of AI technology, such as supporting companies to propose effective solutions to change employees' performance expectations and effort expectations to influence employees' use of AI technology, thereby further positively affecting the company's organizational efficiency.

[0152] like Figure 2 As shown, the model constructed for this application is an extension of the UTAUT (Unified Theory of Technology Acceptance and Use) model, in which variables are added in this embodiment, and the UTAUT model is expanded to cover more influencing factors, such as self-efficacy and trust, to provide a more accurate and comprehensive evaluation. This extended model aims to study how employees' attitudes towards AI technology affect their actual usage behavior, and further analyze the impact of this usage behavior on the organizational structure of the enterprise. The model includes four main independent variables: performance expectancy (PE), effort expectancy (EE), social influence (SI), and facilitating conditions (FC). These independent variables affect employees' AI technology usage behavior (UB), and usage behavior has an effect on organizational impact (OI).

[0153] In the construction of the model, first, the impact of independent variables on employees' AI usage behavior is evaluated through regression analysis. This analysis reveals how employees' attitudes affect their actual usage behavior and measures the extent of these effects. Next, the impact of employees' AI usage behavior on organizational structure is analyzed. Through this step, how usage behavior directly affects organizational structure is explored, and the indirect effect of independent variables on organizational structure through usage behavior is evaluated. This process provides insights into how the actual use of AI technology drives organizational change and improves adaptability. Combining these analyses helps to understand how employees' attitudes towards AI technology affect organizational structure through actual usage behavior, and provides a scientific basis for enterprises to optimize their organizational structure when implementing AI technology.

[0154] Specifically, the newly added variables in this embodiment include organizational impact (OI), which further refines the observed variables of the existing core constructs (such as self-efficacy and trust).

[0155] like Figure 2 As shown, the relationship between each core construct and use behavior (UB) is established, especially the direct impact of use behavior on organizational impact.

[0156] The evaluation and display unit 305 is used to evaluate and display the constructed model.

[0157] The evaluation unit 305 may be specifically an integrated evaluation management platform, in which data can be stored, analyzed, and evaluated. Through the platform, the analysis results can be presented to management and related personnel in the form of charts and reports. A detailed evaluation report is provided, including the scores of each latent variable, analysis of influencing factors, optimization suggestions, etc.

[0158] The evaluation unit 305 performs the following sub-steps:

[0159] Step W1: Evaluate based on the constructed model.

[0160] The evaluation based on the constructed model is to evaluate the direct impact of each latent variable in the six major variables on usage behavior, as well as the direct impact of usage behavior on the organization, so as to obtain specific feedback on the use of artificial intelligence technology (AI) by employees.

[0161] The evaluation results specifically include the scores of each latent variable, the performance of usage behavior, and the estimated impact on the organization.

[0162] The evaluation results include employees' specific feedback on AI. For example, in performance expectations, the evaluation result is that AI technology has significantly improved production efficiency and fault response time, but the learning pressure of employees needs to be further reduced. In effort expectations, the evaluation result is that there are differences in employees' adaptability to AI technology, and more training and support are needed. In social impact, the evaluation result is that inter-departmental collaboration has improved, but the timeliness of information sharing needs to be improved. In promoting conditions, the evaluation result is that employees reflect that training and technical support are insufficient and need to be further strengthened. In job satisfaction, the evaluation result is that after the introduction of AI, the job challenges have increased and satisfaction has improved. In organizational flexibility, the evaluation result is that the frequency of departmental collaboration and the timeliness of information sharing need to be increased to enhance organizational flexibility.

[0163] Step W2: Display the evaluation results.

[0164] The display of evaluation results includes automatic summary of data analysis results and generation of charts.

[0165] The evaluation report includes text description, data tables and visual charts to fully display the evaluation results. The report generation module adopts a template design, supports custom report content and format, and facilitates customization according to different user needs.

[0166] Furthermore, this embodiment can also collect user feedback and continuously optimize according to user feedback and actual operation conditions.

[0167] Specifically, the latent variables can be dynamically adjusted based on user feedback and actual operating conditions for continuous optimization.

[0168] For example, based on employee feedback, the impact on the organizational model (organizational impact) can be optimized as follows: Organizational efficiency: Continue to optimize the application of AI technology to reduce employee learning pressure. Department collaboration: Improve the timeliness of information sharing and enhance inter-departmental collaboration. Job satisfaction: Maintain job challenges and further improve satisfaction. Technology adaptation: Provide more AI technology training for different age groups. Training needs: Increase AI technology training and technical support.

[0169] Continuous optimization can include technical adaptability and providing organizational support. In technical adaptability, customized training and technical support can be provided to help employees adapt to AI technology more quickly. In organizational support, the organization can increase its support and resource investment in the application of AI technology to ensure that employees have sufficient resources to use AI technology.

[0170] You can also show employees the results of feedback processing through internal communication platforms and invite them to continue to provide opinions and suggestions in order to continuously optimize the organizational model.

[0171] This application has the following beneficial effects:

[0172] This application can comprehensively evaluate the application effect of artificial intelligence technology in actual use, including not only the use behavior of employees, but also the impact on organizational structure. At the same time, this application has expanded the UTAUT model to cover more influencing factors, such as self-efficacy and trust, providing a more accurate and comprehensive evaluation. In addition, this application can also continuously optimize the system through the obtained application effects, thereby improving the decision-making quality and application effect.

[0173] Although the present application is described with reference to examples, this is for illustrative purposes only and is not intended to limit the present application, and changes, additions and / or deletions to the embodiments may be made without departing from the scope of the present application.

[0174] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An application evaluation method, characterized in that: The following steps are involved: Determine the model framework; Determine data variables based on the model framework; Conduct data collection based on the determined data variables; Build a model based on the collected data; Evaluate and present based on the constructed model.

2. The application evaluation method according to claim 1, characterized in that: The specific framework of the model is the UTAUT model framework.

3. The application evaluation method according to claim 1, characterized in that: Data variables included: performance expectancy, effort expectancy, social influence, facilitating conditions, usage behavior, and organizational influence.

4. The application evaluation method according to claim 1, characterized in that: The construction of the model based on the collected data includes the following sub-steps: Process the collected data; A model is constructed based on the processed data, and the values ​​of the latent variables of the six major data variables are calculated.

5. The application evaluation method according to claim 1, characterized in that: The evaluation and presentation based on the constructed model includes determining the moderating variables and evaluating the impact of the moderating variables on the walking data variables.

6. An application evaluation system, characterized in that: Specifically include: Model framework determination unit, data variable determination unit, data collection unit, model building unit and evaluation and presentation unit; A model framework determination unit, used for determining the model framework; A data variable determination unit, used for determining data variables according to a model framework; A data collection unit, used for collecting data according to determined data variables; A model building unit, used for building a model based on the collected data; The evaluation and display unit is used to evaluate and display the constructed model.

7. The application evaluation system according to claim 6, characterized in that: In the model framework determination unit, the specific framework of the model is the UTAUT model framework.

8. The application evaluation system according to claim 6, characterized in that: The data variables in the data variable determination unit include: performance expectancy, effort expectancy, social influence, facilitating conditions, usage behavior, and organizational influence.

9. The application evaluation system according to claim 6, characterized in that: The model building unit builds the model based on the collected data, including the following sub-steps: Process the collected data; A model is constructed based on the processed data, and the values ​​of the latent variables of the six major data variables are calculated.

10. The application evaluation system according to claim 6, characterized in that: The evaluation and display unit evaluates and displays the constructed model, including determining the moderating variables and evaluating the impact of the moderating variables on the walking data variables.

Images