Software evaluation method

Abstract

The invention relates to a software evaluation method. The method comprises the following steps: compiling a product usability scale through an SPSS exploratory factor analysis method; constructing an expert consistency evaluation table by utilizing a Delphi method; obtaining a software overall feeling score in an online questionnaire form; obtaining a software efficiency experience score and a software performance experience score according to the user usage behavior data; the product usability scale and the expert consistency evaluation table serve as questionnaires, software usability scores and software design normative scores are obtained through online questionnaire survey, and software evaluation scores are obtained through weighted summation of software overall feeling scores, efficiency experience scores, performance experience scores and operation experience scores. According to the characteristics of the life insurance industry, a product usability scale and an expert consistency evaluation table suitable for life insurance industry personnel are compiled, and the problems that an index system in the prior art is universal, and indexes and scales which are based on the index system and can be suitable for tool product characteristics in the insurance industry are lacked are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of software evaluation, and particularly relates to a software evaluation method. Background Art

[0002] In the current development process of software tools for financial institutions at all levels such as banks, insurance companies, and securities companies, "customer-centric" has become the core goal, and "user experience" has been widely mentioned. User experience refers to all the feelings of users before, during, and after using a product or system, including emotions, preferences, cognitive impressions, physiological and psychological reactions, behaviors, and achievements. User experience is a subjective perception of users. Developers need to accurately measure the user experience performance of current products and systems in order to manage the experience and improve it targeted through effective measures.

[0003] The commonly used experience index systems on the market currently have the following deficiencies: 1) The index system is established around the final goals and results, lacking process indicators, which is not conducive to subsequent disassembly, positioning, and tracing of problems and cannot effectively solve practical problems. 2) The index system is relatively general, lacking indicators and scales based on the index system that can be applicable to the characteristics of tool products in the insurance industry. 3) The measurement method is single, only collecting users' subjective evaluations through simple questionnaires or scales. Summary of the Invention

[0004] The purpose of the present invention is to provide a software evaluation method to overcome the deficiencies of the above-mentioned existing technologies.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] The present invention provides a software evaluation method on the one hand, including the following steps:

[0007] Step S1: Compile a product usability scale through the SPSS exploratory factor analysis method;

[0008] Step S2: Use the Delphi method to construct an expert consensus evaluation form applicable to the life insurance industry;

[0009] Step S3: Obtain the overall perception index score of insurance customers through the overall perception index of insurance customers in the form of an online questionnaire, obtain the overall perception index score of insurance agents through the overall perception index of insurance agents in the form of an online questionnaire, and obtain the overall perception score of the software according to the overall perception index score of insurance customers and the overall perception index score of insurance agents;

[0010] Step S4: Deploy and embed points in the software through a third-party data management tool to capture users' usage behavior data;

[0011] Step S5: Score the efficiency experience of the software based on the user usage behavior data to obtain the efficiency experience score of the software, and score the performance experience of the software based on the user usage behavior data to obtain the performance experience score of the software;

[0012] Step S6: Score the usage situation and usage depth of the software based on the user usage behavior data to obtain the usage situation score and usage depth score of the software. Use the product usability scale as a questionnaire to obtain the product usability scale data through an online questionnaire survey. Obtain the usability score of the software based on the product usability scale data. Use the expert consistency evaluation form as a questionnaire to obtain the expert consistency evaluation form data through an online questionnaire survey. Obtain the design standardization score of the software based on the expert consistency evaluation form data. Obtain the operation experience score of the software by weighted summation of the usage situation score, usage depth score, usability score, and design standardization score;

[0013] Step S7: Obtain the evaluation score of the software by weighted summation of the overall feeling score, efficiency experience score, performance experience score, and operation experience score of the software, and determine whether the software meets the life insurance industry according to the evaluation score of the software.

[0014] Further, the specific steps of step S1 are as follows:

[0015] Step S11: Compile an initial questionnaire according to the existing basic questions of the usability scale;

[0016] Step S12: Add an option of whether it is understood to each question of the initial questionnaire, and adjust the initial questionnaire according to the score of the whether it is understood option to obtain the adjusted questionnaire;

[0017] Step S13: Use the adjusted questionnaire as a questionnaire to obtain the scoring data of the adjusted questionnaire through an online questionnaire form. Input the scoring data of the adjusted questionnaire into SPSS for exploratory factor analysis, explore the factor structure, identify potential higher-order dimensions, obtain the standardized loading coefficient of each question of the adjusted questionnaire, delete the questions with a standardized loading coefficient less than the first preset value, use the Cronbach's coefficient to evaluate the internal consistency coefficient of each question for the remaining scoring data of the adjusted questionnaire, eliminate the questionnaire questions according to the internal consistency coefficient of each question, and use the remaining questions as the product usability scale.

[0018] Further, the adjustment of the initial questionnaire according to the score of the whether it is understood option in step S12 includes the following steps:

[0019] If the score of the whether it is understood option is within the first preset range, delete the corresponding question. If the score of the whether it is understood option is within the second preset range, modify the corresponding question.

[0020] Further, in step S13, the remaining adjusted questionnaire data is used to evaluate the internal consistency coefficient of each question by using Cronbach's coefficient, and the questionnaire questions are eliminated according to the internal consistency coefficient of each question, which specifically includes the following steps:

[0021] Calculate the total Cronbach's coefficient based on all the questions in the remaining adjusted questionnaire data;

[0022] Individually eliminate each question in the questionnaire and calculate the new Cronbach's coefficient;

[0023] Subtract the new Cronbach's coefficient from the total Cronbach's coefficient to obtain the internal consistency coefficient of the eliminated question. If the internal consistency coefficient is greater than or equal to the second preset value, delete the question from the questionnaire. If the internal consistency coefficient is less than the second preset value, keep the question.

[0024] Further, the formula for calculating Cronbach's coefficient is:

[0025]

[0026] where α is Cronbach's coefficient, N is the number of questionnaire questions, is the average covariance between the scores of each question in the questionnaire, is the average variance of the scores of each question in the questionnaire.

[0027] Further, in step S3, the overall perception index of insurance customers is the net promoter score, and the overall perception index of insurance agents is the satisfaction value;

[0028] The specific steps for obtaining the overall perception score of the software based on the overall perception index score of insurance customers and the overall perception index score of insurance agents are as follows:

[0029] Classify according to the scores of the overall perception index score of insurance customers and the overall perception index score of insurance agents. The classification includes: satisfied customers, complainers, and detractors;

[0030] Count the number of each classification, and use the result of (the number of satisfied customers - the number of complainers / the number of detractors) / the total number of responses as the overall perception score of the software.

[0031] Further, the user usage behavior data includes: user click volume, page view volume, user access number, dwell time on each page, one-time completion rate of functions, one-time completion time of functions, page loading time, page crash times, and function error rate.

[0032] Further, the specific steps for scoring the efficiency experience of the software based on the user usage behavior data to obtain the efficiency experience score of the software are as follows:

[0033] Obtain the one-time completion rate and one-time completion duration of functions in the user usage behavior data, calculate the standard deviation of the one-time completion rate and one-time completion duration of functions from the average of the same behavior data in the past three months, and use the calculated standard deviation as the efficiency experience score of the software;

[0034] The performance experience of the software is scored based on the user usage behavior data to obtain the performance experience score of the software, which specifically includes the following steps:

[0035] Obtain the page loading duration, page crash times, and function error rate in the user usage behavior data, calculate the standard deviation of the page loading duration, page crash times, and function error rate from the average of the same behavior data in the past three months, and use the calculated standard deviation as the performance experience score of the software.

[0036] Furthermore, the usage situation score is obtained based on the user click volume, page access volume, and user access number in the user usage behavior data, and the usage depth score is obtained based on the residence duration of each page in the user usage behavior data.

[0037] Furthermore, the product usability scale is used as a questionnaire to obtain product usability scale data through online questionnaire surveys, and the usability score of the software is obtained based on the product usability scale data. The expert consistency evaluation form is used as a questionnaire to obtain expert consistency evaluation form data through online questionnaire surveys, and the design standardization score of the software is obtained based on the expert consistency evaluation form data, which specifically includes the following steps:

[0038] Use the product usability scale as a questionnaire and distribute it online to obtain product usability scale data. Based on the scoring scores of each question in the product usability scale data, calculate the usability score of the software through the following formula:

[0039]

[0040] Among them, Qi is the scoring score of question i, k is the total number of questions, Wi is the question weight, N is the number of users in the questionnaire survey, and M is the number of experts in the questionnaire survey;

[0041] The question weight Wi is determined based on the internal consistency coefficient of each question. The smaller the internal consistency coefficient, the larger the question weight Wi;

[0042] Use the expert consistency evaluation form as a questionnaire and distribute it online to obtain expert consistency evaluation form data. Based on the scoring scores of each question in the expert consistency evaluation form data, calculate the design standardization score of the software through the following formula:

[0043]

[0044] Among them, N is the total number of all questions in the expert consistency evaluation form, n is the number of questions with scores ranging from 4 to 5, m is the number of questions in the expert consistency evaluation form that are consistent with the life insurance experience specification, s i is the score of all question i, w i is the weight of question i.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] (1) The present invention optimizes the indicators that do not conform to the characteristics of life insurance products, and structurally adjusts the traditional usability and consistency scoring forms through a scientifically verified method, solving the situation that the measurement scale does not conform to the current situation of the life insurance industry, resulting in inaccurate measurement.

[0047] (2) The present invention incorporates the evaluation of the experience usability and ease of use of requirements into the acceptance link before going online, making up for the missing link of experience quality evaluation in the quality control before going online. If the evaluation result does not meet the standard, a warning is given according to the degree of problem impact or it is not allowed to go online, and an evaluation analysis report is issued, pointing out the problem points, and optimizing and improving the problems in the rapid iteration after going online.

[0048] (3) The evaluation results of the present invention are comparable and can be used for rapid iteration and optimization. Evaluation scores are respectively carried out before and after optimization and going online, and the change in scores before and after can be compared to evaluate the optimization effect through the attribution of process indicators. It can also be used for stage experience evaluation, selecting benchmark competitors in the industry, evaluating experience gaps, advantages and disadvantages, understanding the current position of the product in the industry, providing directions and suggestions for subsequent improvement, and quantifying the effects and values of stage experience work.

[0049] (4) The present invention conducts exploratory factor analysis through SPSS, optimizes and adjusts the questionnaire based on the standardized loading coefficients of each questionnaire question, determines the three main dimensions of the product usability scale: learnability, operability, and fault tolerance, eliminates invalid questions in the questionnaire through the Cronbach's coefficient, improves the internal consistency of the product usability scale, ensures that each question can effectively measure the relevant characteristics of the research object, thereby improving the reliability and validity of the data. Finally, for the questionnaire that passes the Bartlett test, confirmatory factor analysis is used to detect the scale structure and the correlation between each question again. Through these three steps, the main structure of the finally obtained product usability scale, the three dimensions of learnability, operability, and fault tolerance, and each question meet the professional requirements of scale compilation and also meet the actual use needs.

[0050] (5) The advantage of constructing the expert consensus evaluation form by using the Delphi method in the present invention is that it can ensure the questionnaire design is more accurate and comprehensive through multiple rounds of feedback and correction of expert opinions. By collecting and summarizing the independent opinions of domain experts, the Delphi method helps to eliminate single opinions or biases, thus providing more scientific and professional questionnaire content. Since experts can modify or adjust their opinions after each round of feedback, this iterative correction process helps to reach a consensus and improves the effectiveness and credibility of the content of the expert consensus evaluation form. In addition, the Delphi method can effectively identify and avoid ambiguity and unclear issues in questionnaire design, enhancing the accuracy and pertinence of questionnaire questions. Through the systematic expert feedback process, the Delphi method can also improve the overall quality of the expert consensus evaluation form, ensure that it can effectively measure the research objectives, reduce errors and biases, and ensure that the survey results are more representative and universal.

[0051] (6) By using the three process indicators of operation experience, efficiency experience, and performance experience, the present invention solves the problems in the prior art that the indicator system is established around the final goals and results, lacks process indicators, is not conducive to subsequent disassembly, positioning, and traceability of problems, and cannot effectively solve practical problems.

[0052] (7) According to the characteristics of the life insurance industry, the present invention has compiled a product usability scale and an expert consensus evaluation form suitable for personnel in the life insurance industry, solving the problems in the prior art that the indicator system is relatively general and lacks indicators and scales based on the indicator system that can be applicable to the characteristics of tool products in the insurance industry.

[0053] (8) By deploying data collection points in software through a third-party data management tool, capturing user usage behavior data, and evaluating the software based on the user usage behavior data, the present invention solves the problems in the prior art such as only collecting users' subjective evaluations through simple questionnaires or scales. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is the method flow chart of the present invention;

[0055] Figure 2 is the evaluation index model diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0057] Embodiment 1:

[0058] On the one hand, this embodiment provides a software evaluation method for the life insurance industry, as Figure 1 shown, which includes the following steps:

[0059] Step S1: Compile a product usability scale through SPSS exploratory factor analysis;

[0060] Step S1 specifically includes the following steps:

[0061] Step S11: Compile an initial questionnaire based on the existing basic questions of the usability scale;

[0062] Step S12: Add an option of whether it is understood to each question of the initial questionnaire, and adjust the initial questionnaire according to the scores of the whether it is understood option to obtain an adjusted questionnaire;

[0063] Adjusting the initial questionnaire according to the scores of the whether it is understood option includes the following steps:

[0064] If the score of the whether it is understood option is within the first preset range, delete the corresponding question; if the score of the whether it is understood option is within the second preset range, modify the corresponding question.

[0065] Step S13: Take the adjusted questionnaire as a questionnaire to obtain the scoring data of the adjusted questionnaire through an online questionnaire form. Input the scoring data of the adjusted questionnaire into SPSS for exploratory factor analysis, explore the factor structure, identify potential high-order dimensions, obtain the standardized loading coefficients of each question of the adjusted questionnaire, delete the questions with standardized loading coefficients less than the first preset value, use the Cronbach's coefficient to evaluate the internal consistency coefficients of each question for the remaining adjusted questionnaire scoring data, eliminate the questionnaire questions according to the internal consistency coefficients of each question, and use the remaining questions as the product usability scale.

[0066] In step S13, using the Cronbach's coefficient to evaluate the internal consistency coefficients of each question for the remaining adjusted questionnaire data and eliminating the questionnaire questions according to the internal consistency coefficients of each question specifically includes the following steps:

[0067] Calculate the overall Cronbach's coefficient based on all the questions of the remaining adjusted questionnaire data;

[0068] Individually eliminate each question in the questionnaire and calculate the new Cronbach's coefficient;

[0069] Subtract the new Cronbach's coefficient from the overall Cronbach's coefficient to obtain the internal consistency coefficient of this question. If the internal consistency coefficient is greater than or equal to the second preset value, delete this question from the questionnaire; if the internal consistency coefficient is less than the second preset value, retain this question.

[0070] The calculation formula of the Cronbach's coefficient is:

[0071]

[0072] Among them, α is the Cronbach's coefficient, N is the number of questionnaire items, is the average covariance between the scores of each questionnaire item, is the average variance of the scores of each questionnaire item.

[0073] Step S2: Use the Delphi method to construct an expert consensus evaluation form applicable to the life insurance industry;

[0074] Step S3: Obtain the scores of the overall perception indicators of insurance customers through the overall perception indicators of insurance customers in the form of an online questionnaire, obtain the scores of the overall perception indicators of insurance agents through the overall perception indicators of insurance agents in the form of an online questionnaire, and obtain the overall software perception score based on the scores of the overall perception indicators of insurance customers and the overall perception indicators of insurance agents;

[0075] In Step S3, the overall perception indicator of insurance customers is the Net Promoter Score, and the overall perception indicator of insurance agents is the satisfaction value;

[0076] Obtaining the overall software perception score based on the scores of the overall perception indicators of insurance customers and the overall perception indicators of insurance agents specifically includes the following steps:

[0077] Classify according to the scores of the overall perception indicators of insurance customers and the overall perception indicators of insurance agents. The classification includes: satisfied customers, complainers, and detractors;

[0078] Count the number of each classification, and use the result of (the number of satisfied customers - the number of complainers / the number of detractors) / the total number of responses as the overall software perception score.

[0079] Step S4: Deploy and embed points in the software through a third-party data management tool to capture user usage behavior data. The user usage behavior data includes: user click volume, page view volume, number of user visits, dwell time on each page, one-time completion rate of functions, one-time completion duration of functions, page loading duration, number of page crashes, and function error rate.

[0080] Step S5: Score the software efficiency experience based on the user usage behavior data to obtain the software efficiency experience score, and score the software performance experience based on the user usage behavior data to obtain the software performance experience score;

[0081] Scoring the software efficiency experience based on the user usage behavior data to obtain the software efficiency experience score specifically includes the following steps:

[0082] Obtain the one-time completion rate and one-time completion duration of functions in the user usage behavior data, calculate the standard deviation of the one-time completion rate and one-time completion duration of functions from the average of the same behavior data in the past three months, and use the calculated standard deviation as the software efficiency experience score;

[0083] Score the software performance experience based on the user usage behavior data to obtain the software performance experience score, which specifically includes the following steps:

[0084] Obtain the page loading duration, page crash times, and function error rate in the user usage behavior data, calculate the standard deviation of the page loading duration, page crash times, and function error rate from the average of the same behavior data in the past three months, and use the calculated standard deviation as the software performance experience score.

[0085] Step S6: Score the software usage situation and usage depth based on the user usage behavior data to obtain the software usage situation score and usage depth score; the usage situation score is obtained from the user click volume, page access volume, and user access number in the user usage behavior data, and the usage depth score is obtained from the stay duration of each page in the user usage behavior data.

[0086] Use the product usability scale as a questionnaire to obtain product usability scale data through online questionnaire surveys, obtain the software usability score based on the product usability scale data, use the expert consistency evaluation form as a questionnaire to obtain expert consistency evaluation form data through online questionnaire surveys, obtain the software design standardization score based on the expert consistency evaluation form data, and obtain the software operation experience score by weighted summing the usage situation score, usage depth score, usability score, and design standardization score;

[0087] Use the product usability scale as a questionnaire to obtain product usability scale data through online questionnaire surveys, obtain the software usability score based on the product usability scale data, use the expert consistency evaluation form as a questionnaire to obtain expert consistency evaluation form data through online questionnaire surveys, which specifically includes the following steps:

[0088] Use the product usability scale as a questionnaire to distribute through online questionnaires to obtain product usability scale data. According to the scoring scores of each question in the product usability scale data, calculate the software usability score through the following formula:

[0089]

[0090] where Qi is the scoring score of question i, k is the total number of questions, Wi is the question weight, N is the number of users in the questionnaire survey, and M is the number of experts in the questionnaire survey;

[0091] The question weight Wi is determined according to the internal consistency coefficient of each question. The smaller the internal consistency coefficient, the larger the question weight Wi.

[0092] Use the expert consistency evaluation form as a questionnaire and distribute it online to obtain the data of the expert consistency evaluation form. According to the scores of each question in the data of the expert consistency evaluation form, calculate the software design specification score through the following formula:

[0093]

[0094] where N is the number of all questions in the expert consistency evaluation form, n is the number of questions with scores between 4 and 5, m is the number of questions in the expert consistency evaluation form that are consistent with the life insurance experience specification, s i is the score of all question i, w i is the weight of question i.

[0095] Step S7: Obtain the software evaluation score by weighted summation of the software overall feeling score, efficiency experience score, performance experience score, and operation experience score, and judge whether the software meets the life insurance industry according to the software evaluation score.

[0096] Example 2:

[0097] The parts not mentioned in this example are the same as those in Example 1.

[0098] This example provides a theoretical model for software experience evaluation in the life insurance industry to solve the problem of lacking process indicators, such as Figure 2 shown, including:

[0099] Four first-level dimensions: overall feeling, operation experience, efficiency experience, and performance experience. Among them, the overall feeling is the result indicator, and the operation experience, efficiency experience, and performance experience are the process indicators. Among them, the overall feeling has 2 second-level dimensions: satisfaction and effort. The operation experience has 4 second-level dimensions: usage, usage depth, ease of use, and design specification.

[0100] Compilation of the "Product Ease of Use Scale":

[0101] Step 1: Compile the initial questionnaire: Refer to the basic question statements of the traditional ease of use scale, and initially screen out 17 ease of use questions related to the use of life insurance agent tools in combination with the current status of insurance digital products to compile the initial questionnaire.

[0102] Step 2: Questionnaire optimization: Considering that the overall age of the current traditional individual insurance agent team is relatively old and there are certain difficulties in understanding content such as Internet jargon, in the design of the initial scale, in addition to the 1-5 level scoring options, an option for whether the question text is understood is added to adjust and eliminate the question statements that are difficult for users to understand and increase the friendliness of the final scale.

[0103] Step 3: Data collection: 274 test data from agents and experience experts were collected through online questionnaires.

[0104] Step 4: Data analysis: Exploratory and confirmatory factor analyses were performed on the questionnaire results to eliminate invalid questions and dimensions. The initial questionnaire data was input into SPSS for exploratory factor analysis (EFA) to explore the factor structure and identify potential higher-order dimensions. The results showed that the absolute value of the standardized loading coefficient of 3 questions in the initial questionnaire was less than 0.4, indicating a weak measurement relationship, which in turn affected the discriminant validity among the three main dimensions of learnability, operability, and fault tolerance. The Cronbach's coefficient was used to evaluate the internal consistency of each question, and invalid factors were eliminated. The adjusted model structure passed the Bartlett test, with a KMO value of 0.949, meeting the requirements of confirmatory factor analysis (CFA). Moreover, the factor loading coefficients of the items in the three dimensions of the scale met the professional dimensions preset for usability.

[0105] Among them, using the Cronbach's coefficient to evaluate the internal consistency of each question and eliminating invalid factors includes the following steps:

[0106] Calculate the overall Cronbach's coefficient based on all the questions in the remaining adjusted questionnaire data;

[0107] Individually eliminate each question in the questionnaire and calculate the new Cronbach's coefficient;

[0108] Subtract the new Cronbach's coefficient from the overall Cronbach's coefficient to obtain the internal consistency coefficient of the question. If the internal consistency coefficient is greater than or equal to the second preset value, delete the question from the questionnaire. If the internal consistency coefficient is less than the second preset value, retain the question.

[0109] The formula for calculating the Cronbach's coefficient is:

[0110]

[0111] where α is the Cronbach's coefficient, N is the number of questions in the questionnaire, is the average covariance between the scores of each question in the questionnaire, is the average variance of the scores of each question in the questionnaire.

[0112] Step 5: Finalize the scale: Based on the data analysis results and combined with the usage scenarios of salespersons, all the questions were formed into an ease-of-use measurement tool with 3 dimensions, namely operability, fault tolerance, and understandability, and a total of 14 questions.

[0113] The optimized ease-of-use scale can evaluate the experience quality in the acceptance stage before the product goes online, and provide ease-of-use details in cooperation with usability testing to guide product optimization.

[0114] Compilation of "Expert Consistency Evaluation Form":

[0115] The focus of consistency evaluation is the degree of compliance between the final online state of the product and the design specifications, so as to ensure the continuity of the user experience after the product is launched. In the compilation stage of the evaluation form, the Delphi method is used to construct life insurance evaluation indicators applicable to the usage scenarios of life insurance agents' tools based on industry standards.

[0116] Step 1: Form an expert group: Establish an expert group with positions covering experience designers and product managers.

[0117] Step 2: Preliminary screening: Screen the first- and second-level indicator dimensions in the traditional cloud computing consistency self-check form according to the differences between life insurance agents' tools and cloud computing software. Considering the high requirements of life insurance agents' tools for mobile convenience, multi-scene switching, etc., select the more general overall style, general framework, common scenarios, and components as the basis for the first-level indicators. At the same time, adjust the original "Others" to "Copywriting" to meet the scenario requirements of multiple instructions, prompts, and terms in life insurance agents' tools, forming an evaluation form framework composed of 4 first-level indicators. Since the interfaces and interaction elements of mobile tools and PC tools are quite different, the second-level indicators in the traditional cloud computing consistency self-check form are not directly adopted.

[0118] Step 3: Index improvement: Define the definitions and required contents of the first-level indicators. For example, the overall style refers to the overall visual interface presentation that users see, including fonts, colors, styles, device adaptation, etc. On the basis of clear definitions, compile the corresponding second-level indicators in combination with the principles of independence and integrity of each indicator.

[0119] Step 4: Index refinement: Provide the prepared background materials and consultation forms to the experts, and invite the expert group to discuss, score, analyze data, and verify the contents, scopes, expression methods of each standard, and weights of the first- and second-level indicators.

[0120] Step 5: Multiple verifications: Conduct a new round of discussion, score, data analysis, and verification on the results. After 3 rounds, reach a consensus.

[0121] Step 6: Finalize the draft: The evaluation form covers 4 major contents: overall style, general framework, common scenarios / components, and copywriting. There are a total of 17 second-level indicators and 22 consistency standards. And a certain weight of specification reuse rate is added to the scoring formula to improve the accuracy and objectivity of the final consistency score.

[0122] A method for evaluating the experience of software products in the life insurance industry, which solves the problems of single data collection method and mainly subjective evaluation in existing experience evaluation.

[0123] The indicator system of this embodiment includes 4 first-level indicator dimensions, namely overall feeling, operation experience, performance experience, and efficiency experience.

[0124] Among them, the overall feeling is the result indicator, and the operation experience, performance experience, and efficiency experience are 3 process indicators. The problem is disassembled and verified for the result indicator through the process indicators.

[0125] The overall feeling is the satisfaction degree of the user with the entire product after use. The data of the satisfaction degree is different according to the different users of the product. If the user is an insurance agent, the overall feeling is measured by the user satisfaction; if the user is an insurance customer, the overall feeling is measured by the NPS result. These two indicator data are both obtained by the user filling in the online questionnaire embedded in the life insurance digital tool. The satisfaction questionnaire requires the user to select according to the question "Are you satisfied with this xxx (specific operation or business scenario)?", and the NPS questionnaire requires the user to select according to the question "How likely are you to recommend xxx to your friends or family?". Calculate the overall score based on (the number of satisfied people - the number of complainers / the number of detractors) / the total number of responses, and obtain the score as the overall satisfaction dimension of the product.

[0126] The operation experience includes 4 parts of data: usage situation, usage depth, ease of use, and design standardization. Among them, the usage situation includes the number of user clicks, page views, and the number of user visits. The usage depth includes the page stay duration. Both dimensions are obtained by monitoring user behavior data through system data logging. Deploy data logging for each page of the function to be monitored through a third-party data management tool, and capture user behavior data in real time during the observation time. Take the average of the usage situation and usage depth data within the observation time and the standard deviation from the average of the same indicators in the past 3 months as the scores for the usage situation and usage depth dimensions. Higher than the average or lower than the average by 0.3 standard deviations is 5 points, 0.3 - 0.5 standard deviations is 4 points, 0.5 - 1 standard deviation is 3 points, 1 - 2 standard deviations is 2 points, and 2 standard deviations and above is 1 point; convert it into a final score of 1 - 5; the ease of use is based on the score of the "Product Ease of Use Scale" to measure the ease of use, understandability, and fault tolerance of the product. Invite the expert group and the user group to conduct an operation experience on the target product, and complete the scoring of the ease of use scale according to the real usage feelings. Obtain the data through the online questionnaire distribution. The final score of the recovered scoring results is calculated through the following formula:

[0127]

[0128] Among them, Qi is the scoring score of question i, k is the total number of questions, Wi is the weight of the question, N is the number of users in the questionnaire survey, and M is the number of experts in the questionnaire survey;

[0129] The topic weight Wi is determined according to the internal consistency coefficient of each topic. The smaller the internal consistency coefficient, the larger the topic weight Wi.

[0130] Based on the "Expert Consistency Evaluation Form" for design standardization, measure the consistency of the product in four dimensions: overall style, general framework, common scenarios / components, and copywriting norms. By recruiting experts and scoring according to the expert consistency evaluation form, obtain the scores. The total score calculation formula for this dimension is as follows:

[0131]

[0132] Among them, N represents the number of all standards in the consistency scale, n is the number of those that meet the standards (i.e., the score is 4 - 5 points), n / N is the pass rate, m is the number of standards in the consistency scale that are consistent with the life insurance experience norms, m / N is the reuse rate of life insurance experience norms, s is the scoring situation of all standards, and it measures the average value of the performance of each standard in the formula. The pass rate, the average value of the performance of each standard, and the reuse rate of life insurance norms each account for weights of 0.3, 0.4, and 0.3, and the result is multiplied by the full score of 5 points to form the final result.

[0133] After obtaining the data of the above four dimensions, convert according to the weights of each dimension (by default, all are 25% without special requirements) to form the total score of the product operation experience.

[0134] The efficiency experience refers to the one-time completion rate and completion duration when users use the product. By deploying data points through a third-party data management tool, capture user usage behavior data, and calculate the user retention situation and the time required from the product function home page to the final function page; the performance experience includes page loading duration, page crash times, and function error reporting rate, and perform corresponding deployment and monitoring through a dedicated APP performance monitoring tool. The scores of both indicators are obtained by taking the standard deviation of the data during the observation period and the average of the same-behavior data in the past 3 months as the score of this dimension. Higher than the average or lower than the average by 0.3 standard deviations is 5 points, 0.3 - 0.5 standard deviations is 4 points, 0.5 - 1 standard deviation is 3 points, 1 - 2 standard deviations is 2 points, and 2 standard deviations and above is 1 point; convert to the final score of 1 - 5.

[0135] Each of the 4 major experience dimensions will result in a score of 1 - 5. Through the formula: PXS = 0.3 * overall feeling + 0.25 * operation experience + 0.2 * efficiency experience + 0.15 * design experience + 0.1 * performance experience, obtain the final PXS score of the product.

[0136] Put the overall score into the experience design measurement standard to determine the current position of the product in terms of experience. This experience design measurement standard is formulated in combination with the life insurance situation.

[0137] 1 - 2 points indicate usability, meaning that the user operation has no obvious lags and the performance is stable, and the usage rate of the PXUI design specification is relatively low.

[0138] 2 - 3 points indicate ease of use, meaning that the user can complete tasks smoothly and each element basically conforms to the PXUI design specification.

[0139] 3 - 4 points indicate good usability, meaning that the user can efficiently complete tasks without assistance, the information structure is reasonable, and the interaction process is clear. 4 - 5 points indicate emotionalization, meaning that the user obtains satisfaction during use.

[0140] If the above - mentioned functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer - readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read - only memories (ROM, Read - Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs, etc., all kinds of media that can store program codes.

[0141] The above - mentioned is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A software evaluation method, characterized in that: The following steps are involved: Step S1: Compile a product usability scale through SPSS exploratory factor analysis; Step S2: Use the Delphi method to construct an expert consensus evaluation table suitable for the life insurance industry; Step S3: obtaining the insurance customer overall feeling index score through the insurance customer overall feeling index through an online questionnaire, obtaining the insurance agent overall feeling index score through the insurance agent overall feeling index through an online questionnaire, and obtaining the software overall feeling score based on the insurance customer overall feeling index score and the insurance agent overall feeling index score; Step S4: Use a third-party data management tool to deploy tracking points in the software and capture user usage behavior data; Step S5: scoring the efficiency experience of the software according to the user usage behavior data to obtain the efficiency experience score of the software, and scoring the performance experience of the software according to the user usage behavior data to obtain the performance experience score of the software; Step S6: scoring the usage and usage depth of the software according to the user usage behavior data, obtaining the usage score and usage depth score of the software, using the product usability scale as a questionnaire to obtain product usability scale data through an online questionnaire survey, obtaining the software usability score according to the product usability scale data, using the expert consistency evaluation form as a questionnaire to obtain expert consistency evaluation form data through an online questionnaire survey, obtaining the software design specification score according to the expert consistency evaluation form data, and obtaining the software operation experience score by weighted summing the usage score, usage depth score, usability score and design specification score; Step S7: Obtain the software evaluation score by weighted summing up the software's overall experience score, efficiency experience score, performance experience score and operation experience score, and determine whether the software is suitable for the life insurance industry based on the software's evaluation score.

2. A software evaluation method according to claim 1, characterized in that: The step S1 specifically includes the following steps: Step S11: compile an initial questionnaire based on the basic questions of the existing usability scale; Step S12: adding an understanding option to each question in the initial questionnaire, and adjusting the initial questionnaire according to the scores of the understanding options to obtain an adjusted questionnaire; Step S13: Use the adjusted questionnaire as a questionnaire to obtain the adjusted questionnaire scoring data through an online questionnaire, input the adjusted questionnaire scoring data into SPSS for exploratory factor analysis, explore the factor structure, identify potential high-order dimensions, obtain the standardized load coefficient of each question in the adjusted questionnaire, delete the questions with a standardized load coefficient less than a first preset value, use the remaining adjusted questionnaire scoring data to evaluate the internal consistency coefficient of each question using the Cronbach coefficient, eliminate the questionnaire questions according to the internal consistency coefficient of each question, and use the remaining questions as the product usability scale.

3. A software evaluation method according to claim 2, characterized in that: The step S12 of adjusting the initial questionnaire according to the scoring of the options includes the following steps: If the score of the option whether to understand is within the first preset range, the corresponding question is deleted; if the score of the option whether to understand is within the second preset range, the corresponding question is modified.

4. A software evaluation method according to claim 2, characterized in that: In step S13, the remaining adjusted questionnaire data is evaluated using Cronbach's coefficient to evaluate the internal consistency coefficient of each question, and the questionnaire questions are eliminated according to the internal consistency coefficient of each question, which specifically includes the following steps: The total Cronbach's alpha was calculated based on all the questions in the remaining adjusted questionnaire data; Each question in the questionnaire was removed individually and a new Cronbach's alpha was calculated; The total Cronbach's coefficient is subtracted from the new Cronbach's coefficient to obtain the internal consistency coefficient of the eliminated question. If the internal consistency coefficient is greater than or equal to the second preset value, the question is deleted from the questionnaire. If the internal consistency coefficient is less than the second preset value, the question is retained.

5. A software evaluation method according to claim 4, characterized in that: The Cronbach coefficient calculation formula is: Among them, α is Cronbach's coefficient, N is the number of questionnaire questions, is the average covariance between the scores of each question in the questionnaire, The average variance of the scores for each question in the questionnaire.

6. A software evaluation method according to claim 1, characterized in that: In step S3, the overall perception index of the insurance customer is the net recommendation value, and the overall perception index of the insurance agent is the satisfaction value; The step of obtaining the overall feeling score of the software according to the overall feeling index score of the insurance customer and the overall feeling index score of the insurance agent specifically comprises the following steps: Classify the insurance customers according to their overall perception index scores and the insurance agents' overall perception index scores, the classifications including: satisfied, complaining, and detracting; Count the number of people in each category, and use the result of (number of satisfied people - number of complainers / number of detractors) / total number of responses as the overall experience score of the software.

7. A software evaluation method according to claim 1, characterized in that: The user usage behavior data includes: user clicks, page visits, number of user visits, duration of stay on each page, function one-time completion rate, function one-time completion time, page loading time, number of page crashes, and function error rate.

8. A software evaluation method according to claim 1 or 7, characterized in that: Scoring the efficiency experience of the software according to the user usage behavior data to obtain the efficiency experience score of the software specifically includes the following steps: Obtain the function one-time completion rate and function one-time completion time from the user usage behavior data, calculate the standard deviation of the function one-time completion rate and function one-time completion time and the average of the same behavior data in the past three months, and use the calculated standard deviation as the software efficiency experience score; Scoring the performance experience of the software according to the user usage behavior data to obtain the performance experience score of the software specifically includes the following steps: Obtain page loading time, page crash times, and function error rate from user usage behavior data, calculate the standard deviation of page loading time, page crash times, function error rate and the average of the same behavior data in the past three months, and use the calculated standard deviation as the performance experience score of the software.

9. A software evaluation method according to claim 1, characterized in that: The usage score is obtained based on the number of user clicks, page views, and user visits in the user usage behavior data, and the usage depth score is obtained based on the length of time the user stays on each page in the user usage behavior data.

10. A software evaluation method according to claim 1 or 4, characterized in that: The method of using the product usability scale as a questionnaire to obtain product usability scale data through an online questionnaire survey, obtaining the software usability score according to the product usability scale data, using the expert consistency evaluation form as a questionnaire to obtain expert consistency evaluation form data through an online questionnaire survey, and obtaining the software design specification score according to the expert consistency evaluation form data specifically includes the following steps: The product usability scale is used as a questionnaire through an online questionnaire to obtain product usability scale data. According to the scores of each question in the product usability scale data, the usability score of the software is calculated by the following formula: Among them, Qi is the score of question i, k is the total number of questions, Wi is the weight of the question, N is the number of users surveyed, and M is the number of experts surveyed; The question weight Wi is determined according to the internal consistency coefficient of each question. The smaller the internal consistency coefficient, the greater the question weight Wi; The expert consistency evaluation form is used as a questionnaire to distribute online questionnaires to obtain the expert consistency evaluation form data. According to the scores of each question in the expert consistency evaluation form data, the design standardization score of the software is calculated by the following formula: Where N is the number of all questions in the expert consistency evaluation table, n is the number of questions with a score of 4 to 5 points, m is the number of questions in the expert consistency evaluation table that are consistent with the life insurance experience standards, and s i is the score of all questions i, w i is the weight of question i.

Images