Product evaluation method and device, equipment, storage medium and product

Abstract

The invention discloses a product evaluation method and device, equipment, a storage medium and a product, and relates to the technical field of product evaluation.The product evaluation method comprises the steps that a target index used for product evaluation is obtained, and the target index is obtained through index decomposition based on an experience analysis model of a preset layer; the target indexes comprise indexes of different dimensions; determining target log data corresponding to the target index through a preset database based on the target index; and converting the target log data corresponding to the target indexes of different dimensions through a preset index scoring algorithm to obtain evaluation scores for transverse comparison, and generating a product evaluation report based on the evaluation scores. According to the method, the problems that analysis conclusions of different indexes are always mutually independent, a set of complete product evaluation system is difficult to form, and the accuracy of product evaluation is reduced can be avoided.

Description

Technical Field

[0001] The present application relates to the technical field of product evaluation, and in particular to a product evaluation method, device, equipment, storage medium and product. Background Art

[0002] Quantitative evaluation of corporate product experience refers to the process of quantitative analysis of the user experience of enterprise-level products (such as enterprise software, financial services, etc.). This quantitative evaluation can help companies better understand user needs, optimize product design, improve user experience, and ultimately promote business growth.

[0003] In related technologies, various objective data about customers’ use of products, such as access depth, behavior path, and task results, are collected, and a certain aspect of the customer’s use process is restored through the collected data. However, the analysis conclusions of different indicators are often independent of each other, making it difficult to form a complete product evaluation system, which reduces the accuracy of product evaluation. Summary of the invention

[0004] The main purpose of this application is to provide a product evaluation method, device, equipment, storage medium and product, aiming to solve the technical problem that the analysis conclusions of different indicators are often relatively independent of each other, it is difficult to form a complete product evaluation system, and the accuracy of product evaluation is reduced.

[0005] To achieve the above objectives, the present application proposes a product evaluation method, device, equipment, storage medium and product. The product evaluation method includes:

[0006] Obtaining a target indicator for product evaluation, wherein the target indicator is obtained by decomposing an indicator based on an experience analysis model of a preset layer, and the target indicator includes indicators of different dimensions;

[0007] Based on the target indicator, determine the target log data corresponding to the target indicator through a preset database;

[0008] The target log data corresponding to the target indicators of different dimensions are converted through a preset indicator scoring algorithm to obtain evaluation scores for horizontal comparison, and a product evaluation report is generated based on the evaluation scores.

[0009] In one embodiment, before the step of obtaining the target indicator for product evaluation, the following steps are included:

[0010] Obtain real-time log data uploaded by the client based on tracking points;

[0011] The real-time log data is stored in a preset database, and a mapping relationship is established between the preset indicator information in the preset database and the real-time log data to obtain a preset mapping relationship;

[0012] Based on the preset mapping relationship, data is screened on the preset database to determine the target log data corresponding to the target indicator.

[0013] In one embodiment, the preset mapping relationship includes an offline mapping relationship, and before the step of obtaining the real-time log data uploaded by the client based on the embedding point, the following steps are included:

[0014] Obtain historical log data uploaded by the client based on tracking points;

[0015] The historical log data is stored in a preset database, and a mapping relationship is established between the preset indicator information in the preset database and the historical log data to obtain an offline mapping relationship.

[0016] In one embodiment, after the step of determining the target log data corresponding to the target indicator through a preset database based on the target indicator, the method further includes:

[0017] Based on the target log data, determine the task log data and the error log data;

[0018] Based on the task log data, the ratio of the number of completed tasks to the total number of tasks is calculated to obtain the task completion rate;

[0019] Based on the task log data, calculate the average completion time and total completion time of all completed tasks to obtain the completion time;

[0020] Based on the error log data, the number of errors occurring per unit time is calculated to obtain an error density;

[0021] Based on the task completion rate, the completion time and the error density, a comprehensive analysis result of the task experience is obtained;

[0022] Based on the error log data, obtain a task error analysis result;

[0023] The task experience comprehensive analysis result and the task error analysis result are used as the task experience analysis result.

[0024] In one embodiment, the evaluation score includes a clarity score, and the step of converting the target log data corresponding to the target indicators of different dimensions through a preset indicator scoring algorithm to obtain an evaluation score for horizontal comparison includes:

[0025] Get the sampled page images;

[0026] Using a preset index scoring algorithm, the element focus and form clarity of the page images are scored to obtain an initial clarity score for each page;

[0027] The initial clarity scores are weighted and calculated to obtain clarity scores.

[0028] In one embodiment, before the step of obtaining real-time log data, the method further includes:

[0029] A target field is added to the real-time system log data to obtain the added real-time system log data, wherein the target field includes the batch number, the batch quantity, the quantity to be uploaded and the quantity uploaded, so that the user can monitor the data volume based on the target field to obtain the monitoring result.

[0030] In addition, to achieve the above purpose, the present application also proposes a product evaluation device, which includes:

[0031] An acquisition module, used to acquire target indicators for product evaluation, wherein the target indicators are obtained by decomposing indicators based on the experience analysis model of a preset layer, and the target indicators include indicators of different dimensions;

[0032] A determination module, used to determine the target log data corresponding to the target indicator through a preset database based on the target indicator;

[0033] The calculation module is used to convert the target log data corresponding to the target indicators of different dimensions through a preset indicator scoring algorithm to obtain evaluation scores for horizontal comparison, and generate a product evaluation report based on the evaluation scores.

[0034] In addition, to achieve the above objectives, the present application also proposes a product evaluation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the product evaluation method described above.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the product evaluation method described above are implemented.

[0036] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the product evaluation method described above are implemented.

[0037] One or more technical solutions proposed in this application have at least the following technical effects:

[0038] Compared with the related art, in which the analysis conclusions of different indicators are often relatively independent of each other, it is difficult to form a complete product evaluation system, and the accuracy of product evaluation is reduced, the present application obtains target indicators for product evaluation, wherein the target indicators are obtained by decomposing indicators based on a preset layer of experience analysis model, and the target indicators include indicators of different dimensions; based on the target indicators, the target log data corresponding to the target indicators are determined through a preset database; through a preset indicator scoring algorithm, the target log data corresponding to the target indicators of different dimensions are converted to obtain an evaluation score for horizontal comparison, and based on the evaluation score, a product evaluation report is generated. It can be understood that the present application obtains target indicators of different dimensions by decomposing indicators based on a five-layer experience analysis model, obtains the corresponding target log data of the target indicators through a preset database, converts the target log data through an indicator scoring algorithm, and obtains an evaluation score that can be compared horizontally, so that the analysis conclusions of different indicators are not independent. After obtaining the evaluation score, an evaluation report is generated, and then, the problem that the analysis conclusions of different indicators are often relatively independent of each other, it is difficult to form a complete product evaluation system, and the accuracy of product evaluation is reduced can be avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 A flowchart of the first embodiment of the product evaluation method of the present application is provided;

[0042] Figure 2 This is the overall system architecture diagram of the product evaluation method for this application;

[0043] Figure 3 This is the architecture diagram of the experience analysis model of the product evaluation method for this application;

[0044] Figure 4 This is a full cycle flow chart of data processing for the product evaluation method of this application;

[0045] Figure 5 A specific flow chart of the product evaluation method for this application;

[0046] Figure 6A flow chart of the second embodiment of the product evaluation method of this application;

[0047] Figure 7 This is a page diagram of the product evaluation method for this application;

[0048] Figure 8 This is a field description diagram of the product evaluation method for this application;

[0049] Fig. 9 This is a schematic diagram of the module structure of the product evaluation device according to the embodiment of the present application;

[0050] Fig.10 Schematic diagram of the device structure of the hardware operating environment involved in the product evaluation method in the embodiment of the present application.

[0051] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] The main solution of the embodiment of the present application is: obtaining target indicators for product evaluation, wherein the target indicators are obtained by decomposing indicators based on an experience analysis model of a preset layer, and the target indicators include indicators of different dimensions; based on the target indicators, determining target log data corresponding to the target indicators through a preset database; converting the target log data corresponding to the target indicators of different dimensions through a preset indicator scoring algorithm to obtain evaluation scores for horizontal comparison, and generating a product evaluation report based on the evaluation scores.

[0055] In related technologies, various objective data about customers’ use of products, such as access depth, behavior path, and task results, are collected, and a certain aspect of the customer’s use process is restored through the collected data. However, the analysis conclusions of different indicators are often independent of each other, making it difficult to form a complete product evaluation system, which reduces the accuracy of product evaluation.

[0056] This application obtains target indicators of different dimensions by decomposing indicators based on a five-layer experience analysis model, obtains target log data corresponding to the target indicators through a preset database, converts the target log data through an indicator scoring algorithm, and obtains evaluation scores that can be compared horizontally, so that the analysis conclusions of different indicators are not independent. After obtaining the evaluation scores, an evaluation report is generated, and further, it can be avoided that the analysis conclusions of different indicators are often relatively independent of each other, it is difficult to form a complete product evaluation system, and the accuracy of product evaluation is reduced.

[0057] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes a product evaluation device as an example to illustrate this embodiment and the following embodiments.

[0058] Based on this, the present application embodiment provides a product evaluation method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the product evaluation method of the present application.

[0059] In this embodiment, please refer to Figure 2 , Figure 2 The overall system architecture diagram is shown in FIG. 1 . The product evaluation method includes steps S100 to S300 :

[0060] Step S100, obtaining a target indicator for product evaluation, wherein the target indicator is obtained by decomposing an indicator based on an experience analysis model of a preset layer, and the target indicator includes indicators of different dimensions;

[0061] It is understandable that the preset experience analysis model is a five-layer experience analysis model, which is a five-layer experience analysis model based on user evaluation of public products. The five-layer experience analysis model includes performance experience, character experience, participation, clarity and satisfaction. Each layer of the experience analysis model is divided into corresponding evaluation indicators of different dimensions. Please refer to Figure 3 , Figure 3 The architecture diagram of the experience analysis model is disclosed. The product evaluation device obtains the target indicators for product evaluation that are split through the five-layer experience analysis model.

[0062] Step S200, based on the target indicator, determine the target log data corresponding to the target indicator through a preset database;

[0063] It should be noted that the preset database is a database storing log data corresponding to the indicator. After obtaining the target indicator, the product evaluation device obtains the target log data corresponding to the target indicator in the preset database based on the target indicator.

[0064] In a feasible implementation manner, the following steps are included before step S100:

[0065] Obtain real-time log data uploaded by the client based on tracking points;

[0066] It is understandable that the product evaluation device is provided with a client, a server, a distributed stream processing platform (kafka) and a unified log platform. The client is provided with a tracking point software development kit (SDK), and the tracking point SDK can build the tracking point generation capability into the front-end code of each product. The server is provided with a tracking point server (MDServer). When the product evaluation device acquires logs, it acquires them in batches, and the historical system log data is the system log data acquired in batches before the product evaluation. Before obtaining the target indicators for product evaluation, the product evaluation device's tracking point SDK acquires the real-time log of the product, encrypts the acquired real-time log, and sends it to the tracking point receiving service. After the tracking point receiving service is decrypted and verified, Kafka sends the historical system log data to the unified log platform for consumption. The unified log platform is provided with a scheduling center (Logstash) and a business processing module (EXT). Kafka sets different topics according to different product systems to distinguish different types of log data. The Logstash of the product evaluation device schedules Logstash to consume different Kafka partition data in real time according to different Topics and their sharding configurations, and sends it to the EXT module to extract indicator information from the log to obtain real-time log data.

[0067] The real-time log data is stored in a preset database, and a mapping relationship is established between the preset indicator information in the preset database and the real-time log data to obtain a preset mapping relationship;

[0068] It is understandable that the product evaluation device is equipped with a consumption cluster ES (Elasticsearch). The preset database includes a real-time database. The real-time database is a quasi-real-time indicator warehouse for storing indicators with high real-time requirements such as performance and user engagement. The quasi-real-time indicator warehouse is processed by the consumption cluster ES and can support real-time monitoring scenario requirements such as performance alarms and real-time online user numbers. The product evaluation device stores the real-time log data of the consumption cluster ES in the real-time database in the preset indicator database, and establishes a mapping relationship between the preset indicator information in the real-time database and the real-time log data. Please refer to Figure 4 , Figure 4 This is a flowchart of the entire data processing cycle.

[0069] Based on the preset mapping relationship, data is screened on the preset database to determine the target log data corresponding to the target indicator.

[0070] It should be noted that the product evaluation device is equipped with a public experience analysis management platform. The product evaluation device queries the target indicator data corresponding to the target indicator through the database management system (ClickHouse) based on the preset mapping relationship. The public experience analysis management platform of the product evaluation device obtains the target indicator data for product evaluation of different indicators through real-time mapping relationships. Please refer to Figure 5 , Figure 5 A specific flow chart.

[0071] In a feasible implementation manner, the step of obtaining the real-time log data uploaded by the client based on the embedding point includes the following steps:

[0072] Obtain historical log data uploaded by the client based on tracking points;

[0073] It is understandable that the tracking SDK of the product evaluation device obtains the historical logs of the product, encrypts the obtained historical logs, and sends them to the tracking receiving service. After decryption and verification by the tracking receiving service, Kafka sends the historical logs to the unified log platform for consumption. The Logstash of the product evaluation device schedules Logstash to consume different Kafka partition data in real time according to different Topics and their sharding configurations, and sends it to the EXT module to extract indicator information from the logs to obtain historical log data.

[0074] The historical log data is stored in a preset database, and a mapping relationship is established between the preset indicator information in the preset database and the historical log data to obtain an offline mapping relationship.

[0075] It should be noted that the preset database includes an offline database, which is used to store data such as task completion rate and completion time, which have large computational requirements but low real-time requirements. Therefore, when conducting product evaluation, it is necessary to obtain historical log data from the offline database for product evaluation. The product evaluation device is also provided with an offline consumption cluster (Hive), which is used as a data warehouse to store original embedded point log data. The product evaluation device writes the obtained offline indicator data into the Hive of the offline indicator library by date partition, and cleans the original data in Hive, such as removing invalid records and filling missing values, etc. The cleaned data is imported into the offline database, and a mapping relationship with the historical log data is established for each preset indicator information.

[0076] Step S300, converting the target log data corresponding to the target indicators of different dimensions through a preset indicator scoring algorithm to obtain evaluation scores that can be compared horizontally, and generating a product evaluation report based on the evaluation scores.

[0077] It should be noted that the product evaluation device converts the target log data corresponding to the target indicators of different dimensions through a preset indicator scoring algorithm, obtains the evaluation scores corresponding to the different indicators, and generates a product evaluation report based on the evaluation scores.

[0078] In a feasible implementation manner, based on the target indicator, after the step of determining the target log data corresponding to the target indicator through a preset database, the following steps are also included:

[0079] Based on the target log data, determine the task log data and the error log data;

[0080] It is understandable that the task log data records the start time, end time and status of the task. The error log records the error information that occurs during the execution of the task. The product evaluation device determines the task log data and the error log data from the target log data.

[0081] Based on the task log data, the ratio of the number of completed tasks to the total number of tasks is calculated to obtain the task completion rate;

[0082] It should be noted that the task completion rate represents the completion of the task. The product evaluation device calculates the ratio of the number of completed tasks to the total number of tasks to obtain the task completion rate.

[0083] Based on the task log data, calculate the average completion time and total completion time of all completed tasks to obtain the completion time;

[0084] It is understandable that the completion time is the comprehensive time of task execution. The product evaluation device calculates the average completion time and the total completion time to obtain the completion time.

[0085] Based on the error log data, the number of errors occurring per unit time is calculated to obtain an error density;

[0086] It should be noted that the error density is used to characterize the error situation. The product evaluation device calculates the number of errors that occur per unit time to obtain the error density.

[0087] Based on the task completion rate, the completion time and the error density, a comprehensive analysis result of the task experience is obtained;

[0088] It is understandable that the comprehensive analysis of task experience includes analyzing the task completion situation, including information such as task completion rate, completion time, and error density. The product evaluation device uses the task completion rate, completion time, and error density as the comprehensive analysis result of the task experience.

[0089] Based on the error log data, obtain a task error analysis result;

[0090] It should be noted that task error analysis is used to count business errors encountered by customers during task execution, helping product teams identify high-frequency errors and explore opportunities for optimization and improvement. Based on error logs, product evaluation equipment classifies errors by type, counts the number of occurrences of each error type, and identifies the error types with higher frequency, and uses the above information as the result of task error analysis.

[0091] The task experience comprehensive analysis result and the task error analysis result are used as the task experience analysis result.

[0092] It is understandable that the task experience analysis result is the result obtained by performing the task experience analysis function. The product evaluation device uses the task experience comprehensive analysis result and the task error analysis result as the task experience analysis result.

[0093] In particular, the product evaluation device can also analyze other specific information of the five-layer experience analysis model, as follows:

[0094] Performance experience analysis:

[0095] 1. Comprehensive analysis of page performance: Displays page performance-related data in multiple dimensions, including the performance of each link of page loading, the proportion of fast / slow opening, the ranking of long-loading pages, etc., and can identify abnormal page loading problems.

[0096] 2. Comprehensive analysis of static resources: Pay attention to the status of static resources, support sorting by static resource size / loading time, and identify abnormal static resources.

[0097] 3. White screen abnormality analysis: monitor white screen abnormalities and the impact of white screen on users.

[0098] 4.TOPURL (page) analysis: aggregate URLs by different dimensions for analysis, expose abnormal URLs, and assist in performance analysis to support sorting by visits / average visit duration / number of users.

[0099] 5. Comprehensive analysis of API (interface) calls: used to monitor the success rate, time consumption, errors, etc. of web API calls.

[0100] 6. Page error analysis: monitor the product’s web page errors and the impact on users by page dimension.

[0101] 7. Network and environment analysis: Statistics on the network, operating system and browser type when users use the product.

[0102] Other analysis:

[0103] 1. Comprehensive analysis of user journey: Analyze the user's operation path, error reports, etc. during product use according to user dimensions and timeline, so as to facilitate the location and tracking of individual user's usage behavior and anomalies.

[0104] 2. Comprehensive analysis of user visits: collect and compile core usage data of users using the product, including cumulative visit statistics and daily activity trend statistics.

[0105] 3. Comprehensive analysis of user visits: collect and compile core usage data of users using the product, including cumulative visit statistics and daily activity trend statistics.

[0106] 4. User access data analysis: Analyze the user access data of the product, and support sorting by user visits / number of pages opened / average visit duration, so as to facilitate the discovery and tracking of users with abnormal access.

[0107] 5. Task path analysis tool: automatically identifies the core path of the task, provides the function of viewing the path node leakage data, and can clearly display the operation path distribution through an intuitive Sankey diagram, assisting the product team in finding user loss nodes.

[0108] 6. Comprehensive analysis of user journey: Analyze the user's operation path, error reports, etc. during product use according to user dimensions and timeline, so as to facilitate the location and tracking of individual user's usage behavior and anomalies.

[0109] In this embodiment, the product evaluation device obtains the target indicator for product evaluation, and calculates the evaluation score for the target indicator through the offline indicator data and the offline indicator data corresponding to the target indicator in the preset indicator database, through a preset algorithm, to obtain the evaluation score for evaluating the product experience, and generates an evaluation report, forming a complete product evaluation system, which improves the accuracy of product evaluation.

[0110] The present application embodiment provides a product evaluation method, referring to Figure 6 , Figure 6 This is a flow chart of the second embodiment of the product evaluation method of the present application.

[0111] In a feasible implementation manner, the step of calculating the target indicator data by a preset algorithm to obtain an evaluation score includes steps A10 to A30:

[0112] Step A10: Obtain sampled page images;

[0113] It should be noted that the page images are obtained by professional designers through sampling and inspection of key pages of the product according to certain sampling rules. The product evaluation equipment scores the element focus and form clarity according to standard specifications, and performs weighted calculation according to the following page types to finally obtain the page clarity score. Please refer to Figure 7 , Figure 7 This is a page diagram.

[0114] Step A20: Score the element focus and form clarity of the page image using a preset index scoring algorithm to obtain an initial clarity score for each page;

[0115] It is understandable that the product evaluation device converts the indicators of different dimensions into percentage scores that can be compared and scored horizontally through static / dynamic interval conversion and sigmoid neural network activation algorithm. For product evaluation devices, the clarity of the product is equal to the clarity score of the sampled page multiplied by the load percentage of the page.

[0116] Step A30: performing weighted calculation on the initial clarity score to obtain a clarity score;

[0117] It should be noted that the product evaluation device performs weighted summation of initial clarity scores of different pages to obtain a clarity score.

[0118] Specifically, assuming that there are three pages uploaded in the product, namely the workbench, the purchase list page, and the purchase details page, and the exposure of the workbench during the statistical period is 1000 times, the purchase list page is 2000 times, and the purchase details page is 500 times, then the total AI clarity score of the product = workbench AI clarity score x (1000 / 1000+2000+500) + purchase list page AI clarity score x (2000 / 1000+2000+500) + purchase details page AI clarity score x (500 / 1000+2000+500).

[0119] Other indicators and scoring algorithms are as follows:

[0120] 1. Performance experience

[0121] (1) Page loading score, calculation method: static interval linear.

[0122] 1. First Contentful Paint (FCP):

[0123] Page dimension: The FCP of a single page load is calculated, and the average of all scores is used as the FCP score of the page. The benchmark is temporarily set at 3000ms (3 seconds), and the scoring interval is [0,3000]. The scoring algorithm is as follows:

[0124] Take the FCP value samples of all page loadings, remove abnormal samples first, and keep only the samples within [0,60000ms]. Take the FCP value t of the sample, in ms, and convert it into a percentage score according to the formula (3000-t) / 3000. If it is greater than 3s, it will be scored as 0, and the FCP score of a single load will be calculated. The average FCP score of all the loading samples of the page is taken as the FCP score of the page.

[0125] 2. Page loading time:

[0126] Page dimension: Calculate the score based on the PageLoad of a single page load, and then calculate the average of all scores as the PageLoad score of the page. The benchmark is temporarily set at 4000ms (4 seconds), and the scoring interval is [0,4000]. The scoring rules are as follows:

[0127] The algorithm is the same as FCP. First, abnormal samples are removed and only samples within [0,60000ms] are retained. The remaining samples are first calculated as single load scores, and then converted into percentage scores according to the formula (4000-t) / 4000. Samples longer than 4s are uniformly scored as 0. The average Pageload score of all page load samples is taken as the PageLoad score of the page.

[0128] 3. Page loading asynchronous request duration:

[0129] Page dimension: Calculate the score based on the asynchronous request performance of a single page load, and then calculate the average of all scores as the asynchronous request performance score of the page.

[0130] 3.1 First calculate the maximum return time of the asynchronous request for a single page single load, take the API-Invoke asynchronous request captured within the time interval of [Behavior.EntryTime, Behavior.EntryTime+ElapsedTime+1s] for a single page rendering (with the same PageLoadID), and exclude

[0131] For samples with endTime-startTime>60000ms, the rest are used as valid request samples for page rendering. According to the wooden barrel principle, the maximum APIInvoke.endTime of the asynchronous request sample is taken (the last asynchronous request returned), and the APIInvoke.endTime-Behavior.EntryTime time interval is calculated, that is, the maximum return duration of the asynchronous request for this loading (assumed to be t).

[0132] 3.2 Calculate the score of the asynchronous request duration for a single load. The benchmark time is temporarily set at 2000ms (2 seconds), and the scoring interval is [0, 2000]. Convert it into a percentage score according to the formula (2000-t) / 2000. If it is greater than 2s, it will be scored as 0. Then calculate the average score of the asynchronous request duration of all loading samples of the page. 2s is the temporary threshold. It can be optimized according to actual data later. For the loading time of single-page applications, since there is no standard automated detection and calculation solution in the industry, Jianwei uses a DOM change monitoring solution similar to Alibaba, which also has certain errors. It is necessary to explain it where the platform-related indicators are displayed. The final loading performance score of the page is the weighted average of the three duration scores, with the following weights: FCP: 2, LOAD: 3, asynchronous request duration: 1, function / service unit dimension: the average of all page performance scores under the function / service unit, product / system dimension: the average of all page performance scores under the product / system.

[0133] 2. Mission Accomplished

[0134] (1) Task completion rate, calculation method: direct scoring method.

[0135] Since the task success rate currently uses binary statistics (there are only two results: success - 1 and failure - 0), the task success rate can be directly used as the score. The scoring rules are as follows:

[0136] Task dimension: Take all valid samples of the task, calculate the task success rate, and adjust the judgment rule of valid samples to operationTimes>3 (<=3 is considered to be pure browsing users with no intention of doing tasks, and the sample proportion is roughly 20-30%).

[0137] Function / service unit dimension: the number of valid completed samples of all tasks under the function / service unit divided by the number of valid samples of all tasks.

[0138] Product / system dimension: the number of valid completed samples of all tasks under the product / system divided by the number of valid samples of all tasks.

[0139] Since the completion rate uses binary points (1 if completed, 0 otherwise), the result is unrelated to the task attributes (difficulty, process). Therefore, when calculating the completion rate score of a function or product, the sample calculations of all tasks should be combined, rather than calculating them separately by task and then calculating the average.

[0140] (2) Task completion time, calculation method: dynamic interval linear conversion.

[0141] Task Dimensions:

[0142] 1. Remove abnormal samples: For all time samples of successful tasks, first calculate the arithmetic mean and standard deviation, and exclude samples other than [mean - 2*standard deviation, mean + 2*standard deviation].

[0143] 2. Determine the reference interval:

[0144] Take the minimum value of the samples after (1) (assumed to be m) as the lower limit of the interval; calculate the arithmetic mean of the remaining samples (assumed to be x), and take twice the arithmetic mean as the upper limit of the interval. The complete reference interval is [m, 2x].

[0145] 3. After excluding abnormal samples, convert each remaining sample (assumed to be t) into a percentage score according to (2x-t) / (2x-m). Samples greater than 2x are uniformly scored as 0.

[0146] 4. Use the scores of all samples calculated by (3) to make a weighted average according to different intervals: [100,90):4, [90,80):3, [80,60]:2, (60,0]:1. ​​Calculate the average value as the score of the task completion time.

[0147] Function / service unit dimension: the average of the completion time scores of all tasks under the function / service unit.

[0148] Product / system dimension: The average of the completion time scores of all tasks under the product / system.

[0149] The completion time is related to the task attributes. When calculating the score, it should be calculated separately by task and then the average score is calculated. This will help expose low-scoring tasks.

[0150] (3) Task error density, calculation method: static interval linear conversion.

[0151] The benchmark is tentatively set at 0.5 per task (only business errors are counted, system errors are generally non-blocking and are not included in the scoring for the time being), the scoring interval is [0, 0.5], and the scoring rules are as follows:

[0152] Task dimension: First calculate the task error density (all successful and failed sample errors need to be counted), assume that the error density is e, and then convert it into a percentage score according to the formula (0.5-e) / 0.5. If the error density is greater than 0.5, it will be recorded as 0 points.

[0153] Function / service unit dimension: the average error density score of all tasks under the function / service unit.

[0154] Product / system dimension: The average error density score of all tasks under the product / system.

[0155] The error density is related to the task attributes. When calculating the score, it should be calculated separately by task and then the average score is calculated. This will help expose low-scoring tasks.

[0156] The total score of the task completion indicator is calculated by weighted average. According to the current analysis dimension, the scores of the task completion rate (weight: 2), duration (weight: 1), and error density (weight: 1) are calculated, and the weighted average is calculated as the total score.

[0157] 3. Satisfaction calculation:

[0158] The data source of satisfaction is the customer feedback platform and questionnaire system within the industry. The data is collected according to product dimensions, and the scores of different systems are summarized and calculated through a certain weight ratio to finally get the satisfaction score. Among them, the preset weights are satisfaction (2), clarity (1), task experience (2), and performance experience (1).

[0159] In a feasible implementation manner, the step of obtaining real-time system log data further includes the following steps:

[0160] A target field is added to the real-time system log data to obtain the added real-time system log data, wherein the target field includes the batch number, the batch quantity, the quantity to be uploaded and the quantity uploaded, so that the user can monitor the data volume based on the target field to obtain the monitoring result.

[0161] It should be noted that the product evaluation equipment is also equipped with a data warehouse to monitor the data volume. The product evaluation equipment ensures the integrity of the buried point data through batch checking (Batch-Check) to reduce the leakage of buried points. At present, the overall loss rate can be controlled at about 1.5%. The specific plan is described as follows:

[0162] The product evaluation device adds 4 new fields to the tracking log, and the values ​​of these fields are recalculated for each tracking session (APPSessionID). Please refer to Figure 8 , Figure 8 This is a field description diagram. The specific fields are as follows:

[0163] Batch No: The upload batch number of this log, starting from 1 and increasing;

[0164] Batch Size: The number of logs in the upload batch to which this log belongs. Since the client uploads logs in batches (packages), Batch Size is the total number of logs uploaded in this batch.

[0165] Number of logs to be uploaded (ToBeSent): The number of logs to be uploaded that are cached locally by the client when uploading the batch to which this log belongs;

[0166] HasBeenSent: The number of previously uploaded logs when the batch to which this log belongs is uploaded.

[0167] Similarly, the collection server can add a new field id similar to the concept of APPSessionID based on this solution: container-id.

[0168] In the actual calculation process, when the maximum HasBeenSent is counted according to the session dimension, the sum of HasBeenSent, ToBeSent and BatchSize is the client transmission data under the session dimension. When the maximum HaseBeenSent is counted according to the server id: container-id dimension, the sum of HasBeenSent, ToBeSent and BatchSize is the client transmission data under the server.

[0169] Finally, the total number of transmissions of all sessions on that day is summed up to get the total number of transmissions of the client. The total number of transmissions requested by all current containers is the total number of transmissions of the server. Finally, by requesting all data of the day through the data warehouse, the total number of Hive areas of the data warehouse can be obtained. By monitoring the three totals, the leakage of each segment in the transmission link can be obtained separately, which is convenient for accurate management of data leakage.

[0170] In this embodiment, the product evaluation device calculates the indicator data of different indicators through a preset algorithm to obtain an evaluation score for the overall evaluation of the product experience, and ensures the integrity of the buried data through batch inspection to reduce buried data leakage.

[0171] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the product evaluation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0172] This application also provides a product evaluation device, please refer to Fig. 9 , the product evaluation device comprises:

[0173] An acquisition module 10 is used to acquire a target indicator for product evaluation, wherein the target indicator is obtained by decomposing an indicator based on an experience analysis model of a preset layer, and the target indicator includes indicators of different dimensions;

[0174] A determination module 20, configured to determine target log data corresponding to the target indicator through a preset database based on the target indicator;

[0175] The calculation module 30 is used to convert the target log data corresponding to the target indicators of different dimensions through a preset indicator scoring algorithm to obtain evaluation scores for horizontal comparison, and generate a product evaluation report based on the evaluation scores.

[0176] Optionally, the acquisition module includes:

[0177] The real-time module is used to obtain real-time log data uploaded by the client based on the embedding point; store the real-time log data in a preset database, and establish a mapping relationship between the preset indicator information in the preset database and the real-time log data to obtain a preset mapping relationship; based on the preset mapping relationship, screen the data in the preset database to determine the target log data corresponding to the target indicator.

[0178] Optionally, the real-time module includes:

[0179] The offline module is used to obtain the historical log data uploaded by the client based on the embedding point; store the historical log data in a preset database, and establish a mapping relationship between the preset indicator information in the preset database and the historical log data to obtain an offline mapping relationship.

[0180] The monitoring module is used to add a target field to the real-time system log data to obtain the added real-time system log data, wherein the target field includes the batch number, the batch quantity, the quantity to be uploaded and the quantity uploaded, and the data volume is monitored based on the target field to obtain the monitoring result.

[0181] Optionally, the determining module includes:

[0182] The analysis module is used to determine task log data and error log data based on target log data; calculate the ratio of the number of completed tasks to the total number of tasks based on the task log data to obtain the task completion rate; calculate the average completion time and the total completion time of all completed tasks based on the task log data to obtain the completion time; calculate the number of errors occurring per unit time based on the error log data to obtain the error density; obtain a comprehensive analysis result of task experience based on the task completion rate, the completion time and the error density; obtain a task error analysis result based on the error log data; and use the comprehensive analysis result of task experience and the task error analysis result as the task experience analysis result.

[0183] Optionally, the calculation module includes:

[0184] The scoring module is used to obtain the sampled page images; score the element focus and form clarity of the page images through a preset indicator scoring algorithm to obtain an initial clarity score for each page; and perform weighted calculation on the initial clarity scores to obtain clarity scores.

[0185] The product evaluation device provided by the present application adopts the product evaluation method in the above embodiment, which can solve the technical problem of product evaluation. Compared with the prior art, the beneficial effects of the product evaluation device provided by the present application are the same as the beneficial effects of the product evaluation method provided by the above embodiment, and the other technical features in the product evaluation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0186] The present application provides a product evaluation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the product evaluation method in the above-mentioned embodiment one.

[0187] Reference below Fig.10 , which shows a schematic diagram of the structure of a product evaluation device suitable for implementing the embodiment of the present application. The product evaluation device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The product evaluation device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0188] like Fig.10As shown, the product evaluation device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the product evaluation device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the product evaluation device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a product evaluation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0189] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0190] The product evaluation device provided by the present application adopts the product evaluation method in the above embodiment to solve the technical problem of product evaluation. Compared with the prior art, the beneficial effects of the product evaluation device provided by the present application are the same as the beneficial effects of the product evaluation method provided by the above embodiment, and the other technical features in the product evaluation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0191] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0192] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0193] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the product evaluation method in the above-mentioned embodiment.

[0194] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0195] The computer-readable storage medium may be included in the product evaluation device; or may exist independently without being assembled into the product evaluation device.

[0196] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the product evaluation device, the product evaluation device: obtains target indicators for product evaluation, wherein the target indicators are obtained by decomposing indicators based on an experience analysis model of a preset layer, and the target indicators include indicators of different dimensions; based on the target indicators, determines the target log data corresponding to the target indicators through a preset database; through a preset indicator scoring algorithm, converts the target log data corresponding to the target indicators of different dimensions to obtain evaluation scores for horizontal comparison, and generates a product evaluation report based on the evaluation scores.

[0197] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0198] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0199] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0200] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned product evaluation method, and can solve the technical problems of product evaluation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the product evaluation method provided in the above-mentioned embodiment, and will not be repeated here.

[0201] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned product evaluation method when executed by a processor.

[0202] The computer program product provided by this application can solve the technical problem of product evaluation. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the product evaluation method provided by the above embodiment, which will not be repeated here.

[0203] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A product evaluation method, characterized in that: The product evaluation method comprises: Obtaining a target indicator for product evaluation, wherein the target indicator is obtained by decomposing an indicator based on an experience analysis model of a preset layer, and the target indicator includes indicators of different dimensions; Based on the target indicator, determine the target log data corresponding to the target indicator through a preset database; The target log data corresponding to the target indicators of different dimensions are converted through a preset indicator scoring algorithm to obtain evaluation scores for horizontal comparison, and a product evaluation report is generated based on the evaluation scores.

2. The product evaluation method according to claim 1, characterized in that: The step of obtaining the target indicator for product evaluation includes: Obtain real-time log data uploaded by the client based on tracking points; The real-time log data is stored in a preset database, and a mapping relationship is established between the preset indicator information in the preset database and the real-time log data to obtain a preset mapping relationship; Based on the preset mapping relationship, data is screened on the preset database to determine the target log data corresponding to the target indicator.

3. The product evaluation method according to claim 2, characterized in that: The preset mapping relationship includes an offline mapping relationship, and the step of obtaining the real-time log data uploaded by the client based on the embedding point includes: Obtain historical log data uploaded by the client based on tracking points; The historical log data is stored in a preset database, and a mapping relationship is established between the preset indicator information in the preset database and the historical log data to obtain an offline mapping relationship.

4. The product evaluation method according to claim 1, characterized in that: After the step of determining the target log data corresponding to the target indicator through a preset database based on the target indicator, the method further includes: Based on the target log data, determine the task log data and the error log data; Based on the task log data, the ratio of the number of completed tasks to the total number of tasks is calculated to obtain the task completion rate; Based on the task log data, calculate the average completion time and total completion time of all completed tasks to obtain the completion time; Based on the error log data, the number of errors occurring per unit time is calculated to obtain an error density; Based on the task completion rate, the completion time and the error density, a comprehensive analysis result of the task experience is obtained; Based on the error log data, obtain a task error analysis result; The task experience comprehensive analysis result and the task error analysis result are used as the task experience analysis result.

5. The product evaluation method according to claim 1, characterized in that: The evaluation score includes a clarity score, and the step of converting the target log data corresponding to the target indicators of different dimensions by a preset indicator scoring algorithm to obtain the evaluation score for horizontal comparison includes: Get the sampled page images; Using a preset index scoring algorithm, the element focus and form clarity of the page images are scored to obtain an initial clarity score for each page; The initial clarity scores are weighted and calculated to obtain clarity scores.

6. The product evaluation method according to claim 2, characterized in that: The step of obtaining real-time log data also includes: A target field is added to the real-time system log data to obtain the added real-time system log data, wherein the target field includes a batch number, a batch quantity, a quantity to be uploaded, and a quantity uploaded, and the data volume is monitored based on the target field to obtain a monitoring result.

7. A product evaluation method and device, characterized in that: The device comprises: An acquisition module, used to acquire target indicators for product evaluation, wherein the target indicators are obtained by decomposing indicators based on the experience analysis model of a preset layer, and the target indicators include indicators of different dimensions; A determination module, used to determine the target log data corresponding to the target indicator through a preset database based on the target indicator; The calculation module is used to convert the target log data corresponding to the target indicators of different dimensions through a preset indicator scoring algorithm to obtain evaluation scores for horizontal comparison, and generate a product evaluation report based on the evaluation scores.

8. A product evaluation device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the product evaluation method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the product evaluation method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the product evaluation method according to any one of claims 1 to 6 are implemented.

Images