Easy-to-use evaluation method and device, electronic equipment and program product

Abstract

The invention provides a usability evaluation method and device, electronic equipment and a program product, and relates to the technical field of artificial intelligence. The method comprises the steps of obtaining index values of a to-be-evaluated artificial intelligence content generation AIGC application at a plurality of evaluation indexes, wherein the plurality of evaluation indexes comprise an experience perception index, an intelligent degree index, a safety compliance index and a scene coverage index; a usability score for the AIGC application is determined based on the index values of the AIGC application at the plurality of evaluation indexes. According to the method, the usability of the AIGC application can be evaluated from different dimensions based on different types of evaluation indexes, and the reliability of evaluating the usability of the AIGC application can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a usability evaluation method, device, electronic equipment and program product. Background Art

[0002] In recent years, with the development of artificial intelligence technology, large model technology with deep learning as the core has made breakthrough progress. This breakthrough has improved the performance of machine learning in many fields such as natural language processing and image recognition.

[0003] Major manufacturers have keenly captured this development trend and have intensively launched and iterated artificial intelligence generated content (AIGC) application products based on large models. These AIGC application products have a wide range of applicable scenarios, covering AI dialogue, writing, learning, media creation, office business and other fields, greatly meeting the diverse functional needs of users.

[0004] At present, there are a large number of existing AIGC application products with varying quality, which makes it difficult for users to intuitively understand and select AIGC applications. Summary of the invention

[0005] The present application provides a usability evaluation method, device, electronic device and program product, which can evaluate the usability of AIGC applications from different dimensions based on different types of evaluation indicators, and can improve the reliability of the usability evaluation of AIGC applications.

[0006] In a first aspect, the present application provides an ease of use evaluation method, the method comprising: obtaining index values ​​of an artificial intelligence content generation AIGC application to be evaluated at multiple evaluation indicators; the multiple evaluation indicators include: experience perception indicators, intelligence level indicators, security and compliance indicators, and scenario coverage indicators; based on the index values ​​of the AIGC application at multiple evaluation indicators, determining the usability score of the AIGC application.

[0007] The usability evaluation method provided in this application evaluates the AIGC application to be evaluated based on multiple evaluation indicators, and obtains the index values ​​of the application to be evaluated at multiple evaluation indicators, wherein the multiple evaluation indicators cover the key aspects of AIGC applications in experience perception, intelligence level, security compliance, and scenario coverage. Furthermore, the index values ​​of the application to be evaluated at multiple evaluation indicators are calculated, and the evaluation index values ​​of multiple dimensions can be combined to obtain the usability score of the AIGC application to be evaluated. This method combines the evaluation indicators of AI models on the basis of traditional application evaluation indicators, so that the evaluation of AIGC applications is more in line with the evaluation criteria of artificial intelligence models, thereby improving the reliability of the usability evaluation of AIGC applications.

[0008] A possible implementation method is that multiple evaluation indicators include first-level evaluation indicators; the first-level evaluation indicators include: experience perception indicators, intelligence level indicators, security compliance indicators, and scenario coverage indicators; each first-level evaluation indicator includes multiple second-level evaluation indicators; each second-level evaluation indicator includes at least one third-level evaluation indicator.

[0009] In a possible implementation method, the secondary evaluation indicators of experience perception indicators include: functional completeness indicators, performance reliability indicators, user interface indicators, and customer service indicators; the tertiary evaluation indicators of functional completeness indicators include: functional diversity indicators, which are used to evaluate whether AIGC applications can provide generated content in different content forms; functional innovation indicators, which are used to evaluate whether AIGC applications use AI technology when generating content; functional extensibility indicators, which are used to evaluate whether AIGC applications allow the integration of other tools or data sources. The tertiary evaluation indicators of performance reliability indicators include: response speed indicators, which are used to evaluate the response speed of AIGC applications; compatibility indicators, which are used to evaluate the compatibility of AIGC applications with different devices or operating systems. The tertiary evaluation indicators of user interface indicators include: operation convenience indicators, which are used to evaluate the simplicity of the AIGC application user interface; operation personalization indicators, which are used to evaluate whether AIGC applications support personalized customization of generated content according to user needs. The tertiary evaluation indicators of customer service indicators include: user satisfaction indicators, which are used to evaluate user satisfaction of AIGC applications in application stores or community forums; customer service indicators, which are used to evaluate whether AIGC applications provide customer support services and usage documents.

[0010] In a possible implementation method, the secondary evaluation indicators of the intelligent degree index include: generated content quality index and intelligent agent service index. The tertiary evaluation indicators of the generated content quality index include: content correctness index, which is used to evaluate whether the grammar and logic of the generated content of the AIGC application are correct; content authenticity index, which is used to evaluate whether the generated content of the AIGC application is consistent with the actual facts; content timeliness index, which is used to evaluate whether the generated content of the AIGC application is consistent with the latest information; content relevance index, which is used to evaluate the degree of relevance between the generated content of the AIGC application and the prompt input by the user; content stability index, which is used to evaluate whether the AIGC application can output the generated content corresponding to the semantics of the prompt when there are typos in the prompt input by the user. The tertiary evaluation indicators of the intelligent agent service index include: anthropomorphism index, which is used to evaluate whether the AIGC application supports the creation of anthropomorphic intelligent agents; learning ability index, which is used to evaluate whether the intelligent agent created by the AIGC application can learn according to the user's historical prompts; prediction ability index, which is used to evaluate whether the intelligent agent created by the AIGC application can predict the user's needs and provide corresponding solutions.

[0011] In a possible implementation, the secondary evaluation indicators of security and compliance indicators include: data security secondary indicators, data compliance secondary indicators, and privacy protection secondary indicators. The tertiary evaluation indicators of data security secondary indicators include: data security level 3 indicators, which are used to evaluate whether the AIGC application adopts security measures to ensure the security of user data during data processing. The tertiary evaluation indicators of data compliance secondary indicators include: data compliance level 3 indicators, which are used to evaluate whether the AIGC application can generate guiding content corresponding to sensitive words when there are sensitive words in the prompt words input by the user. The tertiary evaluation indicators of privacy protection secondary indicators include: privacy protection level 3 indicators, which are used to evaluate whether the AIGC application has a privacy protection strategy.

[0012] A possible implementation method is that the secondary evaluation indicators of the scenario coverage indicators include: application indicators in the field of natural language processing and application indicators in the field of computer vision. The tertiary evaluation indicators of the application indicators in the field of natural language processing include: text generation indicators, which are used to evaluate whether AIGC applications have the ability to generate articles, press releases, or blog content; machine translation indicators, which are used to evaluate whether AIGC applications have the ability to translate texts in different languages; sentiment analysis indicators, which are used to evaluate whether AIGC applications have the ability to analyze sentiment tendencies based on text; dialogue and chat indicators, which are used to evaluate whether AIGC applications have the ability to interact with users using natural language; text summary indicators, which are used to evaluate whether AIGC applications have the ability to generate summaries based on documents; speech recognition and generation indicators, which are used to evaluate whether AIGC applications have the ability to convert between speech and text; code generation indicators, which are used to evaluate whether AIGC applications have the ability to generate code. The three-level evaluation indicators of application indicators in the field of computer vision include: image recognition and classification indicators, which are used to evaluate whether AIGC applications have the ability to recognize and classify objects in images; image generation indicators, which are used to evaluate whether AIGC applications have the ability to generate and modify images; object detection and tracking indicators, which are used to evaluate whether AIGC applications have the ability to detect and track objects in videos.

[0013] A possible implementation method is to obtain the index values ​​of the AIGC application at multiple evaluation indicators of the artificial intelligence content to be evaluated, including: obtaining the index values ​​of all third-level evaluation indicators. For all third-level evaluation indicators of the same second-level evaluation indicator, the index values ​​of the third-level evaluation indicators are weighted and summed to determine the index value of the corresponding second-level evaluation indicator. For all second-level evaluation indicators of the same first-level evaluation indicator, the index values ​​of the second-level evaluation indicators are weighted and summed to determine the index values ​​of the AIGC application at multiple first-level evaluation indicators.

[0014] A possible implementation method is to determine the usability score of the AIGC application based on the index values ​​of the AIGC application at multiple evaluation indicators, including: performing weighted summation on the index values ​​of the AIGC application at multiple primary evaluation indicators to obtain the usability score of the AIGC application.

[0015] A possible implementation method is that when weighted summing is performed on the indicator values ​​at multiple first-level evaluation indicators, the weight of each first-level evaluation indicator is determined in the following manner: multiple weighted questionnaires are obtained; each weighted questionnaire includes the relative scale between any two first-level evaluation indicators in multiple first-level evaluation indicators. The relative scale in each weighted questionnaire is used as an element in the matrix to obtain the priority judgment matrix corresponding to each weighted questionnaire. The priority judgment matrix corresponding to each weighted questionnaire is normalized to obtain the normalized matrix corresponding to each weighted questionnaire. The normalized matrix corresponding to each weighted questionnaire is normalized column averaged to obtain the weight set corresponding to each weighted questionnaire, and the weight set includes the weights of multiple first-level evaluation indicators. The weights of the same first-level evaluation indicator in all weight sets are averaged to obtain the weight of each first-level evaluation indicator.

[0016] In a second aspect, the present application provides a usability evaluation device, which includes various functional modules used in the method described in the first aspect above.

[0017] In a third aspect, the present application provides a computer program product, including: computer instructions; when the computer instructions are executed on an electronic device, the electronic device implements the method described in the first aspect above.

[0018] In a fourth aspect, the present application provides an electronic device, comprising: a processor and a memory; the memory stores instructions executable by the processor; when the processor is configured to execute the instructions, the electronic device implements the method described in the first aspect above.

[0019] In a fifth aspect, the present application provides a readable storage medium, which includes: software instructions; when the software instructions are executed in an electronic device, the electronic device implements the method described in the first aspect above.

[0020] The beneficial effects of the second to fifth aspects mentioned above can be referred to the first aspect and will not be elaborated on again. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A schematic diagram of a usability evaluation method provided in an embodiment of the present application;

[0023] Figure 2A flow chart of a method for obtaining an indicator value of a primary evaluation indicator provided in an embodiment of the present application;

[0024] Figure 3 A flow chart of a method for obtaining the weight of a primary evaluation index provided in an embodiment of the present application;

[0025] Figure 4 A flowchart of another usability evaluation method provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of a usability evaluation device provided in this application;

[0027] Figure 6 A schematic diagram of the composition of an electronic device provided in this application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] It should be noted that, in the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0030] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. are not limiting the quantity and execution order.

[0031] AIGC application refers to the use of artificial intelligence technology, based on deep learning, natural language processing, computer vision and other technical means, through the learning and analysis of massive data, can automatically generate or assist in the generation of application products in various forms of content such as text, images, audio, video, code, etc.

[0032] As can be seen from the background technology, thanks to the breakthrough progress of artificial intelligence technology, the number of existing AIGC applications continues to grow, and relying on the support of big data, the generated content of AIGC applications has been greatly improved in richness and diversity. The quality of different AIGC applications varies. For example, the performance parameters of the large model they rely on, the quality, scale and annotation accuracy of the large model training data may differ. This difference is difficult for users to capture, which leads to users often judging the ease of use of the application through simple trials and subjective feelings when choosing AIGC applications.

[0033] Therefore, how to scientifically and objectively evaluate the usability of AIGC applications is an urgent problem to be solved in the current field of artificial intelligence technology.

[0034] Based on this, the present application provides an ease of use evaluation method, which can combine the evaluation indicators of traditional applications and AI models, conduct a multi-dimensional investigation of AIGC applications, and then evaluate the usability of AIGC applications, which can improve the reliability of the evaluation of AIGC applications.

[0035] The usability evaluation method provided in this application can be applied to electronic devices (or computing devices), where the electronic device can be a server cluster composed of multiple servers, or a single server, or a computer, or a processor or processing chip in a server or computer, etc. The embodiments of this application do not limit the specific device form of the electronic device.

[0036] like Figure 1 A flow chart of a usability evaluation method provided in an embodiment of the present application is provided. The method can be applied to the above-mentioned electronic devices, such as Figure 1 As shown, the method includes S101-S102:

[0037] S101. Obtain the index values ​​of the artificial intelligence content to be evaluated and generate the AIGC application at multiple evaluation indicators.

[0038] Among them, multiple evaluation indicators include: experience perception indicators, intelligence level indicators, security and compliance indicators, and scenario coverage indicators.

[0039] In some embodiments, methods for obtaining the index values ​​of AIGC applications at multiple evaluation indexes may include the following: questionnaire survey, material review, and test verification.

[0040] Questionnaire survey: A questionnaire is designed based on the evaluation indicators and distributed to the survey population of AIGC applications, including users of AIGC applications, experts in the AI ​​field, suppliers of AIGC applications, etc. After collecting the questionnaires and obtaining the survey results, the survey results are quantified to obtain the index values ​​of AIGC applications at the corresponding evaluation indicators.

[0041] Material review: review the functional documents, technical documents, industry reports and other relevant materials provided by the AIGC application supplier, and quantify the review results to obtain the index values ​​of the AIGC application at the evaluation indicators corresponding to the relevant materials.

[0042] Test verification: design test cases based on the evaluation indicators, use predetermined test methods and test tools to test the AIGC application according to the test cases, and quantify the test results to obtain the indicator values ​​of the AIGC application at the corresponding evaluation indicators.

[0043] It should be noted that, for an evaluation indicator, one of the above-mentioned indicator value acquisition methods can be selected according to the evaluation content of the evaluation indicator.

[0044] It should also be noted that different types of evaluation indicators may cause the survey results, review results, and test results in the above-mentioned acquisition methods to be on different dimensions and cannot be directly compared due to different evaluation contents. Therefore, it is necessary to quantify these results according to certain numerical conversion rules so that the final indicator values ​​have unified standards and comparability.

[0045] One possible implementation method is to use a questionnaire survey to obtain the indicator values ​​of the AIGC application in the experience perception category. Experience perception indicators are closely related to the user's actual usage experience, intuitive understanding of functions, and experience of services. Because this type of indicator is more biased towards personal subjective feelings, the feedback from only a few individuals cannot reflect the actual experience perception of the AIGC application. Therefore, a questionnaire survey can be used to survey the AIGC application user group, which can eliminate the impact of individual subjective differences with rich sample data.

[0046] Another possible implementation method is to use the material review method to obtain the indicator values ​​of security compliance indicators. Security compliance indicators include data security, data compliance, and privacy protection. The functional documents, technical documents and other materials provided by the AIGC application supplier can reflect its security measures in various aspects such as data storage, transmission, and use. By reviewing these materials, the indicator values ​​of AIGC applications in security compliance indicators can be directly obtained.

[0047] Another possible implementation method is to use the test verification method to obtain the indicator value of the scenario coverage indicator. Scenario coverage indicators involve applications in the field of natural language processing (such as text generation, machine translation, conversation chat, etc.) and computer vision applications (such as image recognition and classification, image generation, etc.). By using predetermined test methods and test tools, comprehensive tests are carried out on AIGC applications for specific tasks in different scenarios, which can comprehensively judge its functional implementation and performance in various scenarios.

[0048] S102: Determine the usability score of the AIGC application based on the index values ​​of the AIGC application at multiple evaluation indicators.

[0049] Specifically, after obtaining the index values ​​of the AIGC application in the experience perception index, intelligence index, security compliance index, and scenario coverage index, it is necessary to calculate the comprehensive score of each index value as the usability score of the AIGC application. The specific process can be found below. Figure 2 , I will not go into details here.

[0050] It can be seen from steps S101-S102 that the above scheme can select appropriate indicator value acquisition methods for different types of evaluation indicators, and use numerical conversion rules to process results of different dimensions. It can evaluate the usability of AIGC applications from different dimensions, thereby improving the reliability of the evaluation results.

[0051] The following is a detailed introduction to different types of evaluation indicators.

[0052] In some embodiments, the multiple evaluation indicators may include first-level evaluation indicators, which include: experience perception indicators, intelligence level indicators, safety compliance indicators, and scenario coverage indicators. Each first-level evaluation indicator includes multiple second-level evaluation indicators, and each second-level evaluation indicator includes at least one third-level evaluation indicator.

[0053] Specifically, the secondary indicators of experience perception indicators include: functional completeness indicators, performance reliability indicators, user interface indicators, and customer service indicators. Among them:

[0054] The three-level evaluation indicators of the functional completeness index include: functional diversity index, functional innovation index, and functional extensibility index. The functional diversity index is used to evaluate whether the AIGC application can provide generated content in different content forms. The functional innovation index is used to evaluate whether the AIGC application uses AI technology when generating content. The functional extensibility index is used to evaluate whether the AIGC application allows the integration of other tools or data sources.

[0055] The three-level evaluation indicators of performance reliability indicators include: response speed indicator and compatibility indicator. The response speed indicator is used to evaluate the response speed of AIGC applications. The compatibility indicator is used to evaluate the compatibility of AIGC applications with different devices or operating systems.

[0056] The three-level evaluation indicators of the user interface index include: operation convenience index and operation personalization index. The operation convenience index is used to evaluate the simplicity of the AIGC application user interface. The operation personalization index is used to evaluate whether the AIGC application supports personalized customization of generated content according to user needs.

[0057] The three-level evaluation indicators of customer service indicators include: user satisfaction indicators and customer service indicators. User satisfaction indicators are used to evaluate the user satisfaction of AIGC applications in application stores or community forums. Customer service indicators are used to evaluate whether AIGC applications provide customer support services and usage documents.

[0058] Specifically, the secondary evaluation indicators of the degree of intelligence include: generated content quality indicators and intelligent body service indicators.

[0059] The three-level evaluation indicators of the generated content quality index include: content correctness index, content authenticity index, content timeliness index, content relevance index, and content stability index. The content correctness index is used to evaluate whether the grammar and logic of the generated content of the AIGC application are correct. The content authenticity index is used to evaluate whether the generated content of the AIGC application is consistent with the actual facts. The content timeliness index is used to evaluate whether the generated content of the AIGC application is consistent with the latest information. The content relevance index is used to evaluate the degree of relevance between the generated content of the AIGC application and the prompt input by the user. The content stability index is used to evaluate whether the AIGC application can output generated content corresponding to the semantics of the prompt when there are typos in the prompt input by the user.

[0060] The three-level evaluation indicators of the intelligent agent service index include: anthropomorphism degree index, learning ability index, and prediction ability index. The anthropomorphism degree index is used to evaluate whether the AIGC application supports the creation of anthropomorphic intelligent agents. The learning ability index is used to evaluate whether the intelligent agent created by the AIGC application can learn based on the user's historical prompts. The prediction ability index is used to evaluate whether the intelligent agent created by the AIGC application can predict the user's needs and provide corresponding solutions.

[0061] Specifically, the secondary evaluation indicators of security and compliance indicators include: data security secondary indicators, data compliance secondary indicators, and privacy protection secondary indicators.

[0062] The third-level evaluation indicators of the second-level data security indicators include: the third-level data security indicators are used to evaluate whether the AIGC application adopts security measures to ensure the security of user data during data processing.

[0063] The third-level evaluation indicators of the second-level data compliance indicators include: the third-level data compliance indicators are used to evaluate whether the AIGC application can generate guiding content corresponding to sensitive words when there are sensitive words in the prompt words input by the user.

[0064] The third-level evaluation indicators of the second-level privacy protection indicators include: the third-level privacy protection indicators are used to evaluate whether AIGC applications have privacy protection strategies.

[0065] Specifically, the secondary evaluation indicators of the scene coverage category include: application indicators in the field of natural language processing and application indicators in the field of computer vision.

[0066] The three-level evaluation indicators of the application indicators in the field of natural language processing include: text generation indicators, machine translation indicators, sentiment analysis indicators, dialogue chat indicators, text summary indicators, speech recognition and generation indicators, and code generation indicators. Text generation indicators are used to evaluate whether AIGC applications have the ability to generate articles, press releases, or blog content. Machine translation indicators are used to evaluate whether AIGC applications have the ability to translate texts in different languages. Sentiment analysis indicators are used to evaluate whether AIGC applications have the ability to analyze sentiment tendencies based on text. Dialogue chat indicators are used to evaluate whether AIGC applications have the ability to interact with users using natural language. Text summary indicators are used to evaluate whether AIGC applications have the ability to generate summaries based on documents. Speech recognition and generation indicators are used to evaluate whether AIGC applications have the ability to convert between speech and text. Code generation indicators are used to evaluate whether AIGC applications have the ability to generate code.

[0067] The three-level evaluation indicators of the application indicators in the field of computer vision include: image recognition and classification indicators, image generation indicators, and object detection and tracking indicators. Image recognition and classification indicators are used to evaluate whether the AIGC application has the ability to recognize and classify objects in the image. Image generation indicators are used to evaluate whether the AIGC application has the ability to generate and modify images. Object detection and tracking indicators are used to evaluate whether the AIGC application has the ability to detect and track objects in the video. Table 1 is a summary table of the first-level evaluation indicators, second-level evaluation indicators, and third-level evaluation indicators in the above multiple evaluation indicators.

[0068] Table 1

[0069]

[0070]

[0071]

[0072] It should be understood that by systematically dividing the evaluation indicators into experience perception, intelligence level, security compliance, and scenario coverage, and grading them according to the importance of each indicator in reflecting the characteristics of AIGC applications and the degree of refinement in specific business scenarios, the evaluation of AIGC applications can be more targeted and operational. This refined evaluation indicator system can go deep into each functional level of AIGC applications, and thus can more comprehensively capture the advantages and disadvantages of the AIGC applications to be evaluated in different dimensions.

[0073] In some embodiments, as shown in Table 1, the first-level evaluation index includes multiple second-level evaluation indexes, and the second-level evaluation index includes at least one third-level evaluation index. It can be seen that the index value of the first-level evaluation index can be obtained by calculating the index value of the corresponding third-level evaluation index. Then, the process of obtaining the index value of the AIGC application at multiple evaluation indicators in step S101 is as follows: Figure 2 As shown, it may specifically include:

[0074] S201. Obtain the index values ​​of all three-level evaluation indexes.

[0075] Specifically, the indicator values ​​of AIGC applied to all three-level evaluation indicators can be obtained through the indicator value acquisition method in the aforementioned step S101 (questionnaire, material review, test verification).

[0076] S202: For all the third-level evaluation indicators of the same second-level evaluation indicator, perform weighted summation on the indicator values ​​of the third-level evaluation indicators to determine the indicator value of the corresponding second-level evaluation indicator.

[0077] Specifically, the process can be expressed as:

[0078]

[0079] Among them, F i,j represents the index value of the jth secondary evaluation index under the i-th primary evaluation index, F i,j,k represents the index value of the kth third-level evaluation index under the jth second-level evaluation index, w i,j,k It represents the weight of the kth third-level evaluation index under the jth second-level evaluation index. l represents the number of third-level evaluation indicators under the jth second-level evaluation index.

[0080] For example, the process of obtaining the index value of the functional completeness index (a secondary evaluation index) in Table 1 is used as an example. The functional completeness index includes the functional diversity index, the functional innovation index, and the functional extensibility index. Assume that the weights of the diversity index, the functional innovation index, and the functional extensibility index are 0.4, 0.3, and 0.3, respectively, and their index values ​​are 2 points, 3 points, and 3 points, respectively. Then the calculation process of the index value of the functional completeness index is: (0.4×2)+(0.3×3)+(0.3×3)=2.2 points.

[0081] S203: For all the second-level evaluation indicators of the same first-level evaluation indicator, perform weighted summation on the indicator values ​​of the second-level evaluation indicators to determine the indicator values ​​of AIGC applied to multiple first-level evaluation indicators.

[0082] Specifically, the process can be expressed as:

[0083]

[0084] Among them, F i represents the index value of the i-th primary evaluation index, F i,j represents the index value of the jth secondary evaluation index under the ith primary evaluation index, w i,j represents the weight of the jth secondary evaluation indicator under the ith first-level evaluation indicator. m represents the number of secondary evaluation indicators under the ith first-level evaluation indicator.

[0085] For example, the process of obtaining the index value of the experience perception index (belonging to the first-level evaluation index) in Table 1 is taken as an example. The experience perception index includes a functional completeness index, a performance reliability index, a user interface index, and a customer service index. Assume that the weights of the functional completeness index, the performance reliability index, the user interface index, and the customer service index are 0.2, 0.3, 0.25, and 0.25, respectively, and their index values ​​are 2 points, 3 points, 3 points, and 2 points, respectively. Then the calculation process of the index value of the experience perception index is: (0.2×2)+(0.3×3)+(0.25×3)+(0.25×2)=2.55 points.

[0086] In some embodiments, in the process of determining the usability score of the AIGC application based on the indicator values ​​at multiple evaluation indicators in step S102, the indicator values ​​of the AIGC application at multiple primary evaluation indicators can be weighted summed to obtain the usability score of the AIGC application.

[0087] Specifically, the process can be expressed as:

[0088]

[0089] Among them, F represents the usability score of AIGC application, Fi represents the index value of the i-th level evaluation index, w i It represents the weight of the i-th first-level evaluation index. n represents the number of first-level evaluation indicators in the evaluation index.

[0090] In a possible implementation, when performing weighted summation on the index values ​​at multiple primary evaluation indexes, the weights of the primary evaluation indexes preset in the electronic device may be used for calculation.

[0091] Another possible implementation method is to determine the weight of each first-level evaluation indicator by using a questionnaire when performing weighted summation on the indicator values ​​at multiple first-level evaluation indicators. The process is as follows: Figure 3 As shown, the method may include:

[0092] S301. Obtain multiple weighted questionnaires.

[0093] Each weighted questionnaire includes a relative scale between any two first-level evaluation indicators among multiple first-level evaluation indicators.

[0094] Specifically, a relative scale is a numerical scale used to measure the relative importance of two things. In this application, it refers to the numerical values ​​given in the weighted questionnaire to indicate the relative importance between any two first-level evaluation indicators among multiple first-level evaluation indicators.

[0095] In some embodiments, the survey group of the weighted questionnaire can be experts in the field of artificial intelligence technology. As a cutting-edge application in the field of artificial intelligence, AIGC technology involves technical principles, algorithm architecture, application scenarios and potential risks, which are highly professional and complex. At present, experts in the field of artificial intelligence technology are highly professional and authoritative in terms of knowledge reserves, practical experience and the degree of mastery of industry dynamics. By surveying them through the designed weighted questionnaire, we can fully tap the professional insights of experts and obtain more accurate weight data of evaluation indicators.

[0096] For example, Table 2 is the result of a weighted questionnaire filled in by an industry expert. As shown in Table 2, the intersection unit of each row and each column represents the relative scale of the first-level evaluation index corresponding to the row relative to the first-level evaluation index corresponding to the column.

[0097] Taking the experience perception index (column) relative to the intelligence degree index (row) as an example, the value of the intersection unit is 1 / 4, which means that the importance of the experience perception index is 1 / 4 of the intelligence index.

[0098] Taking the experience perception indicators (columns) relative to the security compliance indicators (rows) as an example, the value of the intersection unit is 2, which means that the importance of the experience perception indicators is twice that of the security compliance indicators.

[0099] Table 2

[0100] Experience Perception Degree of intelligence Safety and Compliance Scene coverage Experience Perception 1 4 1 / 2 1 Degree of intelligence 1 / 4 1 1 / 2 1 / 3 Safety and Compliance 2 2 1 1 Scene coverage 1 3 1 1

[0101] S302: Taking the relative scale in each weighted questionnaire as an element in a matrix, a priority judgment matrix corresponding to each weighted questionnaire is obtained.

[0102] The priority judgment matrix is ​​a two-dimensional matrix obtained by filling the relative scale in a weighted questionnaire into the matrix. The priority judgment matrix can present the relative importance relationship between multiple first-level evaluation indicators.

[0103] For example, based on the questionnaire results shown in Table 2, a priority judgment matrix is ​​established, as shown in formula (4):

[0104]

[0105] S303 , normalizing the priority judgment matrix corresponding to each weighted questionnaire to obtain a normalized matrix corresponding to each weighted questionnaire.

[0106] Among them, normalization is an operation that transforms matrix elements according to specific rules, with the purpose of making the elements in the matrix comparable and with a unified metric standard, which is convenient for subsequent data analysis and calculation. In actual operation, a specific mathematical transformation is implemented for the priority judgment matrix, mapping each element in the matrix to a specific interval (usually [0,1]), while keeping the relative size relationship between the elements unchanged. The matrix obtained after processing is called a normalized matrix.

[0107] In some embodiments, the normalization process may include the following methods: column sum normalization method, row sum normalization method, maximum-minimum normalization method, and vector normalization method.

[0108] One possible implementation method is to use the column sum normalization method to normalize the priority judgment matrix. This method normalizes the sum of the elements in each column to 1, so that each element has a unified metric, which makes it easier to compare the importance of different indicators.

[0109] Exemplarily, the column and normalization method is used to perform positive programming processing on formula (4). First, the sum of the elements in each column in formula (4) is calculated: 1+1 / 4+2+1=4.5, 4+1+2+3=10, 1 / 2+1 / 2+1+1=3, 1+1 / 3+1+1≈3.333. Then, the row element is divided by the sum of the column elements of the column corresponding to the row element. The calculation process of the first row element is: 1÷4.5=0.235, 4÷10=0.4, 1 / 2÷3≈0.167, 1÷3.333=0.3, and so on. The normalized matrix obtained by the column elements of the second to fourth rows can be obtained as shown in formula (5).

[0110]

[0111] S304, performing normalized row averaging processing on the normalized matrix corresponding to each weighted questionnaire, to obtain a weight set corresponding to each weighted questionnaire.

[0112] It should be noted that the normalized matrix is ​​a process in which the elements of the rows or columns in the matrix are converted into numerical values ​​with specific proportional relationships through positive programming, and then the normalized matrix is ​​subjected to normalized row average processing to integrate these local proportional relationships into a comprehensive indicator reflecting the overall characteristics.

[0113] It should also be noted that for the normalized matrix obtained by the column-sum normalization method, it is necessary to perform a normalized row average, while for the normalized matrix obtained by the row-sum normalization method, it can perform a normalized column average.

[0114] For example, the specific process of normalizing the column average of the normalized matrix in formula (5) is given as an example. First, the sum of the elements in each column of the priority judgment matrix in formula (5) is calculated as follows: 0.154+0.120+0.333+0.167=0.774, 0.194, 0.615+0.180+0.333+0.5=1.928, 0.077+0.240+0.167+0.167=0.651, 0.154+0.160+0.167+0.167=0.648. Then, the average value of each column is calculated, 0.774 / 4=0.1935, 1.928 / 4=0.482, 0.651 / 4=0.16275, 0.648 / 4=0.162, and finally the weight set corresponding to the questionnaire in Table 2 above is obtained (0.1935, 0.482, 0.16275, 0.162).

[0115] S305: Calculate the average value of the weights of the same first-level evaluation index in all weight sets to obtain the weight of each first-level evaluation index.

[0116] Exemplarily, there are 5-point questionnaire results. After being converted into a weight set, they are respectively: (0.289, 0.112, 0.345, 0.254), (0.276, 0.109, 0.365, 0.25), (0.283, 0.114, 0.369, 0.234), (0.278, 0.108, 0.371, 0.234), (0.262, 0.111, 0.369, 0.258). Calculate the average value of the elements with the same subscript in this weight set as the weight of the first-level evaluation index corresponding to this subscript. For example, calculate the average value of the elements with subscript 1 in the weight set. 0.289 + 0.276 + 0.278 + 0.262 ≈ 0.278. Then the element with subscript 1 corresponds to the experience perception type index, and further the weight of the corresponding experience perception type index can be determined.

[0117] As can be seen from the above steps S301 - S305, by analyzing multiple questionnaire results to obtain the weights of multiple first-level evaluation indexes, this method can absorb the opinions of different groups on the importance of AIGC application evaluation indexes, and through the standardized processing and summarization of these diversified opinions, calculate the weights of the evaluation indexes, which can effectively eliminate the one-sidedness of individual judgments and make the weights of AIGC application evaluation indexes more objective and accurate.

[0118] Moreover, the above method for obtaining the weights of the first-level evaluation indexes based on questionnaires is also applicable to the process of obtaining the weights of the second-level and third-level evaluation indexes. It is possible to conduct a relative scale survey on all the second-level evaluation indexes belonging to the same first-level evaluation index, and conduct a relative scale survey on all the third-level evaluation indexes belonging to the same second-level evaluation index. The specific process can refer to S301 - S305 and will not be elaborated here.

[0119] As shown in Table 3, this is the weights of the first-level evaluation indexes, second-level evaluation indexes, and third-level evaluation indexes calculated based on the above steps S301 - S305 in the embodiments of the present application. In the present application, steps S201 - S203 can use the evaluation index weights shown in Table 3 for relevant calculations.

[0120] Table 3

[0121]

[0122]

[0123] In some embodiments, after determining the usability score of the AIGC application, it is also possible to conduct a grade assessment on the usability of the AIGC and generate a usability evaluation report. As Figure 4 shown, after step S102, S103 and S104 are also included:

[0124] S103. Determine the usability level of the AIGC application based on the usability score of the AIGC application.

[0125] Specifically, a set of usability levels can be divided according to the usability score of the AIGC application, each usability level corresponds to a scoring interval, and the usability level corresponding to the AIGC application is determined by judging the scoring interval into which the usability score of the AIGC application falls.

[0126] One possible implementation method is to put the usability score of the AIGC application into the usability grade score range to determine the usability grade of the AIGC application.

[0127] Another possible implementation method is that the usability score of the AIGC application can also be in the form of a ratio. In this case, the specific implementation is: the index values ​​of multiple first-level evaluation indicators of the AIGC application are judged against the index qualification threshold. If the index value of the first-level evaluation indicator is greater than or equal to its corresponding qualification threshold, the evaluation indicator is marked as qualified. Then, among the multiple evaluation indicators, the ratio of the number of qualified evaluation indicators to the total number of multiple evaluation indicators is calculated, and the ratio is used as the usability score of the AIGC application. The usability score is placed in the usability score range to determine the usability level of the AIGC application.

[0128] For example, assume that the usability level and scoring range are as follows: the general level scoring range is [0.1] points, the usability level (1,2] points, and the good level (2,3] points, and the weights of the experience perception index, the intelligence level index, the security compliance index, and the scenario coverage index are: 0.2, 0.3, 0.3, 0.2, and their index values ​​(the full score of the index value is 3 points) are: 2 points, 1 point, 3 points, 2 points. First, determine the usability score of the AIGC application: (2×0.2)+(1×0.3)+(3×0.3)+(2×0.2)=2 points, and then, according to the usability numerical score of the AIGC application of 2 points, the usability level of the AIGC application is the usability level.

[0129] For another example, assume that the usability level and scoring range are as follows: general level (0%, 25%] points, ease of use level (25%, 75%] points, easy-to-use level (75%, 100%] points, the index values ​​of experience perception indicators, intelligence level indicators, security compliance indicators, and scenario coverage indicators (assuming the full score is 100 points) are: 50 points, 60 points, 70 points, 80 points, and the qualified threshold of the above evaluation indicators is 60 points. First, compare the index value of each evaluation indicator with the qualified threshold of 60 points, and the number of qualified evaluation indicators is 3. Then calculate the proportion of qualified evaluation indicators in multiple evaluation indicators as 3 / 4×100%=75%. The usability level of the AIGC application is the ease of use level.

[0130] S104. Output an evaluation report of the AIGC application.

[0131] Specifically, after determining the usability level of the AIGC application, an evaluation report on the usability of the AIGC application can be generated. The evaluation report includes: the weights and index values ​​of the AIGC application at multiple evaluation indicators, the usability level of the AIGC application, and the usability analysis of the AIGC application.

[0132] Among them, the usability analysis includes a summary of the advantages and disadvantages of the AIGC application, an analysis of the causes of the disadvantages, and related improvement suggestions.

[0133] It should be understood that through the above steps S103-S104, the evaluation report provides an intuitive and clear usability level and detailed information on each evaluation indicator, which can help users quickly understand whether the AIGC application meets their needs, so as to more efficiently select the appropriate AIGC application. In addition, the supplier of AIGC applications can also clarify the direction of product improvement, optimize the investment of development resources, and enhance product competitiveness based on the analysis of the advantages and disadvantages of the functions in the evaluation report.

[0134] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. It should be easy to realize that the technical goals in this field are combined with the units and algorithm steps of each example described in the embodiments disclosed in this article, and the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical goals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0135] In an exemplary embodiment, the present application also provides a usability evaluation device, such as Figure 5 As shown, the usability evaluation device includes: an acquisition module 410 and a processing module 420.

[0136] The acquisition module 410 is used to obtain the index values ​​of the AIGC application to be evaluated at multiple evaluation indicators. The multiple evaluation indicators include: experience perception indicators, intelligence level indicators, security compliance indicators, and scenario coverage indicators.

[0137] The processing module 420 is used to determine the usability score of the AIGC application based on the index values ​​of the AIGC application at multiple evaluation indicators.

[0138] A possible implementation method is that multiple evaluation indicators include first-level evaluation indicators; the first-level evaluation indicators include: experience perception indicators, intelligence level indicators, security compliance indicators, and scenario coverage indicators; each first-level evaluation indicator includes multiple second-level evaluation indicators; each second-level evaluation indicator includes at least one third-level evaluation indicator.

[0139] In a possible implementation method, the secondary evaluation indicators of experience perception indicators include: functional completeness indicators, performance reliability indicators, user interface indicators, and customer service indicators; the tertiary evaluation indicators of functional completeness indicators include: functional diversity indicators, which are used to evaluate whether AIGC applications can provide generated content in different content forms; functional innovation indicators, which are used to evaluate whether AIGC applications use AI technology when generating content; functional extensibility indicators, which are used to evaluate whether AIGC applications allow the integration of other tools or data sources. The tertiary evaluation indicators of performance reliability indicators include: response speed indicators, which are used to evaluate the response speed of AIGC applications; compatibility indicators, which are used to evaluate the compatibility of AIGC applications with different devices or operating systems. The tertiary evaluation indicators of user interface indicators include: operation convenience indicators, which are used to evaluate the simplicity of the AIGC application user interface; operation personalization indicators, which are used to evaluate whether AIGC applications support personalized customization of generated content according to user needs. The tertiary evaluation indicators of customer service indicators include: user satisfaction indicators, which are used to evaluate user satisfaction of AIGC applications in application stores or community forums; customer service indicators, which are used to evaluate whether AIGC applications provide customer support services and usage documents.

[0140] In a possible implementation method, the secondary evaluation indicators of the intelligent degree index include: generated content quality index and intelligent agent service index. The tertiary evaluation indicators of the generated content quality index include: content correctness index, which is used to evaluate whether the grammar and logic of the generated content of the AIGC application are correct; content authenticity index, which is used to evaluate whether the generated content of the AIGC application is consistent with the actual facts; content timeliness index, which is used to evaluate whether the generated content of the AIGC application is consistent with the latest information; content relevance index, which is used to evaluate the degree of relevance between the generated content of the AIGC application and the prompt input by the user; content stability index, which is used to evaluate whether the AIGC application can output the generated content corresponding to the semantics of the prompt when there are typos in the prompt input by the user. The tertiary evaluation indicators of the intelligent agent service index include: anthropomorphism index, which is used to evaluate whether the AIGC application supports the creation of anthropomorphic intelligent agents; learning ability index, which is used to evaluate whether the intelligent agent created by the AIGC application can learn according to the user's historical prompts; prediction ability index, which is used to evaluate whether the intelligent agent created by the AIGC application can predict the user's needs and provide corresponding solutions.

[0141] In a possible implementation, the secondary evaluation indicators of security and compliance indicators include: data security secondary indicators, data compliance secondary indicators, and privacy protection secondary indicators. The tertiary evaluation indicators of data security secondary indicators include: data security level 3 indicators, which are used to evaluate whether the AIGC application adopts security measures to ensure the security of user data during data processing. The tertiary evaluation indicators of data compliance secondary indicators include: data compliance level 3 indicators, which are used to evaluate whether the AIGC application can generate guiding content corresponding to sensitive words when there are sensitive words in the prompt words input by the user. The tertiary evaluation indicators of privacy protection secondary indicators include: privacy protection level 3 indicators, which are used to evaluate whether the AIGC application has a privacy protection strategy.

[0142] A possible implementation method is that the secondary evaluation indicators of the scenario coverage indicators include: application indicators in the field of natural language processing and application indicators in the field of computer vision. The tertiary evaluation indicators of the application indicators in the field of natural language processing include: text generation indicators, which are used to evaluate whether AIGC applications have the ability to generate articles, press releases, or blog content; machine translation indicators, which are used to evaluate whether AIGC applications have the ability to translate texts in different languages; sentiment analysis indicators, which are used to evaluate whether AIGC applications have the ability to analyze sentiment tendencies based on text; dialogue and chat indicators, which are used to evaluate whether AIGC applications have the ability to interact with users using natural language; text summary indicators, which are used to evaluate whether AIGC applications have the ability to generate summaries based on documents; speech recognition and generation indicators, which are used to evaluate whether AIGC applications have the ability to convert between speech and text; code generation indicators, which are used to evaluate whether AIGC applications have the ability to generate code. The three-level evaluation indicators of application indicators in the field of computer vision include: image recognition and classification indicators, which are used to evaluate whether AIGC applications have the ability to recognize and classify objects in images; image generation indicators, which are used to evaluate whether AIGC applications have the ability to generate and modify images; object detection and tracking indicators, which are used to evaluate whether AIGC applications have the ability to detect and track objects in videos.

[0143] A possible implementation method, processing module 420, is specifically used to obtain the index values ​​of the artificial intelligence content to be evaluated and generated by AIGC at multiple evaluation indicators, including: obtaining the index values ​​of all three-level evaluation indicators. For all three-level evaluation indicators of the same second-level evaluation indicator, the index values ​​of the three-level evaluation indicators are weighted and summed to determine the index value of the corresponding second-level evaluation indicator. For all second-level evaluation indicators of the same first-level evaluation indicator, the index values ​​of the second-level evaluation indicators are weighted and summed to determine the index values ​​of AIGC applied at multiple first-level evaluation indicators.

[0144] A possible implementation method is to determine the usability score of the AIGC application based on the index values ​​of the AIGC application at multiple evaluation indicators, including: performing weighted summation on the index values ​​of the AIGC application at multiple primary evaluation indicators to obtain the usability score of the AIGC application.

[0145] A possible implementation method, processing module 420, is also used to obtain the weights of multiple first-level evaluation indicators, including: obtaining multiple weighted questionnaires; each weighted questionnaire includes the relative scale between any two first-level evaluation indicators in the multiple first-level evaluation indicators. The relative scale in each weighted questionnaire is used as an element in the matrix to obtain the priority judgment matrix corresponding to each weighted questionnaire. The priority judgment matrix corresponding to each weighted questionnaire is normalized to obtain the normalized matrix corresponding to each weighted questionnaire. The normalized matrix corresponding to each weighted questionnaire is normalized column averaged to obtain the weight set corresponding to each weighted questionnaire, and the weight set includes the weights of multiple first-level evaluation indicators. The weights of the same first-level evaluation indicators in all weight sets are averaged to obtain the weight of each first-level evaluation indicator.

[0146] It should be noted that Figure 5 The division of modules in the example is schematic and is only a logical function division. There may be other division methods in actual implementation. For example, two or more functions may be integrated into one processing module. The above integrated modules may be implemented in the form of hardware or software function modules.

[0147] In an exemplary embodiment, as described above, the computing device may be a computer or a server or other electronic device having a computing function. In this case, the present application also provides an electronic device, Figure 6 The following is a schematic diagram of the composition of the electronic device provided in the embodiment of the present application. Figure 6 As shown, the electronic device includes: a processor 10 , a memory 20 , a communication line 30 , a communication interface 40 , and an input / output interface 50 .

[0148] The processor 10 , the memory 20 , the communication interface 40 and the input / output interface 50 may be connected via a communication line 30 .

[0149] The processor 10 is used to execute the instructions stored in the memory 20 to implement the usability evaluation method provided in the above embodiments of the present application. The processor 10 can be a CPU, a general-purpose processor network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller (MCU) / single-chip microcomputer / single-chip microcomputer, a programmable logic device (PLD) or any combination thereof. The processor 10 can also be any other device with processing functions, such as a circuit, a device or a software module, which is not limited in the embodiments of the present application. In one example, the processor 10 may include one or more CPUs, such as Figure 6 As an optional implementation, the electronic device may include multiple processors, for example, in addition to the processor 10, it may also include a processor 60 ( Figure 6 The dashed line is used as an example.

[0150] The memory 20 is used to store instructions. For example, the instructions may be computer programs. Optionally, the memory 20 may be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage devices, etc., and the embodiments of the present application are not limited to this.

[0151] It should be noted that the memory 20 may exist independently of the processor 10, or may be integrated with the processor 10. The memory 20 may be located inside the electronic device, or may be located outside the electronic device, which is not limited in the embodiment of the present application.

[0152] The communication line 30 is used to transmit information between various components included in the electronic device.

[0153] The communication interface 40 is used to communicate with other devices or other communication networks. The other communication networks may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 40 may be a module, a circuit, a transceiver or any device capable of achieving communication.

[0154] The input / output interface 50 is used to implement human-computer interaction between a user and an electronic device, for example, to implement action interaction or information interaction between a user and an electronic device.

[0155] Exemplarily, the input / output interface 50 may be a mouse, a keyboard, a display screen, or a touch display screen, etc. Action interaction or information interaction between a user and an electronic device may be achieved through a mouse, a keyboard, a display screen, or a touch display screen, etc.

[0156] It should be noted that Figure 6 The structure shown in the figure does not constitute a limitation on the electronic device, except Figure 6 In addition to the components shown, the electronic device may include more or fewer components than shown, or a combination of certain components, or a different arrangement of components.

[0157] In an exemplary embodiment, the embodiment of the present application further provides a computer program product, which includes computer instructions. When the computer instructions are executed in an electronic device, the electronic device implements the method in the aforementioned method embodiment.

[0158] In an exemplary embodiment, the present application also provides a readable storage medium, which includes software instructions. When the software instructions are executed in an electronic device, the electronic device implements the method in the aforementioned method embodiment. The computer-readable storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device.

[0159] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer-executable instructions. When loading and executing computer-executable instructions on a computer, a process or function according to an embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-executable instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer-executable instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center.

[0160] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other changes to the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0161] Although the present application has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely exemplary illustrations of the present application as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0162] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A usability evaluation method, characterized in that: The method comprises: Obtaining index values ​​of the AIGC application for the artificial intelligence content to be evaluated at multiple evaluation indicators; the multiple evaluation indicators include: experience perception indicators, intelligence level indicators, security compliance indicators, and scenario coverage indicators; Based on the index values ​​of the AIGC application at multiple evaluation indicators, the usability score of the AIGC application is determined.

2. The method according to claim 1, characterized in that The multiple evaluation indicators include first-level evaluation indicators; the first-level evaluation indicators include: experience perception indicators, intelligence level indicators, security compliance indicators, and scenario coverage indicators; each first-level evaluation indicator includes multiple second-level evaluation indicators; each second-level evaluation indicator includes at least one third-level evaluation indicator.

3. The method according to claim 2, characterized in that The secondary evaluation indicators of the experience perception indicators include: functional completeness indicator, performance reliability indicator, user interface indicator, and customer service indicator; The three-level evaluation indicators of the functional completeness index include: functional diversity index, which is used to evaluate whether the AIGC application can provide generated content in different content forms; functional innovation index, which is used to evaluate whether the AIGC application applies AI technology when generating content; functional extensibility index, which is used to evaluate whether the AIGC application allows the integration of other tools or data sources; The three-level evaluation indicators of the performance reliability index include: a response speed index, which is used to evaluate the response speed of the AIGC application; a compatibility index, which is used to evaluate the compatibility of the AIGC application with different devices or operating systems; The three-level evaluation indicators of the user interface indicators include: an operation convenience indicator, which is used to evaluate the simplicity of the AIGC application user interface; an operation personalization indicator, which is used to evaluate whether the AIGC application supports personalized customization of generated content according to user needs; The three-level evaluation indicators of the customer service index include: a user satisfaction index, which is used to evaluate the user satisfaction of the AIGC application in the application store or community forum; and a customer service index, which is used to evaluate whether the AIGC application provides customer support services and usage documents.

4. The method according to claim 2, characterized in that: The secondary evaluation indicators of the intelligent degree indicators include: generated content quality indicators and intelligent agent service indicators; The three-level evaluation index of the generated content quality index includes: content correctness index, which is used to evaluate whether the syntax and logic of the generated content of the AIGC application are correct; content authenticity index, which is used to evaluate whether the generated content of the AIGC application is consistent with the actual facts; content timeliness index, which is used to evaluate whether the generated content of the AIGC application is consistent with the latest information; content relevance index, which is used to evaluate the relevance between the generated content of the AIGC application and the prompt input by the user; content stability index, which is used to evaluate whether the AIGC application can output generated content corresponding to the semantics of the prompt when there are typos in the prompt input by the user; The three-level evaluation indicators of the intelligent agent service indicators include: an anthropomorphism degree index, which is used to evaluate whether the AIGC application supports the creation of anthropomorphic intelligent agents; a learning ability index, which is used to evaluate whether the intelligent agent created by the AIGC application can learn according to the user's historical prompts; and a predictive ability index, which is used to evaluate whether the intelligent agent created by the AIGC application can predict user needs and provide corresponding solutions.

5. The method according to claim 2, characterized in that: The secondary evaluation indicators of the security compliance indicators include: data security secondary indicators, data compliance secondary indicators, and privacy protection secondary indicators; The third-level evaluation indicators of the data security secondary indicators include: data security third-level indicators, which are used to evaluate whether the AIGC application adopts security measures to ensure user data security during data processing; The third-level evaluation indicators of the data compliance second-level indicators include: the third-level data compliance indicators, which are used to evaluate whether the AIGC application can generate guiding generated content corresponding to sensitive words when there are sensitive words in the prompt words input by the user; The third-level evaluation indicators of the second-level privacy protection indicators include: a third-level privacy protection indicator, which is used to evaluate whether the AIGC application has a privacy protection strategy.

6. The method according to claim 2, characterized in that: The secondary evaluation indicators of the scenario coverage indicators include: application indicators in the field of natural language processing and application indicators in the field of computer vision; The three-level evaluation indicators of the application indicators in the field of natural language processing include: text generation indicators, which are used to evaluate whether the AIGC application has the ability to generate articles, press releases, or blog content; machine translation indicators, which are used to evaluate whether the AIGC application has the ability to translate texts in different languages; sentiment analysis indicators, which are used to evaluate whether the AIGC application has the ability to analyze sentiment tendencies based on text; dialogue chat indicators, which are used to evaluate whether the AIGC application has the ability to interact with users using natural language; text summary indicators, which are used to evaluate whether the AIGC application has the ability to generate summaries based on documents; speech recognition and generation indicators, which are used to evaluate whether the AIGC application has the ability to convert between speech and text; code generation indicators, which are used to evaluate whether the AIGC application has the ability to generate code; The three-level evaluation indicators of the computer vision field application indicators include: image recognition and classification indicators, which are used to evaluate whether the AIGC application has the ability to recognize and classify objects in images; image generation indicators, which are used to evaluate whether the AIGC application has the ability to generate and modify images; object detection and tracking indicators, which are used to evaluate whether the AIGC application has the ability to detect and track objects in videos.

7. The method according to claim 2, characterized in that: The obtaining of the index values ​​of the AIGC application at multiple evaluation indicators of the artificial intelligence content to be evaluated includes: Get the index values ​​of all three-level evaluation indicators; For all the third-level evaluation indicators of the same second-level evaluation indicator, the indicator values ​​of the third-level evaluation indicators are weighted and summed to determine the indicator value of the corresponding second-level evaluation indicator; For all the secondary evaluation indicators of the same primary evaluation indicator, the indicator values ​​of the secondary evaluation indicators are weighted and summed to determine the indicator values ​​of the AIGC applied to multiple primary evaluation indicators.

8. The method according to claim 7, characterized in that The determining the usability score of the AIGC application based on the index values ​​of the AIGC application at multiple evaluation indexes includes: The index values ​​of the AIGC application at multiple primary evaluation indicators are weighted and summed to obtain the usability score of the AIGC application.

9. The method according to claim 8, characterized in that When weighted summing is performed on the index values ​​at multiple first-level evaluation indexes, the weight of each first-level evaluation index is determined in the following manner: Obtain multiple weighted questionnaires; each weighted questionnaire includes a relative scale between any two first-level evaluation indicators among multiple first-level evaluation indicators; The relative scale in each weighted questionnaire is used as an element in the matrix to obtain the priority judgment matrix corresponding to each weighted questionnaire; The priority judgment matrix corresponding to each weighted questionnaire is normalized to obtain the normalized matrix corresponding to each weighted questionnaire; Performing canonical column averaging processing on the normalized matrix corresponding to each weighted questionnaire to obtain a weight set corresponding to each weighted questionnaire, wherein the weight set includes the weights of each of the plurality of primary evaluation indicators; The weights of the same first-level evaluation indicators in all weight sets are averaged to obtain the weight of each first-level evaluation indicator.

10. A usability evaluation device, characterized in that: The device comprises: an acquisition module and a processing module; The acquisition module is used to obtain the index values ​​of the artificial intelligence content generation AIGC application to be evaluated at multiple evaluation indicators; the multiple evaluation indicators include: experience perception indicators, intelligence level indicators, security compliance indicators, and scenario coverage indicators; The processing module is used to determine the usability score of the AIGC application based on the indicator values ​​of the AIGC application at multiple evaluation indicators.

11. An electronic device, characterized in that: include: Processor and memory; The memory stores instructions executable by the processor; When the processor is configured to execute the instructions, the electronic device implements the method according to any one of claims 1 to 9.

12. A readable storage medium, characterized in that: include: Software instructions; When the software instructions are executed in an electronic device, the electronic device implements the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that include: Computer instructions; When the computer instructions are executed in an electronic device, the electronic device is enabled to implement the method according to any one of claims 1 to 9.

Images