AI virtual character intelligent generation effect analysis and evaluation method

Abstract

The invention belongs to the field of artificial intelligence, and particularly discloses an AI virtual character intelligent generation effect analysis and evaluation method, which comprises the following steps of: firstly, comparing a test text with an expected reply, evaluating a semantic comprehension degree, and ensuring that vocabularies are accurate and similar to sentence structures; secondly, historical dialogues are analyzed, context relevance is evaluated, and communication coherence is ensured; by combining the two items, the language understanding ability is comprehensively judged, and the deficiency is warned in time. Meanwhile, the response duration is monitored stably, and smooth user experience is ensured. In addition, collapse times in a historical period are tracked, operation stability is evaluated, and faults are prevented. Finally, multi-dimensional data such as language understanding, response speed and operation stability are integrated, the intelligent generation effect is comprehensively evaluated, a scientific basis is provided for performance optimization and problem early warning of the virtual character, it is ensured that the overall performance of the virtual character in the application meets the expectation, and potential problems are early warned in time.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and relates to a method for analyzing and evaluating the effect of intelligent generation of an AI virtual character. Background Art

[0002] With the rapid development of artificial intelligence technology, AI avatars, as a new interactive interface and entertainment form, have gradually penetrated into many areas of our lives. From games, social media to education, business, etc., AI avatars provide users with an unprecedented immersive experience with their unique interactive capabilities and personalized expressions. However, how to evaluate the intelligent generation effect of AI avatars has become an urgent problem to be solved.

[0003] For example, a Chinese patent with patent number CN110276436B discloses a method for monitoring a virtual character, which is characterized by comprising: A. creating an identity identifier corresponding to the virtual character; creating a relationship between the identity identifier and its associated person identifier; B. creating different identity type attributes of the virtual character according to different usage scenarios of the virtual character; different identity type attributes are limited to different speech and behavior rules; C. monitoring the speech and behavior information of the virtual character according to the identity identifier of the virtual character and based on the identity type attribute of the virtual character, and when the monitored speech and behavior information is inconsistent with the speech and behavior rules corresponding to the identity type attribute, the virtual character is processed accordingly, and its corresponding associated person identifier is obtained according to the identity identifier. As a result, it is conducive to realizing the monitoring of the virtual character, obtaining its corresponding associated person identifier according to the identity identifier, and timely processing of its irregular behavior.

[0004] The above existing technologies have the following problems: 1. When evaluating the comprehension ability of virtual characters, the existing technologies lack a unified quantitative standard to measure the performance of virtual characters in semantic comprehension. In the absence of a quantitative standard, even if it is found that the virtual character has problems in comprehension, it is difficult to accurately determine whether it is caused by inaccurate vocabulary recognition or other factors, which increases the difficulty of problem diagnosis and resolution.

[0005] 2. Existing virtual character dialogues often find it difficult to fully understand and remember the contextual information of long conversations, and can only process simple, short-term conversation histories. For complex, multi-round conversations, key information is easily lost, resulting in replies that are disconnected from the previous conversation content and lack coherence, which increases the abruptness of the replies and fails to ensure that the virtual character's response is closely connected with the previous conversation content, causing confusion or discomfort to users. Summary of the invention

[0006] In view of this, in order to solve the problems raised in the above background technology, a method for analyzing and evaluating the effect of intelligent generation of AI virtual characters is proposed.

[0007] The purpose of the present invention can be achieved through the following technical solutions: The present invention provides an AI virtual character intelligent generation effect analysis and evaluation method, including: S1, semantic understanding degree assessment: collecting the virtual character's replies based on language test texts in each language scene, extracting the expected replies corresponding to the language test texts in each language scene from the database, analyzing the vocabulary recognition accuracy index and sentence structure similarity coefficient of the virtual character's replies, and then analyzing the semantic understanding degree index Semantic of the virtual character's replies.

[0008] S2. Contextual relevance evaluation: Based on the virtual character's responses based on the language test text in each language scenario and the historical conversation data extracted from the database, the contextual relevance index context of the virtual character is analyzed.

[0009] S3. Language comprehension ability assessment: Based on the semantic comprehension degree index and context relevance degree index of the virtual character’s reply, analyze the virtual character’s language comprehension ability degree index com, determine whether the virtual character’s language comprehension ability is qualified, and issue a warning.

[0010] S4. Response time evaluation: extract the input time and response time of each request of the virtual character from the database, analyze the response time stability index of the virtual character, determine whether the response time stability index of the virtual character is qualified, and issue an early warning.

[0011] S5. Operational stability assessment: extract the number of crashes of the virtual character within a set historical period from the database, analyze the operational stability index of the virtual character, determine whether the operational stability of the virtual character is qualified, and then issue an early warning.

[0012] S6. Intelligent generation effect evaluation: According to the virtual character's language comprehension ability index, response time stability index and operation stability index, analyze the virtual character's intelligent generation effect qualification index, judge whether the virtual character's intelligent generation effect is qualified, and then issue an early warning.

[0013] Compared with the prior art, the present invention has the following beneficial effects: (1) By analyzing the vocabulary recognition accuracy and sentence structure similarity of the virtual character, the present invention can fully and accurately evaluate the semantic understanding ability of the virtual character in different language scenarios. These indicators provide a quantitative standard for the evaluation of semantic understanding ability, making the evaluation results more objective and comparable. High vocabulary recognition accuracy and sentence structure similarity ensure that the virtual character can accurately understand the user's questions and give expected answers, thereby improving the user experience.

[0014] (2) The present invention can evaluate whether the virtual character's responses in multiple language scenarios are consistent with the previous and subsequent conversations, thereby evaluating whether the virtual character can maintain the coherence and logic of the conversation. This coherence is crucial to improving the user experience, because it allows users to feel the naturalness and fluency of the conversation. When the context is highly relevant, it can reduce the abruptness of the response, ensure that the virtual character's response is closely connected with the previous conversation content, and avoid causing confusion or discomfort to the user.

[0015] (3) The present invention extracts the number of crashes of virtual characters within a set historical period from the database, analyzes the operation stability index of the virtual characters, determines whether the operation stability of the virtual characters is qualified, and then issues an early warning. It can promptly discover problems that arise during the operation of the virtual characters to avoid the accumulation of problems that lead to more serious consequences. When the number of crashes exceeds a preset threshold or shows an abnormal growth trend, the early warning mechanism can be quickly triggered to remind relevant personnel to pay attention and take measures, thereby effectively reducing system risks.

[0016] (4) The present invention can more fully and objectively evaluate the intelligent generation effect of virtual characters by comprehensively considering multiple dimensions such as language comprehension ability, response time stability and operation stability. Based on the comprehensive analysis of multi-dimensional data, it can more accurately judge whether the intelligent generation effect of virtual characters meets the preset standards, avoiding the one-sidedness caused by a single indicator. In addition, according to the analysis results of the intelligent generation effect qualification index, it can be clarified in which aspects the virtual characters are deficient, providing a clear direction for technology iteration and optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0018] Figure 1 It is a schematic diagram of the connection of each module of the system of the present invention.

[0019] Figure 2 A dependency tree of the virtual character's answer in the present invention.

[0020] Figure 3 This is the expected reply dependency tree described in the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] See also Figure 1 As shown, the present invention provides an AI virtual character intelligent generation effect analysis and evaluation method, the method comprising: S1, semantic understanding degree assessment: collecting the virtual character's replies based on the language test text in each language scene, extracting the expected replies corresponding to the language test text in each language scene from the database, analyzing the vocabulary recognition accuracy index and sentence structure similarity coefficient of the virtual character's replies, and then analyzing the semantic understanding degree index Semantic of the virtual character's replies.

[0023] Furthermore, the analysis of the vocabulary recognition accuracy index of the virtual character's responses includes: comparing the responses of the virtual character based on the language test text in each language scenario with the corresponding expected responses, obtaining statistically the same vocabulary between the responses of the virtual character based on the language test text in each language scenario and the corresponding expected responses, and summing them up to obtain the total vocabulary that is the same as the responses of the virtual character and the corresponding expected responses.

[0024] Based on the expected responses corresponding to the language test text in each language scenario, the vocabulary size of the expected responses corresponding to the language test text in each language scenario is obtained, and the vocabulary size is summarized to obtain the total vocabulary size of the expected responses.

[0025] The total vocabulary of the virtual character's response is compared with the total vocabulary of the corresponding expected response, and the vocabulary recognition accuracy index of the virtual character's response is obtained, which is recorded as Vocabulary.

[0026] Further, the analysis of the semantic understanding index of the virtual character's answer includes: performing syntactic analysis on the virtual character's answer based on the language test text and the corresponding expected answer in each language scene by using a syntactic analysis tool, and constructing a dependency relationship tree between the virtual character's answer based on the language test text and the corresponding expected answer in each language scene, see Figure 2 and Figure 3 As shown, the number of dependency relationships in the dependency tree of the virtual character's reply based on the language test text and the corresponding expected reply in each language scenario is obtained, and the number of dependency relationships in the dependency tree of the virtual character's reply and the corresponding expected reply is obtained, which are recorded as D1 and D2 respectively.

[0027] The dependency trees of the virtual character's responses based on the language test text and the corresponding expected responses in each language scenario are compared to obtain the number of identical dependencies and the number of dependencies with the same direction in the dependency trees of the virtual character's responses based on the language test text and the corresponding expected responses in each language scenario, and then the number of identical dependencies and the number of dependencies with the same direction in the dependency trees of the virtual character's responses and the corresponding expected responses are obtained, which are recorded as D3 and E1 respectively.

[0028] Analyze the sentence structure similarity coefficient between the virtual character's response and the corresponding expected response

[0029] structure,structure=same*ψ1+consistence*ψ2, where

[0030] Same indicates the same dependency ratio between the virtual character’s answer and the corresponding expected answer. Consistence represents the dependency direction consistency coefficient between the virtual character's answer and the corresponding expected answer ψ1 and ψ2 represent the weights corresponding to the same dependency ratio and dependency direction consistency coefficient of the set virtual character's answer and the corresponding expected answer, respectively, and ψ1+ψ2=1.

[0031] As a preferred embodiment, the ψ1 and ψ2 can be set to 0.6 and 0.4 respectively. The dependency relationship plays a vital role in the sentence structure. It defines the relationship between the various components in the sentence, such as the subject-predicate relationship, the verb-object relationship, etc. Therefore, the proportion of the same dependency relationship directly reflects the structural similarity of the two sentences. Although the dependency direction is also an important aspect of the sentence structure, in some cases, even if the dependency direction is inconsistent, as long as the dependency relationship is correct, it may not affect the basic meaning of the sentence. Therefore, the weight corresponding to the same dependency ratio of the virtual character's reply and the corresponding expected reply is greater than the weight corresponding to the dependency direction consistency coefficient.

[0032] Analyze the semantic understanding degree index Semantic of the virtual character's reply and the corresponding expected reply, Semantic = Vocabulary*χ1+structure*χ2, where χ1 and χ2 represent the weights corresponding to the vocabulary recognition accuracy index and sentence structure similarity coefficient of the virtual character's reply, respectively, and χ1+χ2=1.

[0033] As a preferred embodiment, the χ1 and χ2 can be set to 0.6 and 0.4 respectively. Vocabulary is the basic unit of language. The accuracy of vocabulary recognition directly affects the accuracy of the reply content. Sentence structure is an important means to express complex ideas and logical relationships. However, compared with vocabulary recognition, sentence structure similarity may be slightly less important in semantic understanding because it focuses more on form rather than content. Therefore, the weight corresponding to the vocabulary recognition accuracy index of the virtual character's reply is greater than the weights corresponding to the sentence structure similarity coefficient and the logical consistency coefficient.

[0034] The specific scenario of constructing the dependency tree of the virtual character's response based on the language test text and the corresponding expected response in each language scenario is: for example, language scenario: in a coffee shop, a customer asks the waiter about the type and price of coffee.

[0035] Customer question (expected question): "What kinds of coffee do you have? And how much do they cost?"

[0036] The virtual character replied: "We have latte, Americano and cappuccino here. The price of latte is 30 yuan, Americano is 25 yuan, and cappuccino is 35 yuan."

[0037] Expected answer: "We offer three types of coffee: latte, Americano, and cappuccino. The price of latte is 30 yuan, Americano is 25 yuan, and cappuccino is 35 yuan."

[0038] Said Figure 2 and Figure 3 The dependencies in the text include: ROOT (root node), SBV (subject-predicate relationship), VOB (verb-object relationship), ATT (attribute-predicate relationship), COO (parallel relationship), MT (function word component) and IC (clause structure).

[0039] Among them, ATT (attribute-center relationship) indicates the relationship between the attributive and the central word. For example, "latte" is the attributive of "coffee", which indicates the type of coffee; MT (function word component) is usually used to indicate the relationship between function words and the central word, such as punctuation marks, auxiliary words, etc. For example, "," and "." indicate pauses and ends in a sentence; IC (clause structure) is used to indicate two independent or related single sentences. For example, "the price of latte..." can be regarded as an independent description of the price of latte, which has a certain connection with the main sentence "we provide..." but is structurally independent.

[0040] The dependencies with the same direction specifically mean that in the dependency tree, not only the types of dependencies are the same, but also the directions of the nodes corresponding to the dependencies and the arrows between the nodes are consistent.

[0041] By analyzing the vocabulary recognition accuracy and sentence structure similarity of the virtual character, the present invention can fully and accurately evaluate the semantic comprehension ability of the virtual character in different language scenarios. These indicators provide a quantitative standard for the evaluation of semantic comprehension ability, making the evaluation results more objective and comparable. High vocabulary recognition accuracy and sentence structure similarity ensure that the virtual character can accurately understand the user's questions and give expected answers, thereby improving the user experience.

[0042] S2. Contextual relevance evaluation: Based on the virtual character's responses based on the language test text in each language scenario and the historical conversation data extracted from the database, the contextual relevance index context of the virtual character is analyzed.

[0043] Furthermore, the analyzing of the context relevance index of the virtual character includes: extracting historical dialogue segments based on language test texts in each language scene from the historical dialogue data.

[0044] Through the pre-trained word embedding model, the virtual character's responses based on the language test text and the corresponding historical dialogue fragments in each language scenario are converted into sentence vectors.

[0045] Substitute the sentence vectors of the virtual characters' responses based on the language test text and the corresponding historical dialogue fragments in each language scene into the cosine similarity calculation formula to obtain the cosine similarity of the sentence vectors of the virtual characters' responses based on the language test text and the corresponding historical dialogue fragments in each language scene, and then obtain the semantic similarity of the virtual characters, which is recorded as similarity.

[0046] As a preferred embodiment, the specific process of obtaining the semantic similarity of the virtual character is: substituting the cosine similarity of the sentence vectors of the virtual character's reply based on the language test text and the corresponding historical dialogue fragment in each language scene into the average calculation formula, obtaining the average cosine similarity of the sentence vectors of the virtual character's reply and the corresponding historical dialogue fragment, and recording it as the semantic similarity of the virtual character.

[0047] Based on the preset list of topic keywords, the total number of topic keywords in the replies of the virtual characters based on the language test text and the corresponding historical dialogue segments in each language scenario are counted and summarized to obtain the total number of topic keywords in the replies of the virtual characters and the corresponding historical dialogue segments, which are recorded as reply and theme respectively.

[0048] The responses of the virtual characters based on the language test text in each language scene are compared with the corresponding historical dialogue segments to obtain the number of identical topic keywords in the responses of the virtual characters based on the language test text and the corresponding historical dialogue segments in each language scene. These are summarized to obtain the number of identical topic keywords in the responses of the virtual characters and the corresponding historical dialogue segments, which is recorded as match.

[0049] Analyze the theme consistency coefficient of the virtual character Theme,

[0050] Analyze the context relevance index context of the virtual character, context = similarity*φ1+Theme*φ2, where φ1 and φ2 represent the weights corresponding to the semantic similarity and theme consistency coefficient of the virtual character respectively, and φ1+φ2=1.

[0051] As a preferred embodiment, φ1 and φ2 can be set to 0.6 and 0.4 respectively. Semantic similarity directly reflects whether the virtual character's reply understands the user's meaning, which is the basis for ensuring smooth conversation. Thematic consistency ensures the coherence and logic of the conversation and avoids the virtual character's reply from deviating from the topic. However, compared with semantic similarity, its influence is slightly smaller. Therefore, the weight corresponding to the semantic similarity of the virtual character is greater than the weight corresponding to the topic consistency coefficient.

[0052] The present invention can evaluate whether the virtual character's responses in multiple language scenarios are consistent with the previous and subsequent conversations, and whether the virtual character can maintain the coherence and logic of the conversation. This coherence is crucial to improving the user experience, because it makes the user feel that the conversation is natural and smooth. When the context is highly relevant, it can reduce the abruptness of the response, ensure that the virtual character's response is closely connected with the previous conversation content, and avoid confusion or discomfort to the user.

[0053] S3. Language comprehension ability assessment: Based on the semantic comprehension degree index and context relevance degree index of the virtual character’s reply, analyze the virtual character’s language comprehension ability degree index com, determine whether the virtual character’s language comprehension ability is qualified, and issue a warning.

[0054] The specific calculation formula for analyzing the language comprehension ability index of the virtual character is: comprehension = Semantic*β1+context*β2, where β1 and β2 represent the weights corresponding to the semantic comprehension index and context relevance index of the virtual character's reply, respectively, and β1+β2=1.

[0055] As a preferred embodiment, the β1 and β2 can be set to 0.6 and 0.4 respectively. Semantic understanding is the core of language understanding, which is directly related to whether the virtual character can accurately capture and understand the meaning of the user's input information. Contextual relevance refers to whether the virtual character can fully consider the context of the question when answering the question, so as to give a more appropriate and coherent answer. Although it is not as good as semantic understanding in directly determining the accuracy of the answer, it plays an important role in improving user experience and enhancing the naturalness of the answer. Therefore, the weight corresponding to the semantic understanding degree index of the virtual character's answer is greater than the weight corresponding to the contextual relevance degree index.

[0056] S4. Response time evaluation: extract the input time and response time of each request of the virtual character from the database, analyze the response time stability index of the virtual character, determine whether the response time stability index of the virtual character is qualified, and issue an early warning.

[0057] The step of judging whether the stability of the virtual character's response time is qualified includes: subtracting the input time from the response time of each request of the virtual character to obtain the response time of each request of the virtual character, and recording it as time g , g=1,2,3,...G.

[0058] Substitute the response time of each virtual character's request into the average calculation formula to obtain the average response time of the virtual character, which is recorded as

[0059] Analyze the virtual character's response time stability index response, G represents the number of input requests.

[0060] The response time stability index of the virtual character is compared with the preset response time stability index threshold. If the response time stability index of the virtual character is less than the preset response time stability index threshold, the response time stability of the virtual character is unqualified, and an early warning is issued.

[0061] By analyzing the response time stability index, the present invention can ensure that the virtual character provides timely and stable responses to users, thereby improving user satisfaction and willingness to use. Once the response time fluctuates abnormally or exceeds a preset threshold, the early warning mechanism can be triggered quickly, so that the problem can be paid attention to and handled in time to avoid the problem from expanding.

[0062] S5. Operational stability assessment: extract the number of crashes of the virtual character within a set historical period from the database, analyze the operational stability index of the virtual character, determine whether the operational stability of the virtual character is qualified, and then issue an early warning.

[0063] The number of days in a set historical cycle is compared with the number of crashes of the virtual character in the set historical cycle to obtain the crash frequency of the virtual character, which is recorded as Collapse.

[0064] The allowed crash frequency of the virtual character is extracted from the database and recorded as Analyze the operational stability index of the virtual character.

[0065] The determining whether the operation stability of the virtual character is qualified includes: comparing the operation stability index of the virtual character with a preset operation stability index threshold; if the operation stability index of the virtual character is less than the preset operation stability index threshold, the operation stability of the virtual character is unqualified, and an early warning is issued.

[0066] The present invention extracts the number of crashes of virtual characters within a set historical period from a database, analyzes the operating stability index of the virtual characters, determines whether the operating stability of the virtual characters is qualified, and then issues an early warning. It can promptly discover problems that arise during the operation of the virtual characters to avoid accumulation of problems that lead to more serious consequences. When the number of crashes exceeds a preset threshold or shows an abnormal growth trend, the early warning mechanism can be quickly triggered to remind relevant personnel to pay attention and take measures, thereby effectively reducing system risks.

[0067] S6. Intelligent generation effect evaluation: According to the virtual character's language comprehension ability index, response time stability index and operation stability index, analyze the virtual character's intelligent generation effect qualification index, judge whether the virtual character's intelligent generation effect is qualified, and then issue an early warning.

[0068] The specific formula for analyzing the qualified degree index of the intelligent generation effect of the virtual character is: Effect = α1*comprehension+α2*response+α3*Operational, wherein α1, α2 and α3 respectively represent the weights corresponding to the virtual character's language comprehension ability index, response time stability index and operational stability index, and α1+α2+α3=1.

[0069] As a preferred embodiment, the α1, α2 and α3 can be set to 0.5, 0.25 and 0.25 respectively. Language understanding is the basis of the interaction between the virtual character and the user, which directly determines whether the virtual character can accurately understand the user's needs and give an appropriate response. The stability of the response time is an important indicator for measuring the reaction time and quality of the virtual character. The operation stability is a key factor to ensure that the virtual character can provide services continuously and reliably. Therefore, the weight corresponding to the virtual character's language comprehension ability index is greater than the weights corresponding to the response time stability index and the operation stability index.

[0070] The judging whether the intelligent generation effect of the virtual character is qualified includes: comparing the qualified degree index of the intelligent generation effect of the virtual character with a preset qualified degree index threshold of the intelligent generation effect; if the qualified degree index of the intelligent generation effect of the virtual character is less than the preset qualified degree index threshold of the intelligent generation effect, the intelligent generation effect of the virtual character is unqualified, and then an early warning is issued.

[0071] The present invention can more fully and objectively evaluate the intelligent generation effect of virtual characters by comprehensively considering multiple dimensions such as language comprehension ability, response time stability and operation stability. Based on the comprehensive analysis of multi-dimensional data, it can more accurately judge whether the intelligent generation effect of the virtual character meets the preset standard, avoiding the one-sidedness caused by a single indicator. In addition, according to the analysis results of the intelligent generation effect qualification index, it can be clarified in which aspects the virtual character has deficiencies, providing a clear direction for technology iteration and optimization.

[0072] The above contents are merely examples and explanations of the concept of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the present invention, they shall all fall within the protection scope of the present invention.

Claims

1. A method for analyzing and evaluating the effect of intelligent generation of AI virtual characters, characterized in that: The method includes: S1. Semantic understanding degree assessment: Collect the responses of the virtual characters based on the language test text in each language scenario, extract the expected responses corresponding to the language test text in each language scenario from the database, analyze the vocabulary recognition accuracy index and sentence structure similarity coefficient of the virtual character's responses, and then analyze the semantic understanding degree index of the virtual character's responses; S2, context relevance evaluation: Based on the virtual character's responses based on the language test text in each language scenario and the historical dialogue data extracted from the database, the context relevance index context of the virtual character is analyzed; S3. Language comprehension ability assessment: Based on the semantic comprehension degree index and context relevance degree index of the virtual character's reply, the language comprehension ability degree index of the virtual character is analyzed to determine whether the language comprehension ability of the virtual character is qualified and issue an early warning; S4, response time evaluation: extract the input time and response time of each request of the virtual character from the database, analyze the response of the virtual character's response time stability index, determine whether the virtual character's response time stability index is qualified, and issue an early warning; S5. Operational stability assessment: extract the number of crashes of the virtual character in the set historical period from the database, analyze the operational stability index of the virtual character, determine whether the operational stability of the virtual character is qualified, and then issue an early warning; S6. Intelligent generation effect evaluation: According to the virtual character's language comprehension ability index, response time stability index and operation stability index, analyze the virtual character's intelligent generation effect qualification index, judge whether the virtual character's intelligent generation effect is qualified, and then issue an early warning.

2. The method for analyzing and evaluating the effect of intelligent generation of AI virtual characters according to claim 1, characterized in that: The analysis of the vocabulary recognition accuracy index of the virtual character's response includes: Comparing the responses of the virtual characters based on the language test text in each language scenario with the corresponding expected responses, obtaining the same vocabulary in the responses of the virtual characters based on the language test text in each language scenario and the corresponding expected responses, and summing them up to obtain the same total vocabulary in the responses of the virtual characters and the corresponding expected responses; Based on the expected responses corresponding to the language test text in each language scenario, the vocabulary of the expected responses corresponding to the language test text in each language scenario is obtained, and the vocabulary is summarized to obtain the total vocabulary of the expected responses; The total vocabulary of the virtual character's response is compared with the total vocabulary of the corresponding expected response, and the vocabulary recognition accuracy index of the virtual character's response is obtained, which is recorded as Vocabulary.

3. The method for analyzing and evaluating the effect of intelligent generation of AI virtual characters according to claim 2, characterized in that: The analysis of the semantic understanding index of the virtual character's reply includes: Perform syntactic analysis on the virtual character's answer based on the language test text and the corresponding expected answer in each language scene by using a syntactic analysis tool, construct a dependency tree of the virtual character's answer based on the language test text and the corresponding expected answer in each language scene, and then obtain the number of dependency relationships in the dependency tree of the virtual character's answer based on the language test text and the corresponding expected answer in each language scene, and then obtain the number of dependency relationships in the dependency tree of the virtual character's answer and the corresponding expected answer, which are recorded as D1 and D2 respectively; Compare the dependency trees of the virtual character's answer based on the language test text and the corresponding expected answer in each language scenario, and obtain the number of identical dependencies and the number of dependencies with the same direction in the dependency tree between the virtual character's answer based on the language test text and the corresponding expected answer in each language scenario, and then obtain the number of identical dependencies and the number of dependencies with the same direction in the dependency tree between the virtual character's answer and the corresponding expected answer, which are recorded as D3 and E1 respectively; Analyze the sentence structure similarity coefficient between the virtual character's response and the corresponding expected response structure,structure=same*ψ1+consistence*ψ2, where Same indicates the same dependency ratio between the virtual character’s answer and the corresponding expected answer. Consistence represents the dependency direction consistency coefficient between the virtual character's answer and the corresponding expected answer ψ1 and ψ2 represent the weights corresponding to the same dependency ratio and dependency direction consistency coefficient of the set virtual character's answer and the corresponding expected answer, respectively, and ψ1+ψ2=1; Analyze the semantic understanding degree index Semantic of the virtual character's reply and the corresponding expected reply, Semantic = Vocabulary*χ1+structure*χ2, where χ1 and χ2 represent the weights corresponding to the vocabulary recognition accuracy index and sentence structure similarity coefficient of the virtual character's reply, respectively, and χ1+χ2=1.

4. The method for analyzing and evaluating the effect of intelligent generation of AI virtual characters according to claim 1, characterized in that: The analyzing the context relevance index of the virtual character includes: Extract historical conversation segments based on language test texts in each language scenario from historical conversation data; Through the pre-trained word embedding model, the virtual character's responses and corresponding historical dialogue fragments based on the language test text in each language scenario are converted into sentence vectors; Substitute the reply of the virtual character based on the language test text in each language scene and the sentence vector of the corresponding historical dialogue fragment into the cosine similarity calculation formula to obtain the cosine similarity of the reply of the virtual character based on the language test text in each language scene and the sentence vector of the corresponding historical dialogue fragment, and then obtain the semantic similarity of the virtual character, which is recorded as similarity; Based on the preset theme keyword list, the total number of theme keywords in the virtual character's reply based on the language test text and the corresponding historical dialogue fragments in each language scene are counted respectively, and they are summarized to obtain the total number of theme keywords in the virtual character's reply and the corresponding historical dialogue fragments, which are recorded as reply and theme respectively; Compare the answers of the virtual characters based on the language test text in each language scene with the corresponding historical dialogue segments, obtain the number of identical topic keywords in the answers of the virtual characters based on the language test text in each language scene and the corresponding historical dialogue segments, summarize them, and obtain the number of identical topic keywords in the answers of the virtual characters and the corresponding historical dialogue segments, which is recorded as match; Analyze the theme consistency coefficient of the virtual character Theme, Analyze the context relevance index context of the virtual character, context = similarity*φ1+Theme*φ2, where φ1 and φ2 represent the weights corresponding to the semantic similarity and theme consistency coefficient of the virtual character respectively, and φ1+φ2=1.

5. The method for analyzing and evaluating the effect of intelligent generation of AI virtual characters according to claim 1, characterized in that: The specific calculation formula for analyzing the language comprehension ability index of the virtual character is: com=Semantic*β1+context*β2, wherein β1 and β2 represent the weights corresponding to the semantic comprehension index and context relevance index of the virtual character's reply, respectively, and β1+β2=1.

6. The method for analyzing and evaluating the effect of intelligent generation of AI virtual characters according to claim 1, characterized in that: The step of judging whether the virtual character's response time stability index is qualified includes: Subtract the input time from the response time of each request of the virtual character to get the response time of each request of the virtual character, which is recorded as time g , g = 1, 2, 3, ... G; Substitute the response time of each virtual character's request into the average calculation formula to obtain the average response time of the virtual character, which is recorded as Analyze the virtual character's response time stability index response, G represents the number of input requests; The response time stability index of the virtual character is compared with the preset response time stability index threshold. If the response time stability index of the virtual character is less than the preset response time stability index threshold, the response time stability index of the virtual character is unqualified, and an early warning is issued.

7. The method for analyzing and evaluating the effect of intelligent generation of AI virtual characters according to claim 1, characterized in that: The analyzing the operation stability index of the virtual character includes: Compare the number of days in the set historical cycle with the number of crashes of the virtual character in the set historical cycle to obtain the crash frequency of the virtual character, which is recorded as Collapse; The allowed crash frequency of the virtual character is extracted from the database and recorded as Analyze the operational stability index of the virtual character.

8. The method for analyzing and evaluating the effect of intelligent generation of AI virtual characters according to claim 1, characterized in that: The determining whether the running stability of the virtual character is qualified includes: The running stability index of the virtual character is compared with the preset running stability index threshold. If the running stability index of the virtual character is less than the preset running stability index threshold, the running stability of the virtual character is unqualified, and an early warning is issued.

9. The method for analyzing and evaluating the effect of intelligent generation of AI virtual characters according to claim 1, characterized in that: The specific formula for analyzing the qualified degree index of the intelligent generation effect of the virtual character is: Effect = α1*com+α2*response+α3*Operational, wherein α1, α2 and α3 respectively represent the weights corresponding to the virtual character's language comprehension ability index, response time stability index and operation stability index, and α1+α2+α3=1.

10. The method for analyzing and evaluating the effect of intelligent generation of AI virtual characters according to claim 1, characterized in that: The determining whether the intelligent generation effect of the virtual character is qualified includes: The intelligence generation effect qualification index of the virtual character is compared with the preset intelligence generation effect qualification index threshold. If the intelligence generation effect qualification index of the virtual character is less than the preset intelligence generation effect qualification index threshold, the intelligence generation effect of the virtual character is unqualified, and an early warning is issued.

Images