A method and system for generating virtual human sign language

Abstract

The present specification relates to the field of information technology, and in particular to a method and system for generating virtual human sign language, the method comprising: identifying whether a preset condition for a response is met, and in response to the preset condition for a response being met, obtaining an original text based on the content of the preset condition for the response; determining keywords related to the original text based on a knowledge graph; classifying the original text, and determining a simplified text based on the classification to which the original text belongs; the simplified text being able to reflect the intention of the original text; and determining a target sign language based on at least one of the keywords and the simplified text.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a method and system for generating virtual human sign language. Background Art

[0002] With the development of science and technology, virtual people are becoming more and more common in human life. We can see virtual people in various industries, such as robots used to provide consulting services in the service industry, 3D or flat virtual characters performing in the entertainment industry, etc. However, since the behavior of virtual people is usually pre-set by programs, when virtual people talk to users, they behave stiffly or even weirdly, which makes the user experience bad.

[0003] Therefore, it is hoped that a method for generating virtual human sign language can be provided, so that the virtual human can make appropriate sign language when talking, optimize the communication expression effect of the virtual human, and improve the user experience. Summary of the invention

[0004] One of the embodiments of this specification provides a method for generating virtual human sign language. The method for generating virtual human sign language includes: identifying whether a response preset condition is met, and in response to the response preset condition being met, obtaining an original text based on the content of the response preset condition; determining keywords related to the original text based on a knowledge graph; classifying the original text, and determining a simplified text based on the classification to which the original text belongs; the simplified text can reflect the intention of the original text; and determining a target sign language based on at least one of the keywords and the simplified text.

[0005] One of the embodiments of this specification provides a virtual human sign language generation system. The virtual human sign language generation system includes: a judgment module, used to identify whether a response preset condition is met, and in response to the response preset condition being met, obtain the original text based on the content of the response preset condition; a keyword determination module, used to determine keywords related to the original text based on the knowledge graph; a simplified text determination module, used to classify the original text, and determine the simplified text based on the classification to which the original text belongs; the simplified text can reflect the intention of the original text; a target sign language determination module, used to determine the target sign language based on at least one of the keywords and the simplified text.

[0006] One of the embodiments of this specification provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, any of the above-mentioned methods for generating virtual human sign language is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0008] Figure 1 is a schematic diagram of an application scenario of a virtual human sign language generation system according to some embodiments of this specification;

[0009] Figure 2 is an exemplary flow chart of a method for generating virtual human sign language according to some embodiments of this specification;

[0010] Figure 3 is another exemplary flow chart of the method for generating virtual human sign language according to some embodiments of this specification;

[0011] Figure 4 is an exemplary flow chart of determining keywords based on a knowledge graph according to some embodiments of this specification;

[0012] Figure 5 It is an exemplary module diagram of a virtual human sign language generation system according to some embodiments of this specification. DETAILED DESCRIPTION

[0013] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0014] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0015] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0016] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0017] With the widespread use of virtual humans, people hope that virtual humans can be more vivid and more similar to humans in appearance and behavior. When talking to virtual humans, users hope that they can not only explain through voice, but also make corresponding body movements according to the voice content like humans, so as to enhance the user experience.

[0018] In some embodiments, the behaviors that the virtual person should perform in response to different conversation contents can be pre-set on the virtual person, so that the virtual person can make corresponding sign language while answering the user. However, due to the complexity of the language, the conversation content cannot be exhausted, and a large memory space is required to store the huge conversation content and corresponding behavior data. Therefore, the general virtual person only sets corresponding sign language for certain fixed sentences. However, the method of setting sign language for only a few fixed sentences lacks a systematic solution design, and can only heuristically design a few regular sign language solutions, which makes the sign language movements made by the virtual person when expressing the content of the speech very limited, and cannot meet the user's high degree of freedom in conversation needs.

[0019] In view of this, in some embodiments, the communication expression effect of the virtual person can be optimized by extracting the speech expression content of the virtual person and determining the sign language to be performed by the virtual person based on the speech expression content.

[0020] Figure 1 Schematic diagram of application scenarios of the virtual human sign language generation system according to some embodiments of this specification. Figure 1 As shown, the application scenario of the virtual human sign language generation system may include a server 110, a virtual human 120, a database 130, a network 140, and the like.

[0021] The server 110 refers to a system with computing capabilities. The server 110 may include a processing device to determine the sign language that the virtual person is to make based on the speech expression content of the virtual person. For example, the server 110 may obtain the original text of the virtual person 120, process the obtained original text, and determine the target sign language that the virtual person is to make based on the processing result. The virtual person talks to the user based on the processed target sign language. For more information about the original text and the target sign language, see Figure 2 The related descriptions will not be repeated here.

[0022] The virtual person 120 may be an object that can interact with the user through program control, and its manifestation forms include mobile phones, tablet computers, desktop computers, laptop computers, etc. In some embodiments, it may also be a robot device (such as a service robot that provides consulting services to users), a three-dimensional character or a virtual idol realized by VR / AR / MR technology, etc. In some embodiments, the virtual person 120 may include a sign language execution module, wherein the sign language execution module may be a module for the virtual person 120 to execute sign language, such as a robotic arm, an arm, etc.; the virtual person 120 may execute sign language related to the verbal expression content through the sign language execution module. For example, the virtual person makes a like sign 120-1, a pointing sign 120-2, and an applause sign 120-3 through a robotic arm.

[0023] The database 130 may be a device that provides data support. For example, the server 110 may extract keywords from the original text based on the original text and the data in the database 130. For another example, the database 130 may store a knowledge graph for use by the server 110.

[0024] The network 140 can connect the various components of the system and / or connect the system with external resources. The network 140 allows the various components to communicate with each other and with other parts outside the system. For example, the virtual person 120 transmits the speech expression content to the server 110 through the network 140 for processing. The server 110 obtains the data in the database 130 through the network 140 to process the received speech expression content, and then sends the sign language corresponding to the processed speech expression content to the virtual person 120 through the network 140.

[0025] Figure 2 is an exemplary flow chart of a method for generating virtual human sign language according to some embodiments of this specification. In some embodiments, one or more steps in process 200 may be performed by Figure 1 The server 110 in the embodiment is executed. Figure 2 As shown, process 200 may include the following steps:

[0026] Step 210, identifying whether the preset conditions for answering are satisfied, and in response to the preset conditions for answering being satisfied, obtaining the original text based on the content of the preset conditions for answering. In some embodiments, step 210 may be performed by Figure 5 The judgment module 510 in is executed.

[0027] The preset response condition may be a preset condition for the virtual person to communicate and express. For example, for a virtual person used for reception, the preset response condition may be opening the door. When the door is recognized to be opened, it is determined that the preset response condition is satisfied. For another example, the preset response condition may be whether the user communicates with the virtual person. When the user is recognized to communicate with the virtual person, it is determined that the preset response condition is satisfied. In some embodiments, the preset response condition may be a preset wake-up word or a preset voiceprint.

[0028] The original text may be the content that the virtual person wants to express. In some embodiments, the virtual person may identify the content of the response preset condition, and obtain the original text to be responded to for the response preset condition based on the content of the response preset condition. For example, when the door is opened, the welcome message "Welcome to my home" may be used as the original text. For another example, when it is recognized that the user wants to buy a certain product, the "Place an order to purchase" prompting the user to pay may be used as the original text. In some embodiments, the original text may be constructed in the form of text.

[0029] In some embodiments, the virtual person may also generate original text through a machine learning model. For example, the machine learning model may process the acquired voice of the conversation between the user and the virtual person to obtain the original text for answering the user. For example, when the user greets the virtual person with "Hello", the virtual person correspondingly determines the original text as "Welcome to my home".

[0030] Step 220, based on the knowledge graph, determine the keywords related to the original text. In some embodiments, step 220 can be performed by Figure 5 The keyword determination module 520 in is executed.

[0031] The knowledge graph may include the relationships between entities and the attributes of entities.

[0032] Keywords can be words that describe specific content in the original text. In some embodiments, keywords can be verbs and / or names in the original text. For a piece of original text, a word segmenter can be used to divide the original text into at least one word. For each word, its attributes are searched in the knowledge graph. When the attribute of the word is a verb or a noun, the word is used as a keyword. For example, continuing with the above example, for the original text "Welcome to my home", the knowledge graph shows that "you", "me" and "home" are nouns, so the three words "you", "me" and "home" can be used as keywords.

[0033] In some embodiments, entity information in the original text can also be obtained based on the knowledge graph, and keywords can be determined based on the entity information. For more information about entity information and determining keywords, see Figure 3 The related descriptions will not be repeated here.

[0034] Step 230, classify the original text and determine the simplified text based on the classification to which the original text belongs. In some embodiments, step 230 may be performed by Figure 5 The simplified text determination module 530 in is executed.

[0035] Simplified text can reflect the intention of the original text, and intention can be understood as the intention to achieve a certain goal.

[0036] In some embodiments, the original text can be classified according to the intention, and appropriate words can be determined based on the classification and combined into a simplified text. For example, the original text "you and I are together" expresses the intention of the connection between you and me, so it can be classified into the "contact" class, and its simplified text can include "you contact me".

[0037] In some embodiments, repeated words in the original text may be removed, and the text after removal may be classified, and the simplified text may be determined based on the classification. For example, in "place an order to purchase", "place an order" and "purchase" have similar meanings, so "place an order to purchase" may be simplified to "place an order" or "purchase", and then classified into the "payment" category, and its simplified text may include "payment".

[0038] In some embodiments, simplified text can also be determined based on image schema classification of the original text. For more information on image schema classification and determination of simplified text, see Figure 3 The related descriptions will not be repeated here.

[0039] Step 240, based on at least one of the keywords and the simplified text, determine the target sign language. In some embodiments, step 240 can be performed by Figure 5 The target sign language determination module 540 is executed.

[0040] In some embodiments, the target sign language may include hand movements made by the virtual person when expressing speech (if the virtual person has parts similar to human "hands" or "arms"); in some embodiments, the target sign language may also include other body movements made by the virtual person when expressing speech (if the virtual person does not have parts similar to human "hands" or "arms"). In some embodiments, the virtual person can express the target sign language and the voice and / or text corresponding to the original text at the same time. For example, when talking to the user, the virtual person can point to the speaker and then to himself in turn to express the sign language of "you and me". In some embodiments, the medium for the virtual person to perform the target sign language may not be limited to parts related to the hands, but may also be a display screen or other devices. For example, the hand movements made by the hands in the display screen, etc.

[0041] In some embodiments, when determining the target sign language based on keywords and / or simplified text, the target sign language may also be determined based on conditions such as user preferences or the duration of candidate sign languages. For more information about user preferences, the duration of candidate sign languages, and determining the target sign language, see Figure 3 The related descriptions will not be repeated here.

[0042] In some embodiments, when based on keywords or simplified text, a target sign language can also be generated through a fourth model. For more information about the fourth model and determining the target sign language, see Figure 3 The related descriptions will not be repeated here.

[0043] Some embodiments in this specification determine the keywords that express the image and the simplified text that express the intention based on the original text at the text level, and determine the target sign language that the virtual person should make when expressing based on the keywords or simplified text, thereby optimizing the communication and expression effect of the virtual person and improving the user experience.

[0044] Figure 3 is another exemplary flow chart of a method for generating virtual human sign language according to some embodiments of this specification. In some embodiments, one or more steps in process 300 may be performed by Figure 1 The server 110 in the embodiment is executed. Figure 3 As shown, process 300 may include the following steps:

[0045] Step 310: Acquire entity information in the original text based on the first model.

[0046] Entity information can be words or phrases related to an entity. For example, "you", "I" and "home" in the original text "Welcome to my home" can be extracted as entity information. Optionally, if only one entity information is selected, "home" can be selected as the entity information of the original text "Welcome to my home".

[0047] The first model can be used to obtain entity information. In some embodiments, the input of the first model is the original text, and the output is the entity information corresponding to the original text. Continuing with the above example, if "Welcome to my home" is input into the first model, the first model can output "you", "me" and "home", or only output "home".

[0048] In some embodiments, the first model type may be a model obtained by training a keyword extraction algorithm, including but not limited to named entity recognition (NER) or TF-IDF algorithm, etc., which is not limited in this specification.

[0049] In some embodiments, the first model can be obtained by training multiple labeled first training samples. For example, multiple labeled first training samples can be input into the initial first model, a loss function can be constructed by using the labels and the results of the initial first model, and the parameters of the initial first model can be iteratively updated based on the loss function. When the loss function of the initial first model meets the preset conditions, the model training is completed and the trained first model is obtained. Among them, the preset conditions can be that the loss function converges, the number of iterations reaches a threshold, etc. It should be noted that in some other embodiments, the first model can also be trained by an unsupervised learning algorithm.

[0050] In some embodiments, the first training sample may include multiple conversations; the first training sample may be extracted from daily communication, and its label may be a keyword in each conversation. In some embodiments, the label may be obtained at least by manual annotation.

[0051] Step 320, determining keywords based on the entity information.

[0052] For more information about keywords, see Figure 2 The related descriptions will not be repeated here.

[0053] In some other embodiments, information related to an entity may be used as a keyword. For example, for the keyword "order", the entity "money" related thereto may be used as its keyword.

[0054] In some embodiments, it is also possible to determine whether the entity information is a figurative description based on the knowledge graph, and determine keywords based on the determination result. For more information about determining keywords based on the knowledge graph, see Figure 4 The related descriptions will not be repeated here.

[0055] Some embodiments of the present specification can improve the efficiency and accuracy of keyword extraction by using the first model to extract entity information from the original text and determining keywords based on the entity information.

[0056] Step 330, obtaining image schema classification of the original text based on the second model.

[0057] Image schema classification can be a variety of classification results obtained by classifying text. In some embodiments, various types of image schema classification can be determined by cognitive linguistics, including but not limited to starting point-path-target, center-edge, part-whole, container, connection, power, linear and / or balance diagrams, etc.

[0058] In some embodiments, the imagery of the original text may be identified, and then the original text may be classified based on the imagery of the original text. For example, for the original text "you and I are together", the imagery to be expressed is the connection between "you" and "me", so the text may be classified into the "connection" category. Another example is "I am at home", which may be classified into the "container" category.

[0059] The second model can be used to obtain image schema classification. In some embodiments, the input of the second model is the original text, and the output is the image schema classification of the original text. For example, the original text "I am with you" is input into the second model to obtain the probability values ​​of all classification results corresponding to the original text, such as the probability of "container" classification is 47%, the probability of "connection" classification is 80%, the probability of "strength" classification is 3%, etc.

[0060] In some embodiments, the second model can output a classification with the highest probability. For example, for the above original text "You and I are together", the second model can use the "connection" classification with the highest probability as the output of the second model. In some embodiments, the second model can set a preset output threshold and output the probability of the intentional schema classification greater than the preset threshold. For example, the preset threshold can be 40%, and for the above original text "You and I are together", the "container" and "connection" classifications can be output. For another example, the preset output threshold can be set to 90%. At this time, there is no output that meets the conditions, and the image schema classification can be not output.

[0061] In some embodiments, the type of the second model may be a pre-trained language model (Language model pre-training) based on NLP (natural language processing) to implement classification tasks such as a BERT model or an XLNET model.

[0062] In some embodiments, the second model can be obtained through supervised training or unsupervised training using multiple second training samples. For example, multiple labeled second training samples can be input into an initial second model (such as a BERT pre-trained language model), a loss function can be constructed using the labels and the results of the initial second model, and the parameters of the initial second model can be iteratively updated based on the loss function. When the loss function of the initial second model meets the preset conditions, the model training is completed and a trained second model is obtained. Among them, the preset conditions may be that the loss function converges, the number of iterations reaches a threshold, etc. In some embodiments, similar to the first training sample, the second training sample may include multiple conversation contents. The second training sample can be extracted through daily communication, and the label can be one or more image schema classifications corresponding to each conversation determined by manual classification.

[0063] Step 340, determine simplified text based at least on the image schema classification.

[0064] In some embodiments, the image schema classification can be used to partially replace the content in the original text that is similar to the meaning expressed by the classification or generate new simplified content. For example, for the original text "You and I are together", its image schema classification is "connection", and "You and I are together" can be simplified to "I connect you" based on the original text and the image schema classification.

[0065] In some embodiments, the image schema classification and the original text may also be input into a third model, and the simplified text may be output through the third model.

[0066] For example, the third model can recompile the sentence based on the original text "You and I are together" and the image schema classification "connection" corresponding to the original text to obtain a simplified text "You and I" with simpler expression.

[0067] In some embodiments, the type of the third model may be a model including an Encoder-decoder structure, which may include but is not limited to a Seq2Seq model and a BERT model.

[0068] In some embodiments, the third model can be obtained through supervised training or unsupervised training using multiple third training samples.

[0069] Some embodiments in this specification rewrite the original text to obtain a simplified text with a simpler structure, so that the content of the text is more focused, the structure of the original text is simplified, and irrelevant words are reduced, which is more conducive to the subsequent conversion of the text content into the target sign language.

[0070] Step 350: Generate target sign language based on at least one of the keywords and the simplified text through a fourth model.

[0071] The fourth model can be used to determine the target sign language. In some embodiments, due to the setting of preset thresholds or strategies, steps 320 and 340 may not generate keywords or simplified texts. The input of the fourth model is keywords and / or simplified texts, and the output can be the target sign language determined according to the keywords and / or simplified texts. For example, for the original text "I am with you", its keywords can be "I", "you" and "together", and its simplified text can be "I connect you". The keywords or simplified text of the original text are input into the fourth model. The fourth model outputs the first candidate target sign language "pointing fingers to oneself to indicate 'I' and then pointing to the speaker to indicate 'you', and then using a hug to indicate 'together'" according to the keywords "I connect you", and outputs the second candidate target sign language "pointing fingers to oneself to indicate 'I' and then pointing to the speaker to indicate 'you', and then sliding back and forth between the speaker and oneself to indicate 'connection'" according to the simplified text "I connect you".

[0072] In some embodiments, the fourth model may be a model for converting text into sign language, such as a GODIVA model. In some embodiments, the fourth model may be trained using a fourth training sample. For example, the sign language and the corresponding text may be obtained through media information or sign language courses with sign language, and the model may be trained using the fourth training sample.

[0073] In some embodiments, the fourth model may not have suitable sign language output for the input keywords and / or simplified text, or the generated target sign language may not meet the user's preferences. In this case, the keywords and / or simplified text can be redetermined based on the original text. In the process of redetermining the keywords and / or simplified text, a more relaxed generation strategy can be followed (such as lowering the aforementioned preset threshold) so that the fourth model can output the target sign language based on the redetermined keywords and / or simplified text.

[0074] Some embodiments in this specification determine the target sign language through the fourth model, thereby improving the efficiency and accuracy of generating the target sign language.

[0075] In some embodiments, the manner of determining the target sign language may include: determining at least one of a keyword and a simplified text based on a user preference.

[0076] The user preference may be the user's preference for the sign language made by the virtual person. For example, an older user may prefer a sign language with a longer duration.

[0077] In some embodiments, the user preference may be determined based on the user's historical behavior. For example, the user preference may be determined by counting the basis for forming the target sign language corresponding to the user's historical selection (e.g., whether the target sign language is generated based on keywords or simplified text).

[0078] In some embodiments, the virtual person may determine whether to use keywords or simplified text to assist speech expression when communicating with the user based on the user's preference for the target sign language formed by keywords and the target sign language formed by simplified text.

[0079] In some embodiments, the server 110 may obtain user preferences based on the recommendation information.

[0080] The recommendation information may be a system for recommending target sign language for users. In some embodiments, the recommendation information may automatically recommend the target sign language for the user when the virtual person expresses verbally with the user, and update the recommendation information based on the user's feedback on the recommended target sign language. For example, for the original text "place an order to buy", the keywords "money" and "downward" may be obtained. The recommendation information may randomly display the target sign language formed based on "money" or "downward" to user A, and user A may feedback "like" or "dislike" to form a preference for user A. For another example, for user group B, it is possible that a part of user group B tends to form a target sign language based on "money", and another part tends to form a target sign language based on "downward". The recommendation information may set a threshold based on the user group B's liking rate for these two target sign languages, such as 50%. If the user group B's liking rate for the target sign language formed based on "money" exceeds 50%, "money" is used as the target sign language made by the virtual person to the user group B when expressing "place an order", and "money" is used as the user group B's preference for this type of text (such as placing an order, purchasing, paying, etc.).

[0081] In some embodiments, there may be multiple keywords and simplified texts determined based on the original text, and the recommendation information may determine the keywords and simplified texts generated subsequently based on the user's feedback on the target sign language formed by the multiple simplified texts or keywords. For example, for "place an order", the keyword may be "money" or "downward", and the recommendation information may randomly select keywords (such as money) to generate the target sign language. When the user communicates with the virtual person, if the user feedback does not like the target sign language, the target sign language of "place an order" may be formed for the user based on "downward" in the future.

[0082] The target sign language is determined based on at least one of the determined keywords or the simplified text.

[0083] In some embodiments, users may have different preferences for target sign language generated based on keywords or simplified texts, so the virtual person may also determine how to generate the target sign language for the user based on the user's preference for the target sign language generated based on keywords or simplified texts. For example, user C prefers target sign language generated based on keywords, so the virtual person may use the target sign language generated based on keywords when communicating with user C.

[0084] Some embodiments in this specification improve the user experience by determining the target sign language based on the user's preference.

[0085] In some embodiments, the method of determining the target sign language may further include: generating candidate sign languages ​​based on keywords and simplified texts respectively.

[0086] The candidate sign language may be a sign language that is expected to be selected as the target sign language. In some embodiments, the candidate sign language may include a first candidate target sign language generated based on the keyword and a second candidate target sign language generated based on the simplified text.

[0087] In some embodiments, the first candidate target sign language and the second candidate target sign language may be generated by a fourth model.

[0088] The target sign language is determined based on the duration of the candidate sign languages.

[0089] The duration of the sign language may refer to the time consumed by the virtual person to make the sign language. The duration of the candidate sign language may refer to the time consumed by the virtual person to make the candidate sign language.

[0090] In some embodiments, in order to ensure the simplicity of the action, a candidate sign language with a shorter duration can be selected as the target sign language. In some embodiments, a candidate sign language with a suitable duration can also be selected as the target sign language based on the user's preference, such as selecting a longer duration sign language for a hearing-impaired user.

[0091] Figure 4 is an exemplary flow chart of determining entity information and keywords based on a knowledge graph according to some embodiments of this specification. In some embodiments, one or more steps in process 400 may be performed by Figure 1 The server 110 in the embodiment is executed. Figure 4 As shown, process 400 may include the following steps:

[0092] Step 410, judging whether the entity information is a figurative description based on the knowledge graph.

[0093] The knowledge graph includes nodes and edges between nodes. In some embodiments, nodes correspond to entities, and edges between nodes reflect the relationship between entities. Therefore, the relationship between entities, attributes of entities, and other information can be reflected through the knowledge graph. For example, the knowledge graph may include entities "order" and "money". The edge formed by "order" and "money" can be used to reflect the relationship between "order" and "money", including but not limited to the relationship of "money is a tool for ordering". In some embodiments, nodes in the knowledge graph may also include attribute values. The edges formed between entities and attribute values ​​may be attributes corresponding to attribute values. The attributes of entities can be reflected through entities, attributes, and attribute values. For example, for the entity "home" in the knowledge graph, since "home" is an image expression, "home" has the attribute of "image expression", so the attribute value of "home" for the attribute "whether it is image" is "yes"; similarly, for the entity "order", since "order" is not an image expression, "order" does not have the attribute of "image expression", so the attribute value of "order" for the attribute "whether it is image" is "no". In some embodiments, the knowledge graph can be constructed based on expert experience.

[0094] In some embodiments, the nodes and edges in the knowledge graph that reflect the relationship between entities can be called relationship triples, which can be recorded as (entity, relationship, entity). For example, the relationship triple in the knowledge graph that reflects "the tool for placing an order is money" can include the nodes "place an order", "money" and the relationship "tool", which can be recorded as (place an order, tool, money). In some embodiments, the nodes and edges in the knowledge graph that reflect the attributes of an entity can be called attribute triples, which can be recorded as (entity, attribute, attribute value). For example, the nodes "home", "is" and the edge "whether it is an image" that reflect "home as an image expression" in the knowledge graph can be recorded as (home, whether it is an image, yes).

[0095] Figurative description can be a figurative description, and the opposite of figurative description can be an abstract description. For example, for "home", it is figurative, so "home" can be considered as a figurative description. For another example, for "placing an order", it is abstract, so "placing an order" cannot be considered as a figurative description.

[0096] In some embodiments, whether the entity information is a figurative description can be determined through graph query.

[0097] Step 420: If the entity information is a figurative description, determine that the entity information is a keyword. For example, for the entity "home", the attribute triple (home, whether it is figurative, yes) can be determined through a graph query to determine whether it is figurative. In response to the attribute value being "yes", "home" can be extracted as a keyword.

[0098] Step 430: If the entity information is a non-figurative description, other figurative description entities related to the entity information are determined as keywords based on the knowledge graph. For example, for the entity "place an order", the attribute triple (place an order, whether it is figurative, no) indicating whether it is figurative can be queried in the knowledge graph. Since the attribute value is "no", "place an order" cannot be directly used as a keyword. It is necessary to determine other figurative description entities related to "place an order" as its keywords based on the knowledge graph.

[0099] Other image description entities may be other image entity information related to the entity information, for example, by querying the attribute value of whether the entity represented by other nodes connected to the node is image.

[0100] In some embodiments, other figurative description entities related to entity information can be determined through the relationship triples in the knowledge graph. For example, for the entity "place an order", its relationship triples can be queried through graph association as (place an order, tool, money) and (place an order, tendency, place), so "money" and "place" can be extracted, and the attribute triples of "money" and "place" can be queried. The attribute triples of "money" and "place" are (money, whether it is figurative, yes) and (place, whether it is figurative, yes), so "money" and "place" can be used as other figurative description entities related to "place an order".

[0101] In some embodiments, it may happen that multiple queries fail to find other figurative entities related to the entity information. Therefore, in order to avoid infinite searches or searches that take too long, a search upper limit can be set so that the virtual person can respond within a certain time frame. The search upper limit can be limited by the number of traversed edges, such as limiting the query to no more than 4 edges.

[0102] In some embodiments, it may happen that the search limit is reached but no other image description entities related to the entity information are found, i.e., the search fails. In some embodiments, when the search fails, other entity information related to the entity information can be used as new entity information, and other image description entities related to the new entity information can be searched based on the new entity information, or no output can be selected.

[0103] Some embodiments of the present description can improve the accuracy and search efficiency of querying other figurative description entities related to non-figural description entity information by searching for other figurative description entities related to entity information through a knowledge graph.

[0104] Some embodiments of this specification use the knowledge graph to determine whether entity information is a figurative description, and use the figurative description entity as a keyword, and use other figurative description entities of non-figurative description entities as keywords, so that target sign language can be generated based on the figurative description entity. Since the figurative description entity is easier to express in sign language, the vividness of the subsequently generated target sign language is improved, making the generated target sign language easier for users to accept. Some embodiments of this specification use the knowledge graph to search for the attributes of the entity and other figurative description entities, so that complex calculations are not required, thereby improving query efficiency.

[0105] Figure 5 is an exemplary module diagram of a virtual human sign language generation system according to some embodiments of this specification. In some embodiments, one or more modules in the system 500 may be arranged in Figure 1 In the server 110. Figure 5 As shown, the system 500 may include the following modules:

[0106] The judgment module 510 is used to identify whether the preset condition of the response is met, and in response to the preset condition of the response being met, obtain the original text based on the content of the preset condition of the response. For more information about the preset condition of the response, the original text and obtaining the original text, see Figure 2 The related descriptions will not be repeated here.

[0107] The keyword determination module 520 is used to determine keywords related to the original text based on the knowledge graph. For more information about keywords and keyword determination, see Figure 2 The related descriptions will not be repeated here.

[0108] The simplified text determination module 530 is used to classify the original text and determine the simplified text based on the classification to which the original text belongs; the simplified text can reflect the intention of the original text. For more information about simplified text and determination of simplified text, see Figure 2 The related descriptions will not be repeated here.

[0109] The target sign language determination module 540 is used to determine the target sign language based on at least one of the keyword and the simplified text. For more information about the target sign language and determining the target sign language, see Figure 2 The related descriptions will not be repeated here.

[0110] The embodiment of the present specification also provides a computer-readable storage medium. The storage medium stores computer instructions, and when a computer reads the computer instructions in the storage medium, the computer implements the aforementioned virtual human sign language generation method.

[0111] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0112] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0113] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0114] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0115] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicates that the numbers allow for

[0116] ±20% change. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the characteristics required by individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0117] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification are hereby incorporated by reference in their entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this specification (currently or later attached to this specification) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0118] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A method for generating virtual human sign language, comprising: Identifying whether a preset response condition is satisfied, and in response to the preset response condition being satisfied, acquiring an original text to be responded to the preset response condition based on the content of the preset response condition; Acquire entity information in the original text based on the first model; Based on the attribute triples in the knowledge graph, whether the entity information is a figurative description is determined; the nodes in the knowledge graph correspond to entities, and the edges between the nodes reflect the relationship between the entities; If the attribute value is yes, determining that the entity information is a keyword; If the attribute value is no, searching for other image description entities related to the entity information as the keyword based on the relationship triples in the knowledge graph; If the search fails, other entity information related to the entity information is used as new entity information, and other image description entities related to the new entity information are searched based on the new entity information; Based on the second model, obtaining the image schema classification of the original text; Determine a simplified text based on the original text and the image schema classification to which it belongs; Based on at least one of the keyword and the simplified text, a target sign language is determined, where the target sign language at least includes hand movements made by a virtual person when performing speech expression.

2. The method according to claim 1, wherein determining the simplified text based on the original text and the image schema classification to which it belongs comprises: The image schema classification and the original text are input into a third model to obtain the simplified text.

3. The method according to claim 1, wherein determining the target sign language based on at least one of the keyword and the simplified text comprises: determining at least one of the keyword and the simplified text based on a user preference; The target sign language is determined based on at least one of the determined keywords or the simplified text.

4. The method of claim 1, wherein determining the target sign language based on at least one of the keyword and the simplified text comprises: Generate candidate sign languages ​​based on the keywords and the simplified text respectively; The target sign language is determined based on the duration of the candidate sign language.

5. The method of claim 1, wherein determining the target sign language based on at least one of the keyword and the simplified text comprises: The target sign language is generated based on at least one of the keyword and the simplified text through a fourth model.

6. A virtual human sign language generation system, comprising: A judgment module, used to identify whether a preset response condition is satisfied, and in response to the preset response condition being satisfied, obtain an original text to be responded to the preset response condition based on the content of the preset response condition; A keyword determination module is used to obtain entity information in the original text based on the first model; determine whether the entity information is an image description based on the attribute triple of whether it is image in the knowledge graph; the nodes in the knowledge graph correspond to entities, and the edges between nodes reflect the relationship between entities; If the attribute value is yes, determining that the entity information is a keyword; If the attribute value is no, then searching for other image description entities related to the entity information as the keyword based on the relationship triples in the knowledge graph; if the search fails, taking the other entity information related to the entity information as new entity information, and searching for other image description entities related to the new entity information based on the new entity information; A simplified text determination module is used to obtain the image schema classification of the original text based on the second model; and determine the simplified text based on the original text and the image schema classification to which it belongs; The target sign language determination module is used to determine the target sign language based on at least one of the keywords and the simplified text, and the target sign language at least includes the hand movements made by the virtual person when expressing speech.

7. A computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the virtual human sign language generation method as described in any one of claims 1 to 5.

Images