Virtual image synthesis method, device, computing device and storage medium
Through the combination of multimodal input data and intelligent dialogue module, basic virtual image and interactive features are generated, which solves the problem of real-time online virtual image synthesis in the existing technology, and realizes personalized virtual image output and interaction in offline scenarios Sexual services.
Patent Information
- Application Number
- CN202110875935.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-30
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-07-30
AI Technical Summary
The existing virtual image synthesis technology requires business personnel to be online in real time to achieve dynamic synthesis, and lacks interactive and personalized services in offline scenarios.
By obtaining multimodal input data of business personnel, including personal information, human body images and historical service audio and video, the network generates basic virtual images using pre-built multimodal virtual images, and combining intelligent dialogue modules and interactive feature output modules, Automatically generate virtual avatar video frames, without the need for business personnel to be online in real time.
It realizes personalized virtual image output in offline scenarios and enhances the interaction between virtual image and customers, improving service flexibility and customer experience.
Smart Images

Figure CN113590078B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a virtual image synthesis method, device, computing equipment and storage medium. Background Art
[0002] With the rapid development of the information age, most companies or organizations generally use virtual images to provide services in order to improve service quality and customer experience. However, current virtual images generally display a set fixed image. In instant messaging services, there is a lack of interactivity between the virtual image and the user.
[0003] Currently, in order to solve the above problems, the agent's virtual video is output in real time and dynamically by combining the business personnel's biometric features such as expressions and movements in the actual online service video. However, this solution still relies on the real-time online presence of the business personnel, which makes the current synthesis of virtual images limited. Summary of the invention
[0004] The embodiments of the present invention provide a method, apparatus, computing device and storage medium for synthesizing a virtual image, so as to solve the limitation problem that the dynamic synthesis of the current virtual image requires the business personnel to be online in real time.
[0005] A virtual image synthesis method, comprising:
[0006] Acquire multimodal input data of business personnel; wherein the multimodal input data includes personal information, human body images, and historical service audio and video;
[0007] Inputting the personal information, the human image of the business personnel, and the historical service audio and video into a pre-built multimodal virtual image generation network for multimodal recognition to output a basic virtual image of the business personnel;
[0008] If the current conversation scene is an offline scene, based on the basic virtual image, the intelligent conversation module is called to perform initial service and receive conversation data in real time;
[0009] identifying the current dialogue stage based on the dialogue data by the multimodal avatar generation network, so as to output interaction features corresponding to the dialogue stage;
[0010] Based on the interaction features and the basic virtual image, a virtual image video frame at the current moment is synthesized.
[0011] A virtual image synthesis device, comprising:
[0012] A multimodal input acquisition module, used to acquire multimodal input data of business personnel; wherein the multimodal input data includes personal information, human body images, and historical service audio and video;
[0013] A basic virtual image synthesis module is used to input the personal information, the human image of the business personnel, and the historical service audio and video into a pre-built multimodal virtual image generation network for multimodal recognition to output the basic virtual image of the business personnel;
[0014] An intelligent dialogue module, for invoking the intelligent dialogue module to perform initial services based on the basic virtual image and receiving dialogue data in real time if the current dialogue scene is an offline scene;
[0015] An interaction feature output module, configured to identify the current dialogue stage based on the dialogue data through the multimodal avatar generation network, so as to output interaction features corresponding to the dialogue stage;
[0016] The dynamic virtual image synthesis module is used to synthesize the virtual image video frame at the current moment based on the interactive features and the basic virtual image.
[0017] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned virtual image synthesis method when executing the computer program.
[0018] A computer storage medium stores a computer program, which implements the steps of the above-mentioned virtual image synthesis method when executed by a processor.
[0019] In the above-mentioned virtual image synthesis method, device, computing device and storage medium, by extracting multimodal inputs of different business personnel, multimodal recognition is performed through a pre-constructed multimodal virtual image generation network, and the basic virtual image of the business personnel is output to achieve personalized virtual image output; then, when the current dialogue scene is an offline scene, the intelligent dialogue module is called based on the basic virtual image to perform initial service, and dialogue data is received in real time, so that the dialogue stage at the current moment is identified through the multimodal virtual image generation network according to the actually received dialogue data, so as to automatically generate the interaction features of the basic virtual image at the dialogue stage, so that the business personnel can serve the customers through the virtual image without being online in real time; finally, based on the interaction features and the basic virtual image, the virtual image video frame at the current moment is synthesized to enhance the interactivity between the virtual image and the customer. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0021] Figure 1 is a schematic diagram of an application environment of a virtual image synthesis method in one embodiment of the present invention;
[0022] Figure 2 is a flow chart of a method for synthesizing a virtual image in one embodiment of the present invention;
[0023] Figure 3 yes Figure 2 A specific flow chart of step S202;
[0024] Figure 4 yes Figure 2 A specific flow chart of step S204;
[0025] Figure 5 yes Figure 4 A specific flow chart of step S402;
[0026] Figure 6 is a flow chart of a method for synthesizing a virtual image in one embodiment of the present invention;
[0027] Figure 7 is a flow chart of a method for synthesizing a virtual image in one embodiment of the present invention;
[0028] Figure 8 is a schematic diagram of a virtual image synthesis device in one embodiment of the present invention;
[0029] Fig. 9 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION
[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0031] The virtual image synthesis method can be applied in Figure 1In an application environment, a computer device communicates with a server via a network. The computer device may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented as a standalone server.
[0032] In one embodiment, if Figure 2 As shown, a virtual image synthesis method is provided, and the method is applied in Figure 1 The server in the example is used as an example to illustrate the process, including the following steps:
[0033] S201: Acquire multimodal input data of a business person; wherein the multimodal input data includes personal information, human body images, and historical service audio and video.
[0034] Among them, personal information includes but is not limited to the age, gender, and rank of the business personnel. The human body image refers to the full-body image of the business personnel, and the historical service audio and video includes the video of the business personnel's historical service and the corresponding audio. It can be understood that the expression of the dynamic virtual image of the same business personnel needs to be described or expressed through multimodal features to ensure the accuracy of the virtual image output.
[0035] S202: Inputting personal information, body images of business personnel, and historical service audio and video into a pre-built multimodal virtual image generation network for multimodal recognition to output a basic virtual image of the business personnel.
[0036] Among them, the multimodal virtual image generation network is a multimodal knowledge graph constructed by unsupervised learning of multimodal data corresponding to multiple business personnel. The multimodal knowledge graph includes virtual human entity nodes, service theme entity nodes and link relationships between entities, wherein the virtual human entity link has personal information features, action features and expression features. The service theme entity link has action features, expression features, reply content, intonation features and emotional features at different times during the service process, and the intonation feature link has lip shape features. It can be understood that in the multimodal virtual image generation network, intonation, emotion, etc. can be expressed according to the corresponding classification labels, for example, intonation includes rising tone, falling tone, flat tone or melody, etc., and emotional features include pleasure, anger, sadness, etc. During the service process, the emotional feature will change with the customer's emotions.
[0037] Specifically, by inputting personal information, body images of business personnel, and historical service audio and video into a pre-built generative network of a multimodal virtual image generation network, a basic virtual image of the business personnel can be generated in the multimodal virtual image generation network according to the multimodal input data and the link relationship between the entity nodes in the graph.
[0038] S203: If the current dialogue scene is an offline scene, based on the basic virtual image, the intelligent dialogue module is called to perform initial service and receive dialogue data in real time.
[0039] Specifically, since the currently generated virtual image is a static image, when providing services, the interactive features of different service stages at different times can be generated based on the actual communication link service subject entity. During the initial service, since the current dialogue scene is an offline scene, it is necessary to call the intelligent dialogue module to provide initial services to customers in order to receive dialogue data in real time, so as to identify the service stage (i.e., dialogue stage) based on the dialogue data of the actual communication, thereby generating interactive features of different dialogue stages at different times, and enhancing the interactivity between the virtual image and the customer. At the same time, since the virtual image and the interactive features during the service process can be generated offline, it is not necessary for business personnel to be online in real time to provide services to customers through the virtual image.
[0040] Furthermore, if the current dialogue scene is an online scene, the service is initially provided based on the basic virtual image, and the voice data of the business personnel is received in real time; then, the interaction features of the business personnel at the current moment are extracted through the existing open source model, such as facial expression features, body movement features, reply content features, emotional features, intonation features and lip shape features, and based on the interaction features and the basic virtual image, the virtual image video frame at the current moment is synthesized, that is, when the current dialogue scene is an online scene, the virtual image is synthesized through the real feedback of the business personnel.
[0041] S204: Identify the current dialogue stage based on the dialogue data through the multimodal avatar generation network to output the interaction features corresponding to the dialogue stage.
[0042] The interactive features include, but are not limited to, facial expressions, body movements, response content, emotional features, intonation, and lip shape. The dialogue phase includes, but is not limited to, warm-up greetings, talking about the company, talking about the industry, talking about yourself, needs analysis, solution communication, or objection handling.
[0043] Specifically, the conversation data can be filtered to filter out punctuation marks and special characters to obtain a valid character string; the valid character string is then input into a pre-built multi-mode matching tree for multi-mode matching to determine the current conversation stage; finally, the multi-modal virtual image generation network is used to generate interaction features at the current conversation stage based on the conversation stage.
[0044] S205: Based on the interactive features and the basic virtual image, synthesize the virtual image video frame at the current moment.
[0045] Specifically, the interaction features at the current moment are mapped to the basic virtual image to synthesize the virtual image video frame at the current moment, and then as the service process continues to advance, the virtual image video frame at the next moment is dynamically synthesized until the service ends and the virtual image service video stream is output.
[0046] In this embodiment, by extracting multimodal inputs of different business personnel, multimodal recognition is performed through a pre-constructed multimodal virtual image generation network, and the basic virtual image of the business personnel is output to achieve personalized virtual image output; then, when the current conversation scene is an offline scene, the intelligent conversation module is called based on the basic virtual image to perform initial service, and the conversation data is received in real time, so that the multimodal virtual image generation network can identify the current conversation stage according to the actually received conversation data, thereby automatically generating the interaction features of the basic virtual image in the conversation stage, and the business personnel can provide services to customers through the virtual image without the need for real-time online service; finally, based on the interaction features and the basic virtual image, the virtual image video frame at the current moment is synthesized to enhance the interactivity between the virtual image and the customer.
[0047] In one embodiment, if Figure 3 As shown, in step S202, personal information, human images of business personnel, and historical service audio and video are input into a pre-built multimodal virtual image generation network for multimodal recognition to output a basic virtual image of the business personnel, which specifically includes the following steps:
[0048] S301: Obtaining the global encoding of the business personnel's personal information in the multimodal virtual image generation network.
[0049] Specifically, when constructing a multimodal avatar generation network, a global encoding identifier (embedding) is corresponding to different personal information to describe different personal information in the multimodal avatar generation network. The encoding identifiers for different personal information can be encoded according to preset rules, such as female-000, male-001.
[0050] S302: Using a first feature point positioning model to identify a first modal feature corresponding to the human body image.
[0051] Among them, the first feature point positioning model can be an MTCNN feature point positioning model, which is used to locate facial key points and limb key points in a human body image to obtain a first modal feature, which includes a first facial feature and a first limb feature.
[0052] Specifically, MTCNN consists of three cascade networks, namely P-net, R-Net and O-net. It is a step-by-step search and positioning process. First, the input human body image is resized to obtain images of different sizes to form an image pyramid; then the image pyramid is passed through a shallow P-net network to quickly screen out multiple potential candidate frames for the face, and the redundant candidate frames are removed by the NMS non-maximum suppression method, and the remaining candidate frames are passed through R-Net to obtain the most likely face frame position; finally, the positioning coordinates of the mouth are output through the O-net neural network, so that according to the positioning coordinates of the mouth, the mouth shape image corresponding to the frame image is returned and scaled to a fixed size (n*n) matrix. It can be understood that the network structure and processing process for P-net, R-Net and O-net are consistent with the traditional MTCNN network, and will not be repeated here.
[0053] S303: Using the second feature point positioning model to identify the second modal features corresponding to the historical service video.
[0054] Among them, the second feature point positioning model can adopt the Lanmarks feature point positioning model, which is used to detect the key points of the face and the key points of the body in the historical service video to obtain the first modal feature, which includes the first face feature and the first body feature of the business personnel communicating with the customer at different times. At this time, the first face feature and the first body feature have time sequence, that is, they express the expressions, movements, emotions and lip shapes of the business personnel at different times during the service process.
[0055] It can be understood that by adopting different positioning models to locate the facial features and body features of corresponding business personnel from different modalities, the expressive ability of the virtual image can be improved, thereby improving the accuracy of virtual image synthesis.
[0056] S304: Using the intonation recognition model, identify the third modal features corresponding to the historical service audio.
[0057] Specifically, the voiceprint features of the business personnel in the historical service audio are extracted through the intonation recognition model, and the intonation classification is performed on the voiceprint signs to obtain the intonation classification label, which is used as the third modal feature.
[0058] S305: Predicting based on the global coding, the first modality feature, the second modality feature, and the third modality feature through a multimodal virtual image generation network, and outputting a basic virtual image of the business personnel.
[0059] Specifically, by inputting the global code, the first modal feature, the second modal feature and the third modal feature into a multimodal virtual image generation network, a basic virtual image of the business personnel can be generated in the multimodal virtual image generation network according to the global code, the first modal feature, the second modal feature and the third modal feature. The basic virtual image is described by facial features, body features and intonation features.
[0060] In one embodiment, if Figure 4 As shown, in step S204, the multimodal avatar generation network is used to identify the current dialogue stage based on the dialogue data to output the interaction features corresponding to the dialogue stage, which specifically includes the following steps:
[0061] S401: Filter the conversation data to obtain a valid character string.
[0062] Specifically, a regular expression pair may be used. A valid character string refers to a character string that does not contain special characters (such as spaces) and / or punctuation marks, obtained after matching the original character string using a regular expression matching algorithm. In this embodiment, a regular expression (such as ^[\u4e00-\u9fa5_a-zA-Z0-9]+$) may be used to match the conversation data to obtain a valid character string that does not contain special characters and / or punctuation marks, filter out interference factors such as special characters and / or punctuation marks, and increase the efficiency of subsequent multi-mode matching.
[0063] S402: Input the valid character string into a pre-built multi-pattern matching tree to perform multi-pattern matching to determine the current dialogue stage.
[0064] The success pointer refers to the pointer of the successful path constructed by pre-using the currently open source named entity recognition model to identify the service entity words in the historical service video. The failure pointer refers to the pointer of the failed path constructed when the pointer of the success pointer does not match the character to be matched. The root node is a node that does not contain any characters.
[0065] The multi-pattern matching tree refers to the Trie tree, word search tree or prefix tree, which is a multi-branch tree structure used for fast retrieval, such as the dictionary tree of English letters is a 26-branch tree, and the dictionary tree of numbers is a 10-branch tree. Specifically, a multi-pattern matching algorithm can be used to match the conversation subject words in the valid character string based on the multi-pattern matching tree. Among them, the multi-pattern matching algorithm includes but is not limited to the AC (Aho-Corasick, multi-pattern matching algorithm) algorithm. The AC (Aho-Corasick) algorithm is a dictionary matching algorithm, which is used to search for strings in the dictionary in the input text. The algorithm uses a finite automaton to cleverly convert character comparison into state transfer. The time complexity is linear and the algorithm is efficient.
[0066] S403: Outputting the interaction features corresponding to the current dialogue stage based on the dialogue stage through the multimodal avatar generation network.
[0067] Specifically, by inputting the dialogue stage into the multimodal avatar generation network and matching the dialogue stage corresponding to the service subject entity, the interaction features of the service subject entity link are obtained, thereby generating the interaction features at the current dialogue stage.
[0068] In one embodiment, if Figure 5 As shown, in step S402, a valid character string is input into a pre-built multi-pattern matching tree for multi-pattern matching to determine the current dialogue stage, which specifically includes the following steps:
[0069] S501: traverse the valid character string, and perform keyword matching on the characters to be matched in the valid character string in sequence based on the multi-pattern matching tree.
[0070] S502: If there is a first matching character that is successfully matched in the multi-mode matching tree at the previous moment, the first matching character is used as the matching start node at the current moment.
[0071] S503: According to the path of the success pointer, point the success pointer to the specific character corresponding to the next node; wherein the next node is one or more nodes adjacent to the matching start node in the path of the success pointer.
[0072] Among them, the first matching character is the character to be matched that was successfully matched in the multi-mode matching tree at the previous moment, and serves as the starting node of the match at the current moment, that is, the matching starting node at the current moment. The specific character is the next one or more nodes adjacent to the matching starting node pointed to by the success pointer in the multi-mode matching tree with the first matching character as the matching starting node at the current moment. The path of the success pointer refers to the path pointed to by the pointer of the success pointer. Specifically, the server first reads the characters to be matched in the valid string according to the order of the sequence labels of the characters to be matched, and then searches in the multi-mode matching tree; if there are characters to be matched in the multi-mode matching tree and the match is successful, the character to be matched is used as the first matching character, and the first matching character is used as the matching starting node at the current moment, and the success pointer is pointed to the specific character corresponding to the next node according to the path in the multi-mode matching tree; if it does not exist, the next character of the first matching character in the valid string is read for matching.
[0073] S504: If the specific character matches the second matching character, the second matching character is used as the matching start node at the current moment; wherein the second matching character refers to the next sequential character of the first matching character in the characters to be matched.
[0074] S505: Repeat the steps of pointing the success pointer to the specific character corresponding to the next node according to the path of the success pointer in the multi-pattern matching tree, until the success pointer points to the target termination node and the failure pointer corresponding to the termination node points to the root node, and the target termination node is obtained.
[0075] Exemplarily, for example, if the valid character string is sher, assuming that the character to be matched (i.e., "s") that was successfully matched at the previous moment exists in the multi-mode matching tree, then the character to be matched "s" is used as the first matching character, and the first matching character is used as the matching start node at the current moment. According to the path of the success pointer in the multi-mode matching tree, the success pointer is pointed to the specific character (such as "a" and "h") corresponding to the next node, and the second matching character is the next sequential character "h" of the first matching character. The second matching character matches the specific character "h", and the second matching character is used as the matching start node at the current moment. Steps S502-S504 are repeated, and the above process is continued until the failure pointer of the target termination node points to the root node.
[0076] S506: Traverse the path of the success pointer based on the root node and the target end node to determine the dialogue stage corresponding to the dialogue theme word.
[0077] Specifically, taking the application of this method in the field of medical insurance as an example, in the process of communicating and interacting with customers using virtual images, different dialogue stages may be included, such as warm-up greetings, talking about the company, talking about the industry, talking about oneself, demand analysis, program communication, objection handling, etc. Among them, different dialogue stages correspond to different dialogue keywords for identifying dialogue stages. For example, when the dialogue stage is the demand analysis stage, its corresponding dialogue keywords include but are not limited to keywords such as illness, mortgage, expenditure or medical expenses.
[0078] Exemplarily, for ease of understanding, the execution process of steps S501-S506 is explained here by taking step S505 as an example. The server reads the next sequential character of the first matching character, that is, the second matching character corresponding to the next sequential label (that is, "h"), and matches the second matching character "h" with the specific character obtained in step S502. Assuming that the specific character is "a" at this time, it can be seen that the character to be matched "h" does not match the specific character "a", then steps S501-S506 are repeated to read the next character to be matched "e", and steps S502-S506 are executed until the last character to be matched in the valid string is detected, and matching stops.
[0079] Furthermore, assuming that the specific characters are "a" and "h" at this time, it can be seen that the character "h" to be matched matches the specific character "h", and the second matching character is used as the matching starting node at the current moment; S502-S504 are repeatedly executed to match the specific characters with the characters to be matched corresponding to the next sequence label in the valid string, that is, the specific character string "e" and "r" are matched with the character to be matched "e" corresponding to the next sequence label. It can be seen that the character to be matched "e" matches the specific character "e", and assuming that "e" is the end node, the word obtained from the root node to the end node is recorded. As the conversation theme word, i.e., "she", since "e" is the termination node at this time, according to the failure path, the failure pointer of the termination node "e" points to the failed character node "e", and according to the success path, the next node "r" of the failed character node "e" is used as a specific character to match the to-be-matched character "r" corresponding to the fourth sequence label in the valid string. Assuming that "r" is the target termination node, the conversation theme word "sher" is obtained, and the execution continues according to the pointing of the failure pointer. Assuming that the failure pointer of the target termination node "r" points to the root node (i.e., the root node), the loop is terminated at this time. It can be understood that each time the pointer points to the termination node in the loop, the word obtained from the root node to the termination node will be recorded as the conversation theme word.
[0080] In one embodiment, if Figure 6 As shown, after step S205, the method further includes the following steps:
[0081] S601: Evaluate the service effect of each actual service audio and video according to a plurality of preset evaluation factors, and obtain a score value corresponding to each evaluation factor.
[0082] Among them, evaluation factors refer to evaluation factors that determine service effects, including but not limited to communication quality, repeated conversations, ineffective conversations, the proportion of different emotions in the entire conversation, and service transaction rates. Customer emotions include but are not limited to pleasure, unhappiness, and anger. Customer emotions can be described by the customer's voice features and micro-expressions. Ultimately, the customer's emotional features at different times can be described by combining voice features and micro-expressions, for example, by setting the preset weights of these two dimensional features for weighted calculation, or by generating a model recognition of multimodal emotion recognition through machine learning or deep learning, which is not limited here.
[0083] Specifically, the scoring rules corresponding to different evaluation factors can be set in advance to obtain the scoring value corresponding to each evaluation factor, which is not limited here. For example, for the evaluation factor of repeated conversations, multiple levels (such as level 2) of scoring values can be set in advance, and each level corresponds to an evaluation standard. When the number of repeated conversations reaches the level 1 standard, the scoring value corresponding to the level 1 standard is obtained; when the number of repeated conversations reaches the level 2 standard, the scoring value corresponding to the level 1 standard is obtained.
[0084] S602: Construct a corresponding service behavior profile based on the score value corresponding to each evaluation factor.
[0085] Among them, the service behavior portrait is a matrix with N rows and M columns; N represents the number of actual service audio and video; M represents the number of evaluation factors.
[0086] S603: Based on the score value corresponding to each evaluation factor, calculate the covariance matrix corresponding to the service behavior profile.
[0087] Specifically, the covariance matrix can be calculated using the following formula: Among them, a k represents the Kth column of evaluation factors; M is the number of evaluation factors, such as communication quality, repeated conversations, ineffective conversations, proportion of different emotions or transaction rate after service; u represents the difference with a k The score mean of the corresponding evaluation factor column. C represents the covariance matrix, X represents the service behavior profile (N-row M-column matrix), and N represents the number of actual service audio and video. The actual service audio and video can be collected through a scheduled task to collect actual service samples within a period of time, which is not limited here.
[0088] It can be understood that in the above formula, The matrix X is converted to a zero-mean matrix, and then the corresponding covariance matrix is calculated based on the zero-mean matrix and its matrix transpose.
[0089] S604: Decompose the covariance matrix to obtain eigenvalues and corresponding eigenvectors.
[0090] Specifically, the covariance matrix can be decomposed by SVD singular value decomposition. Singular value decomposition (SVD decomposition for short) is an important matrix decomposition in linear algebra. The singular value decomposition operation can effectively reduce the dimension of large quantities of data to reduce the amount of calculation and save calculation time. Specifically, the server performs singular value decomposition on the covariance matrix to obtain two unitary matrices and a semi-positive definite diagonal matrix. The values on the diagonal of the semi-positive definite diagonal matrix are singular values (i.e., eigenvalues). The singular values generally contain N (N is greater than 2) and are arranged in order from large to small according to the singular values. Singular values can represent the important information implicit in the matrix, and the importance is positively correlated with the size of the singular value. It can be understood that the larger the singular value, the greater the amount of effective information contained in the singular value; conversely, the smaller the singular value, the less effective information contained in the singular value, and it is determined that it contains more noise. The server performs singular value decomposition on the covariance matrix to obtain eigenvalues and corresponding eigenvectors, and can intuitively observe the extent of effective information contained in the eigenvalues, making it easier to perform noise reduction on the samples.
[0091] S605: Filter multiple actual service audios based on the feature values to obtain incremental samples, so as to update the multimodal virtual image generation network according to the incremental samples.
[0092] Among them, the feature vectors are arranged into a matrix in rows from large to small and from top to bottom according to their eigenvalues, and the eigenvectors with eigenvalues in the top N or greater than a preset threshold are retained to filter out negative samples with less effective information, that is, actual service audio and video with poor service effects, to obtain incremental samples, which is convenient for updating the multimodal virtual image generation network according to the incremental samples.
[0093] It can be understood that the incremental samples only contain positive samples with good service effects, and the multimodal avatar generation network is updated according to the historical service audio and video and conversation data in the positive samples to ensure that the subsequent multimodal avatar generation network outputs effective learning of the conversation content and corresponding emotions during the service communication stage. Specifically, the TransE algorithm can be used to perform incremental feature learning on the multimodal avatar generation network, which is not limited here.
[0094] In one embodiment, if Figure 7 As shown, after step S205, the method further includes the following steps:
[0095] S701: Obtaining the satisfaction of business personnel with the basic virtual image.
[0096] S702: Using a basic virtual image whose satisfaction reaches a preset threshold as a training virtual image; using the training virtual image and the multimodal input data of the business personnel corresponding to the training virtual image as training samples to perform supervised learning on the multimodal virtual image generation network and update the multimodal virtual image generation network.
[0097] Among them, since the multimodal virtual image generation network is obtained through unsupervised learning training, the generated multimodal virtual image generation network is coarse-grained, so it is necessary to determine the accuracy of the multimodal virtual image generation network through subsequent satisfaction evaluation of actual business personnel. Therefore, in this case, the basic virtual image (i.e., as a real annotation) with the satisfaction reaching the preset threshold is used as the training virtual image, and the training virtual image and the multimodal input data of the corresponding business personnel corresponding to the training virtual image are used as training samples to supervise the learning of the multimodal virtual image generation network, thereby updating the multimodal virtual image generation network. Specifically, the TransE algorithm can be used to supervise the learning of the multimodal virtual image generation network, which is not limited here.
[0098] Furthermore, when incremental samples are obtained, such as the incremental samples obtained in S605 or S702, when the atlas is updated, the update can be performed on the basis of the original multimodal virtual image generation network, or the incremental samples obtained in these two dimensions can be intersected to obtain updated incremental samples and perform end-to-end regression learning based on a deep learning network or machine learning, which is not limited here.
[0099] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
[0100] In one embodiment, a virtual image synthesis device is provided, and the virtual image synthesis device corresponds to the virtual image synthesis method in the above embodiment. Figure 8 As shown, the avatar synthesis device includes a multimodal input acquisition module 10, a basic avatar synthesis module 20, an intelligent dialogue module 30, an interactive feature output module 40 and a dynamic avatar synthesis module 50. The functional modules are described in detail as follows:
[0101] The multimodal input acquisition module 10 is used to acquire multimodal input data of the business personnel; wherein the multimodal input data includes personal information, human body images, and historical service audio and video;
[0102] The basic virtual image synthesis module 20 is used to input the personal information, the human image of the business personnel, and the historical service audio and video into a pre-built multimodal virtual image generation network for multimodal recognition to output the basic virtual image of the business personnel;
[0103] The intelligent dialogue module 30 is used to call the intelligent dialogue module for initial service based on the basic virtual image and receive dialogue data in real time if the current dialogue scene is an offline scene;
[0104] An interaction feature output module 40, configured to identify the current dialogue stage based on the dialogue data through the multimodal avatar generation network, so as to output interaction features corresponding to the dialogue stage;
[0105] The dynamic virtual image synthesis module 50 is used to synthesize the virtual image video frame at the current moment based on the interaction characteristics and the basic virtual image.
[0106] Specifically, the basic virtual image synthesis module includes a global encoding unit, a first modality extraction unit, a second modality extraction unit, a third modality extraction unit and a basic virtual image synthesis unit.
[0107] A global encoding unit, used for obtaining the global encoding of the personal information of the business personnel in the multimodal virtual image generation network;
[0108] A first modality extraction unit, configured to identify a first modality feature corresponding to the human body image by using a first feature point positioning model;
[0109] A second modality extraction unit, configured to identify a second modality feature corresponding to the historical service video using a second feature point positioning model;
[0110] A third modality extraction unit, configured to use a voice recognition model to identify a third modality feature corresponding to the historical service audio;
[0111] A basic virtual image synthesis unit is used to predict based on the global encoding, the first modal feature, the second modal feature and the third modal feature through the multimodal virtual image generation network, and output the basic virtual image of the business personnel.
[0112] Specifically, the interaction feature output module includes a filtering unit, a dialogue stage determination unit and an interaction feature output unit.
[0113] A filtering unit, used for filtering the conversation data to obtain a valid character string;
[0114] A dialogue stage determination unit, used for inputting the valid character string into a pre-built multi-mode matching tree for multi-mode matching to determine the dialogue stage at the current moment;
[0115] The interactive feature output unit is used to output the interactive features corresponding to the dialogue stage at the current moment based on the dialogue stage through the multimodal virtual image generation network.
[0116] Specifically, the multi-mode matching tree includes a success pointer, a failure pointer and a root node; the dialogue stage determination unit includes a keyword matching subunit, a matching start node determination subunit, a specific character pointing subunit, a specific character matching subunit, a loop matching subunit and a dialogue stage determination subunit.
[0117] A keyword matching subunit, used for traversing the valid character string and performing keyword matching on characters to be matched in the valid character string in sequence based on the multi-mode matching tree;
[0118] A matching start node determination subunit, configured to use the first matching character as the matching start node at the current moment if there is a first matching character that has been successfully matched in the multi-mode matching tree at the previous moment;
[0119] A specific character pointing subunit, used to point the success pointer to the specific character corresponding to the next node according to the path of the success pointer; wherein the next node is the next one or more nodes adjacent to the matching start node in the path of the success pointer;
[0120] A specific character matching subunit, used for, if the specific character matches a second matching character, using the second matching character as a matching start node at the current moment; wherein the second matching character refers to the next sequential character of the first matching character in the characters to be matched;
[0121] A loop matching subunit is used to repeatedly execute the step of pointing the success pointer to the specific character corresponding to the next node according to the path of the success pointer in the multi-mode matching tree, until the success pointer points to the target termination node and the failure pointer corresponding to the termination node points to the root node, thereby obtaining the target termination node.
[0122] The dialogue stage determination subunit is used to traverse the path of the success pointer based on the root node and the target termination node to determine the dialogue stage corresponding to the dialogue theme word.
[0123] Specifically, the virtual image synthesis device also includes a scoring module, a behavior portrait construction module, a covariance matrix calculation module, a matrix decomposition module and a network optimization module.
[0124] A scoring module, used to evaluate the service effect of each actual service audio and video according to a plurality of preset evaluation factors, and obtain a scoring value corresponding to each of the evaluation factors;
[0125] A behavior profile building module, used to build a corresponding service behavior profile according to the score value corresponding to each evaluation factor;
[0126] A covariance matrix calculation module, used to calculate the covariance matrix corresponding to the service behavior profile based on the score value corresponding to each evaluation factor;
[0127] A matrix decomposition module, used to decompose the covariance matrix to obtain eigenvalues and corresponding eigenvectors;
[0128] The network optimization module is used to filter the multiple actual service audios based on the feature values to obtain incremental samples, so as to update the multimodal virtual image generation network according to the incremental samples.
[0129] Specifically, the virtual image synthesis device also includes an image satisfaction acquisition module and a network optimization module.
[0130] An image satisfaction acquisition module is used to acquire the satisfaction of the business personnel with the basic virtual image;
[0131] The network optimization module is used to use the basic virtual image whose satisfaction reaches a preset threshold as a training virtual image; and use the training virtual image and the multimodal input data of the business personnel corresponding to the training virtual image as training samples to supervise the learning of the multimodal virtual image generation network and update the multimodal virtual image generation network.
[0132] The specific definition of the virtual image synthesis device can be found in the definition of the virtual image synthesis method above, which will not be repeated here. Each module in the above virtual image synthesis device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0133] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a computer storage medium and an internal memory. The computer storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the computer storage medium. The database of the computer device is used to store data generated or obtained during the execution of the virtual image synthesis method, such as a multimodal virtual image generation network. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a virtual image synthesis method is implemented.
[0134] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the virtual image synthesis method in the above embodiment are implemented, such as Figure 2 Steps S201-S205 shown, or Figures 3 to 7 Alternatively, when the processor executes the computer program, the functions of each module / unit in the embodiment of the virtual image synthesis device are realized, for example Figure 8 The functions of the modules / units shown are not described here in detail to avoid repetition.
[0135] In one embodiment, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the virtual image synthesis method in the above embodiment are implemented, such as Figure 2 Steps S201-S205 shown, or Figures 3 to 7 Alternatively, when the computer program is executed by a processor, the functions of each module / unit in the embodiment of the virtual image synthesis device are realized, for example, Figure 8 The functions of the modules / units shown are not described here in detail to avoid repetition.
[0136] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0137] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0138] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.
Claims
1. A method for synthesizing a virtual image, characterized in that: include: Acquire multimodal input data of business personnel; wherein the multimodal input data includes personal information, human body images, and historical service audio and video; Inputting the personal information, the human image of the business personnel, and the historical service audio and video into a pre-built multimodal virtual image generation network for multimodal recognition to output a basic virtual image of the business personnel; If the current conversation scene is an offline scene, based on the basic virtual image, the intelligent conversation module is called to perform initial service and receive conversation data in real time; Based on the dialogue data, the multimodal avatar generation network identifies the dialogue stage at the current moment to output the interaction features corresponding to the dialogue stage; Based on the dialogue data, the multimodal avatar generation network identifies the dialogue stage at the current moment to output the interaction features corresponding to the dialogue stage, including filtering the dialogue data to obtain a valid character string; inputting the valid character string into a pre-constructed multi-mode matching tree for multi-mode matching to determine the dialogue stage at the current moment; Based on the dialogue stage, the multimodal avatar generation network outputs the interaction features corresponding to the dialogue stage at the current moment; The multi-mode matching tree includes a success pointer, a failure pointer and a root node; the valid character string is input into the pre-constructed multi-mode matching tree for multi-mode matching to determine the current dialogue stage, including traversing the valid character string and, based on the multi-mode matching tree, sequentially performing keyword matching on the characters to be matched in the valid character string; if there is a first matching character that has been successfully matched in the multi-mode matching tree at the previous moment, the first matching character is used as the matching start node at the current moment; according to the path of the success pointer, the success pointer is pointed to a specific character corresponding to the next node; wherein the next node is the character in the path of the success pointer that matches the matching character. the next one or more nodes adjacent to the starting node; if the specific character matches the second matching character, the second matching character is used as the matching starting node at the current moment; wherein the second matching character refers to the next sequential character of the first matching character in the characters to be matched; repeatedly executing the step of pointing the success pointer to the specific character corresponding to the next node according to the path of the success pointer in the multi-mode matching tree, until the success pointer points to the target termination node and the failure pointer corresponding to the termination node points to the root node, and the target termination node is obtained; based on the root node and the target termination node, traversing according to the path of the success pointer, determine the dialogue stage corresponding to the dialogue theme word; Based on the interaction features and the basic virtual image, a virtual image video frame at the current moment is synthesized.
2. The method for synthesizing a virtual image according to claim 1, wherein: The personal information, the human image of the business personnel, and the historical service audio and video are input into a pre-built multimodal virtual image generation network for multimodal recognition to output the basic virtual image of the business personnel, including: Obtaining a global encoding of the personal information of the business personnel in the multimodal avatar generation network; Using a first feature point positioning model to identify a first modal feature corresponding to the human body image; Using a second feature point positioning model to identify a second modal feature corresponding to the historical service video; Using a voice recognition model, identifying a third modality feature corresponding to the historical service audio; The multimodal virtual image generation network predicts based on the global encoding, the first modal feature, the second modal feature and the third modal feature, and outputs a basic virtual image of the business personnel.
3. The method for synthesizing a virtual image according to claim 1, wherein: After synthesizing the virtual image video frame at the current moment based on the interaction feature and the basic virtual image, the method further includes: Evaluate the service effect of each actual service audio and video according to a plurality of preset evaluation factors, and obtain a score value corresponding to each of the evaluation factors; According to the score value corresponding to each evaluation factor, a corresponding service behavior profile is constructed; Based on the score value corresponding to each evaluation factor, calculate the covariance matrix corresponding to the service behavior profile; Decomposing the covariance matrix to obtain eigenvalues and corresponding eigenvectors; Based on the feature value, a plurality of the actual service audios are filtered to obtain incremental samples, so as to update the multimodal virtual image generation network according to the incremental samples.
4. The method for synthesizing a virtual image according to claim 1, wherein: After synthesizing the virtual image video frame at the current moment based on the interaction feature and the basic virtual image, the virtual image synthesis method further includes: Obtaining satisfaction of the business personnel with the basic virtual image; The basic virtual image whose satisfaction reaches a preset threshold is used as a training virtual image; and The training virtual image and the multimodal input data of the business personnel corresponding to the training virtual image are used as training samples to perform supervised learning on the multimodal virtual image generation network and update the multimodal virtual image generation network.
5. A virtual image synthesis device, characterized in that: include: A multimodal input acquisition module, used to acquire multimodal input data of business personnel; wherein the multimodal input data includes personal information, human body images, and historical service audio and video; A basic virtual image synthesis module is used to input the personal information, the human image of the business personnel, and the historical service audio and video into a pre-built multimodal virtual image generation network for multimodal recognition to output the basic virtual image of the business personnel; An intelligent dialogue module, for invoking the intelligent dialogue module to perform initial services based on the basic virtual image and receiving dialogue data in real time if the current dialogue scene is an offline scene; An interactive feature output module, configured to identify the current dialogue stage based on the dialogue data through the multimodal avatar generation network, so as to output the interactive features corresponding to the dialogue stage; the interactive feature output module includes a filtering unit, a dialogue stage determination unit and an interactive feature output unit; The filtering unit is used to filter the conversation data to obtain a valid character string; The dialogue stage determination unit is used to input the valid character string into a pre-built multi-mode matching tree to perform multi-mode matching to determine the dialogue stage at the current moment; The interaction feature output unit is used to output the interaction feature corresponding to the dialogue stage at the current moment based on the dialogue stage through the multimodal avatar generation network; The multi-mode matching tree includes a success pointer, a failure pointer and a root node; the dialogue stage determination unit includes a keyword matching subunit, a matching start node determination subunit, a specific character pointing subunit, a specific character matching subunit, a loop matching subunit and a dialogue stage determination subunit; The keyword matching subunit is used to traverse the valid character string and perform keyword matching on the characters to be matched in the valid character string in sequence based on the multi-mode matching tree; The matching start node determination subunit is used to use the first matching character as the matching start node at the current moment if there is a first matching character that has been successfully matched in the multi-mode matching tree at the previous moment; The specific character pointing subunit is used to point the success pointer to the specific character corresponding to the next node according to the path of the success pointer; wherein the next node is the next one or more nodes adjacent to the matching start node in the path of the success pointer; The specific character matching subunit is used to use the second matching character as the matching start node at the current moment if the specific character matches the second matching character; wherein the second matching character refers to the next sequential character of the first matching character in the characters to be matched; The loop matching subunit is used to repeatedly execute the step of pointing the success pointer to the specific character corresponding to the next node according to the path of the success pointer in the multi-pattern matching tree, until the success pointer points to the target termination node and the failure pointer corresponding to the termination node points to the root node, thereby obtaining the target termination node; The dialogue stage determination subunit is used to traverse the path of the success pointer based on the root node and the target end node to determine the dialogue stage corresponding to the dialogue theme word; The dynamic virtual image synthesis module is used to synthesize the virtual image video frame at the current moment based on the interactive features and the basic virtual image.
6. The virtual image synthesis device according to claim 5, characterized in that: The basic virtual image synthesis module includes: A global encoding unit, used for obtaining the global encoding of the personal information of the business personnel in the multimodal virtual image generation network; A first modality extraction unit, configured to identify a first modality feature corresponding to the human body image by using a first feature point positioning model; A second modality extraction unit, configured to identify a second modality feature corresponding to the historical service video using a second feature point positioning model; A third modality extraction unit, configured to use a voice recognition model to identify a third modality feature corresponding to the historical service audio; A basic virtual image synthesis unit is used to predict based on the global encoding, the first modal feature, the second modal feature and the third modal feature through the multimodal virtual image generation network, and output the basic virtual image of the business personnel.
7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the virtual image synthesis method according to any one of claims 1 to 5 are implemented.
8. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the virtual image synthesis method as described in any one of claims 1 to 5 are implemented.
