Virtual human intelligent customer service method and system based on digital virtual technology and generative AI

Through digital virtual technology and generative AI, combined with three-dimensional modeling and large language model, the problem of invisible interaction between traditional intelligent customer service is solved, intelligent and visual customer service is realized, and the level of intelligent question-and-answer and interactive experience is improved.

CN119089933APending Publication Date: 2024-12-06CHINA CITIC BANK CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202411023437.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Traditional intelligent customer service lacks intelligence and cannot effectively handle customer consultations beyond preset issues. The interactive experience is not vivid, the response is lagging and there is a lack of visual interaction.

Method used

Using digital virtual technology and generative AI, through three-dimensional modeling and bone binding and deformation driving, combined with large language models, we build an intelligent question-and-answer knowledge base to realize the visualization and intelligent interaction of intelligent customer service, and support multi-modal input and output.

Benefits of technology

It has improved the intelligent Q&A level of smart customer service, enhanced the naturalness and vividness of interaction with customers, expanded the application scenarios, and provided a richer and more accurate service experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089933A_ABST
    Figure CN119089933A_ABST
Patent Text Reader

Abstract

The invention relates to a virtual human intelligent customer service method and system based on a digital virtual technology and generative AI, and the method comprises the steps: formulating an intelligent customer service image, and carrying out the three-dimensional modeling of the intelligent customer service; the intelligent customer service is driven on the basis of skeleton binding and deformation and AI driving, wherein action and expression driving is carried out on the intelligent customer service on the basis of skeleton binding and deformation; using AI to drive the intelligent customer service; the intelligent customer service is rendered; and constructing an intelligent question and answer knowledge base based on the large language model, and performing fine tuning on the large language model. The intelligent question and answer level is improved through the large language model, multi-scene question and answer interaction is supported, the matching degree of the face and the mouth shape is improved through the training model, the emotion factors are added, so that the intelligent customer service expression is more natural, and the user experience is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a virtual human intelligent customer service method based on digital virtual technology and generative AI. Background Art

[0002] Intelligent customer service is a new technical means of customer service in the current financial industry. It is a business assistant that can provide customers with interactive question-and-answer services based on artificial intelligence, knowledge management and other technologies. However, traditional intelligent customer service is not smart enough and can only give corresponding answers to some pre-given questions. If you encounter many questions that are beyond the scope, you need to transfer to manual customer service; it lacks attractive image interaction, and the interaction with customers during the question-and-answer process is poor, the experience is poor, and it lacks a vivid and attractive image and real-time interactive capabilities. The response is delayed and the answers are irrelevant.

[0003] To address these issues, with the development and maturity of new technologies such as digital twins, human-computer interaction technology, and AIGC, digital virtual human intelligent customer service can be used to assist customers in handling banking business, 3D modeling can be used to create exquisite and moving images for digital intelligent customer service, AI generation algorithms can be used to drive digital intelligent customer service, and large models can be used to provide digital intelligent customer service with intelligent question and answer processing functions. This digital human intelligent customer service that integrates multiple cutting-edge technologies can greatly improve the customer service experience, enhance the intelligence of digital customer service, and expand the coverage of intelligent customer service services. Summary of the invention

[0004] In order to address the shortcomings of the existing technology, the present invention adopts digital virtual technology to model and drive intelligent customer service, and at the same time connects to a large model as the underlying intelligent question-and-answer engine, making the interaction process between intelligent customer service and customers more natural and the answers more accurate.

[0005] To achieve the above objectives, the technical solutions adopted by the present invention include:

[0006] A virtual human intelligent customer service method based on digital virtual technology and generative AI, characterized by comprising:

[0007] S1. Formulate an intelligent customer service image and perform three-dimensional modeling of the intelligent customer service;

[0008] S2. Driving the intelligent customer service based on skeleton binding and deformation and AI driving, including:

[0009] Driving the actions and expressions of the intelligent customer service based on bone binding and deformation;

[0010] Using AI to drive the intelligent customer service;

[0011] S3. Rendering the intelligent customer service;

[0012] S4. Build an intelligent question-answering knowledge base based on the large language model, and fine-tune the large language model.

[0013] Furthermore, the use of AI to drive the intelligent customer service includes:

[0014] Create a pre-built model to generate a synchronized data set for voice and intelligent customer service models;

[0015] Using the speech and intelligent customer service model synchronization data set to train an expression generation model;

[0016] Emotional latent variables are learned from the speech and intelligent customer service model synchronization data set, and the expression generation model is optimized according to the emotional latent variables.

[0017] Furthermore, the creation of a pre-generated model to generate a speech and intelligent customer service synchronization data set includes:

[0018] The encoder-decoder network is used as the pre-generation model architecture to extract audio features;

[0019] The encoder is used to learn to convert the audio features into low-frequency embeddings, and the decoder is used to map the embeddings into a high-dimensional space.

[0020] Furthermore, the method of using the speech and intelligent customer service model synchronization data set to train the expression generation model includes:

[0021] Annotating phoneme information in the speech and intelligent customer service model synchronization data set;

[0022] Use AAM algorithm to extract facial expression correlation coefficient;

[0023] Train an expression generation model according to the phoneme information and the speech and intelligent customer service model synchronization data set, and output corresponding facial expression coefficients;

[0024] The corresponding facial expression coefficient is transferred to the face of the intelligent customer service for redirection.

[0025] Furthermore, learning emotional latent variables from the speech and intelligent customer service model synchronization data set, and optimizing the expression generation model according to the emotional latent variables, includes:

[0026] Add frequency analysis layer, pronunciation analysis layer, and vertex output layer to the expression generation model;

[0027] The frequency analysis layer includes an autocorrelation layer and a convolution layer, in which the autocorrelation coefficient of the audio is extracted using linear prediction coding to obtain an audio feature map, and the audio feature map is convoluted to obtain an audio feature vector;

[0028] Learning emotional latent variables to form an emotional state matrix, extracting correlation features of adjacent sequence frames in the pronunciation analysis layer, selecting any set of vectors of the emotional state matrix to input into the pronunciation analysis layer, and outputting a feature vector including audio features and emotional features;

[0029] The vertex output layer includes two fully connected layers, wherein the mapping from audio features to facial expression coefficients is obtained in the first fully connected layer, and the mapping from the facial expression coefficients to the coordinate values ​​of each vertex is obtained in the second fully connected layer.

[0030] Furthermore, the intelligent question-answering knowledge base is constructed based on the large language model, and the large language model is fine-tuned, including:

[0031] Collect professional knowledge, integrate the professional knowledge into question-answer pairs, and build an intelligent question-answer knowledge base;

[0032] Use knowledge-enhanced pre-training to embed the expertise into a large language model;

[0033] The large language model is fine-tuned by instruction fine-tuning technology.

[0034] The present invention also relates to a virtual human intelligent customer service system based on digital virtual technology and generative AI, which is characterized by comprising:

[0035] An image customization module, used to formulate an image of an intelligent customer service and perform three-dimensional modeling of the intelligent customer service;

[0036] A driving module, used to drive the intelligent customer service based on skeleton binding and deformation and AI driving, including: driving the intelligent customer service for actions and expressions based on skeleton binding and deformation; driving the intelligent customer service using AI;

[0037] A rendering module, used to render the intelligent customer service;

[0038] The question-answering module is used to build an intelligent question-answering knowledge base based on the large language model and to fine-tune the large language model.

[0039] The present invention also relates to a computer-readable storage medium, characterized in that a computer program is stored on the storage medium, and the computer program implements the above method when executed by a processor.

[0040] The present invention also relates to an electronic device, characterized in that it comprises a processor and a memory;

[0041] The memory is used to store program instructions;

[0042] The processor is used to execute the above method by calling program instructions.

[0043] The present invention also relates to a computer program product, comprising a computer program and / or instructions, characterized in that the computer program and / or instructions implement the steps of the above method when executed by a processor.

[0044] The beneficial effects of the present invention are:

[0045] The virtual human intelligent customer service method based on digital virtual technology and generative AI described in the present invention can improve the intelligent question and answer level of digital intelligent customer service, support a wider range of application scenarios after docking with the large model, and can not only answer questions for customers in financial business scenarios, but also support question and answer interactions in life scenarios; the intelligent customer service of the present invention has a beautiful and vivid image, and can drive the digital customer service in real time according to different scenarios and interactive content, so that the digital intelligent customer service is closer to human expression and emotions in interaction and expression, and is more vivid and attractive in the process of interacting with customers, getting rid of the mechanical sense of traditional intelligent customer service, and at the same time, the interactive interface will also be transformed from a dialog box page to an interactive interface form with the digital intelligent customer service image as the main body, thereby greatly improving the customer service experience; the present invention supports multi-modal input and output, the input can support voice, text and other forms, and the output end can perform multi-modal output according to customer needs, such as pictures, videos, voice, files, etc., the content is richer and more accurate, and can support the implementation of more scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0047] Figure 1 It is a flow chart of the virtual human intelligent customer service method based on digital virtual technology and generative AI of the present invention;

[0048] Figure 2 This is a schematic diagram of the structure of a virtual human intelligent customer service system based on digital virtual technology and generative AI of the present invention;

[0049] Figure 3 The figure is a schematic structural diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. It should be understood that the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments, and these embodiments are only used to illustrate the present application and are not used to limit the scope of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0051] like Figure 1 As shown, the embodiment of the present invention discloses a virtual human intelligent customer service method based on digital virtual technology and generative AI, and the steps of the method include:

[0052] Step S1. Formulate an intelligent customer service image and perform three-dimensional modeling on the intelligent customer service.

[0053] In the embodiment of the present invention, the image customization of the intelligent customer service can be divided into the following steps:

[0054] (1) Design the original image of the intelligent customer service according to the needs, including the head, body, clothing, hairstyle, etc.;

[0055] (2) Based on the designed original painting, use modeling tools such as Maya and 3ds max to perform three-dimensional modeling of the intelligent customer service;

[0056] (3) Apply texture maps to the intelligent customer service image to make it look more realistic;

[0057] (4) Add animation to the intelligent customer service image so that it can interact with expressions and actions;

[0058] (5) Post-production of the intelligent customer service image, including rendering, synthesis, and optimization.

[0059] Step S2: driving the intelligent customer service based on skeleton binding and deformation and AI driving, including:

[0060] Driving the actions and expressions of the intelligent customer service based on bone binding and deformation;

[0061] Utilize AI to drive the intelligent customer service.

[0062] In an embodiment of the present invention, the intelligent customer service is driven by actions and expressions based on skeleton binding and deformation. First, the skeleton is associated with the mesh, where the skeleton can be a simple line or a complex model, and the mesh is the surface of the model, which is composed of vertices, edges and faces. The skeleton and the mesh are associated through skeleton nodes, where the skeleton nodes are small points located on the skeleton, and the movement of the skeleton can be controlled by moving the points.

[0063] In the embodiment of the present invention, the mesh can be transformed from one shape to another by deformation, and each shape corresponds to a form of the mesh, which can be used to create various expressions, such as smiling, frowning, opening the mouth, etc.

[0064] Through bone binding and deformation, virtual human intelligent customer service can perform various actions. For some basic scenes, fixed actions are generally produced using bone binding and deformation. The produced actions are placed in the action library. According to different scene requirements, the redirection algorithm in the rendering engine is used to drive the virtual human intelligent customer service, making the virtual human intelligent customer service actions realistic.

[0065] Preferably, AI is used to drive the intelligent customer service, and driving data of the virtual human intelligent customer service is generated in real time according to the generated answer text content and scene effects.

[0066] Current speech-driven lip shape and facial animation generation technologies have the following drawbacks:

[0067] (1) There is a lack of open-source speech and 3D model synchronization datasets, and self-collection of data requires recording professional actors’ performance videos through professional motion capture software, which is costly. Existing speech-driven facial animation generation technology is not very practical, and requires sufficient training samples to obtain better judgment results;

[0068] (2) Most studies focus only on speech-driven 3D facial lip animation, ignoring speech-driven facial animation, which results in the virtual human faces being dull and lacking feedback on expression information. Many speech-driven facial animation generation technologies cannot reflect the most realistic state of the human face, and may even produce the uncanny valley effect.

[0069] (3) Deep learning-based lip shape and facial animation synthesis suffers from the problem of insufficient animation continuity, frequent jumps, and poor animation smoothness and naturalness. Usually, the speech is ahead of the generated video frames.

[0070] In view of the above defects, an embodiment of the present invention provides a method for driving the intelligent customer service using AI, including:

[0071] Create a pre-built model to generate a synchronized data set for voice and intelligent customer service models;

[0072] Using the speech and intelligent customer service model synchronization data set to train an expression generation model;

[0073] Emotional latent variables are learned from the speech and intelligent customer service model synchronization data set, and the expression generation model is optimized according to the emotional latent variables.

[0074] Specifically, an embodiment of the present invention provides a method for creating a pre-generated model and generating a speech and intelligent customer service model synchronization data set, including:

[0075] The encoder-decoder network is used as the pre-generation model architecture to extract audio features;

[0076] The encoder is used to learn to convert the audio features into low-frequency embeddings, and the decoder is used to map the embeddings into a high-dimensional space.

[0077] In the embodiment of the present invention, the encoder includes four convolutional layers and two fully connected layers, the decoder is a fully connected layer with a linear activation function, and deepSpeech can be used to extract audio features for audio input with a length of N seconds.

[0078] This step can solve the problem of the lack of open source speech and 3D model synchronization datasets. During the training process, the model can be allowed to learn various realistic speaking styles by adjusting the topic labels. The final output of the model is a facial expression animation video consistent with the speech, which can be used to provide video data for training the expression generation model.

[0079] Specifically, the embodiment of the present invention provides a method for training an expression generation model using the speech and intelligent customer service model synchronization data set, including:

[0080] Annotating phoneme information in the speech and intelligent customer service model synchronization data set;

[0081] Use AAM algorithm to extract facial expression related parameters;

[0082] Train an expression generation model according to the phoneme information and the speech and intelligent customer service model synchronization data set, and output corresponding facial expression parameters;

[0083] The corresponding facial expression parameters are transferred to the face of the intelligent customer service for redirection.

[0084] This step can solve the problem of lack of authenticity in voice-driven facial animation. By adopting the AAM (Active Appearance Model) algorithm and combining it with a sliding window predictor, natural movements and visual collaborative pronunciation effects can be accurately generated at a low cost. Among them, the use of a sliding window can enhance the stability of the previous and next frames of the model.

[0085] In the embodiment of the present invention, taking a single expression coefficient as an example, the process of using a sliding window to perform model training is as follows:

[0086] Step 1: First, the input phoneme frame sequence is divided into continuous overlapping subsequences according to a sliding window of length 5;

[0087] Step 2: Take the sequence in each window as input, pass it to the feedforward neural network, and output the expression parameter sequence of corresponding length;

[0088] Step 3: Use frame-by-frame averaging and fusion to obtain the expression parameters corresponding to each frame.

[0089] Preferably, the embodiment of the present invention adds learning of emotional latent variables, which can solve the impact of emotional changes on the expression driving effect during the voice driving process. By adding a frequency analysis layer, a pronunciation analysis layer, and a vertex output layer to the expression generation model, the expression generation model is optimized to achieve emotion processing.

[0090] In an embodiment of the present invention, the frequency analysis layer may include an autocorrelation layer (autocorrelation) with fixed parameters and five convolution layers (convolution). Among them, the autocorrelation layer uses linear predictive coding (LPC) to extract the first k autocorrelation coefficients of the audio. The autocorrelation coefficient describes the energy spectrum distribution of the resonance peak of the original audio signal, so the autocorrelation coefficient is used to describe the characteristics of the audio. The autocorrelation layer takes multiple frames of audio signals as input, and after processing, it outputs a 2D audio feature map of W*H, where W represents the number of frames and H represents the dimension of the autocorrelation coefficient. The 2D audio feature map is then subjected to five layers of 1*3 convolution processing, the autocorrelation coefficient is compressed, and finally a feature vector of W*1 is output. The dimension of the audio feature vector output by this layer is 256*W*1, where 256 is the number of channels of the feature map. The network structure of the frequency analysis layer is shown in Table 1.

[0091] Table 1

[0092] Layer type Kernel Stride Outputs Activation Autocorrelation - - 1*64*32 - Convolution 1*3 1*2 72*64*16 ReLU Convolution 1*3 1*2 108*64*8 ReLU Convolution 1*3 1*2 162*64*4 ReLU Convolution 1*3 1*2 243*64*2 ReLU Convolution 1*2 1*2 256*64*1 ReLU

[0093] In the embodiment of the present invention, the pronunciation analysis layer includes five convolution layers, each of which has a convolution kernel of 3*1, which can be used to extract the associated features of adjacent sequence frames in time sequence and output a pronunciation feature map. The pronunciation analysis layer also receives the emotional state extracted by training, with a vector length of E, and jointly learns the relationship with the expression with the pronunciation feature map. The network structure of the pronunciation analysis layer is shown in Table 2.

[0094] Table 2

[0095] Layer type Kernel Stride Outputs Activation Conv+concat 3*1 2*1 (256+E)*32*1 ReLU Conv+concat 3*1 2*1 (256+E)*16*1 ReLU Conv+concat 3*1 2*1 (256+E)*8*1 ReLU Conv+concat 3*1 2*1 (256+E)*4*1 ReLU Conv+concat 4*1 4*1 (256+E)*1*1 ReLU

[0096] After being processed by the pronunciation analysis layer, a feature vector with a length of (256+E) is output, where 256 is the audio feature and E is the emotional feature. In the training stage, the input of the emotional vector is a random Gaussian distribution sampling of E dimensions. For input samples with emotions, the feature vectors output by the pronunciation analysis layer are stored during the training process, wherein the E-dimensional feature vector is the emotional state contained in the training sample data that is automatically learned. The emotional feature vectors learned under different emotional data are combined to form an emotional state matrix. During the inference process, the user can select any set of vectors of the matrix as emotional input (rather than random Gaussian distribution sampling) to control the output expression to contain the specified emotion. In an embodiment of the present invention, different emotional vectors can also be used as input by linearly combining them to generate an expression animation containing new emotions. By automatically learning the emotions contained in the data, the ambiguity and inaccuracy caused by the emotional state labels of the manually labeled data set can be avoided.

[0097] In an embodiment of the present invention, the vertex output layer includes two fully connected layers, through which the mapping from features to facial expression vertex coordinates can be realized. The first fully connected layer can realize the mapping from speech features to facial expression coefficients, and the second layer can realize the mapping from expression coefficients to each vertex coordinate value. The weight matrix of the second layer corresponds to the multilinear facial expression model in the traditional expression drive, and can be initialized in advance with 150 pre-calculated PCA components. The network structure of the vertex output layer is shown in Table 3.

[0098] Table 3

[0099]

[0100]

[0101] Step S3: Render the intelligent customer service.

[0102] In the embodiment of the present invention, the intelligent customer service can be rendered in two ways: cloud rendering and end rendering.

[0103] Specifically, cloud rendering refers to moving the rendering of 3D smart customer service from the terminal device to the cloud server. This method can utilize the massive computing resources of the cloud server to improve rendering efficiency and provide users with a higher quality experience. Using cloud rendering to render 3D smart customer service includes user input instructions, the cloud rendering platform obtains instructions from the user end, requests routing, uses a high-bandwidth network connection to upload the 3D smart customer service model to the cloud server, renders the 3D smart customer service on the cloud server, processes frame by frame, performs encoding operations, transmits the encoding to the user end through streaming, and performs decoding operations on the user end, that is, transmits the rendered image back to the terminal device.

[0104] Specifically, end-rendering can also be used to render 3D smart customer service. End-rendering refers to moving the rendering of 3D smart customer service from the server to the terminal device. This method can significantly reduce the use of network bandwidth, improve rendering efficiency, and provide users with a smoother experience. Rendering 3D smart customer service using end-rendering includes using mesh compression, texture compression, animation compression and other technologies to compress the model of 3D smart customer service to a size that can be stored on the terminal device, rendering 3D smart customer service in real time on the terminal device, and transmitting the 3D smart customer service model and texture data from the server to the terminal device.

[0105] Step S4: construct an intelligent question-answering knowledge base based on the large language model, and fine-tune the large language model.

[0106] In the embodiment of the present invention, the training model is applied to the intelligent customer service field of the financial industry. The intelligent customer service needs to be able to understand and answer the customer's financial knowledge questions, and at the same time be able to assist the customer in completing specific business processing, such as account information query, business processing, investment consultation, etc. Specifically, the above functions can be achieved through the following steps:

[0107] Collect professional knowledge, integrate the professional knowledge into question-answer pairs, build an intelligent question-answer knowledge base, and use knowledge-aided pre-training to embed the professional knowledge into a large language model. Professional knowledge may include financial product introductions, trading rules, risk management, laws and regulations, etc.

[0108] Identify the types of tasks that intelligent customer service needs to complete, such as question answering, business guidance, risk assessment, etc. Fine-tune the large language model through instruction tuning technology to enable the model to better understand and perform specific tasks. User feedback (RLHF) can also be used to enhance the model's performance on specific tasks.

[0109] The above functions can also be implemented according to the following steps:

[0110] Use multimodal learning to combine text and voice information to optimize the model's ability to understand the context of long conversations, and maintain the coherence and consistency of conversations by training the model;

[0111] Build personalized user profiles based on the user's interaction history and preferences, and fine-tune the model to generate personalized answers that better meet user needs;

[0112] A dialogue state tracking mechanism is introduced to enable the model to manage multi-round dialogues.

[0113] Therefore, the intelligent customer service based on the large language model in the embodiment of the present invention has more advantages than the existing intelligent customer service, which is specifically reflected in:

[0114] Stronger conversational capabilities: The conversational capabilities of traditional intelligent customer service are mainly based on rules, while the large language model can acquire stronger conversational capabilities by learning a large amount of corpus data, and can understand more complex user questions and give more targeted answers;

[0115] Wider knowledge coverage: The knowledge base of traditional intelligent customer service is usually limited, while the large language model can continuously expand the knowledge base through continuous learning and cover a wider range of user questions;

[0116] The interactive experience is more natural: The conversational style of traditional intelligent customer service is usually mechanical, while the large language model can obtain a more natural conversational style by learning human language habits, making the user communication experience more comfortable.

[0117] like Figure 2 As shown, the embodiment of the present invention discloses a virtual human intelligent customer service system based on digital virtual technology and generative AI, and its structure includes:

[0118] An image customization module, used to formulate an image of an intelligent customer service and perform three-dimensional modeling of the intelligent customer service;

[0119] A driving module, used to drive the intelligent customer service based on skeleton binding and deformation and AI driving, including: driving the intelligent customer service for actions and expressions based on skeleton binding and deformation; driving the intelligent customer service using AI;

[0120] A rendering module, used to render the intelligent customer service;

[0121] The question-answering module is used to build an intelligent question-answering knowledge base based on the large language model and to fine-tune the large language model.

[0122] By using this system, the above-mentioned operation processing method can be executed and the corresponding technical effects can be achieved.

[0123] The embodiment of the present invention also provides a method for interaction between a user and an intelligent customer service, comprising:

[0124] Step 1: The user inputs the requirements into the system through voice or text;

[0125] Step 2: The system uses a large language model to generate corresponding emotional voice response content based on the user's emotions and input;

[0126] Step 3: Input the emotional voice reply content into the trained expression generation model capable of processing emotions to generate corresponding expression data;

[0127] Step 4: Use the redirection algorithm to drive the intelligent customer service so that the intelligent customer service can have corresponding expression changes while replying to the user.

[0128] An embodiment of the present invention also provides a computer-readable storage medium capable of implementing all the steps in the method in the above embodiment, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, all the steps in the method in the above embodiment are implemented.

[0129] The embodiment of the present invention further provides an electronic device for executing the above method, as a device for implementing the method, and its structure is as follows: Figure 3 As shown, the electronic device 300 includes: a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, such as through a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in actual applications, the transceiver 304 is not limited to one. It is understandable that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the specific structure of the electronic device 300. In other embodiments of the present application, the electronic device 300 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. Optionally, the electronic device may also include a display screen 305 for displaying images, or receiving user operation instructions when necessary.

[0130] The processor 301 is applied in the embodiment of the present application to implement the method shown in the above method embodiment. The transceiver 304 may include a receiver and a transmitter. The transceiver 304 is applied in the embodiment of the present application to implement the function of the electronic device of the embodiment of the present application communicating with other devices when executed.

[0131] Processor 301 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0132] The processor 301 may also include one or more processing units, for example, the processor 301 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU). Among them, different processing units may be independent devices or integrated into one or more processors. Among them, the controller may be the nerve center and command center of the electronic device 300. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. A memory may also be set in the processor 301 for storing instructions and data. In some embodiments, the memory in the processor 301 is a high-speed cache memory. The memory may store instructions or data that the processor 301 has just used or circulated. If the processor 301 needs to use the instruction or data again, it may be directly called from the memory. Repeated access is avoided, the waiting time of the processor 301 is reduced, and the efficiency of the system is improved.

[0133] The processor 301 can run the method provided in the embodiment of the present application. The processor 301 can include different devices. For example, when a CPU and a GPU are integrated, the CPU and the GPU can cooperate to execute the method provided in the embodiment of the present application, for example, part of the algorithm in the method is executed by the CPU, and another part of the algorithm is executed by the GPU to obtain faster processing efficiency.

[0134] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0135] The memory 303 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or may include a high-speed random access memory, or may include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0136] Optionally, the memory 303 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the method provided in any embodiment of the present application.

[0137] The memory 303 can be used to store computer executable program codes, which include instructions. The processor 301 executes various functional applications and data processing of the electronic device 300 by running the instructions stored in the memory 303. The memory 303 may include a program storage area and a data storage area. Among them, the program storage area may store the code of the operating system, application program, etc. The data storage area may store data created during the use of the electronic device 300 (such as images, videos, etc. collected by the camera application), etc.

[0138] The memory 303 may also store one or more computer programs corresponding to the method provided in the embodiment of the present application. The one or more computer programs are stored in the above-mentioned memory 303 and are configured to be executed by the one or more processors 301. The one or more computer programs include instructions, and the above-mentioned instructions can be used to execute each step in the above-mentioned corresponding embodiment.

[0139] Of course, the code of the method provided in the embodiment of the present application can also be stored in an external memory. In this case, the processor 301 can run the code of the method stored in the external memory through the external memory interface, and the processor 301 can control the running process.

[0140] The display screen 305 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 300 may include 1 or N display screens 305, where N is a positive integer greater than 1. The display screen 305 can be used to display information input by a user or information provided to a user and various graphical user interfaces (GUI). For example, the display screen 305 can display photos, videos, web pages, or files.

[0141] The electronic device provided in the embodiment of the present application is applicable to any embodiment of the above method. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, which will not be repeated here.

[0142] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0144] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in a box or multiple boxes. Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0146] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A virtual human intelligent customer service method based on digital virtual technology and generative AI, characterized in that: include: S1. Formulate an intelligent customer service image and perform three-dimensional modeling of the intelligent customer service; S2. Driving the intelligent customer service based on skeleton binding and deformation and AI driving, including: Driving the actions and expressions of the intelligent customer service based on bone binding and deformation; Using AI to drive the intelligent customer service; S3. Rendering the intelligent customer service; S4. Build an intelligent question-answering knowledge base based on the large language model, and fine-tune the large language model.

2. The method according to claim 1, characterized in that The using of AI to drive the intelligent customer service includes: Create a pre-built model to generate a synchronized data set for voice and intelligent customer service models; Using the speech and intelligent customer service model synchronization data set to train an expression generation model; Emotional latent variables are learned from the speech and intelligent customer service model synchronization data set, and the expression generation model is optimized according to the emotional latent variables.

3. The method according to claim 2, characterized in that The step of creating a pre-generated model and generating a speech and intelligent customer service synchronization data set includes: The encoder-decoder network is used as the pre-generation model architecture to extract audio features; The encoder is used to learn to convert the audio features into low-frequency embeddings, and the decoder is used to map the embeddings into a high-dimensional space.

4. The method according to claim 2, characterized in that The method of using the speech and intelligent customer service model synchronization data set to train the expression generation model includes: Annotating phoneme information in the speech and intelligent customer service model synchronization data set; Use AAM algorithm to extract facial expression correlation coefficient; Train the expression generation model according to the phoneme information and the speech and intelligent customer service model synchronization data set, and output the corresponding facial expression coefficient; The corresponding facial expression coefficient is transferred to the face of the intelligent customer service for redirection.

5. The method according to claim 2, characterized in that The step of learning the emotion latent variables from the speech and intelligent customer service model synchronization data set and optimizing the expression generation model according to the emotion latent variables includes: Add frequency analysis layer, pronunciation analysis layer, and vertex output layer to the expression generation model; The frequency analysis layer includes an autocorrelation layer and a convolution layer, in which the autocorrelation coefficient of the audio is extracted using linear prediction coding to obtain an audio feature map, and the audio feature map is convoluted to obtain an audio feature vector; Learning emotional latent variables to form an emotional state matrix, extracting correlation features of adjacent sequence frames in the pronunciation analysis layer, selecting any set of vectors of the emotional state matrix to input into the pronunciation analysis layer, and outputting a feature vector including audio features and emotional features; The vertex output layer includes two fully connected layers, wherein the mapping from audio features to facial expression coefficients is obtained in the first fully connected layer, and the mapping from the facial expression coefficients to the coordinate values ​​of each vertex is obtained in the second fully connected layer.

6. The method according to claim 1, characterized in that The step of building an intelligent question-answering knowledge base based on a large language model and fine-tuning the large language model includes: Collect professional knowledge, integrate the professional knowledge into question-answer pairs, and build an intelligent question-answer knowledge base; Use knowledge-enhanced pre-training to embed the expertise into a large language model; The large language model is fine-tuned by instruction fine-tuning technology.

7. A virtual human intelligent customer service system based on digital virtual technology and generative AI, characterized in that: include: An image customization module, used to formulate an image of an intelligent customer service and perform three-dimensional modeling of the intelligent customer service; A driving module, used to drive the intelligent customer service based on skeleton binding and deformation and AI driving, including: driving the intelligent customer service for actions and expressions based on skeleton binding and deformation; driving the intelligent customer service using AI; A rendering module, used to render the intelligent customer service; The question-answering module is used to build an intelligent question-answering knowledge base based on the large language model and to fine-tune the large language model.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store program instructions; The processor is configured to execute the method according to any one of claims 1 to 6 by calling program instructions.

10. A computer program product comprising a computer program and / or instructions, characterized in that: When the computer program and / or the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Virtual digital human-based interaction processing method, system, terminal, equipment and medium

    CN117520498A