Live broadcast digital human interaction problem processing method and computing device

Abstract

The embodiment of the invention provides a live broadcast digital human interaction problem processing method and computing equipment. The live broadcast digital human interaction problem processing method comprises the following steps: obtaining a problem representation vector; performing clustering analysis on the problem representation vector to obtain a plurality of clusters; determining a typical interaction problem corresponding to each cluster based on the clustering problem representation vector of each cluster; processing the typical interaction problem to obtain a corresponding solution audio clip, and obtaining a mouth shape change matrix sequence based on the solution audio clip; generating corresponding solution video frames based on the mouth shape change matrix and the live broadcast digital human source image, and forming a solution video clip by the solution video frames based on a timestamp of the mouth shape change matrix; and sending the solution audio clip and the solution video clip to the user terminal playing the digital human commodity live broadcast. By adopting the scheme of the embodiment of the invention, the common problem of most audience relationships can be replied in a live broadcast playing mode, so that the user can pay more attention to the live broadcast content of the live broadcast digital person.

Description

Technical Field

[0001] The present disclosure relates to the field of digital live broadcast technology, and in particular to a method for processing live broadcast digital human interaction problems and a computing device. Background Art

[0002] In order to achieve low-cost, all-weather, full-product live broadcast display, the field of product network live broadcast has begun to adopt live digital human technology based on large model technology, using live digital humans to introduce and explain product information. The live broadcast process of live digital humans mainly includes two parts: product introduction and user question answering. The product introduction part mainly introduces product performance, price, etc., which can be analyzed according to pre-set scripts, so the generation of this part of the content is relatively reasonable; and the audience question answering part is aimed at real-time questions raised by a large number (even massive) of users, which is not easy to solve. Most of the existing solutions are to send the answer content of the corresponding question to a single user through semantic analysis and small window text answers. Users need to pay attention to the small window while watching the live video, and cannot get reasonable responses to the questions in the live video. Summary of the invention

[0003] The embodiments of the present disclosure provide a method and a computing device for processing live broadcast digital human interaction problems.

[0004] In a first aspect, the present disclosure provides a method for processing a live broadcast digital human interaction problem, comprising:

[0005] Performing semantic vectorization processing on cached interactive questions received during the live broadcast of commodities by the live broadcast digital human, obtaining a question representation vector and buffering and storing the question representation vector;

[0006] Performing cluster analysis on multiple problem representation vectors stored in the buffer to obtain multiple cluster clusters; based on the cluster problem representation vectors of each cluster cluster, determining the typical interaction problems corresponding to each cluster cluster in the cache interaction problem;

[0007] Processing the typical interactive questions to obtain corresponding answer audio clips, and obtaining a lip shape change matrix sequence and a timestamp of each lip shape change matrix in the lip shape change matrix sequence based on the answer audio clips;

[0008] Generate corresponding answer video frames based on the lip shape change matrix in the lip shape change matrix sequence and the live digital human source image, and combine the answer video frames into answer video segments based on the timestamp of the lip shape change matrix;

[0009] The answer audio segment and the answer video segment are sent to the user terminal that plays the live broadcast of the digital human product.

[0010] Optionally, the determining of typical interaction problems corresponding to each cluster in the cache interaction problem based on the cluster problem representation vector of each cluster includes:

[0011] Select the corresponding target feature vector in each cluster respectively;

[0012] Determine the problem representation vector closest to the target feature vector, and take the interaction problem corresponding to the closest problem representation vector as the typical interaction problem;

[0013] The processing of the typical interactive questions to obtain corresponding answer audio clips includes: processing the typical interactive questions corresponding to each cluster respectively to obtain corresponding answer texts;

[0014] An answer audio segment is generated based on the answer text and the pronunciation characteristics of the digital live broadcaster.

[0015] Optionally, the typical interaction problems corresponding to each cluster are processed separately to obtain corresponding answer texts, including:

[0016] Perform vector database search based on the question representation vector corresponding to the typical interactive question to determine whether the search text is obtained;

[0017] When the search text is obtained, the search text is used as the answer text;

[0018] And when the search text is not obtained, a large language model for question answering is used to process typical interactive questions, corresponding model output text is obtained, and the model output text is used as the answer text.

[0019] Optionally, the semantic vectorization processing of the interactive questions received during the live broadcast of the commodity by the live digital human includes:

[0020] Obtaining context questions associated with each interactive question, where the context questions are questions from the same source as the interactive questions;

[0021] Combining the interaction question and the associated context question to obtain a fused question, and performing semantic vectorization processing on the fused question;

[0022] The determining of typical interaction problems corresponding to each cluster in the cache interaction problem includes: determining typical interaction problems corresponding to each cluster in the fusion problem.

[0023] Optionally, before sending the answer audio segment and the answer audio segment to the user terminal playing the live broadcast of the digital human commodity, the method further includes:

[0024] Generate a topic conversion text according to the audio and video content currently played by the user terminal, and the answer text corresponding to the typical interactive question and / or the answer audio clip;

[0025] Processing the topic conversion text to obtain a topic conversion audio clip, and obtaining a lip shape transformation matrix sequence corresponding to the topic conversion based on the topic conversion audio clip, and a timestamp of each lip shape transformation matrix in the lip shape transformation matrix sequence;

[0026] Generate corresponding topic conversion video frames based on the lip conversion matrix in the lip conversion matrix sequence and the live digital human source image, and combine the topic conversion video frames into topic conversion video clips based on the timestamp of the corresponding lip conversion matrix;

[0027] The topic conversion audio segment and the topic conversion video segment are sent to the user terminal that plays the live broadcast of the digital human commodity.

[0028] Optionally, the step of generating a corresponding answer video frame based on the lip shape change matrix in the lip shape change matrix sequence and the live digital human source image includes:

[0029] Process the head pixel area of ​​the person in the live digital human source image to obtain the facial key point pixels and key point feature matrix;

[0030] Obtaining a key point change matrix based on each lip change matrix in the lip change matrix sequence and the key point feature matrix;

[0031] The positions of the facial key point pixels are adjusted based on the key point change matrix to obtain corresponding answer video frames.

[0032] Optionally, the method further includes: determining matching expression features according to the type of live commodity in the commodity live broadcast, and corresponding to an emotion type matrix according to the expression features;

[0033] The obtaining of the key point change matrix based on each lip shape change matrix in the lip shape change matrix sequence and the key point feature matrix includes: obtaining the key point change matrix based on the lip shape change matrix, the key point feature matrix and the emotion type matrix.

[0034] Optionally, the method further includes: determining an associated introduction operation action according to the answer text content associated with the answer audio clip, wherein the introduction operation action is a body movement for introducing the live broadcast product during the live broadcast of the product;

[0035] Generate a corresponding product dynamic display video frame and a corresponding timestamp according to the introduction operation action and the 3D structure of the live product;

[0036] Determine the head posture change matrix sequence and the body movement transformation matrix sequence according to the introduction operation action and the position of the live broadcast product relative to the broadcast digital human;

[0037] Determine a limb action sequence according to the body action transformation matrix sequence and the initial limb image of the live digital human, and generate limb part video frames and corresponding timestamps based on the limb action sequence;

[0038] The step of obtaining a key point change matrix based on each lip shape change matrix in the lip shape change matrix sequence and the key point feature matrix comprises: obtaining the key point change matrix based on the lip shape change matrix, the key point feature matrix and the head posture change matrix in the head posture change matrix sequence;

[0039] The method of adjusting the facial key point pixels of the character based on the key point change matrix to obtain the corresponding answer video frame includes: adjusting the facial key points based on the key point change matrix to obtain the corresponding facial video frame and the corresponding timestamp; splicing and layer merging the facial video frames, body part video frames and product dynamic display video frames with the same timestamp to obtain the corresponding answer video frame.

[0040] In a second aspect, an embodiment of the present disclosure provides a digital human live broadcast interaction device, comprising:

[0041] A question processing unit, used for semantically vectorizing the interactive questions received during the live broadcast of the commodity by the live digital human, obtaining a question representation vector and buffering and storing the question representation vector;

[0042] A typical problem determination unit is used to perform cluster analysis on multiple problem representation vectors stored in the buffer to obtain multiple cluster clusters; based on the cluster problem representation vectors of each cluster cluster, determine the typical interaction problem corresponding to each cluster cluster in the cache interaction problem;

[0043] An audio and video generation unit is used to process the typical interactive questions to obtain corresponding answer audio clips, and obtain a lip shape change matrix sequence and a timestamp of each lip shape change matrix in the lip shape change matrix sequence based on the answer audio clips; and to generate corresponding answer video frames based on the lip shape change matrices in the lip shape change matrix sequence and a live digital human source image, and to combine the answer video frames into answer video clips based on the timestamps of the lip shape change matrices;

[0044] The data sending unit is used to send the answer audio segment and the answer video segment to the user terminal playing the live broadcast of the digital human product.

[0045] In a third aspect, an embodiment of the present disclosure provides a computing device, including a processor and a memory, wherein the memory is used to store a computer program; when the computer program is loaded by the processor, the processor executes the live digital human interaction problem processing method as described above.

[0046] In order to be able to answer the questions that most viewers are concerned about in the form of audio and video playback during the live broadcast, the embodiment of the present disclosure performs cluster analysis on the question representation vectors corresponding to various cached interactive questions, and obtains typical interactive questions among all cached interactive questions through cluster clusters. Subsequently, the computing device generates an audio clip and a video clip for the typical interactive question, and then sends the aforementioned audio and video clip and the video clip for the answer to the user terminal. By adopting the embodiment of the present disclosure, the common questions of most viewers can be answered through live broadcast, so that users can pay more attention to the live content of the live digital human. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work, including:

[0049] Figure 1 is a flow chart of a method for processing live broadcast digital human interaction problems provided by an embodiment of the present disclosure;

[0050] Figure 2 is a flow chart of a method for determining typical interaction problems in some embodiments;

[0051] Figure 3 is a flowchart of some embodiments to obtain answers to speech fragments;

[0052] Figure 4 is a flow chart of a method for generating a video frame of an answer in some embodiments;

[0053] Figure 5 It is a structural schematic diagram of a digital live broadcast human interaction device provided by an embodiment of the present disclosure;

[0054] Figure 6 It is a schematic diagram of the structure of a computing device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0055] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0056] The term "including" and its variations used in this document are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. In this document, relational terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0057] In order to solve the problem that it is difficult for existing live broadcast digital humans to directly answer audience questions through live broadcast videos, the embodiments of the present disclosure provide a new method for processing live broadcast digital human interaction problems. The method for processing live broadcast digital human interaction problems provided by the embodiments of the present disclosure is executed by a computing device that deploys a corresponding software system. In a specific implementation, the computing device can be a device with large computing power such as a server.

[0058] Figure 1 : is a flow chart of a method for processing live broadcast digital human interaction problems provided by an embodiment of the present disclosure. Figure 1 As shown, the method for handling live digital human interaction problems provided by the embodiment of the present disclosure includes S110-S150.

[0059] S110: semantically vectorize the interactive questions received during the live broadcast of the product by the live digital human, obtain the question representation vector, and buffer and store the question representation vector.

[0060] When a digital person is live streaming or explaining a product, he or she will provide the audience with a large amount of information about product features, product prices, and product services. Since the aforementioned information is only macro information and may not involve product details or the focus of some viewers, in this case the audience may have some questions they want to consult or ask. At this time, the audience will enter interactive questions in the terminal device through text input or voice input, and the aforementioned interactive questions will be sent by the terminal device to the background computing device.

[0061] After receiving the aforementioned interactive questions, the background computing device will buffer and store the aforementioned interactive questions for subsequent data processing and analysis. At the same time, the computing device will perform semantic vectorization on the interactive questions to obtain a question representation vector. The question representation vector is a vector that represents the semantics of the interactive question in vector form. In a specific implementation, the computing device may use a method based on a bag of words model, a word vector model, or a deep learning model to convert the interactive question into a question representation vector. Because the focus of the embodiment of the present application is not on how to realize the vectorization of the interactive question, how to vectorize the interactive question and obtain the question representation vector is not specifically analyzed here. For specific content, please refer to the prior art literature.

[0062] In some other embodiments, the computing device may use the following method to vectorize the interactive question and obtain a question representation vector.

[0063] S111: Obtain context questions associated with each interactive question, where the context questions are questions from the same source as the interactive questions.

[0064] S112: Combine the interactive question and the associated context question to obtain a fused question, and perform semantic vectorization processing on the fused question.

[0065] In some embodiments, the audience may use the method of daily interactive dialogue to say the interactive question. At this time, a single interactive question may not be able to determine the true intention of the audience, and multiple interactive questions need to be cross-fused to understand the user's intention. To solve this problem, after obtaining an interactive question, the computing device does not directly process the interactive question, but caches the interactive question. After caching for a period of time, the computing device determines the contextual question associated with it based on the source of the interactive question (specifically, it can be information such as IP address, MAC address, etc.), combines the interactive question and the associated contextual question to obtain a fusion question, and then performs semantic vectorization on the fusion question. The aforementioned combination of the interactive question and the associated contextual question to obtain a fusion question can be to combine the keywords in the interactive question and the contextual question, and use the combined question as a fusion question. In the specific implementation, the questions raised by the audience may be more verbose (that is, there is a large amount of non-critical content information), which is not conducive to subsequent semantic vectorization processing. To solve this problem, a summary extraction operation can be performed on the interactive question to obtain information such as keywords.

[0066] After obtaining the fusion problem, the computing device can use the vectorization processing method mentioned above to obtain a corresponding problem representation vector.

[0067] In the embodiment of the present disclosure, after obtaining a question representation vector corresponding to an interactive question, the question representation vector is not directly processed, but is first stored in a buffer sequence for subsequent data clustering analysis.

[0068] S120: performing cluster analysis on the multiple problem representation vectors stored in the buffer to obtain multiple clusters; based on the cluster problem representation vectors of each cluster, determining typical interaction problems corresponding to each cluster in the cache interaction problems.

[0069] In the disclosed embodiment, the computing device will read the problem representation vector from the buffer sequence and perform cluster analysis on the problem representation vector only after certain conditions are met. The aforementioned certain conditions may be: (1) the live broadcast of the digital human introducing the product is basically finished; (2) the timing starts from the time when the problem representation vector was last clustered, and the timing duration has reached the set duration; (3) the number of problem representation vectors that have been buffered has reached a threshold number, or the buffer storage area for storing the problem representation vector is full.

[0070] The computing device can use clustering methods such as K clustering analysis to perform cluster analysis on the problem representation vector to obtain multiple cluster clusters. In the specific implementation, a reasonable cluster number hyperparameter is set on the premise of obtaining a reasonable cluster cluster. In the specific implementation, the following method can be used to determine the cluster number hyperparameter: (1) determine the cluster number hyperparameter according to the type of question that the audience may mention; (2) make the cluster hyperparameter gradually increase from small to large, determine the cluster clusters under each cluster number hyperparameter, and calculate the cluster error and value under the cluster cluster, and then determine the number of clusters that have elbow cluster error and value based on the cluster error, and use the aforementioned cluster number as the final determined cluster number.

[0071] After all the problem representation vectors are clustered to obtain multiple clusters, the problem representation vectors assigned to each cluster are determined. Then, the cache interaction problems corresponding to the problem representation vectors clustered into a cluster are selected to obtain the typical interaction problems corresponding to the cluster.

[0072] Figure 2 is a flow chart of a method for determining typical interaction problems in some embodiments. Figure 2 As shown, in one embodiment, after obtaining each cluster, the following S121-S122 may be used to determine typical interaction problems.

[0073] S121: Select the corresponding target feature vector in each cluster.

[0074] The target feature vector is a vector selected from all problem representation vectors in a cluster that can represent the specific cluster. In a specific implementation, the target feature vector can be a feature vector corresponding to the center position of the cluster, or a feature vector corresponding to the center position of the cluster and a preset number of edge positions. The aforementioned edge positions are evenly distributed edge positions, and should not be clustered edge positions.

[0075] S122: Determine the problem representation vector closest to the target feature vector, and take the interactive problem corresponding to the closest problem representation vector as the typical interactive problem.

[0076] After determining the aforementioned target feature vector, all the question representation vectors in the cluster can then be screened to determine the question representation vector closest to the target feature vector. After determining the question representation vector, the interactive question corresponding to the question representation vector is used as a typical interactive question. In other embodiments, reverse text conversion processing can also be performed based on the target feature vector to obtain a reverse conversion question, and the reverse conversion question is used as a typical interactive question.

[0077] As analyzed above, in some embodiments, the computing device vectorizes the fusion problem obtained based on the interaction problem to obtain a context interaction vector. Accordingly, the computing device can use the method mentioned in the previous paragraph to determine the typical interaction problem corresponding to each cluster in the fusion problem.

[0078] According to the above method, the typical interaction problems corresponding to each cluster can be obtained.

[0079] S130: Processing typical interactive questions to obtain corresponding answer audio clips, and obtaining a lip shape change matrix sequence based on the answer audio clips, as well as a timestamp of each lip shape change matrix in the lip shape change matrix sequence.

[0080] After obtaining the typical interaction question, the typical interaction question may also be processed to obtain a voice segment for answering the question.

[0081] Figure 3 Flowcharts of some embodiments for obtaining answers to speech fragments. Figure 3 As shown, the computing device may obtain the answer voice segment by using the following S131 - S132 .

[0082] S131: Process the typical interactive problems corresponding to each cluster respectively to obtain the corresponding answer text.

[0083] In a specific implementation, the computing device may adopt various existing methods for answering questions to obtain the corresponding answer text. For example, the computing device may obtain the answer text by data retrieval or by language model processing.

[0084] In some embodiments of the present disclosure, in order to maximize the accuracy of the answer text and reduce system resource overhead, the computing device obtains the answer text using the following method S1311-S1323.

[0085] S1311: Perform a vector database search based on the question representation vector corresponding to the typical interactive question to determine whether the search text is obtained; if so, execute S1312; if not, execute S1313.

[0086] S1312: Use the search text as the answer text.

[0087] S1313: Use the large language model for question answering to process typical interactive questions, obtain corresponding model output text, and use the model output text as the answer text.

[0088] The vector database is a database that is determined in advance based on various information of live broadcast products (price information, product feature information, product material information, product warranty and after-sales information, etc.), and uses vectors as search keys. In the specific implementation, after obtaining the problem representation vector corresponding to the typical interactive question, the problem representation vector is then used as the search key to perform a search to determine whether a storage vector whose Euclidean distance to the aforementioned problem representation vector is less than a preset distance threshold is retrieved; if so, the text associated with the aforementioned storage vector can be used as the search text.

[0089] If there is no such storage vector, the corresponding search text cannot be obtained. The computing device calls the pre-trained large language model, uses the large language model to process typical interactive questions to obtain the corresponding model output text, and uses the model output text as the answer text. The large language model trained in the above-mentioned training is also a model trained with question-answer text data constructed in various ways (in actual applications, the above-mentioned training is mostly based on fine-tuning training of existing large language models).

[0090] S132: Generate an answer audio clip based on the answer text and the pronunciation features of the digital live broadcaster.

[0091] An answer audio clip is generated based on the answer text and the pronunciation features of the digital live broadcaster. The corresponding voice library is determined based on the pronunciation features of the digital live broadcaster, and the answer text is used to search the aforementioned voice library to determine the pronunciation of each character in the answer text, and the aforementioned pronunciations are spliced ​​to generate an answer voice clip (in specific implementation, it may also be necessary to process the answer voice clip according to the voice intonation and legato and broken sound features of the digital live broadcaster).

[0092] In some embodiments, after obtaining the answer audio clip, the lip shape (including upper and lower lips, tongue, chin, etc.) feature matrix under the specific pronunciation feature can be determined based on the pronunciation features of each audio frame in the audio clip. The lip shape feature is compared with the reference lip shape feature matrix of the live digital human under the reference condition to obtain the lip shape change matrix. The lip shape change matrices corresponding to each audio frame are sorted to obtain a lip shape change matrix sequence. At the same time, the timestamps of the audio frames corresponding to the aforementioned lip shape change matrices will also be used as the timestamps of the lip shape change matrices.

[0093] In some other embodiments, after obtaining the answer audio clip, the computing device then uses a pre-trained lip recognition network to perform data processing on the audio frame to be played to obtain a lip change matrix. Specifically, after the computing device uses the pre-trained lip recognition network to process the audio clip to be answered and obtains the morphological features (or morphological change features) of each organ or tissue corresponding to the lip shape, the morphological features of each organ or tissue are combined according to a predetermined data organization rule to obtain the lip change matrix.

[0094] S140: Generate corresponding answer video frames based on the lip shape change matrix in the lip shape change matrix sequence and the live digital human source image, and combine the answer video frames into answer video segments based on the timestamp of the lip shape change matrix.

[0095] Figure 4 4 is a flow chart of a method for generating a video frame of an answer in some embodiments. Figure 4 As shown, in some embodiments, the computing device may adopt the following S141-S143 to generate a corresponding answer video frame based on the lip shape change matrix in the lip shape change matrix sequence and the live digital human source image.

[0096] S141: Processing the head pixel area of ​​the person in the live digital human source image to obtain facial key point pixels and key point feature matrix.

[0097] In a specific implementation, the computing device can use a pre-trained facial feature recognition network to process the live digital human source image, determine the head area of ​​the person in the source image, and process the pixel area of ​​the person's head to obtain the 3D facial features of the person. The 3D facial features of the person are features that characterize the plane position and depth information of the facial organs of the person in the two-dimensional image, which reflects the facial state of the person.

[0098] After obtaining the 3D features of the person's face, the computing device can then extract the key point features in the 3D features of the person's face to obtain key point pixels and a key point feature matrix that characterizes the relative position relationship between key point pixels. Key points are pixels that characterize the basic structure of a person's face and are not affected by changes in expression or viewing angle. For example, key point features may include pixels at the corners of the person's eyes, pixels at the base position of the corners of the mouth, pixels at the nose, and pixels at the base position of the eyebrows.

[0099] In a specific implementation, in order to obtain a key point feature matrix with reference significance, the facial expression of the person in the source image is preferably a neutral expression, so that the obtained key point feature matrix is ​​a matrix representing the neutral expression, thereby facilitating the subsequent reproduction of various types of expressions.

[0100] S142: Obtain a key point change matrix based on each lip shape change matrix and the key point feature matrix in the lip shape change matrix sequence.

[0101] Because the lip shape change matrix reflects the relative change characteristics of the key points, the key point change matrix can be obtained by adding the lip shape change matrix and the key point feature matrix. The key point change matrix represents the relative position relationship of each key pixel after superimposing the lip shape change characteristics.

[0102] S143: Adjust the positions of facial key point pixels based on the key point change matrix to obtain corresponding answer video frames.

[0103] In the specific implementation, after obtaining the lip shape change matrix, the existing restoration model such as RePosNet can be used to process the lip shape change matrix and the live digital human source image, and the pixel is reorganized based on the facial pixel features in the source image to construct the answer video frame. After obtaining each answer audio frame, the answer audio frame is assigned according to the timestamp of the corresponding lip shape change matrix, and the answer audio frames are sorted according to the timestamp, so that the answer video clip can be obtained.

[0104] S150: Sending the answer audio clip and the answer video clip to the user terminal that plays the live broadcast of the digital human product.

[0105] After obtaining the aforementioned answer audio clip and answer video clip, the aforementioned answer audio clip and answer video clip can then be written into the audio and video buffer sequence, and the computing device then sends the aforementioned answer audio clip and answer video clip in a streaming manner to the user terminal that wants to play the live broadcast of the digital human product.

[0106] As analyzed above, in order to be able to answer most audience-related questions in the form of audio and video playback during the live broadcast, the embodiment of the present disclosure performs cluster analysis on the problem representation vectors corresponding to various cached interactive questions, and obtains typical interactive questions among all cached interactive questions through cluster clusters. Subsequently, the computing device responds to the typical interactive questions to generate answer audio clips and answer video clips, and then sends the aforementioned audio and video clips to the user terminal. By adopting the embodiment of the present disclosure, most common audience-related questions can be answered through live broadcast, so that users pay more attention to the live broadcast content of the live digital human.

[0107] Of course, in specific implementations, for specific questions raised by each audience member, the computing device can also generate corresponding answer texts in a conventional manner, and return the answer texts to specific audience members through a small window, so that the audience can obtain more accurate answers.

[0108] To enhance user experience, the digital human should have certain facial features during the live broadcast of commodities, such as being relatively happy (such as live broadcast of selling various fast-moving consumer goods and daily necessities) or relatively rigorous (such as live broadcast of selling books and learning materials). To this end, in some embodiments of the present disclosure, before obtaining the key point change matrix according to the aforementioned S142, the computing device may also perform the following S160.

[0109] S160: Determine matching facial expression features according to the type of live product in the live broadcast of the product, and correspond the emotion type matrix according to the facial expression features.

[0110] In the specific implementation, the representation features of various commodities during commodity live broadcast have been predetermined. The expression features mentioned here are the expression features that the live broadcast digital human needs to maintain for a long time. After obtaining the corresponding expression features, the emotion type matrix corresponding to the expression features can be obtained by querying through the identifier of the representation features.

[0111] When executing the aforementioned S160, the aforementioned S142 is specifically: obtaining a key point change matrix based on the lip shape change matrix, the key point feature matrix and the emotion type matrix. According to the aforementioned analysis, the computing device adds the lip shape change matrix, the key point feature matrix and the emotion type matrix to obtain the key point change matrix.

[0112] Similar to live broadcasts of products by real people, when a digital human is broadcasting a product, in order to achieve a better introduction effect and answer questions in a targeted manner, the corresponding features, corresponding information or corresponding part features of the product should be displayed as much as possible while answering questions from the audience.

[0113] To solve this problem, in some embodiments of the present disclosure, before executing the aforementioned 142, the computing device may also execute the following S170-S200.

[0114] S170: Determine the associated introduction operation action according to the answer text content associated with the answer audio clip.

[0115] As analyzed above, the answer audio clip is generated based on the answer text. The corresponding answer text contains answer information for the product to be evaluated.

[0116] The aforementioned answer information is also information that explains certain features or information of the live broadcast product. For example, when the live broadcast product is a refrigerator, the answer text may be an answer to the internal volume of the refrigerator. During the live broadcast, while introducing the internal volume of the refrigerator, the anchor will open the refrigerator door based on the answer content and intuitively show the audience the internal volume of the refrigerator. In the process of performing the aforementioned operations, the anchor needs to perform corresponding introduction operation actions.

[0117] In order to enable the digital live broadcaster to show the corresponding content of the live broadcast product to the audience in the process of answering questions, just like a real person live broadcast. First, the introduction operation action corresponding to the answer text is determined based on the answer text content. The introduction operation action is the body movement of introducing the live broadcast product during the live broadcast of the product. In the specific implementation, the operation actions for different types of answer texts can be pre-designed, or the introduction action can be planned in real time based on the specific content of the introduction and the 3D structure of the live broadcast product.

[0118] S180: Generate corresponding product dynamic display video frames and corresponding timestamps according to the introduction operation actions and the 3D structure of the live broadcast product.

[0119] In the specific implementation, the corresponding product dynamic display video frame is generated according to the introduction operation action and the 3D structure of the live product. Based on the introduction operation action and the 3D structure of the live product, determine which part of the live product to perform which operation and the operation path. Then, based on the operation object and operation path in the live product, operate the 3D structure of the live product, and render some stages of the operation process to obtain the product dynamic display video frame and the corresponding timestamp.

[0120] S190: Determine the head posture change matrix sequence and the body movement transformation matrix sequence according to the introduction operation action and the position of the live broadcast product relative to the playing digital human.

[0121] The head posture transformation matrix is ​​a matrix that represents the head posture movement of a person. The head posture transformation matrix may include a head rotation matrix and a head translation matrix. The head rotation matrix represents the head rotation state, and the head translation matrix is ​​a matrix that represents the head translation state of a person. The combination of the two represents the head posture movement. The corresponding head posture change matrix sequence is a sequence obtained by sorting the head posture change matrices according to the timestamp.

[0122] The body action transformation matrix is ​​a matrix that represents the changes in the body movements of the character. In the specific implementation, the position of each body movement at a specific time is determined according to the matrix of the operation action and basic action introduced above, and the relative position of the body limbs of the character can be determined, and then the body action transformation matrix can be determined. The corresponding body action transformation matrix sequence is a sequence obtained by sorting the body action transformation matrices according to the timestamp.

[0123] S200: Determine a limb action sequence according to a body action transformation matrix sequence and an initial limb image of a live digital human, and generate limb part video frames and corresponding timestamps based on the limb action sequence.

[0124] Because the body motion transformation matrix reflects the relative change characteristics of the body's limbs and trunk, the limb motion matrix can be obtained by adding the body motion transformation matrix change matrix and the body basic matrix. The limb motion matrix represents the relative position relationship of the limbs and trunk after the body motion transformation is superimposed. In a specific implementation, a pre-trained limb motion model can be used to process the body motion change matrix and the initial limb image, determine the limb motion at each timestamp, and generate limb part video frames and corresponding timestamps according to the limb motion.

[0125] In the case of executing the aforementioned S170-S200, the aforementioned S142 is specifically: based on the mouth shape change matrix, the key point feature matrix and the head posture change matrix in the head posture change matrix sequence, the key point change matrix is ​​obtained. In a specific implementation, the mouth shape change matrix, the key point feature matrix and the head posture change matrix are added together to obtain the key point change matrix.

[0126] Correspondingly, after obtaining the key point change matrix, S143 specifically includes the following S1431-S1432.

[0127] S1431: Adjust the facial key points based on the key point change matrix to obtain corresponding facial video frames and corresponding timestamps.

[0128] The implementation process of S1431 is the same as that of the previous method, except that it only generates facial video frames and response timestamps.

[0129] S1432: Perform splicing and layer merging operations on the face video frames, body part video frames, and product dynamic display video frames with the same timestamp to obtain corresponding answer video frames.

[0130] Because the aforementioned method generates video frames for different parts of the live digital human body and live products, and in the actual live video broadcast process, three different contents need to be displayed in the same video screen. Based on this, in the embodiment of the present disclosure, the face video frames, body part video frames and product dynamic display video frames with the same timestamp are merged to obtain the answer video frame. Specifically, the aforementioned merging includes the splicing processing of non-overlapping parts, and the masking processing and layer merging processing of overlapping parts according to the order of overlap.

[0131] In a specific implementation, the aforementioned generation of facial video frames, body part video frames and product dynamic display video frames for stitching and layer merging operations can also be used to generate video frames in the process of live product introduction.

[0132] The above article introduces the method for handling interactive problems of live digital human. In actual application, before handling interactive problems, live digital human may also introduce products. In order to smoothly switch the playing video to the answer to the interactive problem, the following S310-S340 may be executed before executing the aforementioned S150.

[0133] S310: Generate a topic conversion text based on the audio and video content currently played by the user terminal and the answer text corresponding to the typical interactive question and / or answer audio clip.

[0134] The topic conversion text is used to convert the live broadcast topic from product introduction to question answering. In specific implementation, the topic conversion text can be determined according to the response script, combined with the live broadcast product information introduced in the front and the answering question in the back.

[0135] S320: Process the topic conversion text to obtain a topic conversion audio clip, and obtain a lip shape transformation matrix sequence corresponding to the topic conversion based on the topic conversion audio clip, as well as a timestamp of each lip shape transformation matrix in the lip shape transformation matrix sequence.

[0136] S330: Generate corresponding topic conversion video frames based on the lip conversion matrix in the lip conversion matrix sequence and the live digital human source image, and combine the topic conversion video frames into topic conversion video segments based on the timestamp of the corresponding lip conversion matrix.

[0137] S340: Sending the topic conversion audio clip and the topic conversion video clip to the user terminal that plays the live broadcast of the digital human product.

[0138] The implementation process of S320-S340 is the same as the implementation process of S130-S150 mentioned above, which will not be repeated here. Please refer to the previous description for details.

[0139] The foregoing article analyzes the method for handling the live broadcast digital human interaction problem in the embodiment of the present disclosure. In order to implement the aforementioned method, corresponding data preparation and model training are required before the live broadcast. The following is a brief introduction to the data preparation and model training content before the live broadcast.

[0140] In order to obtain the large language model and vector database mentioned above, it is necessary to first obtain information data about the live broadcast products. In the specific implementation, according to the given network connection, the crawler service can be used to automatically collect the relevant data of the products to be broadcast live, and then: (1) Generate corresponding questions based on the aforementioned data and construct question vectors, and then generate a vector database based on the question vectors and the corresponding live broadcast product data; (2) Perform large language model training based on the relevant data of the live broadcast products to obtain a large language model for processing question feature vectors. In the aforementioned model training process, some live broadcast words also need to be integrated into the model, such as "babies", "family members", "how many meters", etc.

[0141] In addition, in order to obtain the model used to generate facial video frames, body part video frames and product dynamic display video frames in the previous article, it is necessary to use corresponding data to train the corresponding models respectively.

[0142] The disclosed embodiment also provides a digital live broadcast human interaction device. Figure 5 Schematic diagram of the structure of the digital live broadcast human interaction device provided by the embodiment of the present disclosure. Figure 5 As shown, the digital live broadcast human interaction device 500 includes a problem processing unit 501, a typical problem determination unit 502, an audio and video generation unit 503 and a data sending unit 504.

[0143] The question processing unit 501 is used to perform semantic vectorization processing on the interactive questions received during the live broadcast of the commodity by the live digital human, obtain the question representation vector and buffer and store the question representation vector.

[0144] The typical problem determination unit 502 is used to perform cluster analysis on the multiple problem representation vectors stored in the buffer to obtain multiple cluster clusters; based on the cluster problem representation vectors of each cluster cluster, determine the typical interactive problem corresponding to each cluster cluster in the cached interactive problems.

[0145] The audio and video generation unit 503 is used to process typical interactive questions to obtain corresponding answer audio clips, and obtain a lip change matrix sequence and the timestamp of each lip change matrix in the lip change matrix sequence based on the answer audio clips; and, generate corresponding answer video frames based on the lip change matrices in the lip change matrix sequence and the live digital human source image, and combine the answer video frames into answer video clips based on the timestamps of the lip change matrices.

[0146] The data sending unit 504 is used to send the answer audio segment and the answer video segment to the user terminal playing the live broadcast of the digital human product.

[0147] In some embodiments, the typical question determination unit 502 selects the corresponding target feature vector in each cluster respectively; then determines the question representation vector closest to the target feature vector, and takes the interactive question corresponding to the nearest question representation vector as the typical interactive question. Accordingly, the audio and video generation unit 503 processes the typical interactive questions corresponding to each cluster respectively to obtain the corresponding answer text; then generates the answer audio clip based on the answer text and the pronunciation characteristics of the digital live broadcaster.

[0148] In some embodiments, the audio and video generation unit 503 performs a vector database search based on the question representation vector corresponding to the typical interactive question to determine whether a search text is obtained; if a search text is obtained, the search text is used as the answer text; and if a search text is not obtained, a large language model for question answering is used to process the typical interactive question, and the corresponding model output text is obtained, and the model output text is used as the answer text.

[0149] In some embodiments, the question processing unit 501 obtains context questions associated with each interactive question, wherein the context questions are questions from the same source as the interactive question; combines the interactive question and the associated context questions to obtain a fused question, and performs semantic vectorization on the fused question. Accordingly, the typical question determination unit 502 determines the typical interactive questions corresponding to each cluster in the fused question.

[0150] In some embodiments, before sending the answer audio segment and the answer audio segment to the user terminal playing the digital human commodity live broadcast, the audio and video generation unit 503 generates a topic conversion text according to the audio and video content currently played by the user terminal, and the answer text corresponding to the typical interactive question and / or the answer audio segment; then processes the topic conversion text to obtain the topic conversion audio segment, obtains the lip shape transformation matrix sequence corresponding to the topic conversion based on the topic conversion audio segment, and the timestamp of each lip shape transformation matrix in the lip shape transformation matrix sequence; generates the corresponding topic conversion video frame based on the lip shape transformation matrix in the lip shape transformation matrix sequence and the live digital human source image, and composes the topic conversion video frame into a topic conversion video segment based on the timestamp of the corresponding lip shape transformation matrix. Correspondingly, the data sending unit 504 sends the topic conversion audio segment and the topic conversion video segment to the user terminal playing the digital human commodity live broadcast.

[0151] In some embodiments, the audio and video generation unit 503 processes the pixel area of ​​the character's head in the live digital human source image to obtain facial key point pixels and a key point feature matrix; then obtains a key point change matrix based on each lip change matrix and a key point feature matrix in a lip change matrix sequence; finally, adjusts the position of the facial key point pixels based on the key point change matrix to obtain a corresponding answer video frame.

[0152] In some embodiments, the audio and video generation unit 503 determines the matching expression features according to the type of live product in the live broadcast of the product, and corresponds to the emotion type matrix according to the expression features; then obtains the key point change matrix based on the lip shape change matrix, the key point feature matrix and the emotion type matrix.

[0153] In some embodiments, the audio and video generation unit 503 determines the associated introduction operation action according to the answer text content associated with the answer audio clip, where the introduction operation action is the body movement of introducing the live broadcast product during the live broadcast of the product; generates the corresponding product dynamic display video frame and the corresponding timestamp according to the introduction operation action and the 3D structure of the live broadcast product; determines the head posture change matrix sequence and the body movement transformation matrix sequence according to the introduction operation action and the position of the live broadcast product relative to the playing digital human; and determines the body movement sequence according to the body movement transformation matrix sequence and the initial body image of the live broadcast digital human, and generates body part video frames and corresponding timestamps based on the body movement sequence;

[0154] The aforementioned scheme of obtaining the key point change matrix based on each lip change matrix and the key point feature matrix in the lip change matrix sequence is specifically as follows: obtaining the key point change matrix based on the lip change matrix, the key point feature matrix and the head posture change matrix in the head posture change matrix sequence; the aforementioned scheme of adjusting the facial key point pixels of the character based on the key point change matrix to obtain the corresponding answer video frame is specifically as follows: adjusting the facial key points based on the key point change matrix to obtain the corresponding facial video frame and the corresponding timestamp; splicing and layer merging operations are performed on the facial video frames, body part video frames and product dynamic display video frames with the same timestamp to obtain the corresponding answer video frame.

[0155] The embodiment of the present disclosure also provides a computing device for implementing the aforementioned method. Figure 6 Schematic diagram of the structure of the computing device provided by the embodiment of the present disclosure. Figure 6 , which shows a schematic diagram of the structure of a computing device 600 suitable for implementing the embodiments of the present disclosure. Figure 6 The computing device shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present disclosure.

[0156] like Figure 6 As shown, the computing device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory ROM 602 or a program loaded from a storage device 608 to a random access memory RAM 603. Various programs and data required for the operation of the computing device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output I / O interface 605 is also connected to the bus 604.

[0157] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a camera, a microphone, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication devices 609 may allow the computing device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The computing device 600 is shown with various devices, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0158] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0159] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable storage medium, a computer-readable signal medium, or any combination of the above two.

[0160] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0161] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer readable program codes. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0162] In some embodiments, the client and system can communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), internets (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

[0163] The computer-readable medium may be included in the computing device, or may exist independently without being incorporated into the computing device.

[0164] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the tester's computer, partially on the tester's computer, as a separate software package, partially on the tester's computer and partially on a remote computer, or entirely on a remote computer or system. In cases involving a remote computer, the remote computer may be connected to the tester's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0165] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system based on hardware that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0166] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The name of the unit does not, in some cases, constitute a limitation on the unit itself. The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0167] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for solving the problem of live broadcast digital human interaction, characterized in that: include: Performing semantic vectorization processing on cached interactive questions received during the live broadcast of commodities by the live broadcast digital human, obtaining a question representation vector and buffering and storing the question representation vector; Performing cluster analysis on multiple problem representation vectors stored in the buffer to obtain multiple cluster clusters; based on the cluster problem representation vectors of each cluster cluster, determining the typical interaction problems corresponding to each cluster cluster in the cache interaction problem; Processing the typical interactive questions to obtain corresponding answer audio clips, and obtaining a lip shape change matrix sequence and a timestamp of each lip shape change matrix in the lip shape change matrix sequence based on the answer audio clips; Generate corresponding answer video frames based on the lip shape change matrix in the lip shape change matrix sequence and the live digital human source image, and combine the answer video frames into answer video segments based on the timestamp of the lip shape change matrix; The answer audio segment and the answer video segment are sent to the user terminal that plays the live broadcast of the digital human product.

2. The method according to claim 1, characterized in that The clustering problem representation vectors of the clustering clusters are used to determine the typical interaction problems corresponding to the clustering clusters in the cache interaction problem, including: Select the corresponding target feature vector in each cluster respectively; Determine the problem representation vector closest to the target feature vector, and take the interaction problem corresponding to the closest problem representation vector as the typical interaction problem; The processing of the typical interactive questions to obtain corresponding answer audio clips includes: processing the typical interactive questions corresponding to each cluster respectively to obtain corresponding answer texts; An answer audio segment is generated based on the answer text and the pronunciation characteristics of the digital live broadcaster.

3. The method according to claim 2, characterized in that The typical interactive problems corresponding to each cluster are processed respectively to obtain corresponding answer texts, including: Perform vector database search based on the question representation vector corresponding to the typical interactive question to determine whether the search text is obtained; When the search text is obtained, the search text is used as the answer text; And when the search text is not obtained, a large language model for question answering is used to process typical interactive questions, corresponding model output text is obtained, and the model output text is used as the answer text.

4. The method according to claim 3, characterized in that The semantic vectorization processing of the interactive questions received during the live broadcast of the commodity by the live broadcast digital human includes: Obtaining context questions associated with each interactive question, where the context questions are questions from the same source as the interactive questions; Combining the interaction question and the associated context question to obtain a fused question, and performing semantic vectorization processing on the fused question; The determining of typical interaction problems corresponding to each cluster in the cache interaction problem includes: determining typical interaction problems corresponding to each cluster in the fusion problem.

5. The method according to any one of claims 1 to 4, characterized in that: Before sending the answer audio segment and the answer audio segment to the user terminal playing the live broadcast of the digital human commodity, the method further includes: Generate a topic conversion text according to the audio and video content currently played by the user terminal, and the answer text corresponding to the typical interactive question and / or the answer audio clip; Processing the topic conversion text to obtain a topic conversion audio clip, and obtaining a lip shape transformation matrix sequence corresponding to the topic conversion based on the topic conversion audio clip, and a timestamp of each lip shape transformation matrix in the lip shape transformation matrix sequence; Generate corresponding topic conversion video frames based on the lip conversion matrix in the lip conversion matrix sequence and the live digital human source image, and combine the topic conversion video frames into topic conversion video clips based on the timestamp of the corresponding lip conversion matrix; The topic conversion audio segment and the topic conversion video segment are sent to the user terminal that plays the live broadcast of the digital human commodity.

6. The method according to any one of claims 1 to 4, characterized in that: The generating of the corresponding answer video frame based on the lip shape change matrix in the lip shape change matrix sequence and the live digital human source image comprises: Process the head pixel area of ​​the person in the live digital human source image to obtain the facial key point pixels and key point feature matrix; Obtaining a key point change matrix based on each lip change matrix in the lip change matrix sequence and the key point feature matrix; The positions of the facial key point pixels are adjusted based on the key point change matrix to obtain corresponding answer video frames.

7. The method according to claim 6, characterized in that Also includes: Determine matching expression features according to the type of live product in the live broadcast of the product, and correspond the emotion type matrix according to the expression features; The obtaining of the key point change matrix based on each lip shape change matrix in the lip shape change matrix sequence and the key point feature matrix includes: obtaining the key point change matrix based on the lip shape change matrix, the key point feature matrix and the emotion type matrix.

8. The method according to claim 6, characterized in that Also includes: Determining an associated introduction operation action according to the answer text content associated with the answer audio clip, wherein the introduction operation action is a body movement for introducing the live broadcast product during the live broadcast of the product; Generate a corresponding product dynamic display video frame and a corresponding timestamp according to the introduction operation action and the 3D structure of the live product; Determine the head posture change matrix sequence and the body movement transformation matrix sequence according to the introduction operation action and the position of the live broadcast product relative to the broadcast digital human; Determine a limb action sequence according to the body action transformation matrix sequence and the initial limb image of the live digital human, and generate limb part video frames and corresponding timestamps based on the limb action sequence; The step of obtaining a key point change matrix based on each lip shape change matrix in the lip shape change matrix sequence and the key point feature matrix comprises: obtaining the key point change matrix based on the lip shape change matrix, the key point feature matrix and the head posture change matrix in the head posture change matrix sequence; The method of adjusting the facial key point pixels of the character based on the key point change matrix to obtain the corresponding answer video frame includes: adjusting the facial key points based on the key point change matrix to obtain the corresponding facial video frame and the corresponding timestamp; splicing and layer merging the facial video frames, body part video frames and product dynamic display video frames with the same timestamp to obtain the corresponding answer video frame.

9. A digital human live interactive device, characterized in that: include: A question processing unit, used for semantically vectorizing the interactive questions received during the live broadcast of the commodity by the live digital human, obtaining a question representation vector and buffering and storing the question representation vector; A typical problem determination unit is used to perform cluster analysis on multiple problem representation vectors stored in the buffer to obtain multiple cluster clusters; based on the cluster problem representation vectors of each cluster cluster, determine the typical interaction problem corresponding to each cluster cluster in the cache interaction problem; An audio and video generation unit is used to process the typical interactive questions to obtain corresponding answer audio clips, and obtain a lip shape change matrix sequence and a timestamp of each lip shape change matrix in the lip shape change matrix sequence based on the answer audio clips; and to generate corresponding answer video frames based on the lip shape change matrices in the lip shape change matrix sequence and a live digital human source image, and to combine the answer video frames into answer video clips based on the timestamps of the lip shape change matrices; The data sending unit is used to send the answer audio segment and the answer video segment to the user terminal playing the live broadcast of the digital human product.

10. A computing device, characterized in that It comprises a processor and a memory, wherein the memory is used to store a computer program; when the computer program is loaded by the processor, the processor executes the method for processing live digital human interaction problems as described in any one of claims 1-8.

Images