Digital human live broadcast method, apparatus and device, and storage medium

Abstract

The invention discloses a digital human live broadcast method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: carrying out the matching in a preset vector database based on a keyword vector of a to-be-live broadcast commodity, obtaining keyword matching information, and generating a copywriting based on the keyword matching information and a preset prompt word template; generating a first voice stream based on the copywriting by using a voice synthesis model, generating a first video frame based on the first voice stream and the initial digital human image by using a digital human model, and sending the first voice stream and the first video frame to a display device; and judging whether the user comment meets a preset condition, if so, performing matching based on a comment vector of the user comment, processing the obtained comment matching information and the user comment to obtain a comment answer, generating a second voice stream based on the comment answer by using a voice synthesis model, and generating a second video frame based on the second voice stream by using a digital human model. Therefore, the digital human live broadcast efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a digital human live broadcast method, device, equipment and storage medium. Background Art

[0002] With the development of artificial intelligence technology, especially the breakthrough in deep learning, emerging technologies such as virtual reality (VR) and augmented reality (AR) have gradually matured, providing a solid foundation for the research and development of digital humans. However, existing digital human solutions still have some limitations, such as stiff movements and unnatural expressions, which directly affect the user experience.

[0003] From the above, it can be seen that how to improve the efficiency of digital human live broadcast during the digital human live broadcast process is a problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a digital human live broadcast method, device, equipment and storage medium, which can improve the efficiency of digital human live broadcast and enhance the harmony of digital human live broadcast. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a digital human live broadcast method, comprising:

[0006] Encode the keywords corresponding to the products to be broadcast live into keyword vectors, and match them in a preset vector database based on the keyword vectors to obtain keyword matching information, and then use the preset copy to generate a large model and generate copy based on the keyword matching information and the corresponding preset prompt word template;

[0007] Generate a first voice stream based on the text using a speech synthesis model, then generate a first video frame based on the first voice stream and an initial digital human image corresponding to a preset anchor image video using a digital human model, and send the first voice stream and the first video frame to a display device;

[0008] Using a preset comment judgment model to judge whether the user comment meets a preset condition, if the user comment meets the preset condition, encoding the user comment into a comment vector, and matching the comment vector in the preset vector database to obtain comment matching information;

[0009] The comment matching information and the user comment are processed by using a preset answer generation model to obtain a comment answer, and a second voice stream is generated based on the comment answer using the speech synthesis model. Then, a second video frame is generated based on the second voice stream using the digital human model, so that the second video frame is sent to the display device for display.

[0010] Optionally, before encoding the keywords corresponding to the to-be-live-broadcasted product into keyword vectors and matching in a preset vector database based on the keyword vectors, the method further includes:

[0011] Generate a question library to be encoded based on the product information corresponding to the product to be live-broadcasted, historical live-broadcast questions, and answers corresponding to the historical live-broadcast questions; the historical live-broadcast questions include the product origin, product ingredients, and product usage methods of the product to be live-broadcasted;

[0012] The content in the to-be-encoded question library is encoded into data vectors using a preset encoding model, and each of the data vectors is stored in a preset vector database.

[0013] Optionally, the keyword corresponding to the product to be broadcast live is encoded into a keyword vector, and matching is performed in a preset vector database based on the keyword vector to obtain keyword matching information, and then a large model is generated using a preset copy and a copy is generated based on the keyword matching information and a corresponding preset prompt word template, including:

[0014] Encode the keywords corresponding to the product to be broadcast live using the preset encoding model to obtain a keyword vector, and match the keyword vector with the preset vector database to obtain keyword matching information;

[0015] Based on user needs, target style prompt words are obtained from preset prompt word templates, and then a large model is generated using preset texts, and texts corresponding to the target style are generated based on the keyword matching information and the target style prompt words.

[0016] Optionally, the step of generating a first voice stream based on the text using a speech synthesis model, and then generating a first video frame based on the first voice stream and an initial digital human image corresponding to a preset anchor image video using a digital human model includes:

[0017] Generate a speech synthesis model based on the preset host audio using a preset voiceprint cloning technology, and process the text using the speech synthesis model to obtain a first voice stream;

[0018] A digital human model is generated based on a preset host image video using a preset digital human algorithm, and the first voice stream and an initial digital human image corresponding to the preset host image video are encoded using the digital human model to obtain a first video frame.

[0019] Optionally, the step of generating a digital human model based on a preset host image video using a preset digital human algorithm, and encoding the first voice stream and an initial digital human image corresponding to the preset host image video using the digital human model to obtain a first video frame includes:

[0020] Using a preset digital human algorithm and a preset network, and based on a preset anchor image video, a digital human model is generated, so as to use a preset temporal attention mechanism in the digital human model to process a scene matrix corresponding to the preset anchor image video, and obtain a scene processing result;

[0021] Processing the initial digital human image in the preset anchor image video based on the preset multi-objective loss function in the digital human model to obtain a loss function result;

[0022] Processing the first to-be-processed digital human image by using a preset micro-expression generation model in the digital human model to obtain a micro-expression generation result;

[0023] Using a preset global feature extractor in the digital human model to extract features from the preset anchor image video, obtaining a first feature extraction result; the first feature extraction result is a modeling result corresponding to the action and expression of the initial digital human image;

[0024] The preset local feature extractor in the digital human model is used to extract features from the preset anchor image video to obtain a second feature extraction result; the second feature extraction result is a modeling result corresponding to the limbs and face of the initial digital human image;

[0025] The second digital human image to be processed is processed using a preset collaborative constraint mechanism based on the micro-expression generation result, the first feature extraction result, the loss function result and the second feature extraction result to obtain a target digital human image, and a first video frame is generated based on the scene processing result, the first voice stream and the target digital human image.

[0026] Optionally, the using a preset comment judgment model to judge whether the user comment meets a preset condition, and if the user comment meets the preset condition, encoding the user comment into a comment vector, includes:

[0027] Using a preset comment reading interface on the display device to read the current user's comments, and performing semantic analysis on the current user's comments to obtain a semantic analysis result;

[0028] Using a preset comment judgment model to judge whether the semantic analysis result is related to the product to be broadcast live, if the semantic analysis result is not related to the product to be broadcast live, then not answering the user comment corresponding to the semantic analysis result;

[0029] If the semantic analysis result is related to the product to be broadcast live, the preset coding model is used to vectorize the user comments corresponding to the semantic analysis result to obtain a comment vector.

[0030] Optionally, the matching in the preset vector database based on the comment vector to obtain comment matching information includes:

[0031] Perform similarity calculation based on the comment vector calculation and all vector information in the preset vector database to obtain a corresponding similarity calculation result;

[0032] The similarity calculation results are arranged in descending order to obtain a similarity arrangement sequence, and a preset number of review matching information are extracted from the similarity arrangement sequence in a forward-to-backward order.

[0033] In a second aspect, the present application provides a digital human live broadcast device, comprising:

[0034] A copywriting generation module is used to encode the keywords corresponding to the products to be broadcast live into keyword vectors, and match them in a preset vector database based on the keyword vectors to obtain keyword matching information, and then use the preset copywriting to generate a large model and generate copywriting based on the keyword matching information and the corresponding preset prompt word template;

[0035] A first video frame generation module, configured to generate a first voice stream based on the text using a speech synthesis model, and then generate a first video frame based on the first voice stream and an initial digital human image corresponding to a preset anchor image video using a digital human model, and send the first voice stream and the first video frame to a display device;

[0036] A user comment judgment module, used to use a preset comment judgment model to judge whether the user comment meets the preset condition, if the user comment meets the preset condition, encode the user comment into a comment vector, and match it in the preset vector database based on the comment vector to obtain comment matching information;

[0037] The second video frame generation module is used to use a preset answer generation model to process the comment matching information and the user comment to obtain the comment answer, and use the speech synthesis model to generate a second voice stream based on the comment answer, and then use the digital human model to generate a second video frame based on the second voice stream, so as to send the second video frame to the display device for display.

[0038] In a third aspect, the present application provides an electronic device, including:

[0039] Memory, used to store computer programs;

[0040] The processor is used to execute the computer program to implement the aforementioned digital human live broadcast method.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned digital human live broadcast method when executed by a processor.

[0042] As can be seen from the above, before the digital human live broadcast, the present application needs to encode the keywords corresponding to the to-be-live broadcasted product into keyword vectors, and match them in the preset vector database based on the keyword vectors to obtain keyword matching information, and then use the preset copy to generate a large model and generate the copy based on the keyword matching information and the corresponding preset prompt word template; use the speech synthesis model and generate a first voice stream based on the copy, and then use the digital human model and generate a first video frame based on the first voice stream and the initial digital human image corresponding to the preset anchor image video, and send the first voice stream and the first video frame to the display device; use the preset comment judgment model to judge whether the user comment meets the preset conditions, if the user comment meets the preset conditions, then encode the user comment into a comment vector, and match it in the preset vector database based on the comment vector to obtain comment matching information; use the preset answer generation large model to process the comment matching information and the user comment to obtain the comment answer, and use the speech synthesis model to generate a second voice stream based on the comment answer, and then use the digital human model to generate a second video frame based on the second voice stream, so as to send the second video frame to the display device for display.

[0043] It can be seen that the present application first encodes the keywords corresponding to the products to be broadcast live into keyword vectors, and matches them in a preset vector database based on the keyword vectors to obtain keyword matching information, and then uses the preset text to generate a large model and generates text based on the keyword matching information and the corresponding preset prompt word template; then, a speech synthesis model is used to generate a first voice stream based on the text, and then a digital human model is used to generate a first video frame based on the first voice stream and an initial digital human image corresponding to a preset host image video, and the first voice stream and the first video frame are sent to a display device; then, a preset comment judgment model is used to determine whether the user comments meet the preset conditions. If the user comments meet the preset conditions, the user comments are encoded into comment vectors, and matched in a preset vector database based on the comment vectors to obtain comment matching information; finally, a preset answer generation large model is used to process the comment matching information and the user comments to obtain comment answers, and a second voice stream is generated based on the comment answers using the speech synthesis model, and then a second video frame is generated based on the second voice stream using the digital human model, so that the second video frame is sent to the display device for display. In this way, the efficiency of digital human live broadcast is improved, and the harmony of digital human live broadcast is enhanced, thereby enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0045] Figure 1 A flowchart of a digital human live broadcast method disclosed in this application;

[0046] Figure 2 This is a structural schematic diagram of a digital human live broadcast device disclosed in this application;

[0047] Figure 3 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] With the development of artificial intelligence technology, especially the breakthrough in the field of deep learning, emerging technologies such as virtual reality and augmented reality have gradually matured, providing a solid foundation for the research and development of digital humans. However, existing digital human solutions still have some limitations, such as stiff movements and unnatural expressions, which directly affect the user experience. To this end, this application provides a digital human live broadcast method that can improve the efficiency of digital human live broadcast and enhance the harmony of digital human live broadcast.

[0050] See also Figure 1 As shown, the embodiment of the present invention discloses a digital human live broadcast method, comprising:

[0051] Step S11, encode the keywords corresponding to the products to be broadcast live into keyword vectors, and match them in a preset vector database based on the keyword vectors to obtain keyword matching information, and then use the preset copy to generate a large model and generate copy based on the keyword matching information and the corresponding preset prompt word template.

[0052] In this embodiment, before the keywords corresponding to the products to be broadcast live are encoded into keyword vectors, and matching is performed in a preset vector database based on the keyword vectors to obtain keyword matching information, the embodiment of the present application needs to be based on the relevant information of the products to be broadcast live, and organize the common user questions to make them into a question-and-answer knowledge base. In a specific implementation, common user questions include but are not limited to the origin of the product, the ingredients of the product, and the method of using the product. Specifically, before the keywords corresponding to the products to be broadcast live are encoded into keyword vectors, and matching is performed in a preset vector database based on the keyword vectors, it can also include: generating a library of questions to be encoded based on the product information corresponding to the products to be broadcast live, historical live questions, and answers corresponding to the historical live questions; historical live questions include the origin of the products to be broadcast live, the ingredients of the products, and the method of using the products; using a preset encoding model to encode the contents in the library of questions to be encoded into data vectors, and storing each data vector in a preset vector database.

[0053] It is worth mentioning that the preset encoding model can be a BGE-M3 model (BGE-M3 Model, i.e., a general semantic vector model). Any model that can encode the content in the encoding problem library is applicable to this embodiment and will not be listed one by one here.

[0054] Further, after generating the preset vector database, the embodiment of the present application needs to use the keyword vector to match in the preset vector database to obtain keyword matching information, and then use the preset copy to generate a large model and generate copy based on the keyword matching information and the corresponding preset prompt word template, and combine the style prompt words required by the user to generate a variety of styles of copy. Specifically, the keywords corresponding to the to-be-broadcasted product are encoded as keyword vectors, and matched in the preset vector database based on the keyword vector to obtain keyword matching information, and then the preset copy is used to generate a large model and generate copy based on the keyword matching information and the corresponding preset prompt word template, which may include: using the preset encoding model to encode the keywords corresponding to the to-be-broadcasted product to obtain keyword vectors, and matching the keyword vectors with the preset vector database to obtain keyword matching information; obtaining the target style prompt words from the preset prompt word template based on user needs, and then using the preset copy to generate a large model and generate copy corresponding to the target style based on the keyword matching information and the target style prompt words. In a specific embodiment, the style of the copy includes formal style and humorous style, which are not specifically limited here.

[0055] Step S12: Generate a first voice stream using a speech synthesis model based on the text, then generate a first video frame using a digital human model based on the first voice stream and an initial digital human image corresponding to a preset host image video, and send the first voice stream and the first video frame to a display device.

[0056] In this embodiment, after obtaining the copy of a specific style, the embodiment of the present application needs to use a speech synthesis model and generate a first voice stream based on the copy. Subsequently, after obtaining the first voice stream, it is necessary to use a digital human model and generate a first video frame corresponding to the first voice stream based on the first voice stream and the initial digital human image corresponding to the preset host image video. Specifically, using a speech synthesis model and generating a first voice stream based on the copy, and then using a digital human model and generating a first video frame based on the first voice stream and the initial digital human image corresponding to the preset host image video, can include: using a preset voiceprint cloning technology and generating a speech synthesis model based on the preset host audio, so as to process the copy using the speech synthesis model to obtain the first voice stream; using a preset digital human algorithm and generating a digital human model based on the preset host image video, so as to encode the first voice stream and the initial digital human image corresponding to the preset host image video using the digital human model to obtain the first video frame. It is worth mentioning that the speech synthesis model can be CosyVoice (i.e., a multi-language large-scale speech generation model), which is not listed one by one here.

[0057] Furthermore, the digital human model is designed based on the improved UniPose (i.e. unified human pose estimation model) network, which optimizes the shortcomings of the original model in terms of limb incoordination, motion generation and expression synthesis. In addition, by introducing an improved temporal attention mechanism, loss function and feature extraction function, a more natural limb movement expression and delicate expression generation are achieved.

[0058] Specifically, using a preset digital human algorithm and a preset anchor image video to generate a digital human model, and using the digital human model to encode the first voice stream and the initial digital human image corresponding to the preset anchor image video to obtain a first video frame, can include: using a preset digital human algorithm and a preset network, and generating a digital human model based on the preset anchor image video, and using the preset temporal attention mechanism in the digital human model to process the scene matrix corresponding to the preset anchor image video to obtain a scene processing result; based on the preset multi-objective loss function in the digital human model, processing the initial digital human image in the preset anchor image video to obtain a loss function result; processing the first to-be-processed digital human image by using the preset micro-expression generation model in the digital human model to obtain a micro-expression generation result. The method comprises the following steps: generating a result; performing feature extraction on a preset anchor image video by using a preset global feature extractor in the digital human model to obtain a first feature extraction result; the first feature extraction result is a modeling result corresponding to the action and expression of the initial digital human image; performing feature extraction on the preset anchor image video by using a preset local feature extractor in the digital human model to obtain a second feature extraction result; the second feature extraction result is a modeling result corresponding to the limbs and face of the initial digital human image; processing the second digital human image to be processed by using a preset collaborative constraint mechanism and based on the micro-expression generation result, the first feature extraction result, the loss function result and the second feature extraction result to obtain a target digital human image, and generating a first video frame based on the scene processing result, the first voice stream and the target digital human image.

[0059] In a specific implementation, for the global posture of the digital human body, the embodiment of the present application uses an improved temporal attention mechanism to adjust the initial digital human image, thereby maintaining the smoothness of the movement while enhancing the scene relevance of the digital human during live broadcast. The formula expression of the improved temporal attention mechanism is as follows:

[0060] ;

[0061] in, It is a scene-related mask matrix used to enhance the temporal continuity and spatial consistency of the action; is the query matrix, which represents the information that needs attention at present; is the key matrix, which is the reference information used to match the query; is the value matrix; is the scaling factor, is the dimension of the attention head, and is used to normalize the dot product result; are the query, key, and value vectors, respectively.

[0062] In another specific implementation, the embodiment of the present application needs to design a loss function to optimize the digital human generation process using a multi-objective loss function, and the expression formula is as follows:

[0063] ;

[0064] in, To rebuild the losses, To smooth the loss, is the physical constraint loss, is the emotion consistency loss, is the weight coefficient, which can be set according to actual application requirements.

[0065] In addition, in order to more accurately capture the motion and expression features of the digital human, the embodiment of the present application designs a global feature extractor and a local feature extractor. In a specific implementation, the embodiment of the present application can use the global feature extractor to model the overall motion and expression of the digital human, and the formula is as follows:

[0066] ;

[0067] in, is the feature extraction function, For the input data, is the network parameter

[0068] In another specific implementation, in order to optimize the local details of the digital human's limbs and face, the embodiment of the present application can use a local feature extractor to further optimize the local details of the digital human's limbs and face, and the formula is as follows:

[0069] ;

[0070] in, is the local feature extractor, is the adaptive weight, is the network parameter.

[0071] Furthermore, with respect to the micro-expression details of the digital human, the embodiment of the present application uses a micro-expression generation model to optimize and adjust the micro-expression details of the digital human, and the adjustment formula is as follows:

[0072] ;

[0073] in, For Micro-expressions generated at every moment, As the basic expression unit, For real-time calculation of weight functions, It is a mixing function used to fuse multiple basic expressions of digital humans.

[0074] At the same time, the embodiment of the present application establishes a coordinated relationship between the digital human's expression and action by utilizing a collaborative constraint mechanism:

[0075] ;

[0076] in, is the feature distance metric, is the temporal consistency measure, For facial features, For action features, is the balancing factor.

[0077] Step S13: using a preset comment judgment model to judge whether the user comment meets the preset condition; if the user comment meets the preset condition, encoding the user comment into a comment vector, and matching the comment vector in the preset vector database to obtain comment matching information.

[0078] In this embodiment, during the live broadcast of the digital human, the embodiment of the present application needs to perform semantic analysis on the user comments that appear in the live comment area to determine whether the semantic analysis results obtained meet the preset conditions to exclude questions that are not related to the live broadcast. In addition, if the user comments are related to the live broadcast, the embodiment of the present application needs to vectorize and encode the relevant questions, and based on the similarity between the obtained comment vector and the vector information in the preset vector database, select several candidate answers with the highest similarity. Subsequently, customized prompt words are written based on the candidate answers and user comments, and then a more contextual answer is generated with the help of a large model, so that the digital human can promptly and accurately answer the user's comments on the product.

[0079] Specifically, a preset comment judgment model is used to judge whether the user comments meet the preset conditions. If the user comments meet the preset conditions, the user comments are encoded into a comment vector, which may include: using a preset comment reading interface on the display device to read the current user comments, and performing semantic analysis on the current user comments to obtain semantic analysis results; using the preset comment judgment model to judge whether the semantic analysis results are related to the product to be broadcast live, if the semantic analysis results are not related to the product to be broadcast live, then no reply is given to the user comments corresponding to the semantic analysis results; if the semantic analysis results are related to the product to be broadcast live, then the preset encoding model is used to vectorize the user comments corresponding to the semantic analysis results to obtain a comment vector.

[0080] In this embodiment, after obtaining the comment vector, the embodiment of the present application needs to match the comment vector in a preset vector database, and determine the comment matching information based on the similarity result obtained by calculating the similarity with the information in the preset vector database. Specifically, obtaining the comment matching information based on matching the comment vector in the preset vector database may include: calculating the similarity based on the comment vector and all the vector information in the preset vector database to obtain the corresponding similarity calculation result; arranging each similarity calculation result in order from large to small values ​​to obtain a similarity arrangement sequence, and extracting a preset number of comment matching information from the similarity arrangement sequence in order from front to back. In a specific embodiment, the preset number may be three, which is not specifically limited here.

[0081] Step S14: Use a preset answer generation model to process the comment matching information and the user comment to obtain a comment answer, and use the speech synthesis model to generate a second voice stream based on the comment answer, and then use the digital human model to generate a second video frame based on the second voice stream, so as to send the second video frame to the display device for display.

[0082] In this embodiment, after obtaining a preset number of comment matching information, the embodiment of the present application needs to use a preset answer generation model to process the comment matching information and the user comments to obtain comment answers corresponding to the user comments. Further, after obtaining the comment answers, the embodiment of the present application needs to use a speech synthesis model to generate a second voice stream based on the comment answers, and then use a digital human model to generate a second video frame based on the second voice stream, so as to send the second video frame to a display device for display, thereby improving the live broadcast quality and efficiency of the digital human.

[0083] As can be seen from the above, before the digital human live broadcast is performed in the embodiment of the present application, it is first necessary to encode the keywords corresponding to the product to be live broadcast into keyword vectors, and match them in a preset vector database based on the keyword vectors to obtain keyword matching information, and then use the preset copy to generate a large model and generate the copy based on the keyword matching information and the corresponding preset prompt word template; then, use the speech synthesis model to generate a first voice stream based on the copy, and then use the digital human model to generate a first video frame based on the first voice stream and the initial digital human image corresponding to the preset anchor image video, and send the first voice stream and the first video frame to the display device; then, use the preset comment judgment model to judge whether the user comment meets the preset conditions. If the user comment meets the preset conditions, the user comment is encoded into a comment vector, and matched in a preset vector database based on the comment vector to obtain comment matching information; finally, use the preset answer generation large model to process the comment matching information and the user comment to obtain the comment answer, and use the speech synthesis model to generate a second voice stream based on the comment answer, and then use the digital human model to generate a second video frame based on the second voice stream, so as to send the second video frame to the display device for display. In this way, the efficiency of digital human live broadcast is improved, and the harmony of digital human live broadcast is enhanced.

[0084] Accordingly, see Figure 2 As shown, the present application also provides a digital human live broadcast device, including:

[0085] The copywriting generation module 11 is used to encode the keywords corresponding to the products to be broadcast live into keyword vectors, and match them in a preset vector database based on the keyword vectors to obtain keyword matching information, and then use the preset copywriting to generate a large model and generate copywriting based on the keyword matching information and the corresponding preset prompt word template;

[0086] A first video frame generating module 12 is used to generate a first voice stream based on the text using a speech synthesis model, and then generate a first video frame based on the first voice stream and an initial digital human image corresponding to a preset anchor image video using a digital human model, and send the first voice stream and the first video frame to a display device;

[0087] A user comment judgment module 13 is used to use a preset comment judgment model to judge whether the user comment meets the preset condition. If the user comment meets the preset condition, the user comment is encoded into a comment vector, and matched in the preset vector database based on the comment vector to obtain comment matching information;

[0088] The second video frame generating module 14 is used to use a preset answer generation large model to process the comment matching information and the user comment to obtain a comment answer, and use the speech synthesis model to generate a second voice stream based on the comment answer, and then use the digital human model to generate a second video frame based on the second voice stream, so as to send the second video frame to the display device for display.

[0089] As can be seen from the above, before the digital human live broadcast is performed in the embodiment of the present application, it is first necessary to encode the keywords corresponding to the product to be live broadcast into keyword vectors, and match them in a preset vector database based on the keyword vectors to obtain keyword matching information, and then use the preset copy to generate a large model and generate the copy based on the keyword matching information and the corresponding preset prompt word template; then, use the speech synthesis model to generate a first voice stream based on the copy, and then use the digital human model to generate a first video frame based on the first voice stream and the initial digital human image corresponding to the preset anchor image video, and send the first voice stream and the first video frame to the display device; then, use the preset comment judgment model to judge whether the user comment meets the preset conditions. If the user comment meets the preset conditions, the user comment is encoded into a comment vector, and matched in a preset vector database based on the comment vector to obtain comment matching information; finally, use the preset answer generation large model to process the comment matching information and the user comment to obtain the comment answer, and use the speech synthesis model to generate a second voice stream based on the comment answer, and then use the digital human model to generate a second video frame based on the second voice stream, so as to send the second video frame to the display device for display. In this way, the efficiency of digital human live broadcast is improved, and the harmony of digital human live broadcast is enhanced, thereby enhancing the user experience.

[0090] In some specific implementations, the digital human live broadcast device may further include:

[0091] A question library generation unit to be encoded, used to generate a question library to be encoded based on the commodity information corresponding to the commodity to be broadcast live, historical broadcast questions and answers corresponding to the historical broadcast questions; the historical broadcast questions include the commodity origin, commodity ingredients and commodity usage methods of the commodity to be broadcast live;

[0092] The vector database determining unit is used to encode the contents in the to-be-encoded question library into data vectors using a preset encoding model, and store each of the data vectors in a preset vector database.

[0093] In some specific implementations, the copywriting generation module 11 may specifically include:

[0094] A keyword matching information determining unit, configured to encode the keyword corresponding to the to-be-broadcasted product by using the preset encoding model to obtain a keyword vector, and match the keyword vector with the preset vector database to obtain keyword matching information;

[0095] The copywriting generation unit is used to obtain a target style prompt word from a preset prompt word template based on user needs, and then use the preset copywriting to generate a large model and generate a copy corresponding to the target style based on the keyword matching information and the target style prompt word.

[0096] In some specific implementations, the first video frame generating module 12 may specifically include:

[0097] A voice stream generating unit, configured to generate a voice synthesis model based on a preset host audio by using a preset voiceprint cloning technology, so as to process the text by using the voice synthesis model to obtain a first voice stream;

[0098] The first video frame generating subunit is used to generate a digital human model based on a preset host image video using a preset digital human algorithm, so as to encode the first voice stream and an initial digital human image corresponding to the preset host image video using the digital human model to obtain a first video frame.

[0099] In some specific implementations, the first video frame generation module 12 may specifically include:

[0100] A scene processing result generating unit, used to generate a digital human model based on a preset host image video using a preset digital human algorithm and a preset network, and to process a scene matrix corresponding to the preset host image video using a preset temporal attention mechanism in the digital human model to obtain a scene processing result;

[0101] A loss function result generating unit, used for processing the initial digital human image in the preset anchor image video based on the preset multi-objective loss function in the digital human model to obtain a loss function result;

[0102] A micro-expression generation result generating unit, used to process the first to-be-processed digital human image through a preset micro-expression generation model in the digital human model to obtain a micro-expression generation result;

[0103] A first feature extraction result generating unit is used to extract features from the preset anchor image video using a preset global feature extractor in the digital human model to obtain a first feature extraction result; the first feature extraction result is a modeling result corresponding to the action and expression of the initial digital human image;

[0104] A second feature extraction result generating unit is used to extract features from the preset anchor image video through a preset local feature extractor in the digital human model to obtain a second feature extraction result; the second feature extraction result is a modeling result corresponding to the limbs and face of the initial digital human image;

[0105] The second video frame generating subunit is used to process the second to-be-processed digital human image by utilizing a preset collaborative constraint mechanism and based on the micro-expression generation result, the first feature extraction result, the loss function result and the second feature extraction result to obtain a target digital human image, and to generate a first video frame based on the scene processing result, the first voice stream and the target digital human image.

[0106] In some specific implementations, the user comment determination module 13 may specifically include:

[0107] A semantic analysis result determination unit, configured to read a current user comment using a preset comment reading interface on the display device, and perform semantic analysis on the current user comment to obtain a semantic analysis result;

[0108] A semantic analysis result judgment unit, used to judge whether the semantic analysis result is related to the product to be broadcast live by using a preset comment judgment model, and if the semantic analysis result is not related to the product to be broadcast live, then no reply is given to the user comment corresponding to the semantic analysis result;

[0109] The comment vector determination unit is used to use the preset coding model to vectorize the user comments corresponding to the semantic analysis result to obtain the comment vector if the semantic analysis result is related to the product to be broadcast live.

[0110] In some specific implementations, the user comment determination module 13 may specifically include:

[0111] A similarity calculation result generating unit, used to perform similarity calculation based on the comment vector and all vector information in the preset vector database to obtain a corresponding similarity calculation result;

[0112] The similarity calculation result arrangement unit is used to arrange the similarity calculation results in descending order to obtain a similarity arrangement sequence, and extract a preset number of comment matching information from the similarity arrangement sequence in a forward-to-back order.

[0113] Furthermore, the present application also discloses an electronic device. Figure 3This is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the figure cannot be regarded as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the digital human live broadcast method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0114] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0115] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0116] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the digital human live broadcast method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0117] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the digital human live broadcast method disclosed above. The specific steps of the method can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0118] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0119] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0120] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0121] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0122] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A digital human live broadcast method, characterized in that: include: Encode the keywords corresponding to the products to be broadcast live into keyword vectors, and match them in a preset vector database based on the keyword vectors to obtain keyword matching information, and then use the preset copy to generate a large model and generate copy based on the keyword matching information and the corresponding preset prompt word template; Generate a first voice stream based on the text using a speech synthesis model, then generate a first video frame based on the first voice stream and an initial digital human image corresponding to a preset anchor image video using a digital human model, and send the first voice stream and the first video frame to a display device; Using a preset comment judgment model to judge whether the user comment meets a preset condition, if the user comment meets the preset condition, encoding the user comment into a comment vector, and matching the comment vector in the preset vector database to obtain comment matching information; The comment matching information and the user comment are processed by using a preset answer generation model to obtain a comment answer, and a second voice stream is generated based on the comment answer using the speech synthesis model. Then, a second video frame is generated based on the second voice stream using the digital human model, so that the second video frame is sent to the display device for display.

2. The digital human live broadcast method according to claim 1, characterized in that: Before encoding the keywords corresponding to the to-be-broadcasted products into keyword vectors and matching them in a preset vector database based on the keyword vectors, the method further includes: Generate a question library to be encoded based on the product information corresponding to the product to be live-broadcasted, historical live-broadcast questions, and answers corresponding to the historical live-broadcast questions; the historical live-broadcast questions include the product origin, product ingredients, and product usage methods of the product to be live-broadcasted; The contents in the to-be-encoded question library are encoded into data vectors using a preset encoding model, and each of the data vectors is stored in a preset vector database.

3. The digital human live broadcast method according to claim 2, characterized in that: The method includes encoding the keywords corresponding to the commodities to be broadcast live into keyword vectors, matching the keywords in a preset vector database based on the keyword vectors to obtain keyword matching information, and then generating a large model using preset text and generating text based on the keyword matching information and the corresponding preset prompt word template, including: Encode the keywords corresponding to the product to be broadcast live using the preset encoding model to obtain a keyword vector, and match the keyword vector with the preset vector database to obtain keyword matching information; Based on user needs, target style prompt words are obtained from preset prompt word templates, and then a large model is generated using preset texts, and texts corresponding to the target style are generated based on the keyword matching information and the target style prompt words.

4. The digital human live broadcast method according to claim 2, characterized in that: The method of generating a first voice stream based on the text using a speech synthesis model, and then generating a first video frame based on the first voice stream and an initial digital human image corresponding to a preset anchor image video using a digital human model includes: Generate a speech synthesis model based on the preset host audio using a preset voiceprint cloning technology, and process the text using the speech synthesis model to obtain a first voice stream; A digital human model is generated based on a preset host image video using a preset digital human algorithm, and the first voice stream and an initial digital human image corresponding to the preset host image video are encoded using the digital human model to obtain a first video frame.

5. The digital human live broadcast method according to claim 4, characterized in that: The method of generating a digital human model based on a preset host image video by using a preset digital human algorithm, and encoding the first voice stream and an initial digital human image corresponding to the preset host image video by using the digital human model to obtain a first video frame includes: Using a preset digital human algorithm and a preset network, and based on a preset anchor image video, a digital human model is generated, so as to use a preset temporal attention mechanism in the digital human model to process a scene matrix corresponding to the preset anchor image video, and obtain a scene processing result; Processing the initial digital human image in the preset anchor image video based on the preset multi-objective loss function in the digital human model to obtain a loss function result; Processing the first to-be-processed digital human image by using a preset micro-expression generation model in the digital human model to obtain a micro-expression generation result; Using a preset global feature extractor in the digital human model to extract features from the preset anchor image video, obtaining a first feature extraction result; the first feature extraction result is a modeling result corresponding to the action and expression of the initial digital human image; The preset local feature extractor in the digital human model is used to extract features from the preset anchor image video to obtain a second feature extraction result; the second feature extraction result is a modeling result corresponding to the limbs and face of the initial digital human image; The second digital human image to be processed is processed using a preset collaborative constraint mechanism based on the micro-expression generation result, the first feature extraction result, the loss function result and the second feature extraction result to obtain a target digital human image, and a first video frame is generated based on the scene processing result, the first voice stream and the target digital human image.

6. The digital human live broadcast method according to claim 3, characterized in that: The method of using a preset comment judgment model to judge whether the user comment meets a preset condition, and if the user comment meets the preset condition, encoding the user comment into a comment vector includes: Using a preset comment reading interface on the display device to read the current user's comments, and performing semantic analysis on the current user's comments to obtain a semantic analysis result; Using a preset comment judgment model to judge whether the semantic analysis result is related to the product to be broadcast live, if the semantic analysis result is not related to the product to be broadcast live, then not answering the user comment corresponding to the semantic analysis result; If the semantic analysis result is related to the product to be broadcast live, the preset encoding model is used to vectorize the user comments corresponding to the semantic analysis result to obtain a comment vector.

7. The digital human live broadcast method according to any one of claims 1 to 6, characterized in that: The matching in the preset vector database based on the comment vector to obtain comment matching information includes: Perform similarity calculation based on the comment vector calculation and all vector information in the preset vector database to obtain a corresponding similarity calculation result; The similarity calculation results are arranged in descending order to obtain a similarity arrangement sequence, and a preset number of review matching information are extracted from the similarity arrangement sequence in a forward-to-backward order.

8. A digital human live broadcast device, characterized in that: include: A copywriting generation module is used to encode the keywords corresponding to the products to be broadcast live into keyword vectors, and match them in a preset vector database based on the keyword vectors to obtain keyword matching information, and then use the preset copywriting to generate a large model and generate copywriting based on the keyword matching information and the corresponding preset prompt word template; A first video frame generation module, configured to generate a first voice stream based on the text using a speech synthesis model, and then generate a first video frame based on the first voice stream and an initial digital human image corresponding to a preset anchor image video using a digital human model, and send the first voice stream and the first video frame to a display device; A user comment judgment module, used to use a preset comment judgment model to judge whether the user comment meets the preset condition, if the user comment meets the preset condition, encode the user comment into a comment vector, and match it in the preset vector database based on the comment vector to obtain comment matching information; The second video frame generation module is used to use a preset answer generation model to process the comment matching information and the user comment to obtain the comment answer, and use the speech synthesis model to generate a second voice stream based on the comment answer, and then use the digital human model to generate a second video frame based on the second voice stream, so as to send the second video frame to the display device for display.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the steps of the digital human live broadcast method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the steps of the digital human live broadcast method as described in any one of claims 1 to 7 are implemented.

Images