Digital human live broadcast method, device and equipment based on large model, and storage medium

Abstract

The invention provides a digital human live broadcast method, device and equipment based on a large model and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of generative large models, natural language processing, machine learning, information retrieval and the like. The specific implementation scheme is as follows: detecting interaction information for a live broadcast object in live broadcast data; inputting the interaction information for the live broadcast object into a large model to generate an interaction introduction content for the live broadcast object; wherein the interaction introduction content of the live broadcast object is used for controlling the interaction state of the digital person in the live broadcast process. According to the method and the device, the large model can generate the interaction introduction content for the live broadcast object based on the interaction information for the live broadcast object in the live broadcast data, so that the digital human can interact timely and accurately in the live broadcast process.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as generative large models, natural language processing, machine learning, and information retrieval. Background Art

[0002] Currently, live broadcast is an important new media form, and the most common form of live broadcast is real-person live broadcast. With the advent of artificial intelligence (AI), digital human live broadcast has emerged. Digital human is a digital human image close to human image created by digital technology. Digital human can also be called virtual image, digital virtual human, virtual digital human, etc. Live broadcast by digital human can be broadcast for a long time, freeing up manpower for live broadcast. However, the content of digital human live broadcast is usually fixed, making it difficult to interact flexibly with users. Summary of the invention

[0003] The present invention provides a method, device, equipment and storage medium for live broadcasting of digital human based on a large model.

[0004] According to one aspect of the present disclosure, a method for live broadcasting a digital human based on a large model is provided, comprising:

[0005] In the live broadcast data, detecting interactive information for the live broadcast object;

[0006] The interactive information for the live broadcast object is input into the big model to generate interactive introduction content for the live broadcast object; wherein the interactive introduction content for the live broadcast object is used to control the interactive state of the digital person during the live broadcast process.

[0007] According to another aspect of the present disclosure, a digital human live broadcast device based on a large model is provided, comprising:

[0008] A detection module, used to detect interactive information for a live broadcast object in live broadcast data;

[0009] The first generation module is used to input the interactive information for the live broadcast object into the large model to generate interactive introduction content for the live broadcast object; wherein the interactive introduction content of the live broadcast object is used to control the interactive state of the digital person during the live broadcast process.

[0010] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0011] at least one processor; and

[0012] a memory communicatively connected to the at least one processor; wherein,

[0013] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0014] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0015] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0016] According to the present disclosure, the big model can generate interactive introduction content for the live broadcast object based on the interactive information for the live broadcast object in the live broadcast data, so that the digital human can interact in a timely and accurate manner during the live broadcast process.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0019] Figure 1 is a flowchart of a digital human live broadcast method based on a large model according to an embodiment of the present disclosure;

[0020] Figure 2 is a flowchart of a digital human live broadcast method based on a large model according to another embodiment of the present disclosure;

[0021] Figure 3 is a flowchart of a digital human live broadcast method based on a large model according to another embodiment of the present disclosure;

[0022] Figure 4 is a flowchart of a digital human live broadcast method based on a large model according to another embodiment of the present disclosure;

[0023] Figure 5 is a schematic diagram of the basic principle of the digital human live broadcast according to the embodiment of the present disclosure;

[0024] Figure 6 is a schematic diagram of a digital human live broadcast process according to an embodiment of the present disclosure;

[0025] Figure 7 is a flowchart of script generation in an embodiment of the present disclosure;

[0026] Figure 8 is a schematic diagram of the structure of the script of the embodiment of the present disclosure;

[0027] Fig. 9 Schematic diagram of an application scenario of an embodiment of the present disclosure:

[0028] Fig.10 is a flowchart of an intelligent conversation according to an embodiment of the present disclosure;

[0029] Fig.11 is a structural schematic diagram of a digital human live broadcast device based on a large model according to an embodiment of the present disclosure;

[0030] Fig.12 is a structural schematic diagram of a digital human live broadcast device based on a large model according to another embodiment of the present disclosure;

[0031] Fig.13 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0033] Figure 1 1 is a flow chart of a method 100 for live broadcasting a digital human based on a large model according to an embodiment of the present disclosure. In one implementation, the method includes:

[0034] S101, detecting interactive information for a live broadcast object in live broadcast data;

[0035] S102, inputting the interactive information for the live broadcast object into the macro model to generate interactive introduction content for the live broadcast object; wherein the interactive introduction content for the live broadcast object is used to control the interactive state of the digital human during the live broadcast process.

[0036] In the disclosed embodiments, live broadcasting can be used to spread knowledge, answer questions, and sell goods. Digital human live broadcasting can be used for a long time, and personalized speech and timbre can be customized. Based on the professional domain knowledge of the big model, real-time interaction can be carried out with users in the live broadcast room.

[0037] You can first play the scene video of the live broadcast room and the video of the digital human introducing each live broadcast object. Live broadcast objects can include products, items, knowledge, etc. Any object that can be displayed and introduced in the live broadcast room can be considered a live broadcast object, and the present disclosure does not limit this. During the live broadcast process, various interactive information may be generated, such as comment message bodies, gift message bodies, like message bodies, live broadcast room entry messages, human-computer interaction events, etc. Among these interactive information, interactive information for certain live broadcast objects may appear. For example, information requesting the introduction of live broadcast object A appears in the comment message body. For another example, information about liking live broadcast object B appears in the like message body. For another example, human-computer interaction events may include music, applause, laughter, crying, coughing, sneezing, etc.

[0038] The interactive information for the live broadcast object can be input into the big model. The big model can be an artificial intelligence (AI) voice big model, which can generate output voice based on the input information. For example, the big model can identify the demand information for the live broadcast object, such as introducing the interior, introducing the appearance, introducing the parts, etc., based on the interactive information for the live broadcast object, and generate interactive introduction content for the live broadcast object based on the demand information for the live broadcast object. The interactive introduction content may include one or more of the introduction voice content, introduction video content, introduction image content, and introduction text content that meet the demand information. For example, the introduction voice content can be a voice stream in the format of WaveForm (WAV), Moving Picture Experts Group Audio Layer III (MP3), Windows Media Audio (WMA), Advanced Audio Coding (ACC), etc. The introduction video content can be a video stream in the format of flash video (FLASH VIDEO, FLV), hypertext transfer protocol live streaming (Hypertext Transfer Protocol Live Streaming, HLS), moving picture experts group 4 (Moving Picture Experts Group 4, MP4), etc. The above formats are only examples and not limitations. In actual applications, they can be selected according to needs.

[0039] According to the introductory voice content in the interactive introduction content, the interactive voice of the digital human for the live broadcast object can be controlled during the live broadcast. According to the introductory video content in the interactive introduction content, the digital human can display the interactive video of the live broadcast object during the live broadcast, for example, the live broadcast object introduction video originally played while standing can be changed to the live broadcast object introduction video played while partially holding. Introductory image content, introductory text content, etc. can also be inserted during the live broadcast.

[0040] In the disclosed embodiment, a microphone may be used to record the audio stream of the interactive voice to obtain updated live broadcast data, which may then be sent to clients of various live broadcast rooms.

[0041] In the disclosed embodiment, the big model can generate interactive introduction content for the live broadcast object based on the interactive information for the live broadcast object in the live broadcast data, so that the digital human can interact in a timely and accurate manner during the live broadcast process.

[0042] Figure 2 2 is a flow chart of a method 200 for live broadcasting a digital human based on a macro model according to another embodiment of the present disclosure. The method 200 can be used to implement step S102 in the method 100 for live broadcasting a digital human based on a macro model. In one embodiment, the method 200 includes: inputting interactive information for the live broadcast object into the macro model to generate interactive introduction content for the live broadcast object, and further includes:

[0043] S201, loading the session data of the interactive object according to the interactive information;

[0044] S202, inputting the conversation data into the intention recognition big model to obtain the intention recognition result;

[0045] S203, generating a prompt word according to the intention recognition result;

[0046] S204: input the prompt word into the large language model to generate interactive introduction content for the live broadcast object.

[0047] In some scenarios, if only direct keyword matching or basic natural language processing technology is used to parse interactive information, the understanding of complex or ambiguous needs is limited. For complex fields such as automotive question-and-answer scenarios, you can directly connect to the big model, send the query directly to the model, and return the corresponding interactive data through the model. If the intention in the interactive information is not understood accurately, it will be impossible to meet personalized intention needs, and it will be difficult to provide answers with the latest professional knowledge.

[0048] In an embodiment of the present disclosure, after detecting interactive information for a live broadcast object, information about interactive objects that are interested in the live broadcast object can be obtained, such as the identifier of the object that initiated the comment. Based on the identifier of the interactive object, historical session data of the interactive object in the live broadcast data can be searched, such as historical comments, historical likes, and other operations of the interactive object during the live broadcast. The query statement (query) can be improved using the historical session data and the current session data, and then the query statement composed of these session data can be input into the intent recognition big model to obtain the intent recognition result. The intent recognition result may include the intent of the interactive object, such as viewing the interior, viewing the exterior, viewing the local part, etc. The intent recognition big model can be obtained by fine-tuning the initial big model using the intent training data. The intent training data may include text, audio or video and its corresponding intent, etc.

[0049] According to the query, scene information, intent recognition results, etc., prompt words can be intelligently generated. The prompt words are input into the large language model (LLM) used to generate the live object introduction, and the interactive introduction content for the live object can be obtained. The LLM used to generate the live object introduction can be obtained by fine-tuning the initial large model using the live object introduction training data. The live object introduction training data can include text, audio or video and its corresponding live object introduction, etc.

[0050] The intent recognition big model and the LLM used to generate live object introductions can be two independent big models or an integrated big model.

[0051] After the big model recognizes the intention, it can generate more accurate interactive introduction content for the live broadcast object based on the intention, so that the digital human can interact in a timely and accurate manner during the live broadcast.

[0052] In one embodiment, the tool set used to generate the prompt word includes one or more of the following: vector materials; Application Programming Interface (API) materials; customized skills.

[0053] In the disclosed embodiment, some local materials can be preset, and prompt words can be generated based on these materials. In addition, the large model can generate prompt words (Prompt) after the intelligent scheduling tool set obtains the materials based on the input information such as query statements, scene information, and intention recognition results. For example, the large model can schedule vector materials, embed one or more of the above input information based on the vector database, obtain feature vectors, and load materials of feature vectors. Among them, the vector database can be built offline, and the materials that can be loaded by the vector database include one or more of Uniform Resource Locator (URL), database (mysql), API, file (format such as csv, word, excel, pdf), etc. These materials may include pre-loaded live broadcast objects related to videos, images, text, voice, links, files, etc. For another example, the large model can schedule API materials, obtain materials from applications or links such as search and encyclopedia, and then generate prompt words. For another example, the large model can schedule customized skills such as API, chain, intelligent agent, etc. to generate prompt words.

[0054] In the disclosed embodiment, after the prompt words are generated, the prompt words can be input into the LLM for circular reasoning to generate interactive introduction content for the live broadcast object. Various tool sets can be used to obtain rich materials and generate accurate prompt words, which is conducive to using large models to generate more accurate and vivid interactive introduction content.

[0055] In one embodiment, the large language model is obtained by fine-tuning the initial large model based on emotion feature training data; the introductory voice content in the live object introduction content carries the emotion features associated with the intent recognition result.

[0056] In the disclosed embodiment, the initial large model can be fine-tuned using the emotional feature training data and the live broadcast object introduction training data to obtain a large language model. The initial large model can be fine-tuned using the live broadcast object introduction training data first, and then the initial large model can be fine-tuned using the emotional feature training data, or vice versa. After the prompt words are input into the fine-tuned large language model, the emotional features of the interactive object, such as calm, happy, excited, sad, angry, etc., can be identified. Based on the emotional features of the interactive object, the emotional features used for the interaction can be determined. For example, the emotional feature of the interactive object is happy, and the emotional feature used for the interaction is also happy, and the corresponding interactive voice feature can be cheerful. For another example, the emotional feature of the interactive object is sad, and the emotional feature used for the interaction is calm, and the corresponding interactive voice feature can be flat, slow, optimistic, etc.

[0057] In the disclosed embodiment, the large model can generate introductory voice content with emotional features associated with the intention recognition result, so that the digital human can produce a more vivid interactive effect. Therefore, the digital human live broadcast can perform emotional interaction and can perform various emotional expressions, making the live broadcast interaction more vivid.

[0058] Figure 3 3 is a flow chart of a digital human live broadcast method 300 based on a large model according to another embodiment of the present disclosure. The method 300 can be used to implement the digital human live broadcast method 100 based on a large model. In one implementation, the method further includes:

[0059] S301. Generate live object introduction content in a live script based on introduction information of one or more live objects; wherein the introduction information of the one or more live objects includes one or more of video, voice, image, text, and link associated with the one or more live objects.

[0060] In the disclosed embodiment, the introduction information of each live broadcast object that needs to be introduced in a live broadcast can be obtained first. For example, the content of the introduction information of the live broadcast object can include one or more of the name, origin, quality, price, story, etc. of the live broadcast object, and the resource form of the introduction information of the live broadcast object can include one or more of video, voice, image, text, link, etc. Based on the introduction information of the live broadcast object, the large model can be used to generate the introduction content of the live broadcast object in the live broadcast script. In the initial stage of the live broadcast, the digital human can first broadcast based on the introduction content of the live broadcast object. During the live broadcast process, S101 can be executed. If the interactive information of the live broadcast data for the live broadcast object is detected, S102 can be continued to generate interactive introduction content for the live broadcast object, and the interactive state of the digital human during the live broadcast process can be changed.

[0061] The digital human live broadcast of the disclosed embodiment can perform scenario-based interaction according to the characteristics of the live broadcast object, provide fundamental answers to the facts that are consistent with the live broadcast object such as things and products, and can provide more professional answers to the live broadcast object itself.

[0062] In one embodiment, the live broadcast script further includes: opening content and ending content; wherein the opening content includes one or more opening remarks, and the ending content includes one or more ending remarks.

[0063] In the disclosed embodiment, a large model can be used to generate the opening content, live object introduction content, and ending content of the live script. For examples of live object introduction content, see the above description. If the opening content can include multiple opening remarks, the ending content includes multiple ending remarks. The opening remarks broadcasted by the digital person in each live broadcast can be flexibly selected from the multiple opening remarks, and the ending remarks broadcasted by the digital person in each live broadcast can be flexibly selected from the multiple ending remarks, thereby improving the flexibility or fun of the live broadcast.

[0064] Figure 4 4 is a flow chart of a digital human live broadcast method 400 based on a large model according to another embodiment of the present disclosure. The method 400 can be used to implement the digital human live broadcast method 100 based on a large model. In one implementation, the method further includes:

[0065] S401, generating a digital human based on visual data and voice data;

[0066] S402: The digital human and the live broadcast room scene data are combined into the live broadcast data.

[0067] In the disclosed embodiments, visual data of a digital human is generated through visual technology such as a visual big model, and voice data of a digital human is generated through language technology such as a voice big model. The picture and voice of a digital human in a live studio can be generated based on the visual data and voice data. The background, decoration, etc. of the live studio can be drawn based on the live studio scene data through a user interface (UI) to generate a live studio picture. The live studio picture and the picture and voice of the digital human are synthesized into complete live data such as a live video. The live studio picture including the digital human can be generated through visual data, voice data, and live studio scene data.

[0068] Figure 5 Schematic diagram of the basic principle of the digital human live broadcast of the embodiment of the present disclosure, including:

[0069] Generate the video images of digital people through visual technology such as visual big models, generate the audio (timbre) of digital people through language technology such as voice big models, draw the live broadcast room scene through the user interface (UI), and complete the decoration of the live broadcast room. Synthesize the video images, audio and live broadcast room scenes into a complete live video. Send live videos to various live broadcast applications based on the broadcast account, streaming code, live streaming server and live broadcast configuration. The architecture can also include artificial intelligence (AI) big models to generate live broadcast room titles, live broadcast room introductions, etc. And control the atmosphere of the live broadcast room by replying to likes, following guides, and sending gifts.

[0070] Figure 6Schematic diagram of the digital human interaction process of the embodiment of the present disclosure, including:

[0071] Input the comment message body, gift message body, like message body and live broadcast room entry message into the AI ​​voice big model. The AI ​​voice big model processes the above input to obtain a voice stream in WAV format. Record the WAV voice stream into the microphone and output the interactive audio in each live broadcast room.

[0072] Figure 7 The following is a flow chart of script generation according to an embodiment of the present disclosure, including:

[0073] In the preparation stage for the broadcast, AI is used to generate a live broadcast script based on the information of the products on the shelves. The live broadcast script can include an opening statement, an introduction, and a closing statement. The generalization ability of the large model can be used to generate several opening statements, introductions, and closing statements respectively. You can select one of the opening statements, introductions, and closing statements to form a live broadcast script. This can make the live broadcast effect more flexible and vivid.

[0074] Figure 8 This is a schematic diagram of the structure of the introduction script of the embodiment of the present disclosure, including at least one of the text script, audio script and video script of the product. The audio script and / or video script of the product can also be inserted into the text script of the product to generate a product introduction script with audio and video. When the digital human reads the audio or video during the live broadcast, the corresponding audio and video can be played.

[0075] For example, the contents of a text script might include:

[0076] (Opening remarks) Hello friends, I am your host ***, and this is the ** live broadcast room.

[0077] (Introduction) The products I would like to recommend to you include A, B, C... [Product introduction video.mp4].

[0078] (Conclusion) Today’s live broadcast is about to end, and the next live broadcast will be on **.

[0079] When the introduction plays to [Product Introduction Video.mp4], switch to this video to play.

[0080] Fig. 9 It is a schematic diagram of the application scenario of the embodiment of the present disclosure, including scenarios such as standing broadcast, interaction, sitting broadcast, etc. If the product includes a car, the interaction can be human-car interaction.

[0081] Fig.10 : is a flow chart of the intelligent conversation of the embodiment of the present disclosure, including:

[0082] Input the query statement (query) and the conversation ID (ID) into the process controller. The process controller can execute high-level skills first, and execute low-level skills if there is no result. In the process controller, first load the historical session data according to the conversation ID, and improve the query statement (query) according to the conversation history. Then use the big model to identify the intent of the query statement and return the intelligent generation or direct connection to the big model. Then execute the corresponding skills (here it can be intelligently generated or directly connected to the big model) according to the intent recognition result to obtain the generated result. Select from the generated results (here the preferred results do not need to be processed), and only those that can be generated can be directly connected to the big model without recall. Use the preferred results or the big model generation results to update the session data and output.

[0083] Among them, the process of executing skills can include using the underlying materials to obtain the generated results. It is also possible to query statements (query), scene information, etc. for intelligent generation, obtain results and output them. In the process of intelligent generation, the large model can intelligently schedule the tool set to obtain materials and generate prompts (Prompt). The prompt words are input into the large language model (LLMs) for circular reasoning to obtain the large model generation results. The tool set may include a vector database, API materials and solidified customized skills. Among them, the vector database can be built offline, and the materials that can be loaded by the vector database (such as milvus) include one or more of the uniform resource locator (URL), database (mysql), API, file (format such as csv, word, excel, pdf), etc. These materials may include pre-loaded videos, images, text, voice, links, files, etc. related to the live broadcast object. API materials may include materials from applications or links such as search and encyclopedia. Customized skills may include API, chain, intelligent agent, etc.

[0084] In addition, the interactive generalization capability based on the large voice model can support the detection of many common human-computer interaction events such as music, applause, laughter, crying, coughing, sneezing, etc., giving digital humans more accurate emotional interactions.

[0085] The use of the digital human live broadcast of the disclosed embodiment can reduce the cost of live broadcasting, making live broadcasting more abundant and diversified. By providing more personalized and interactive live broadcast content, it can increase market share. It can apply advanced AI technology to showcase the forefront of technology and enhance the image and sense of technology. It can be applied to more business scenarios, expand business scope and revenue channels. Through smarter interaction and personalized content, it can increase user stickiness and loyalty and form a stable user group.

[0086] Fig.111 is a structural diagram of a digital human live broadcast device 1100 based on a large model according to an embodiment of the present disclosure. In one implementation, the device includes:

[0087] The detection module 1110 is used to detect interactive information for a live broadcast object in the live broadcast data;

[0088] The first generation module 1120 is used to input the interactive information for the live broadcast object into the macro model to generate interactive introduction content for the live broadcast object; wherein the interactive introduction content of the live broadcast object is used to control the interactive state of the digital human during the live broadcast process.

[0089] Fig.12 1 is a structural diagram of a digital human live broadcast device 1200 based on a large model according to another embodiment of the present disclosure, the device 1200 includes: a detection module 1210 and a first generation module 1220, the functions of the above modules can refer to the functions of the modules of the re-illumination device in the above embodiment. In one embodiment, the first generation module 1220 includes:

[0090] A loading submodule 1221 is used to load the session data of the interactive object according to the interactive information;

[0091] The recognition submodule 1222 is used to input the conversation data into the intention recognition model to obtain the intention recognition result;

[0092] The first generating submodule 1223 is used to generate a prompt word according to the intention recognition result;

[0093] The second generating submodule 1224 is used to input the prompt word into the large language model to generate interactive introduction content for the live broadcast object.

[0094] In one embodiment, the tool set used to generate the prompt word includes one or more of the following: vector materials; application programming interface API materials; customized skills.

[0095] In one embodiment, the large language model is obtained by fine-tuning the initial large model based on emotion feature training data; the introductory voice content in the live object introduction content carries the emotion features associated with the intent recognition result.

[0096] In one embodiment, Fig.12 As shown, the device also includes:

[0097] The second generation module 1230 is used to generate the live broadcast object introduction content in the live broadcast script based on the introduction information of one or more live broadcast objects; wherein the introduction information of the one or more live broadcast objects includes one or more of the video, voice, image, text, and link associated with the one or more live broadcast objects.

[0098] In one embodiment, the live broadcast script further includes: opening content and ending content; wherein the opening content includes one or more opening remarks, and the ending content includes one or more ending remarks.

[0099] In one embodiment, Fig.12 As shown, the device also includes:

[0100] The third generation module 1240 is used to generate a digital human based on the visual data and the voice data;

[0101] The synthesis module 1250 is used to synthesize the digital human and the live broadcast room scene data into the live broadcast data.

[0102] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0103] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0104] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0105] Fig.13 A schematic block diagram of an example electronic device 1300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0106] like Fig.13 As shown, the device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0107] A number of components in the device 1300 are connected to the I / O interface 1305, including: an input unit 1306, such as a keyboard, a mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a disk, an optical disk, etc.; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0108] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as a digital human live broadcast method based on a large model. For example, in some embodiments, a digital human live broadcast method based on a large model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the digital human live broadcast method based on the large model described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to execute the large model-based digital human live broadcast method in any other appropriate manner (eg, by means of firmware).

[0109] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0110] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0111] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0113] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0114] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0115] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0116] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A digital human live broadcast method based on a large model, comprising: In the live broadcast data, detecting interactive information for the live broadcast object; The interactive information for the live broadcast object is input into the big model to generate interactive introduction content for the live broadcast object; wherein the interactive introduction content for the live broadcast object is used to control the interactive state of the digital human during the live broadcast process.

2. The method according to claim 1, wherein: Inputting the interactive information for the live broadcast object into the macro model to generate interactive introduction content for the live broadcast object, including: Loading session data of the interactive object according to the interactive information; Inputting the conversation data into the intention recognition big model to obtain the intention recognition result; generating a prompt word according to the intention recognition result; The prompt words are input into a large language model to generate interactive introduction content for the live broadcast object.

3. The method according to claim 2, wherein: The tool set used to generate the prompt words includes one or more of the following: vector materials; application programming interface API materials; customized skills.

4. The method according to claim 2 or 3, wherein: The large language model is obtained by fine-tuning the initial large model based on the emotion feature training data; the introductory voice content in the live object introduction content carries the emotion features associated with the intention recognition result.

5. The method according to any one of claims 2 to 4, further comprising: Based on the introduction information of one or more live broadcast objects, the live broadcast object introduction content in the live broadcast script is generated; wherein the introduction information of the one or more live broadcast objects includes one or more of the video, voice, image, text, and link associated with the one or more live broadcast objects.

6. The method according to claim 5, wherein: The live broadcast script also includes: opening content and ending content; wherein the opening content includes one or more opening remarks, and the ending content includes one or more ending remarks.

7. The method according to any one of claims 1 to 6, further comprising: Generate digital humans based on visual and voice data; The digital human and the live broadcast room scene data are combined into the live broadcast data.

8. A digital human live broadcast device based on a large model, comprising: A detection module, used to detect interactive information for a live broadcast object in live broadcast data; The first generation module is used to input the interactive information for the live broadcast object into the large model to generate interactive introduction content for the live broadcast object; wherein the interactive introduction content for the live broadcast object is used to control the interactive state of the digital human during the live broadcast process.

9. The device according to claim 8, wherein: The first generating module comprises: A loading submodule, used for loading the session data of the interactive object according to the interactive information; The recognition submodule is used to input the session data into the intention recognition model to obtain the intention recognition result; A first generation submodule, used to generate a prompt word according to the intention recognition result; The second generation submodule is used to input the prompt word into the large language model to generate interactive introduction content for the live broadcast object.

10. The device according to claim 9, wherein: The tool set used to generate the prompt words includes one or more of the following: vector materials; application programming interface API materials; customized skills.

11. The device according to claim 9 or 10, wherein: The large language model is obtained by fine-tuning the initial large model based on the emotion feature training data; the introductory voice content in the live object introduction content carries the emotion features associated with the intention recognition result.

12. The device according to any one of claims 9 to 11, further comprising: The second generation module is used to generate the live broadcast object introduction content in the live broadcast script based on the introduction information of one or more live broadcast objects; wherein the introduction information of the one or more live broadcast objects includes one or more of the video, voice, image, text, and link associated with the one or more live broadcast objects.

13. The device according to claim 12, wherein: The live broadcast script also includes: opening content and ending content; wherein the opening content includes one or more opening remarks, and the ending content includes one or more ending remarks.

14. The device according to any one of claims 8 to 13, further comprising: A third generation module is used to generate a digital human based on visual data and voice data; A synthesis module is used to synthesize the digital human and the live broadcast room scene data into the live broadcast data.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

18. A large model comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images