Content generation method and device based on large model, and electronic equipment

Abstract

The invention provides a content generation method based on a large model, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, natural language processing, computer vision, large models and the like. According to the specific implementation scheme, in response to received input information, intention recognition is conducted on the input information; in response to a drawing knowledge demand intention identified from the input information, calling a multi-modal large model to generate a drawing knowledge text based on the drawing knowledge demand intention; generating a first driving voice and a first action instruction for driving the virtual image according to the drawing knowledge text; and driving the virtual image to broadcast the drawing knowledge text according to the first driving voice and the first action instruction. The invention further provides a content generation device based on the large model, electronic equipment and a storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the technical fields of deep learning, natural language processing, computer vision, large models, etc. More specifically, the present disclosure provides a content generation method, device, electronic device, storage medium, and computer program product based on a large model. Background Art

[0002] The current online art teaching system products are passive, manual and single-interface in usage or interaction for art teachers and art students. Summary of the invention

[0003] The present disclosure provides a content generation method, device, electronic device, storage medium and computer program product based on a large model.

[0004] According to a first aspect, a large model-based content generation method is provided, the method comprising: in response to receiving input information, performing intent recognition on the input information; in response to identifying a painting knowledge requirement intent from the input information, calling a multimodal large model to generate a painting knowledge text based on the painting knowledge requirement intent; based on the painting knowledge text, generating a first driving voice and a first action instruction for driving a virtual image; and driving the virtual image to broadcast the painting knowledge text based on the first driving voice and the first action instruction.

[0005] According to the second aspect, a content generation device based on a large model is provided, and the device includes: an intention recognition module, which is used to recognize the intention of the input information in response to receiving the input information; a painting knowledge text generation module, which is used to call the multimodal large model to generate a painting knowledge text based on the painting knowledge demand intention in response to identifying the painting knowledge demand intention from the input information; a first driving module, which is used to generate a first driving voice and a first action instruction for driving a virtual image according to the painting knowledge text; and a first broadcasting module, which is used to drive the virtual image to broadcast the painting knowledge text according to the first driving voice and the first action instruction.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0008] According to a fifth aspect, a computer program product is provided, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and wherein the computer program implements the method provided according to the present disclosure when executed by a processor.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic diagram of an exemplary system architecture to which a method and apparatus for generating content based on a large model can be applied according to an embodiment of the present disclosure;

[0012] Figure 2 is a flowchart of a method for generating content based on a large model according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic diagram of the system composition of a content generation method based on a large model according to an embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram of a content generation method based on a large model according to an embodiment of the present disclosure;

[0015] Figure 5 is a schematic diagram of the system architecture of a content generation method based on a large model according to an embodiment of the present disclosure;

[0016] Figure 6 is a schematic diagram of system call interaction of a large model-based content generation method according to an embodiment of the present disclosure;

[0017] Figure 7 is a system schematic diagram of digital human interaction according to an embodiment of the present disclosure;

[0018] Figure 8 This is a schematic diagram of the effect of digital human interaction according to an embodiment of the present disclosure.

[0019] Fig. 9 is a block diagram of a content generation device based on a large model according to an embodiment of the present disclosure;

[0020] Fig.10 4 is a block diagram of an electronic device according to a method for generating content based on a large model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] At present, the teaching of art painting is either a live broadcast of a video of a designated work recorded in advance by the teacher (for example, a video of the key points, steps and techniques of cube sketching), or a live online explanation by the teacher, explaining to students the key points, steps, details and techniques of the current teaching work. For example, if an art sketching teacher wants to teach a sketching class, he must check which existing explanation videos are available in the current teaching system, or which explanation cases he has prepared. He cannot teach flexibly or on demand. This is very unfavorable to arouse students' interest and desire to learn.

[0023] Currently, art painting reviews are basically conducted by teachers after students submit their work, which is inefficient. Teachers must check every key point and detail of each teaching work one by one, which takes up a lot of art teachers' time and energy.

[0024] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0025] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0026] Figure 1 is a schematic diagram of an exemplary system architecture to which a method and apparatus for generating content based on a large model can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0027] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0028] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices, including but not limited to smart phones, tablet computers, laptops, etc.

[0029] The content generation method based on the big model provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the content generation device based on the big model provided in the embodiment of the present disclosure can generally be set in the server 105. The content generation method based on the big model provided in the embodiment of the present disclosure can also be executed by a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the content generation device based on the big model provided in the embodiment of the present disclosure can also be set in a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0030] Figure 2 The present invention is a flowchart of a method for generating content based on a large model according to an embodiment of the present invention.

[0031] like Figure 2 As shown, the large model-based content generation method 200 includes operations S210 to S240.

[0032] In operation S210 , in response to receiving input information, intent recognition is performed on the input information.

[0033] For example, the input information may be text or voice input by a user (such as a trainee). If the input information is voice, the voice may be converted into input text, and intent recognition may be performed on the input text.

[0034] The natural language processing model can be used to identify the intent of the input text. The natural language processing model can be a text classifier that can classify the input text into multiple intents, such as painting knowledge demand intent, painting image generation intent, and painting knowledge retrieval intent.

[0035] The intention of demanding painting knowledge may be the intention of the user to learn painting steps, painting techniques, etc., or the intention of the user to comment on the painting. In the case where the user's intention is to comment on the painting, the input information also includes the painting to be commented on input by the user.

[0036] In operation S220 , in response to identifying the painting knowledge demand intention from the input information, the multimodal large model is called to generate a painting knowledge text based on the painting knowledge demand intention.

[0037] For example, if the user's painting knowledge needs are intended to learn painting steps or painting techniques, the input information can be sent to the multimodal large model, and the multimodal large model will generate a painting teaching text containing painting steps or painting techniques.

[0038] If the user's painting knowledge demand is intended to comment on a painting, the painting input by the user can be sent to the multimodal big model, and the multimodal big model will generate a description text for the painting comment.

[0039] Multimodal big models can include generative big language models, image content understanding models, etc. Generative big models are used to describe components that can learn objects from data through various machine learning methods, and then generate new and completely original content, such as text, pictures, videos, etc. It can help people generate various contents including text, pictures, videos, and 3D images on a large scale at a relatively low cost.

[0040] The image content understanding model is one of the large multimodal models of artificial intelligence, which aims to improve the performance of image understanding, image semantic generation, and dialogue generation by using larger multimodal (image-text) training data and more complex model structures. This type of model performs well in tasks and scenarios such as image content understanding and image content question answering, and can help people better understand images and generate natural language based on images.

[0041] The embodiments of the present disclosure can input the knowledge required for painting, the required painting steps and techniques, and historical painting case data into a multimodal large model, and through fine-tuning train a multimodal painting image understanding large model with the same reasoning ability as a professional painting teacher.

[0042] In operation S230, a first driving voice and a first action instruction for driving the avatar are generated according to the painting knowledge text.

[0043] The painting teaching text or painting review text generated by the multimodal large model can be driven by a virtual image to express it, thus realizing a "painting assistant" or "painting teaching assistant". The virtual image can be, for example, a digital human.

[0044] For example, the painting knowledge text (painting teaching text or painting commentary text) can be converted into a voice (first driving voice), and the first driving voice is used to drive the virtual image to broadcast the above painting knowledge text. Expressions, gestures and other action instructions can also be extracted from the painting knowledge text, and the action instructions are used to drive the virtual image to produce corresponding expressions, gestures and other body movements.

[0045] In operation S240, the avatar is driven to read the painting knowledge text according to the first driving voice and the first action instruction.

[0046] After obtaining the first driving voice and the first action instruction, the virtual image can be driven to broadcast the painting knowledge text based on the first driving voice and the first action instruction. In the process of the virtual image broadcasting the painting knowledge text, the virtual image generates expressions and body movements corresponding to the painting knowledge text, making the expression of the virtual image more natural and real.

[0047] The embodiments of the present disclosure perform intent recognition on input information, and when the intention of painting knowledge demand is recognized, call the multimodal large model to generate painting knowledge text corresponding to the painting knowledge demand, drive the virtual image to broadcast the painting knowledge text, and generate corresponding expressions and actions during the broadcasting process. It can communicate with the user in a dialogue manner, and realize the intelligent explanation of art teaching knowledge and the intelligent presentation of painting work comments.

[0048] In the field of art teaching, the embodiments of the present disclosure enable users to communicate with people through a "painting assistant" generated based on an art teaching knowledge base, a multimodal large model, and a digital human. At the same time, by driving the digital human to simulate the speech and expressions of a real person, the user is helped to complete the explanation of the painting steps or the commentary on the work, and the voice output and result output are performed at the same time. Compared with conventional painting teaching methods, this embodiment can provide explanations and comments on demand at any time.

[0049] Figure 3 It is a schematic diagram of the system composition of a large model-based content generation method according to an embodiment of the present disclosure.

[0050] like Figure 3 As shown, the system of this embodiment includes a digital human interactive terminal 310, a digital human central control platform 320, a multimodal large model 330, a painting knowledge base 340, and a painting business platform 350. The painting business platform 350 can be used to provide users with an entry for uploading painting knowledge. For example, through the painting business platform 350, users can upload professional teaching knowledge courses. The painting knowledge base 340 can contain painting knowledge, painting skills, and painting steps of works covered by various courses. For example, in the sketch painting teaching scenario, the painting knowledge base contains sketch course knowledge, basic art knowledge, sketch painting skills of each work, and sketch painting steps of each work.

[0051] User 301 may be a person who has a need to learn painting, and user 301 may interact with the teaching assistant digital human displayed by the digital human interaction terminal 310. For example, user 301 may input painting needs 302 by voice, and the painting needs 302 may include the need to learn painting knowledge, the need to comment on paintings, the need to generate paintings, and the need to retrieve painting knowledge. The digital human interaction terminal 310 transmits the input voice to the digital human central control platform 320.

[0052] According to an embodiment of the present disclosure, an input voice is converted into an input text; and one of a painting knowledge demand intention, a painting image generation intention, and a painting knowledge retrieval intention is identified from the input text.

[0053] The digital human central control platform 320 can serve as the backend of the digital human interactive terminal 310, convert the input voice into text, that is, the demand description text, perform intent recognition on the demand description text, and recognize that the user's intention is to learn painting, to comment on painting, to generate painting works, or to retrieve painting knowledge.

[0054] The digital human central control platform 320 can also be used as a platform for coordinating the call of the multimodal large model 330, and is used to select the corresponding multimodal large model according to different intentions and generate the corresponding painting knowledge text.

[0055] The multimodal large model 330 can be trained using the painting knowledge base 340, and the multimodal large model 330 with multiple capabilities can be trained using the painting knowledge base 340. For example, through training with data such as art sketching course knowledge, basic art knowledge, sketching painting techniques for each work, and sketching painting steps for each work, a multimodal large model with the ability to produce painting steps and painting technique explanations for various sketching works, a multimodal large model with the ability to produce painting reviews for various works, and a multimodal large model with the ability to produce painting works can be obtained.

[0056] Taking the sketch painting scene as an example, the training corpus of the multimodal large model 330 is the sketch painting knowledge base. Sketch painting knowledge can include multiple types, such as: geometric sketch, still life sketch, facial features sketch, portrait sketch, human body sketch, landscape sketch, etc. Therefore, the sketch painting scene can specifically include geometric sketch scene, still life sketch scene, facial features sketch scene, portrait sketch scene, human body sketch scene, and landscape sketch scene. Each painting scene has the basic knowledge of painting, painting steps, painting techniques, etc. in the scene.

[0057] For example, the first type of training corpus is composed of the description of the need to learn sketching as the prompt, and the text content such as sketching steps, painting techniques, and precautions in the sketching scene as the response. The first type of training corpus is used to fine-tune the large model, and the resulting multimodal large model can generate the response text of the drawing steps, painting techniques, and precautions corresponding to the demand scenario based on the sketching learning needs input by the user.

[0058] For another example, the specific steps of a sketching scene are used as prompts, and the sketching images corresponding to the steps are used as responses to form the second type of training corpus. The second type of training corpus is used to perform fine-tuning training on the large model, and the resulting multimodal large model can generate a painting image corresponding to the painting steps input by the user.

[0059] For another example, a sketch image is used as a prompt, and a commentary description of the work is used as the response to form the third type of training corpus. The third type of training corpus is used to fine-tune the large model, and the resulting multimodal large model can generate corresponding commentary description text based on the painting input by the user.

[0060] After the digital human central control platform 320 performs intent recognition on the demand description text, if it recognizes that the user's intention is to learn painting, it can select a multimodal large model 330 that has the ability to explain painting steps and painting techniques. The demand description text is sent to the multimodal large model 330 to generate an explanation text containing painting steps, painting techniques and precautions.

[0061] If the digital human central control platform 320 recognizes that the user's intention is to comment on the painting, it can select a multimodal large model 330 with the ability to comment on the painting. The demand description text and the painting work are sent to the multimodal large model 330 to generate a comment description text for the painting work.

[0062] If the digital human central control platform 320 recognizes that the user intends to generate a painting image, and the painting requirement description text contains specific painting steps, a multimodal large model 330 capable of producing painting works can be selected, and the painting requirement description text can be sent to the multimodal large model 330 to generate a painting work corresponding to the painting steps contained in the painting requirement description text.

[0063] If the digital human central control platform 320 recognizes that the user's intention is to retrieve painting knowledge, it can call the search engine to retrieve the corresponding painting knowledge from the painting knowledge base 340, and select a large model that has been fine-tuned and trained using the painting knowledge base, such as a large language model, to summarize, generalize and organize the retrieved painting knowledge, and respond in a dialogue manner.

[0064] Under the above-mentioned painting learning intention, painting review intention, and painting knowledge retrieval intention, the painting knowledge explanation text, painting review text, and painting knowledge retrieval results generated by the multimodal large model 330 are all texts, which can be collectively referred to as painting knowledge texts.

[0065] The painting knowledge text generated by the multimodal large model 330 can be sent to the digital human central control platform 320. The digital human central control platform 320 can convert the painting knowledge text into driving voice, extract action instructions such as expressions and gestures, and drive the digital human to broadcast the painting knowledge text based on the driving voice and action instructions, so as to realize the art sketch large model driving the digital human to communicate with the sketch students.

[0066] The embodiment of the present disclosure provides a new dialogue-based intelligent art auxiliary teaching solution for online art sketching teachers or students through a painting knowledge base, combined with a multimodal large model and digital human technology. The system assists art teaching in two ways. One way is: "Humanized intelligent explanation" displays the digital human image of the sketch painting teaching assistant through the front end, simulates the voice of a real person through speech synthesis technology, and drives the digital human to communicate with students through voice based on the art sketch teaching knowledge base and the multimodal large model to achieve the purpose of assisting teaching staff to complete art teaching explanation. The second way is: "Humanized intelligent comment on sketch works" inputs the student's sketch work (photos or scans of the painting) into the multimodal large model, and through model training, it can infer and simulate the "work comment report generation" of the work. Then drive the digital human to present to the outside through these signals, so that the student can be informed of the comment on his work in a timely manner in the form of a human report as a comment teacher.

[0067] Figure 4 It is a schematic diagram of a content generation method based on a large model according to an embodiment of the present disclosure.

[0068] like Figure 4 As shown, the input information is subjected to intent recognition, and the user's intent is identified as one of painting learning intent, painting review intent, painting knowledge retrieval intent, and painting image generation intent.

[0069] According to an embodiment of the present disclosure, in response to the painting knowledge demand intention being the painting learning intention, the painting teaching big model is called to generate a painting teaching text, including: parsing the painting learning intention to determine that the painting learning intention is one of basic painting knowledge learning and painting step learning; and based on one of the basic painting knowledge learning and painting step learning, calling the painting teaching big model to generate one of the basic painting knowledge text and the painting step text.

[0070] When the user's intention is to learn painting, a multimodal large model with the ability to explain basic painting knowledge and painting steps can be called to generate a text of basic painting knowledge or a text of painting steps.

[0071] For example, the painting learning intention can be further analyzed to determine whether the user's painting learning intention is to learn basic painting knowledge or painting steps. If the user's painting learning intention is to learn basic painting knowledge, it can be further determined which scene of painting basic knowledge is to be learned, such as basic art knowledge, basic sketching knowledge, basic geometric sketching knowledge, etc. According to the intention analysis results, the corresponding painting basic knowledge model can be called to generate the corresponding basic knowledge text.

[0072] If the user's painting learning intention is to learn painting steps, the painting step model can be called to generate a painting step text, which can include painting steps, painting key points, painting techniques and precautions in the steps.

[0073] According to an embodiment of the present disclosure, in response to the painting knowledge demand intention being the painting comment intention, the painting comment big model is called to generate a painting comment text, including: sending input information and the painting work to the painting comment big model; and using the painting comment big model to comment on the painting work to obtain the painting comment text.

[0074] When the user's intention is to comment on a painting, a multimodal large model with the ability to comment on a painting can be called, and the comment intention description text and the painting work can be sent to the large model, and the large model generates a painting comment text.

[0075] According to an embodiment of the present disclosure, in response to identifying a painting image generation intention from input information, a multimodal large model is called to generate a painting image based on the painting image generation intention, including: sending painting step information to the multimodal large model; and using the multimodal large model to generate a painting image based on the painting step information.

[0076] When the user intends to generate a painting image, the input information includes painting steps. A multimodal large model capable of producing paintings can be called, and the description text of the intention to generate the painting image and the painting step text can be sent to the multimodal large model to generate a painting image corresponding to the painting steps.

[0077] According to an embodiment of the present disclosure, in response to identifying a painting knowledge retrieval intention from input information, a target knowledge text is retrieved from a painting knowledge base; based on the target knowledge text, a second driving voice and a second action instruction are generated for driving a virtual image; and based on the second driving voice and the second action instruction, the virtual image is driven to broadcast the target knowledge text.

[0078] When the user's intention is to retrieve painting knowledge, the search engine can be called to retrieve the corresponding target knowledge text from the painting knowledge base, and the large language model can be called to summarize and organize it, and output it in the form of a dialogue.

[0079] The basic painting knowledge text, painting step text, painting comment text and target knowledge text generated under the above painting learning intention, painting comment intention and painting knowledge retrieval intention can all be broadcasted and explained in the form of a digital human. For example, the above text can be converted into voice, and action instructions corresponding to the text can be generated, and the digital human can be driven to broadcast according to the voice and action instructions.

[0080] The painting image generated under the above painting image generation intention can be directly returned to the user terminal and displayed to the user.

[0081] According to the embodiments of the present disclosure, a digital human is driven by texts generated intelligently based on a large model to achieve the purpose of providing an art teaching assistant similar to a real person. The assistant can help users provide painting steps and painting skills explanations for art sketching courses; painting steps and painting skills explanations for personalized sketching needs; art sketching course or knowledge retrieval inquiries (language questions and language replies); art sketching works intelligent comments and comment report generation and other services, realizing flexible teaching and on-demand teaching.

[0082] Figure 5 It is a schematic diagram of the system architecture of a content generation method based on a large model according to an embodiment of the present disclosure.

[0083] like Figure 5 As shown in Figure 1, the system architecture includes the basic layer, model layer, application layer and display layer. The basic layer is the basic cloud platform, which is responsible for providing basic hardware and operating system capabilities.

[0084] The model layer includes a multimodal large model and a digital human model. The multimodal large model provides contextual responses and content generation for sketching teaching needs and commentary needs. The multimodal large model can include an art basic knowledge enhancement model, a sketching course knowledge enhancement model, a homework commentary enhancement model, and a painting step enhancement model. Each enhancement model can be obtained by fine-tuning the large model (including the language large model and the visual large model) using the corresponding training data. For example, the art basic knowledge enhancement model is obtained by fine-tuning the large model using art basic knowledge. The enhancement model has the reasoning ability of art basic knowledge and can answer questions about art basic knowledge. Other enhancement models are similar and will not be repeated here.

[0085] The digital human model layer provides basic digital human capabilities. It is a combination of multiple models, including a speech escape model, an image rendering model, a speech synthesis model, a UE (User Experience) drive model, and a video drive synthesis model. The speech escape model is used to convert speech into text. The image rendering model is used to render the digital human. The speech synthesis model is used to convert text into speech. The UE drive model is used to generate action instructions such as expressions and gestures based on text. The video drive synthesis model is used to save the digital human's broadcasting process into a video.

[0086] The application layer provides the combined calling function of the basic digital human applications, and can coordinate the calling of the digital human model and the multimodal large model of the model layer. The application layer includes speech synthesis service, image rendering service, task-driven service, video synthesis service, knowledge retrieval service and large model service.

[0087] For example, the speech synthesis service is used to call the speech synthesis model to synthesize speech. The image rendering service is used to call the image rendering model to generate a digital human. The task-driven service is used to perform intent recognition to determine the user's intent and generate a painting task based on the user's intent. The video synthesis service is used to call the video-driven synthesis model to generate a video. The knowledge retrieval service is used to retrieve the target knowledge from the sketch knowledge base. The large model service is used to call the multimodal large model to perform the painting task.

[0088] The display layer is used to realize the interaction between the teaching assistant digital human and the user through the terminal and the corresponding background management functions. The display layer includes the digital human interaction terminal, the digital human central control platform and the painting business platform. The digital human interaction terminal is used to receive the user's voice input and output the voice answer. The digital human central control platform is used to drive the digital human to communicate with the user. The painting business platform is used to provide users with an entry to upload painting knowledge courses.

[0089] Figure 6 The diagram is a system call interaction diagram of a large model-based content generation method according to an embodiment of the present disclosure.

[0090] like Figure 6 As shown, this embodiment includes a digital human central control platform 610, a multimodal large model 620 and a digital human model 630. The digital human central control platform 610 includes a task driving module, a semantic parsing module, a voice driving module and an image driving module. The multimodal large model 620 includes a speech escape model, an image rendering model, a speech synthesis model, a UE driving model and a video driving synthesis model. The speech escape model is used to convert speech into text. The image rendering model is used to render a digital human. The speech synthesis model is used to convert text into speech. The UE driving model is used to generate action instructions such as expressions and gestures based on text. The video driving synthesis model is used to save the digital human's broadcasting process into a video. The multimodal large model 620 includes an art basic knowledge enhancement model, a sketch course knowledge enhancement model, an assignment review enhancement model and a painting step enhancement model.

[0091] The user interacts with the digital human displayed on the digital human interaction terminal. For example, the user asks a question to the digital human, and the digital human broadcasts the answer to the user. The question asked by the user is sent to the digital human central control platform 610 through the digital human interaction terminal. The task driving module recognizes the intention of the question, determines whether the user's intention is painting learning intention, painting comment intention, painting image generation intention or painting knowledge retrieval intention, and generates corresponding painting tasks according to the intention. Such as painting learning tasks, painting comment tasks, painting work generation tasks, painting knowledge retrieval tasks, etc.

[0092] The semantic parsing module is used to further parse the painting tasks determined by the task driving module. For example, the painting learning task can be further parsed into the art basic knowledge question and answer task, the sketching course knowledge question and answer task, the painting step explanation task, etc.

[0093] According to the painting task analyzed by the semantic analysis module, the corresponding large model in the multimodal model 620 can be called to generate an answer. For example, according to the art basic knowledge question and answer task, the art basic knowledge enhancement model is called. For the painting review task, the homework review enhancement model is called, and so on.

[0094] The answer generated by the multimodal model may include texts on basic painting knowledge, texts on painting steps, etc. The answer text may be sent to the digital human model 630. The digital human model 630 generates and drives the digital human to broadcast the answer text based on the voice drive module and the image drive module, and interacts with the user.

[0095] In the embodiment of the present disclosure, the user consults art sketching and painting problems through the terminal, the digital human generates a teaching assistant image to interact with the user through the terminal, and the digital human central control platform is responsible for coordinating the call of the multimodal large model, and coordinating the corresponding multimodal large model for processing according to different business processing flows, which can improve the model processing efficiency and model processing effect.

[0096] The implementation of the digital human provided by the embodiments of the present disclosure is further explained below.

[0097] The digital human system can include a character image driving module, a voice generation module, an animation generation module, an audio and video synthesis display module, and an interaction module. The character image driving module is used to generate character images. According to the dimension of character graphic resources, it can be divided into two categories: 2D and 3D. From the appearance, it can be divided into cartoon, anthropomorphic, realistic, hyperrealistic and other styles. The voice generation module and the animation generation module can generate corresponding character voices and matching character animations based on text respectively. The audio and video synthesis display module synthesizes voice and animation into videos and then displays them to users. The interaction module enables the digital human to have interactive functions, that is, to identify the user's intention through intelligent technologies such as voice semantic recognition, and determine the subsequent voice and action of the digital human according to the user's current intention, and drive the character to start the next round of interaction. For example, it determines whether the user's intention is a painting demand. If so, the painting demand text is sent to the multimodal large model, and a simple reply is returned to the user: for example, please wait, and the specific steps of XX will be provided to you. If it is not a painting demand, you can retrieve painting knowledge to reply. Or if the user inputs a simple greeting, you can also reply according to the configured words.

[0098] Figure 7 is a system diagram of digital human interaction according to an embodiment of the present disclosure.

[0099] like Figure 7 As shown, this embodiment includes a digital human central control platform 710, a digital human interactive terminal 720, and a user 730. The digital human central control platform 710 includes a voice drive module and an image drive module.

[0100] Text input may refer to inputting the painting knowledge text generated by the multimodal large model or the target knowledge text retrieved into the voice drive module. The voice drive module may convert the input text into voice and drive the virtual image to broadcast the input text based on the voice.

[0101] The command input may be to extract relevant features such as expressions and gestures from the input text and input them into the image driving module. The image driving module generates action commands based on these features and drives the virtual image to produce corresponding actions according to the action commands.

[0102] The above-mentioned voice driving module and image driving module jointly drive the generation of a virtual image digital human.

[0103] The digital human central control platform 710 may send the generated virtual image to the digital human interaction terminal 720 so as to be displayed on the digital human interaction terminal 720. The digital human displayed by the digital human interaction terminal 720 interacts with the user 730.

[0104] Figure 8 It is a schematic diagram of the effect of digital human interaction according to an embodiment of the present disclosure.

[0105] like Figure 8 As shown, the interactive display effect of the digital human in this embodiment can be displayed on the digital human interactive terminal. For example, the digital human interactive terminal displays page 810 and page 820. Page 810 can display a painting knowledge text generated by a multimodal large model, such as the steps of geometric sketching. Page 820 displays a digital human, which can broadcast the text displayed in page 810 and generate corresponding body movements such as expressions and gestures during the broadcasting process.

[0106] According to an embodiment of the present disclosure, the present disclosure also provides a content generation device based on a large model.

[0107] Fig. 9 It is a block diagram of a content generation apparatus based on a large model according to an embodiment of the present disclosure.

[0108] like Fig. 9 As shown, the large model-based content generation device 900 includes an intention recognition module 910, a text generation module 920, a first driving module 930 and a first broadcasting module 940.

[0109] The intention recognition module 910 is used to perform intention recognition on the input information in response to receiving the input information.

[0110] The text generation module 920 is used to generate a painting knowledge text based on the painting knowledge demand intention in response to identifying the painting knowledge demand intention from the input information and calling the multimodal large model.

[0111] The first driving module 930 is used to generate a first driving voice and a first action instruction for driving the virtual image according to the painting knowledge text.

[0112] The first broadcasting module 940 is used to drive the virtual image to broadcast the painting knowledge text according to the first driving voice and the first action instruction.

[0113] According to an embodiment of the present disclosure, the input information includes an input voice. The intention recognition module 910 includes a conversion unit and an intention recognition unit.

[0114] The conversion unit is used to convert the input speech into input text.

[0115] The intention recognition unit is used to recognize one of the painting knowledge demand intention, the painting image generation intention, and the painting knowledge retrieval intention from the input text.

[0116] The painting knowledge demand intention includes painting learning intention and painting review intention; the multimodal big model includes at least one of the painting teaching big model and the painting review big model. The text generation module 920 includes a teaching text generation unit and a review text generation unit.

[0117] The teaching text generation unit is used to call the painting teaching model to generate a painting teaching text in response to the painting knowledge demand intention being a painting learning intention.

[0118] The review text generating unit is used for calling the painting review big model to generate the painting review text in response to the painting knowledge demand intention being the painting review intention.

[0119] The teaching text generation unit includes a parsing subunit and a teaching text generation subunit.

[0120] The parsing subunit is used to parse the painting learning intention and determine that the painting learning intention is one of the painting basic knowledge learning and the painting step learning.

[0121] The teaching text generation subunit is used to call the painting teaching model to generate one of the basic painting knowledge text and the painting step text according to one of the basic painting knowledge learning and the painting step learning.

[0122] The input information includes paintings. The review text generation unit includes a sending subunit and a review text generation subunit.

[0123] The sending subunit is used to send the input information and the painting works to the painting review model.

[0124] The review text generation subunit is used to review the painting works using the painting review model to obtain the painting review text.

[0125] The macro-model-based content generation apparatus 900 further includes a painting image generation module.

[0126] The painting image generation module is used to call the multimodal large model to generate a painting image based on the painting image generation intention in response to identifying the painting image generation intention from the input information.

[0127] The input information includes painting step information. The painting image generation module includes a sending unit and a painting image generation unit.

[0128] The sending unit is used to send the painting step information to the multimodal large model.

[0129] The painting image generating unit is used to generate a painting image based on the painting step information using the multimodal large model.

[0130] The large model-based content generation device 900 further includes a retrieval module, a second driving module and a second broadcasting module.

[0131] The retrieval module is used to retrieve target knowledge text from the painting knowledge base in response to identifying the painting knowledge retrieval intention from the input information.

[0132] The second driving module is used to generate a second driving voice and a second action instruction for driving the virtual image according to the target knowledge text.

[0133] The second broadcasting module is used to drive the virtual image to broadcast the target knowledge text according to the second driving voice and the second action instruction.

[0134] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0135] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0136] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0137] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0138] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a content generation method based on a large model. For example, in some embodiments, the content generation method based on a large model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the content generation method based on the large model described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the large model-based content generation method in any other appropriate manner (eg, by means of firmware).

[0139] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0140] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0141] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0143] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0144] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0145] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0146] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A content generation method based on a large model, comprising: In response to receiving input information, performing intent recognition on the input information; In response to identifying a painting knowledge demand intention from the input information, calling a multimodal large model to generate a painting knowledge text based on the painting knowledge demand intention; Generating a first driving voice and a first action instruction for driving the virtual image according to the painting knowledge text; as well as The virtual image is driven to read the painting knowledge text according to the first driving voice and the first action instruction.

2. The method according to claim 1, wherein: The input information includes input speech; In response to receiving the input information, performing intent recognition on the input information includes: Converting the input speech into input text; as well as One of a painting knowledge demand intention, a painting image generation intention, and a painting knowledge retrieval intention is identified from the input text.

3. The method according to claim 1 or 2, wherein: The painting knowledge demand intention includes painting learning intention and painting review intention; the multimodal big model includes at least one of a painting teaching big model and a painting review big model; in response to identifying the painting knowledge demand intention from the input information, calling the multimodal big model to generate a painting knowledge text based on the painting knowledge demand intention includes: In response to the painting knowledge demand intention being the painting learning intention, calling the painting teaching model to generate a painting teaching text; and In response to the painting knowledge demand intention being the painting review intention, the painting review big model is called to generate a painting review text.

4. The method according to claim 3, wherein: In response to the painting knowledge demand intention being the painting learning intention, calling the painting teaching model to generate a painting teaching text includes: Analyzing the painting learning intention to determine that the painting learning intention is one of learning basic painting knowledge and learning painting steps; and According to one of the basic painting knowledge learning and the painting step learning, the painting teaching model is called to generate one of the basic painting knowledge text and the painting step text.

5. The method according to claim 3, wherein: The input information includes paintings; the response to the painting knowledge demand intention being the painting review intention, calling the painting review big model to generate the painting review text includes: Sending the input information and the painting to the painting review model; and The painting review model is used to review the painting work to obtain the painting review text.

6. The method according to claim 2, further comprising: In response to identifying the painting image generation intention from the input information, a multimodal large model is called to generate a painting image based on the painting image generation intention.

7. The method according to claim 6, wherein: The input information includes painting step information; in response to identifying the painting image generation intention from the input information, calling the multimodal large model to generate a painting image based on the painting image generation intention includes: Sending the painting step information to the multimodal macro model; and The painting image is generated based on the painting step information using the multimodal large model.

8. The method according to claim 2, further comprising: In response to identifying a painting knowledge retrieval intention from the input information, retrieving a target knowledge text from a painting knowledge base; generating a second driving voice and a second action instruction for driving the virtual image according to the target knowledge text; and The virtual image is driven to broadcast the target knowledge text according to the second driving voice and the second action instruction.

9. A content generation device based on a large model, comprising: an intention recognition module, configured to perform intention recognition on the input information in response to receiving the input information; A text generation module, configured to generate a painting knowledge text based on the painting knowledge demand intention in response to identifying the painting knowledge demand intention from the input information and calling the multimodal large model; A first driving module, used for generating a first driving voice and a first action instruction for driving the virtual image according to the painting knowledge text; as well as The first broadcasting module is used to drive the virtual image to broadcast the painting knowledge text according to the first driving voice and the first action instruction.

10. The device according to claim 9, wherein: The input information includes input speech; The intention recognition module includes: A conversion unit, used for converting the input speech into input text; as well as The intention recognition unit is used to recognize one of the painting knowledge demand intention, the painting image generation intention, and the painting knowledge retrieval intention from the input text.

11. The device according to claim 9 or 10, wherein: The painting knowledge demand intention includes painting learning intention and painting review intention; the multimodal big model includes at least one of the painting teaching big model and the painting review big model; the text generation module includes: A teaching text generating unit, configured to call the painting teaching model to generate a painting teaching text in response to the painting knowledge demand intention being the painting learning intention; and The review text generating unit is used for calling the painting review big model to generate a painting review text in response to the painting knowledge demand intention being the painting review intention.

12. The device according to claim 11, wherein The teaching text generating unit comprises: A parsing subunit, configured to parse the painting learning intention and determine that the painting learning intention is one of painting basic knowledge learning and painting step learning; and The teaching text generation subunit is used to call the painting teaching model to generate one of the basic painting knowledge text and the painting step text according to one of the basic painting knowledge learning and the painting step learning.

13. The device according to claim 11, wherein: The input information includes paintings; the review text generation unit includes: A sending subunit, configured to send the input information and the painting to the painting review model; and The review text generating subunit is used to review the painting work by using the painting review model to obtain the painting review text.

14. The apparatus according to claim 9, further comprising: A painting image generation module is used to call a multimodal large model to generate a painting image based on the painting image generation intention in response to identifying the painting image generation intention from the input information.

15. The device according to claim 14, wherein: The input information includes painting step information; the painting image generation module includes: a sending unit, configured to send the painting step information to the multimodal large model; and A painting image generating unit is used to generate the painting image based on the painting step information by using the multimodal large model.

16. The apparatus according to claim 10, further comprising: A retrieval module, configured to retrieve a target knowledge text from a painting knowledge base in response to identifying a painting knowledge retrieval intention from the input information; A second driving module, used to generate a second driving voice and a second action instruction for driving the virtual image according to the target knowledge text; as well as The second broadcasting module is used to drive the virtual image to broadcast the target knowledge text according to the second driving voice and the second action instruction.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8.

19. A computer program product, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images