Content generation method and apparatus

Abstract

A content generation method, applied to a terminal device. The method comprises: acquiring input data of a content generation model; uploading the input data to a server via a first routing path; receiving first content returned by the server, wherein the first content is generated by a first content generation model deployed on the server; and when an anomaly occurs in acquiring the first content via the first routing path, inputting, via a second routing path, the input data into a second content generation model deployed locally to obtain second content, wherein the number of parameters of the second content generation model is less than the number of parameters of the first content generation model. According to the content generation method provided by the present application, when a response of a content generation model deployed on a server side is abnormal, a miniaturized content generation model deployed locally is used to generate the required content, thereby preventing a content generation service from being affected when a network is abnormal or a server is abnormal.

Description

Content generation method and device

[0001] This application claims priority to Chinese patent application number 202311188823.7, filed on September 14, 2023, entitled “A content generation method and device,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the technical field of terminal devices, and in particular to a content generation method and apparatus. Background Art

[0003] AI-generated content (AIGC) utilizes artificial intelligence (AI) to generate content. It represents a new content production method, following professional-generated content (PGC) and user-generated content (UGC), creating a new form of digital content generation and interaction. The core concept of AIGC technology is to use AI algorithms to generate content with a certain level of creativity and quality. By training models and learning from large amounts of data, AIGC can generate relevant content based on the input conditions and guidance. For example, by inputting keywords, descriptions, or text, AIGC can generate matching articles, images, audio, and more.

[0004] In existing AIGC services (such as ChatGPT and Midjourney), models are often deployed in the cloud. Users input data through a client deployed on their terminal devices, and the AIGC model deployed in the cloud responds to the user's input and returns generated content. However, when the terminal is in an environment with no network signal or poor network signal, the AIGC service fails and users cannot receive the generated content in a timely manner.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a content generation method, which can enable a terminal to continue to respond to content generation requests in the event of network anomalies.

[0007] In a first aspect, the present application provides a content generation method, which is applied to a terminal device, and the method includes: obtaining input data of a content generation model (which may be referred to as an AIGC model); uploading the input data to a server through a first routing path; receiving first content returned by the server, where the first content is generated by a first content generation model deployed on the server in response to the input data, and the first content generation model is a generative large model; when an exception occurs when obtaining the first content through the first routing path, the input data is input into a second content generation model deployed locally through a second routing path to obtain second content; wherein the number of parameters of the second content generation model is less than the number of parameters of the first content generation model.

[0008] The content generation method provided by the present application deploys a miniaturized content generation model on the terminal and a large content generation model on the server side. When the large generative model deployed remotely (i.e., the server side) responds normally, the content generation model deployed on the server side is used to generate content to obtain high-quality content. When the content generation model deployed on the server side responds abnormally, the locally deployed miniaturized content generation model is used to generate the required content, thereby ensuring that content generation services can continue to be obtained when network or server abnormalities occur.

[0009] The abnormality in obtaining the first content through the first routing path can be understood as: abnormal content acquisition caused by abnormal response of the generative large model deployed at the remote end, such as one or more of abnormal content quality output by the generative large model in response to input data and abnormal time of content output by the generative large model in response to input data.

[0010] For example, due to the large number of parameters in generative models (often reaching hundreds of billions or trillions), they may experience "intelligence decline" after a training update. This can lead to reduced accuracy of generated answers, irrelevance to the question, and other issues. Another example is a generative model deployed in the cloud to provide content generation services to the public. When too many users access it, insufficient computing power may cause the model's inference time to be excessive, leading to extended response times for users.

[0011] In another example, the exception can also be understood as an abnormal situation that causes the terminal to fail to receive a response from the generative large model, or fail to receive a response from the generative large model within a preset time, such as one or more of an abnormality in the communication link between the terminal device and the server, an abnormality in the response output of the first content generation model deployed on the server, and an abnormality in the server.

[0012] Optionally, the second content generation model can be a generative model with a smaller number of parameters obtained by compressing the first content generation model. For example, the second content generation model is a model obtained by subjecting the first content generation model to compression processing such as knowledge distillation, quantization, and pruning.

[0013] In another example, the first content generation model and the second content generation model may also be generative models that are trained separately, but the first content generation model and the second content generation model respond to the same input data and output content in the same format. For example, the first content generation model and the second content generation model respond to the input data and both output content in image format, or both output content in text format, etc.

[0014] In one possible implementation, the server is a cloud server, that is, the generative large model is deployed in the cloud to provide content generation services to users.

[0015] In one possible implementation, an auxiliary small model is also deployed on the terminal device, and the auxiliary small model includes a multi-layer neural network; the second content generation model includes a multi-layer intermediate layer neural network and an output layer; the input data is input into the second content generation model deployed locally through a second routing path, and a specific implementation of obtaining the second content is: the input data is used as the input of the second content generation model and the auxiliary small model respectively, the input data is processed by the multi-layer neural network to obtain a first processing result, and the input data is processed by the multi-layer intermediate layer neural network to obtain a second processing result; the output layer processes the first processing result and the second processing result, and outputs the second content.

[0016] By setting up an auxiliary small model on the terminal, the auxiliary small model and the locally deployed second content generation model work together to achieve responsive output that meets some user customization needs. For example, different users may have different preferences. For example, some users may want to interact in modern Chinese. In this case, an auxiliary small model can be trained with modern Chinese. After the auxiliary small model is applied to the locally deployed second content generation model, the output layer of the locally deployed second content generation model outputs modern Chinese that meets the user's preferences. However, some users may prefer "drag and drop" Chinese and like to interact in ancient Chinese. In this case, an auxiliary small model can be trained with ancient Chinese. After the auxiliary small model is applied to the locally deployed second content generation model, the output layer of the locally deployed second content generation model outputs modern Chinese that meets the user's preferences.

[0017] The first content generation model deployed on the server can be called the remote AIGC large model, and the second content generation model deployed locally on the terminal can be called the local AIGC small model. It can be understood that the second content generation model is called the local AIGC small model simply because it has fewer parameters and a smaller size than the remote AIGC large model. For example, the local AIGC small model has billions of parameters, 2 to 4 GB of storage overhead, and 10 to 20 teraflops of computing power.

[0018] The parameter scale of the remote AIGC large model is greater than that of the local AIGC small model. For example, the parameter scale of the remote AIGC large model is in the hundreds of billions to trillions, while the parameter scale of the local AIGC small model is in the billions. The storage overhead of the remote AIGC large model is 700G, and the computing power overhead is 100+PTFLOPS, while the storage overhead of the local AIGC small model is 2 to 4G, and the computing power overhead is 10-20TFLOPS.

[0019] In an example, the local AIGC small model is an AIGC model obtained by processing the remote AIGC large model through knowledge distillation, quantization, pruning, etc. The response output of the local AIGC small model is relatively simpler than the response output of the remote AIGC large model, but the response quality is still within the user's acceptable range.

[0020] In another possible implementation, the input data includes a prompt; a specific implementation of obtaining the input data of the AIGC model is: receiving user input data; obtaining input text based on the user input data; obtaining context information of the input text; and obtaining a prompt based on the input text and the context information.

[0021] In this possible implementation, the prompt words obtained through context information and input text can more accurately reflect the user's intention, so that the AIGC model can output output that is more consistent with the user's intention.

[0022] In another possible implementation, a specific implementation of obtaining the context information of the input text is: obtaining the user's historical operation records and historical chat records; and obtaining the context information based on the historical operation records and historical chat records.

[0023] Exemplarily, the user's emotional information and preferences are extracted from the operation records (such as web browsing records) and chat records stored in the terminal, and corresponding tags are generated. These tags constitute the context information of the input text.

[0024] In another possible implementation, a specific implementation of obtaining a prompt word based on input text and context information is: using the input text and context information as input to a prompt word generation model, and outputting a prompt word; wherein the prompt word generation model is trained based on a training data set, the training data set includes multiple training samples with labels, the training samples include text sentences and several text words, and the labels include prompt words corresponding to the text sentences and several text words.

[0025] By inputting input text and context information into the trained prompt word generation model, prompt words that conform to the grammatical rules of the AIGC model are obtained. Users do not need to learn the grammar or specifications of each AIGC model prompt word, which reduces the user's learning and time costs. Users can directly interact with the terminal according to their daily conversation habits and obtain the response output of the AIGC model that conforms to their own intentions.

[0026] Optionally, the prompt word generation model is a text generation network, such as a multi-layer perception (MLP) network.

[0027] In a second aspect, the present application provides a content generation device, which is deployed on a terminal device and includes an acquisition module, a sending module, a receiving module and an input module, wherein the acquisition module is used to acquire input data of a content generation model; the sending module is used to upload the input data to a server through a first routing path; the receiving module is used to receive first content returned by the server, where the first content is generated by a first content generation model deployed on the server in response to the input data, and the first content generation model is a generative large model; the input module is used to input the input data to a second content generation model deployed locally through a second routing path when an exception occurs when acquiring the first content through the first routing path, to obtain second content, and the exception is related to the first content generation model; wherein the number of parameters of the second content generation model is less than the number of parameters of the first content generation model.

[0028] In another possible implementation, the anomaly includes one or more of an anomaly in the quality of content output by the generative large model in response to the input data and an anomaly in the timing of the content output by the generative large model in response to the input data.

[0029] In another example, the exception may be one or more of an exception in the communication link between the terminal device and the server, an exception in the response output of the first content generation model deployed on the server, and an exception in the server.

[0030] Optionally, the server is a cloud server.

[0031] In another possible implementation, an auxiliary small model is also deployed on the terminal device, which includes a multi-layer neural network; the second content generation model includes a multi-layer intermediate layer neural network and an output layer; the input data is input into the second content generation model deployed locally through a second routing path, and a specific implementation of obtaining the second content is: the input data is used as the input of the second content generation model respectively, the input data is processed by the multi-layer neural network to obtain a first processing result, and the input data is processed by the multi-layer intermediate layer neural network to obtain a second processing result; the output layer processes the first processing result and the second processing result, and outputs the second content.

[0032] In another possible implementation, the input data includes a prompt word; the acquisition module is specifically used to: receive the data input by the user; obtain the input text based on the data input by the user; obtain context information of the input text; and obtain the prompt word based on the input text and the context information.

[0033] In another possible implementation, obtaining context information of the input text includes: obtaining historical operation records and historical chat records of the user; and obtaining context information based on the historical operation records and historical chat records.

[0034] In another possible implementation, the context information includes one or more of the user's emotion information and the user's preference information.

[0035] In another possible implementation, a specific implementation of obtaining a prompt word based on input text and context information is: using the input text and context information as input to a prompt word generation model, and outputting a prompt word; wherein the prompt word generation model is trained based on a training data set, the training data set includes multiple training samples with labels, the training samples include text sentences and several text words, and the labels include prompt words corresponding to the text sentences and several text words.

[0036] Optionally, the prompt word generation model is a multi-layer perceptron network.

[0037] In a third aspect, an embodiment of the present application provides a computing device comprising a memory and a processor, wherein the memory stores instructions, and when the instructions are executed by the processor, the method described in the first aspect is implemented.

[0038] In a fourth aspect, an embodiment of the present application provides a computing device comprising a memory and a processor, wherein the memory stores instructions, and when the instructions are executed by the processor, the method described in the first aspect is implemented.

[0039] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described in the first aspect is implemented.

[0040] In a sixth aspect, an embodiment of the present application further provides a computer program or a computer program product, which includes instructions that, when executed, cause a computer to execute the method described in the first aspect.

[0041] In a seventh aspect, an embodiment of the present application further provides a chip comprising at least one processor and a communication interface, wherein the processor is used to execute the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in this specification, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings described below are only the multiple embodiments disclosed in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] The following is a brief introduction to the drawings required for describing the embodiments or prior art.

[0044] FIG1 shows a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment disclosed herein;

[0045] FIG2 is a flow chart of a content generation method provided in an embodiment of the present application;

[0046] FIG3 is a schematic diagram of interacting with a terminal via voice;

[0047] FIG4 is a schematic diagram showing a process of generating prompt words according to an embodiment of the present application;

[0048] FIG5 shows a schematic block diagram of content generation;

[0049] Figure 6 is a schematic diagram of the relationship between the subsidiary small model and the local model;

[0050] FIG7 shows a system block diagram of a specific implementation of the content generation method provided in an embodiment of the present application;

[0051] FIG8 is a schematic structural diagram of a content generation device provided in an embodiment of the present application;

[0052] FIG9 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] AI technology has reached a new stage, and leveraging it to provide users with a better intelligent experience is a growing trend. For example, Google Assistant and Stable Diffusion. Google Assistant is an AI-powered virtual assistant capable of two-way communication, allowing users to interact with it through natural language / voice, enhancing the intelligence of terminal devices. Stable Diffusion is a deep learning model that generates images from text descriptions. However, both Google Assistant and Stable Diffusion are deployed on terminal devices and rely primarily on local computing power. Due to insufficient terminal computing power, the content output quality of Google Assistant and Stable Diffusion is inferior to that of larger models.

[0054] Currently, AI models are basically concentrating on super-large models, and the demand for computing power is growing exponentially. However, terminal chips are subject to multiple constraints such as area, power consumption, heat dissipation and process. Their computing power growth rate is difficult to match the rapidly growing demand for AI computing, making it difficult for products like Google Assistant to provide user-satisfied capabilities.

[0055] Another approach to addressing the computing bottleneck on the terminal side is to leverage end-to-end cloud collaboration and, with the help of high-speed network connections, offload some of the load to remote cloud servers. However, this approach also suffers from over-reliance on the network. In locations with unstable network connections, such as subways and elevators, data transmission delays are significantly increased, and even interruptions are possible, significantly impacting the user experience.

[0056] For example, ChatGPT and Midtravel are both deployed on the cloud and provide content generation services on the cloud. However, both ChatGPT and Midtravel have encountered service interruptions, resulting in poor user experience.

[0057] In response to the above problems, an embodiment of the present application provides a content generation method, which deploys a local AIGC small model on the terminal and a remote AIGC large model on the server side. When the communication link between the terminal and the server is normally connected, content is generated by the remote AIGC large model deployed on the server side to obtain high-quality content; when the remote AIGC large model deployed on the server side responds abnormally, the locally deployed local AIGC small model is used to generate the required content, so that content generation services can continue to be obtained when network abnormalities or server abnormalities occur.

[0058] The technical solution of the present application is further described in detail below through the accompanying drawings and examples.

[0059] FIG1 illustrates a schematic diagram of an exemplary system 100 in which the various methods and apparatuses described herein may be implemented according to embodiments disclosed herein. As shown in FIG1 , system 100 includes a terminal 110, a server 130, and a network 120 that communicatively connects terminal 110 and server 130. Terminal 110 may run one or more services or software applications capable of executing content generation methods. A small local AIGC model is deployed on terminal 110, and a large remote AIGC model is deployed on server 130.

[0060] In some embodiments, server 130 can also provide other services or software applications including non-virtual environments and virtual environments. In some embodiments, these services can be provided as web-based services or cloud services, such as provided to terminal users under a software-as-a-service (SaaS) model.

[0061] In the system shown in FIG1 , the server 130 may include one or more components that implement the functions performed by the server 130. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user of the operating terminal 110 may sequentially utilize one or more client applications to interact with the server 130 and utilize the services provided by these components. It should be understood that FIG1 is merely an example of a system for implementing the various methods described in the embodiments of the present application, and other different configurations are possible, and the embodiments of the present application are not limited thereto.

[0062] The user can use the content generation client on the terminal 110 to send input data to the server 130. The terminal 110 can provide an interface that enables the user of the terminal 110 to interact with the terminal. The terminal 110 can also output information to the user via this interface. After receiving the input data, the server 130 uses the input data as input to the remote AIGC large model, responds with high-quality content, and feeds the content back to the terminal 110. The terminal 110 displays the generated content to the user through the content generation client. If the terminal 110 does not receive the content feedback from the server 130 within the preset time, the terminal 110 calls the local AIGC small model to generate content corresponding to the input data, and displays the generated content to the user through the content generation client, ensuring that the user continues to receive content generation services.

[0063] Terminals 110 may include various types of computer devices, such as portable handheld devices, general-purpose computers (e.g., personal computers or laptops), workstation computers, wearable devices, and the like. These computer devices may run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux, or Linux-like operating systems (e.g., Google Chrome OS); or various mobile operating systems, such as Microsoft Windows Mobile OS, Windows Phone, Android, and iPhone OS. Portable handheld devices may include cellular phones, smartphones, tablet computers, and personal digital assistants (PDAs). Wearable devices may include head-mounted displays and smart watches.

[0064] The network 120 may be any type of network known to those skilled in the art, and may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, 3G, 4G, 5G, etc. For example, the one or more networks 120 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0065] Server 130 may include one or more general-purpose computers, specialized server computers (e.g., personal computer servers, UNIX servers, terminal servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement or combination. Server 130 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization, such as one or more flexible pools of logical storage devices that may be virtualized to maintain a virtual storage device for the server. In various embodiments, server 130 may run one or more services or software applications that provide the functionality described below.

[0066] The computing units in server 130 may run one or more operating systems including any of the operating systems described above as well as any commercially available server operating systems. Server 130 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, and the like.

[0067] In some embodiments, server 130 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of terminal 110. Server 130 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of terminal 110.

[0068] In some embodiments, server 130 may be a distributed system server or a server integrated with blockchain. Server 130 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host equipped with artificial intelligence technology. A cloud server is a host product within the cloud computing service system that addresses the management difficulties and poor scalability of traditional physical hosts and virtual private servers (VPS) services.

[0069] The system 100 of FIG. 1 may be configured and operated in various ways to enable the application of various methods and devices provided according to embodiments of the present application.

[0070] Figure 2 is a flow chart of a content generation method provided in an embodiment of the present application. This method can be executed by terminal 110 in Figure 1 to enable the terminal to obtain high-quality content generation responses in a good network environment and continue to obtain content generation responses in a poor network environment, thereby improving the user experience. As shown in Figure 2, the content generation method provided in an embodiment of the present application includes at least steps S201 to S204.

[0071] In step S201 , input data of the AIGC model is obtained.

[0072] The AIGC model in the embodiment of the present application may be an AIGC model that responds to and outputs any type of content. For example, the content output by the AIGC model may be text, image, video, audio, or other types of content.

[0073] AIGC models that output different types of content can be applied to different task scenarios. For example, AIGC models that output text content (such as the ChatGPT model, Pangu ChatGPT model, etc.) can be applied to text creation scenarios, such as news writing, writing in a given format, and style rewriting. For example, the user can enter a description or requirement for the target article, and the model will automatically capture the data and create and output text content that meets the user's requirements according to the instructions described by the user. AIGC models that output image content (such as the various AI drawing models that are currently very popular) can be applied to image creation scenarios. For example, the AIGC model can generate image works based on the text description entered by the user. AIGC models that output video content can be applied to video creation scenarios. AIGC models that output audio content can be applied to audio creation scenarios, such as using the AIGC model to generate virtual human voices.

[0074] Taking the AIGC model as an example, the Pangu ChatGPT model uses text as both input and output. Users can interact with the terminal in various ways, inputting text that expresses their intentions, and the terminal can also output the model-generated content to the user in various ways.

[0075] For example, users can interact with the terminal through text. Users directly input text data through the interactive interface provided by the content generation client deployed on the terminal. The content generation client then feeds back the text content generated by the AIGC model to the user in the form of text.

[0076] For another example, the user can interact with the terminal through voice. For example, the user inputs voice content to the terminal through voice, and then the terminal calls the translation engine to translate the voice content into text content. The terminal calls the translation engine to translate the feedback text output by the AIGC model into voice content, and the terminal outputs the voice content to the user (see Figure 3).

[0077] Of course, users can choose the way to interact with the terminal according to their own needs.

[0078] Optionally, the translation engine can implement mutual conversion between voice content and text content by deploying a speech recognition model.

[0079] The input data of the AIGC model is a text prompt word that conforms to the AIGC model paradigm or grammar. The prompt word can be directly input by the user through the interactive interface of the content generation client.

[0080] For example, users can directly input prompt words by interacting with the content generation client deployed on the terminal. For example, Pangu ChatGPT requires a text prompt word as an information input source. The user enters several prompt words through the content generation client. If the input information does not meet the minimum amount of information required for it to perform the task, it will interact with the user through the content generation client to obtain further information. When the amount of information collected meets the necessary conditions for task initiation, Pangu ChatGPT initiates an inference calculation, generates a response, organizes the response according to the user's requirements, and then outputs the organized response to the user.

[0081] However, when users directly input prompt words, they need to learn the paradigm or grammar of the prompt words of the AIGC model, which requires users to incur a lot of learning costs and time costs. The quality of the prompt words directly affects the quality of the content generated by the AIGC model. Therefore, due to different levels of learning, the quality of the prompt words input by users will vary, and the quality of the content output by the AIGC model will vary, which will affect the user experience.

[0082] In another example, to provide a consistent user experience, a prompt word generation model can be used to generate prompt words that conform to the AIGC model grammar or paradigm. For example, the prompt word generation model automatically generates prompt words that conform to the AIGC model grammar or paradigm based on user input text. For example, the prompt word generation model uses the user input text as input and outputs prompt words that conform to the AIGC model grammar or paradigm corresponding to the user input text.

[0083] By inputting the user's input text into the trained prompt word generation model, prompt words that conform to the grammatical rules of the AIGC model are obtained. Users do not need to learn the grammar or specifications of each AIGC model prompt word, which reduces the user's learning and time costs. Users can directly interact with the terminal according to their daily conversation habits and obtain the response output of the AIGC model that conforms to their own intentions.

[0084] It is understandable that the user input text may be text directly input by the user, or text translated and output by a translation engine.

[0085] Optionally, the prompt word generation model is a trained text generation network, for example, the text generation network can be an MLP network. In one example, the prompt word generation model can be trained using a supervised training method, and the training data set includes multiple training samples, each training sample includes a text sentence with a label, and the label is a prompt word corresponding to the text sentence that conforms to the AIGC model grammar or paradigm. The training sample is used as the input of the prompt word generation model, and the predicted prompt word is output. The loss is determined based on the predicted prompt word and the label, and the weight parameters of the prompt word generation model are adjusted based on the loss. The training is iterated until the training completion condition is reached, that is, a trained prompt word generation model is obtained. Optionally, the training completion condition can be reaching a preset training round or model convergence, etc.

[0086] The quality of content generated by the AIGC model based on prompt words is directly positively correlated with the quality of the prompt words. To generate higher-quality content, specific rules and techniques are required to generate higher-quality prompt words. For example, the prompt words must include one or more of the user's current emotional information and preference information.

[0087] In another example, to further obtain higher-quality prompt words and obtain content output that better meets the user's intention, the generation of prompt words is also related to the context of the user's input text. The context includes one or more of the user's current emotional information and the user's preference information.

[0088] The context information of the input text can be extracted from the user's historical operation records and historical chat records. For example, the user's emotional information, preferences, and personal information can be extracted from the operation records (such as web browsing records) and chat records stored in the terminal, and corresponding tags (tags) are generated, such as emotion tags and preference tags. These tags constitute the context information of the input text. In another example, the user's historical operation records can be historical operation records of the previous 24 hours, and the user's historical chat records can also be historical chat records of the previous 24 hours. In the context information extraction, the closer the extracted information is to the current time (the current time can be understood as the time point when the user interacts with the content generation client to input data), the greater the weight, and the farther the time is from the current time, the smaller the weight; for example, the information extracted from the historical operation records and historical chat records 1 hour ago has the largest weight, and the information extracted from the historical operation records and historical chat records 24 hours ago has the smallest weight. Finally, the context information of the input text is obtained by weighting the information extracted from each hour ago. In other words, the closer the historical operation records and historical chat records are, the more representative the user's current emotions, preferences, and personal information.

[0089] Of course, in some other examples, the time granularity of the user's historical operation records and historical chat records can be larger or smaller. For example, the user's historical operation records and historical chat records can be the history records from the current week, and the history records from one to seven days are extracted respectively, and then the context information of the input text is obtained by weighting. Or the historical operation records and historical chat records can be the history records from the current hour, and the history records from 10 to 60 minutes are extracted respectively, and then the context information of the input text is obtained by weighting.

[0090] Optionally, extracting contextual information about the input text from the terminal's stored operation and chat logs can be implemented using an AI semantic model. The terminal's stored operation and chat logs are used as input for the AI ​​language model, which outputs sentiment tags and preference tags to obtain contextual information about the input text.

[0091] Based on the user's input text and the contextual information of the input text, the prompt word of the AIGC model is obtained. Exemplarily, the input text and contextual information are used as inputs of the prompt word generation model, and the prompt word is output; wherein, the prompt word generation model is trained based on a training data set, and the training data set includes multiple training samples with labels, the training samples include text sentences and several text words, and the labels include prompt words corresponding to the text sentences and several text words. The training samples are used as inputs of the prompt word generation model, and the predicted prompt words are output. The loss is determined based on the predicted prompt words and labels, and the weight parameters of the prompt word generation model are adjusted based on the loss. The training is iterated in this way until the training completion condition is met, that is, a trained prompt word generation model is obtained. Optionally, the training completion condition can be reaching a preset training round or model convergence, etc.

[0092] FIG4 is a schematic diagram showing a process of generating prompt words according to an embodiment of the present application.

[0093] After supplementing the prompt words with contextual information, they will be able to more accurately reflect the user's intention and be more in line with the grammatical preferences of the AIGC model, so that the AIGC model can provide output that is more in line with the user's intention, realizing "the AIGC model understands you better."

[0094] For example, the AIGC model outputs an image, and can complete an AI painting task based on a user-entered prompt. The user is currently feeling very angry and wants the AIGC model to create a picture that reflects their current state of mind. The user's input text is "Draw a fiery phoenix on a paulownia tree." Without additional context, the prompts generated might be "fiery, phoenix, paulownia." In response to the prompts, the AIGC model generates a picture of a fiery phoenix on a paulownia tree. However, the user may feel that the phoenix's fire isn't strong enough to express their current anger. By extracting the user's historical operation records and / or chat history and determining that the user's emotion is "angry," the emotion tag is added to the prompts, resulting in the prompts "angry, fiery, phoenix, paulownia" that capture the user's emotion. In response to the prompts, the AIGC model generates a picture of a fiery phoenix on a paulownia tree, radiating blazing flames. The user receives an AI painting that matches their intent, feeling that the AIGC model understands them well, improving their user experience.

[0095] For another example, if the user inputs "Draw a running Sam," the prompt word generated without supplementary contextual information might be "Running, Sam." In response to this prompt word, the AIGC model might generate a picture of a running little boy. However, Sam is the user's pet dog, and the user's true intention is to create a picture of their pet dog running. By extracting the user's historical operation records and / or chat history, we can determine that Sam is actually a pet dog raised by the user, as well as Sam's facial features. Sam's actual information is then added to the prompt word, resulting in the prompt word "Running, Dog," which expresses the user's preference. In response to this prompt word, the AIGC model generates a picture of a running dog, whose facial features match those of Sam, the user's pet dog. The user then obtains an AI painting that meets their specific needs.

[0096] In step S202 , input data is uploaded to a server via a first routing path, so as to generate first content through a remote AIGC large model deployed on the server.

[0097] Optionally, the server may be a cloud server, that is, the remote AIGC large model is deployed in the cloud to provide high-quality content generation services to users.

[0098] When the content generation client obtains a prompt word that can express the user's intention, it preferentially sends the prompt word to the remote AIGC large model deployed in the cloud through the first routing path. The remote AIGC large model responds to the input prompt word and generates output content that meets the user's intention.

[0099] In step S203, the first content returned by the server is received.

[0100] After the remote AIGC large model deployed in the cloud generates content that meets the user's intention through inference, the content is fed back to the terminal, and the first content is displayed to the user through the content generation client deployed in the terminal.

[0101] In step S204, when an exception occurs in obtaining the first content through the first routing path, the input data is input into the local AIGC mini-model deployed locally through the second routing path to obtain the second content.

[0102] The terminal preferentially inputs the prompt word into the remote AIGC large model deployed on the cloud side, which then generates the content the user needs. If an exception occurs, the terminal inputs the prompt word into the locally deployed small AIGC model, which then generates the required content. This ensures that content generation services continue even when the cloud side responds to an exception.

[0103] Abnormal situations can be understood as: abnormal terminal content acquisition due to abnormal response of the generative large model deployed at the remote end, such as one or more of abnormal content quality output by the generative large model in response to input data and abnormal time output of the generative large model in response to input data.

[0104] For example, due to the large number of parameters in generative models (often reaching hundreds of billions or trillions), they may experience "intelligence decline" after a training update. This can lead to reduced accuracy of generated answers, irrelevance to the question, and other issues. Another example is a generative model deployed in the cloud to provide content generation services to the public. When too many users access it, insufficient computing power may cause the model's inference time to be excessive, leading to extended response times for users.

[0105] In another example, the exception may also include an abnormal situation that causes the terminal to fail to receive a response from the remote AIGC large model, or fail to receive a response from the remote AIGC large model within a preset time, such as one or more of an abnormality in the communication link between the terminal device and the server, an abnormality in the response output of the first content generation model deployed on the server, and an abnormality in the server.

[0106] For example, the terminal can detect whether the content generated by the remote AIGC large model is received within a preset time period (for example, the preset time period can be 1 minute) after the prompt word is sent to the cloud side. If it is detected that the content response generated by the remote AIGC large model is not received within the preset time period, the prompt word is input into the locally deployed local AIGC small model, and the locally deployed local AIGC small model is used to generate the required content.

[0107] For another example, the terminal can detect whether the communication link between the terminal and the cloud server is abnormal. For example, when the terminal is in an environment with poor network signal or no network signal, such as an elevator, subway or airplane, the terminal detects that the communication link between the terminal and the cloud server is abnormal, resulting in problems in sending or receiving information. The prompt word is input into the locally deployed local AIGC small model, and the locally deployed local AIGC small model is used to generate the required content.

[0108] For another example, the response output of the remote AIGC large model deployed on the cloud server produces an exception. The terminal receives a message from the cloud side notifying the content generation service of the exception. The terminal inputs the prompt word to the locally deployed local AIGC small model and uses the locally deployed local AIGC small model to generate the required content.

[0109] For another example, an exception occurs on the cloud server where the remote AIGC large model is deployed. The terminal receives the exception notification message from the cloud side, and the terminal inputs the prompt word to the locally deployed local AIGC small model, and uses the locally deployed local AIGC small model to generate the required content.

[0110] Taking Pangu ChatGPT as an example, the remote AIGC large model is used. The local AIGC small model can be referred to as the local model. Figure 5 shows a schematic block diagram of content generation. As shown in Figure 5, the terminal deploys two soft routers, including Router A and Router B. When the terminal receives the prompt word input, both Router A and Router B will prioritize Path A and send the prompt word to Pangu ChatGPT deployed on the cloud side, obtaining high-quality content output by Pangu ChatGPT deployed on the cloud side. When the communication link is abnormal or the response of Pangu ChatGPT on the cloud side is abnormal, Router A and Router B select Path B. The terminal loads the local model, temporarily replacing Pangu ChatGPT on the cloud side, and provides timely feedback output to the user.

[0111] It is understandable that the parameter scale of the remote AIGC large model is greater than that of the local AIGC small model. For example, the parameter scale of the remote AIGC large model is in the hundreds of billions to trillions, while the parameter scale of the local AIGC small model is in the billions. The storage overhead of the remote AIGC large model is 700G, and the computing power overhead is 100+PTFLOPS, while the storage overhead of the local AIGC small model is 2 to 4G, and the computing power overhead is 10-20TFLOPS.

[0112] Optionally, the local AIGC small model is an AIGC model obtained by processing the remote AIGC large model through knowledge distillation, quantization, pruning, etc. The response output of the local AIGC small model is relatively simpler than the response output of the remote AIGC large model, but the response quality is still within the user's acceptable range.

[0113] In another example, the local AIGC small model and the remote AIGC large model can also be generative models trained separately, but the local AIGC small model and the remote AIGC large model respond to the same input data and output content in the same format. For example, the local AIGC small model and the remote AIGC large model respond to the input data and output content in image format or text format, etc.

[0114] In another example, an auxiliary small model is also deployed on the terminal device, which includes a multi-layer neural network; when the communication link is abnormal or the Pangu ChatGPT response on the cloud side is abnormal, the terminal uses the prompt word as the input of the local model and the auxiliary small model respectively, and the prompt word is processed by the multi-layer neural network to obtain a first processing result, and the prompt word is processed by the multi-layer intermediate layer neural network of the local model to obtain a second processing result; the output layer of the local model processes the first processing result and the second processing result, and outputs the second content.

[0115] By setting up an auxiliary model on the terminal, the auxiliary model and the local AIGC model work together to achieve responsive output that meets some customized user needs. For example, different users may have different preferences. For example, some users may prefer to interact in modern Chinese. In this case, an auxiliary model can be trained with modern Chinese. After the auxiliary model is applied to the local AIGC model, the output layer of the local AIGC model will output modern Chinese that meets the user's preferences. However, some users may prefer to interact in classical Chinese. In this case, an auxiliary model can be trained with classical Chinese. After the auxiliary model is applied to the local AIGC model, the output layer of the local AIGC model will output modern Chinese that meets the user's preferences.

[0116] It is understandable that the subsidiary mini-model is similar to a patch of traditional software, providing functions including but not limited to bug fixes, feature enhancements, etc.

[0117] The auxiliary small model can be called the auxiliary part, and the local AIGC small model can be called the main part. The optional auxiliary part only contains some intermediate layer weight information, so its size is generally much smaller than the main part and does not significantly increase the terminal's computational load. A main neural network can simultaneously load multiple auxiliary neural networks to achieve different functions or purposes. The relationship between them is shown in Figure 6.

[0118] After the main and subsidiary neural networks are loaded, input data is fed to each of them simultaneously. After their respective calculations, the results are aggregated at the output layer. The subsidiary neural networks influence the results of the main network by configuring weights at their corresponding layers, ultimately producing personalized output for each user, improving the user experience.

[0119] Figure 7 shows a system block diagram of a specific implementation of the content generation method provided by an embodiment of the present application. As shown in Figure 7, the user interacts with the content generation client on the terminal and inputs chat information by voice. The terminal generates emotion tags and preference tags based on the user's operations and chat records in the local application; the terminal uses the user's input text (the text generated by translating the user's input voice by the translation engine) and the emotion tag and preference tag as the input of the prompt word generation model, outputs prompt words that meet the characteristics of Pangu ChatGPT, sends the prompt words to Pangu ChatGPT on the cloud side via the network, and waits for a response. Pangu ChatGPT on the cloud side generates high-quality content output text in response to the prompt words, and the content output text is translated into voice output to the user by the translation engine. When the cloud side output data transmission or feedback is abnormal, the routing deployed by the terminal will switch to the local model and activate it to provide timely feedback. The output of the local model is relatively simple compared to the output of ChatGPT on the cloud side.

[0120] For example, the output of ChatGPT on the cloud side in response to the prompt word is more detailed and has more words (for example, about 1,000 words), as well as some related web links. The output of the local model in response to the prompt word is more concise and has fewer words (for example, 100 words), and does not include relevant web links.

[0121] Based on the same concept as the aforementioned embodiment of the content generation method, the present embodiment further provides a content generation device 800. This content generation device 800 can be deployed on the terminal 110 shown in FIG1 . When the network environment is good, high-quality content can be obtained through the remote AIGC large model deployed on the cloud side. When the network signal is abnormal, timely content feedback can still be obtained through the local AIGC small model deployed locally. The content generation device 800 includes units or modules for implementing each step of the content generation method shown in FIG2-7 .

[0122] FIG8 is a schematic diagram of the structure of a content generation device provided by an embodiment of the present application. As shown in FIG8 , the content generation device 800 includes at least an acquisition module 801, a sending module 802, a receiving module 803, and an input module 804, wherein the acquisition module 801 is used to acquire input data of a content generation model; the sending module 802 is used to upload the input data to a server via a first routing path; the receiving module 803 is used to receive first content returned by the server, where the first content is generated by a first content generation model deployed on the server in response to the input data, and the first content generation model is a generative large model; the input module 804 is used to input the input data to a second content generation model deployed locally via a second routing path when an exception occurs in acquiring the first content via the first routing path, and obtain second content, where the exception is related to the first content generation model; wherein the number of parameters of the second content generation model is smaller than the number of parameters of the first content generation model.

[0123] In another possible implementation, the anomaly includes one or more of an anomaly in the quality of content output by the generative large model in response to the input data and an anomaly in the timing of the content output by the generative large model in response to the input data.

[0124] In another example, the abnormality may also include: an abnormality in the communication link between the terminal device and the server, an abnormality in the response output of the first content generation model deployed on the server, and an abnormality in the server.

[0125] Optionally, the server is a cloud server.

[0126] In another possible implementation, an auxiliary small model is also deployed on the terminal device, which includes a multi-layer neural network; the second content generation model includes a multi-layer intermediate layer neural network and an output layer; the input data is input into the second content generation model deployed locally through a second routing path, and a specific implementation of obtaining the second content is: the input data is used as the input of the second content generation model respectively, the input data is processed by the multi-layer neural network to obtain a first processing result, and the input data is processed by the multi-layer intermediate layer neural network to obtain a second processing result; the output layer processes the first processing result and the second processing result, and outputs the second content.

[0127] In another possible implementation, the input data includes a prompt word; the acquisition module 801 is specifically used to: receive the data input by the user; obtain the input text based on the data input by the user; obtain context information of the input text; and obtain the prompt word based on the input text and the context information.

[0128] In another possible implementation, obtaining context information of the input text includes: obtaining historical operation records and historical chat records of the user; and obtaining context information based on the historical operation records and historical chat records.

[0129] In another possible implementation, the context information includes one or more of the user's emotion information and the user's preference information.

[0130] In another possible implementation, a specific implementation of obtaining a prompt word based on input text and context information is: using the input text and context information as input to a prompt word generation model, and outputting a prompt word; wherein the prompt word generation model is trained based on a training data set, the training data set includes multiple training samples with labels, the training samples include text sentences and several text words, and the labels include prompt words corresponding to the text sentences and several text words.

[0131] Optionally, the prompt word generation model is a multi-layer perceptron network.

[0132] According to the embodiment of the present application, the content generating device 800 can correspond to executing the method described in the embodiment of the present application, and the above-mentioned and other operations and / or functions of each module in the content generating device 800 are respectively for implementing the corresponding processes of each method in Figures 2-7. For the sake of brevity, they will not be repeated here.

[0133] An embodiment of the present application also provides a terminal device, including at least one processor, a memory, and a communication interface, wherein the processor is used to execute the method described in Figures 2-7.

[0134] FIG9 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application.

[0135] As shown in Figure 9, the terminal device 900 includes at least one processor 901, a memory 902, and a communication interface 903. The processor 901, memory 902, and communication interface 903 are communicatively connected, either via a wired connection (e.g., a bus) or a wireless connection. The communication interface 903 is used to send and / or receive data sent by other devices, such as a terminal sending prompt words to a remote AIGC large model deployed on the cloud side, and receiving the content output by the remote AIGC large model deployed on the cloud side. The memory 902 stores computer instructions, and the processor 901 executes these computer instructions to perform the content generation method in the aforementioned method embodiment. When the generative large model deployed on the remote server side responds normally, the generative large model deployed on the server side generates content, resulting in high-quality content. When the content generation model deployed on the server side responds abnormally, a locally deployed miniaturized content generation model is used to generate the required content, ensuring that content generation services can continue to be provided even when the generative large model deployed on the server side responds abnormally.

[0136] It should be understood that in the embodiment of the present application, the processor 901 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0137] The memory 902 may include a read-only memory and a random access memory, and provides instructions and data to the processor 901. The memory 902 may also include a nonvolatile random access memory.

[0138] The memory 902 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0139] It should be understood that the terminal device 900 according to the embodiment of the present application can execute the method shown in Figures 2-7 in the embodiment of the present application. The detailed description of the implementation of the method is given above and will not be repeated here for the sake of brevity.

[0140] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned method is implemented.

[0141] An embodiment of the present application provides a chip, which includes at least one processor and an interface, wherein the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the method mentioned above.

[0142] An embodiment of the present application provides a computer program or a computer program product, which includes instructions. When the instructions are executed, the computer is caused to perform the above-mentioned method.

[0143] Those skilled in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0144] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0145] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation methods of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A content generation method, characterized in that: Applied to a terminal device, the method comprises: Obtain input data for the content generation model; Uploading the input data to a server via a first routing path; receiving the first content returned by the server, where the first content is generated by a first content generation model deployed on the server, and the first content generation model is a generative large model; When an exception occurs when acquiring the first content through the first routing path, inputting the input data into a second content generation model deployed locally through a second routing path to obtain second content, and the exception is related to the first content generation model; The number of parameters of the second content generation model is smaller than the number of parameters of the first content generation model.

2. The method according to claim 1, characterized in that The anomaly includes: one or more of an anomaly in content quality output by the first content generation model and an anomaly in the time when the first content generation model outputs the first content in response to the input data.

3. The method according to claim 1 or 2, characterized in that: The server is a cloud server.

4. The method according to any one of claims 1 to 3, characterized in that: The terminal device is also deployed with an auxiliary small model, the auxiliary small model includes a multi-layer neural network; the second content generation model includes a multi-layer intermediate layer neural network and an output layer; The step of inputting the input data to a second content generation model deployed locally through a second routing path to obtain second content includes: The input data is used as input of the second content generation model and the subsidiary small model respectively, the input data is processed by the multi-layer neural network to obtain a first processing result, and the input data is processed by the multi-layer intermediate layer neural network to obtain a second processing result; The output layer processes the first processing result and the second processing result, and outputs the second content.

5. The method according to any one of claims 1 to 4, characterized in that: The input data includes prompt words; The step of obtaining input data of the content generation model includes: Receive data input by the user; Based on the data input by the user, obtaining input text; Obtaining context information of the input text; The prompt word is obtained based on the input text and the context information.

6. The method according to claim 5, characterized in that The obtaining of the context information of the input text includes: Obtain the user's historical operation records and historical chat records; The context information is obtained based on the historical operation records and the historical chat records.

7. A content generating device, characterized in that: Deployed on a terminal device, the device includes: An acquisition module, used to obtain input data of a content generation model; A sending module, used for uploading the input data to a server via a first routing path; A receiving module, configured to receive the first content returned by the server, wherein the first content is generated by a first content generation model deployed on the server, and the first content generation model is a generative large model; An input module, configured to input the input data to a second content generation model deployed locally through a second routing path to obtain second content when an exception occurs in acquiring the first content through a first routing path, and the exception is related to the first content generation model; The number of parameters of the second content generation model is smaller than the number of parameters of the first content generation model.

8. The device according to claim 7, characterized in that The anomaly includes: one or more of an anomaly in content quality output by the first content generation model and an anomaly in the time when the first content generation model outputs the first content in response to the input data.

9. The device according to claim 7 or 8, characterized in that The server is a cloud server.

10. The device according to any one of claims 7 to 9, characterized in that: The terminal device is also deployed with an auxiliary small model, the auxiliary small model includes a multi-layer neural network; the second content generation model includes a multi-layer intermediate layer neural network and an output layer; The step of inputting the input data to a second content generation model deployed locally through a second routing path to obtain second content includes: The input data is used as the input of the second content generation model and the subsidiary small model respectively, and the input data passes through the multiple layers The neural network processes the input data to obtain a first processing result, and the input data is processed by the multi-layer intermediate layer neural network to obtain a second processing result; The output layer processes the first processing result and the second processing result, and outputs the second content.

11. The device according to any one of claims 7 to 10, characterized in that: The input data includes prompt words; The acquisition module is specifically used for: Receive data input by the user; Based on the data input by the user, obtaining input text; Obtaining context information of the input text; The prompt word is obtained based on the input text and the context information.

12. The device according to claim 11, characterized in that The obtaining of the context information of the input text includes: Obtain the user's historical operation records and historical chat records; The context information is obtained based on the historical operation records and the historical chat records.

13. A terminal comprising a memory and a processor, characterized in that: Instructions are stored in the memory, and when the instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

14. A chip, characterized in that: The method comprises at least one processor and a communication interface, wherein the processor is configured to execute the method according to any one of claims 1 to 6.

15. A computing device comprising a memory and a processor, characterized in that: Instructions are stored in the memory, and when the instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images