Content generation method and device

Abstract

The invention provides a content generation method, which is applied to terminal equipment and comprises the following steps: acquiring input data of a content generation model; uploading the input data to a server through a first routing path; receiving first content returned by the server, wherein the first content is generated by a first content generation model deployed on the server; when obtaining the first content through the first routing path is abnormal, inputting the input data to a second content generation model deployed locally through a second routing path to obtain second content; wherein the number of parameters of the second content generation model is smaller than the number of parameters of the first content generation model. According to the content generation method provided by the invention, when the response of the content generation model deployed at the server side is abnormal, the required content is generated by adopting the locally deployed miniaturized content generation model, so that the content generation service is prevented from being influenced when the network is abnormal or the server is abnormal.

Description

Technical Field

[0001] The present application relates to the technical field of terminal equipment, and in particular to a content generation method and device. Background Art

[0002] AI generated content (AIGC) uses artificial intelligence technology to generate content. It is a new content production method after professional-generated content (PGC) and user-generated content (UGC), creating a new form of digital content generation and interaction. The core idea of ​​AIGC technology is to use artificial intelligence algorithms to generate content with certain creativity and quality. Through training models and learning from large amounts of data, AIGC can generate relevant content based on the input conditions and guidance. For example, by entering keywords, descriptions, or text, AIGC can generate articles, images, audio, etc. that match them.

[0003] In existing AIGC services (e.g., ChatGPT, Midjourney, etc.), models are often deployed in the cloud. Users input data through the client deployed on the terminal device, and the AIGC model deployed in the cloud responds to the user's input and returns the generated content. However, when the terminal is in an environment with no network signal or poor network signal, the AIGC service is abnormal and users cannot get the generated content in time. Summary of the invention

[0004] The embodiments of the present application provide a content generation method, which can enable a terminal to continue to respond to content generation requests in the event of network anomalies.

[0005] In a first aspect, the present application provides a content generation method, which is applied to a terminal device, and the method includes: obtaining input data of a content generation model (which may be referred to as an AIGC model); uploading the input data to a server through a first routing path; receiving first content returned by the server, the first content being generated by a first content generation model deployed on the server in response to the input data, and the first content generation model being a generative large model; when an exception occurs when obtaining the first content through the first routing path, inputting the input data into a second content generation model deployed locally through a second routing path to obtain second content; wherein the number of parameters of the second content generation model is less than the number of parameters of the first content generation model.

[0006] The content generation method provided by the present application deploys a miniaturized content generation model on the terminal and a large content generation model on the server side. When the large generative model deployed at the remote end (i.e., the server side) responds normally, the content generation model deployed on the server side is used to generate content to obtain high-quality content. When the content generation model deployed on the server side responds abnormally, a locally deployed miniaturized content generation model is used to generate the required content, thereby achieving continuous content generation service when the network or server is abnormal.

[0007] The abnormality in obtaining the first content through the first routing path can be understood as: abnormal content acquisition caused by abnormal response of the generative big model deployed at the remote end, such as one or more of abnormal content quality output by the generative big model in response to input data and abnormal time of content output by the generative big model in response to input data.

[0008] For example, due to the large number of parameters (usually reaching hundreds of billions or trillions), the generative large model may experience "intelligence decline" after a training update, such as a decrease in the accuracy of the generated answers, or the generated answers are irrelevant to the questions, etc. For another example, the generative large model is deployed in the cloud to provide content generation cloud services to the public. When there are too many access users, the generative large model may take too long to infer due to insufficient computing power, which in turn causes users to take too long to get responses.

[0009] In another example, the exception can also be understood as an abnormal situation that causes the terminal to fail to receive a response from the generative large model, or fail to receive a response from the generative large model within a preset time, such as one or more of an abnormality in the communication link between the terminal device and the server, an abnormality in the response output of the first content generation model deployed on the server, and an abnormality in the server.

[0010] Optionally, the second content generation model may be a generative model with a smaller number of parameters obtained by compressing the first content generation model. For example, the second content generation model is a model obtained by subjecting the first content generation model to compression processing such as knowledge distillation, quantization, and pruning.

[0011] In another example, the first content generation model and the second content generation model may also be generative models that are trained separately, but the first content generation model and the second content generation model respond to the same input data and output content in the same format. For example, the first content generation model and the second content generation model respond to the input data and both output content in image format, or both output content in text format, etc.

[0012] In one possible implementation, the server is a cloud server, that is, the generative large model is deployed in the cloud to provide content generation services to users.

[0013] In a possible implementation, an auxiliary small model is also deployed on the terminal device, and the auxiliary small model includes a multi-layer neural network; the second content generation model includes a multi-layer intermediate layer neural network and an output layer; the input data is input into the second content generation model deployed locally through a second routing path, and a specific implementation of obtaining the second content is: the input data is used as the input of the second content generation model and the auxiliary small model respectively, the input data is processed by the multi-layer neural network to obtain a first processing result, and the input data is processed by the multi-layer intermediate layer neural network to obtain a second processing result; the output layer processes the first processing result and the second processing result, and outputs the second content.

[0014] By setting up a subsidiary small model on the terminal, the subsidiary small model and the second content generation model deployed locally work together to achieve response output that meets some customized needs of users. For example, different users may have different preferences. For example, some users may want to interact in modern Chinese. In this case, a subsidiary small model can be trained with modern Chinese. After the subsidiary small model acts on the second content generation model deployed locally, the output layer of the second content generation model deployed locally outputs modern Chinese that meets the user's preferences. However, some users may prefer "drag text" and like to interact in ancient Chinese. In this case, an subsidiary small model can be trained with ancient Chinese. After the subsidiary small model acts on the second content generation model deployed locally, the output layer of the second content generation model deployed locally outputs modern Chinese that meets the user's preferences.

[0015] The first content generation model deployed on the server can be called a remote AIGC large model, and the second content generation model deployed locally on the terminal can be called a local AIGC small model. It can be understood that the second content generation model is called a local AIGC small model only because it has fewer parameters and a smaller volume than the remote AIGC large model. For example, the number of parameters of the local AIGC small model is in the billions, the storage overhead is 2 to 4G, and the computing power overhead is 10-20TFLOP.

[0016] The parameter scale of the remote AIGC large model is greater than that of the local AIGC small model. For example, the parameter scale of the remote AIGC large model is in the hundreds of billions to trillions, while the parameter scale of the local AIGC small model is in the billions. The storage overhead of the remote AIGC large model is 700G, and the computing power overhead is 100+PTFLOPS, while the storage overhead of the local AIGC small model is 2 to 4G, and the computing power overhead is 10-20TFLOPS.

[0017] In an example, the local AIGC small model is an AIGC model obtained after the remote AIGC large model has been processed through knowledge distillation, quantization, pruning, etc. The response output of the local AIGC small model is relatively simpler than that of the remote AIGC large model, but the response quality is still within the user's acceptable range.

[0018] In another possible implementation, the input data includes a prompt; a specific implementation of obtaining the input data of the AIGC model is: receiving data input by a user; obtaining input text based on the data input by the user; obtaining context information of the input text; and obtaining a prompt based on the input text and the context information.

[0019] In this possible implementation, the prompt words obtained through context information and input text can more accurately reflect the user's intention, so that the AIGC model can output output that is more in line with the user's intention.

[0020] In another possible implementation, a specific implementation of obtaining the context information of the input text is: obtaining the user's historical operation records and historical chat records; and obtaining the context information based on the historical operation records and historical chat records.

[0021] Exemplarily, the user's emotional information and preferences are extracted from the operation records (such as web browsing records) and chat records stored in the terminal, and corresponding tags are generated. These tags constitute the context information of the input text.

[0022] In another possible implementation, a specific implementation of obtaining a prompt word based on input text and context information is: using the input text and context information as input of a prompt word generation model, and outputting a prompt word; wherein the prompt word generation model is trained based on a training data set, the training data set includes multiple training samples with labels, the training samples include text sentences and several text words, and the labels include the text sentences and prompt words corresponding to the several text words.

[0023] By inputting the input text and context information into the trained prompt word generation model, prompt words that conform to the grammatical rules of the AIGC model are obtained. Users do not need to learn the grammar or specifications of each AIGC model prompt word, which reduces the user's learning and time costs. Users can directly interact with the terminal according to their daily conversation habits and obtain the response output of the AIGC model that conforms to their intentions.

[0024] Optionally, the prompt word generation model is a text generation network, such as a multi-layer perception (MLP) network.

[0025] In a second aspect, the present application provides a content generation device, which is deployed on a terminal device and includes an acquisition module, a sending module, a receiving module and an input module, wherein the acquisition module is used to acquire input data of a content generation model; the sending module is used to upload the input data to a server through a first routing path; the receiving module is used to receive a first content returned by the server, wherein the first content is generated by a first content generation model deployed on the server in response to the input data, and the first content generation model is a generative large model; the input module is used to input the input data to a second content generation model deployed locally through a second routing path when an exception occurs when acquiring the first content through the first routing path, so as to obtain the second content, and the exception is related to the first content generation model; wherein the number of parameters of the second content generation model is less than the number of parameters of the first content generation model.

[0026] In another possible implementation, the anomaly includes one or more of an anomaly in content quality of the generative large model output in response to input data and an anomaly in time of content output by the generative large model in response to input data.

[0027] In another example, the anomaly may be one or more of an anomaly in the communication link between the terminal device and the server, an anomaly in the response output of the first content generation model deployed on the server, and an anomaly in the server.

[0028] Optionally, the server is a cloud server.

[0029] In another possible implementation, an auxiliary small model is also deployed on the terminal device, which includes a multi-layer neural network; the second content generation model includes a multi-layer intermediate layer neural network and an output layer; the input data is input into the second content generation model deployed locally through a second routing path, and a specific implementation of obtaining the second content is: the input data is used as the input of the second content generation model, the input data is processed by the multi-layer neural network to obtain a first processing result, and the input data is processed by the multi-layer intermediate layer neural network to obtain a second processing result; the output layer processes the first processing result and the second processing result, and outputs the second content.

[0030] In another possible implementation, the input data includes a prompt word; the acquisition module is specifically used to: receive data input by a user; obtain input text based on the data input by the user; obtain context information of the input text; and obtain a prompt word based on the input text and the context information.

[0031] In another possible implementation, obtaining context information of the input text includes: obtaining historical operation records and historical chat records of the user; and obtaining context information based on the historical operation records and historical chat records.

[0032] In another possible implementation, the context information includes one or more of the user's emotion information and the user's preference information.

[0033] In another possible implementation, a specific implementation of obtaining a prompt word based on input text and context information is: using the input text and context information as input of a prompt word generation model, and outputting a prompt word; wherein the prompt word generation model is trained based on a training data set, the training data set includes multiple training samples with labels, the training samples include text sentences and several text words, and the labels include prompt words corresponding to the text sentences and the several text words.

[0034] Optionally, the prompt word generation model is a multi-layer perceptron network.

[0035] In a third aspect, an embodiment of the present application provides a computing device, including a memory and a processor, wherein the memory stores instructions, and when the instructions are executed by the processor, the method described in the first aspect is implemented.

[0036] In a fourth aspect, an embodiment of the present application provides a computing device, including a memory and a processor, wherein the memory stores instructions, and when the instructions are executed by the processor, the method described in the first aspect is implemented.

[0037] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0038] In a sixth aspect, an embodiment of the present application further provides a computer program or a computer program product, wherein the computer program or the computer program product comprises instructions, which, when executed, cause a computer to execute the method described in the first aspect.

[0039] In a seventh aspect, an embodiment of the present application further provides a chip, comprising at least one processor and a communication interface, wherein the processor is used to execute the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of an exemplary system 100 is shown in which various methods and apparatuses described herein may be implemented according to an embodiment disclosed in the present application;

[0041] Figure 2 A flowchart of a content generation method provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of interacting with a terminal by voice;

[0043] Figure 4A schematic diagram showing a process of generating a prompt word according to an embodiment of the present application is shown;

[0044] Figure 5 A schematic block diagram of content generation is shown;

[0045] Figure 6 It is a schematic diagram of the relationship between the subsidiary small model and the local model;

[0046] Figure 7 A system overall block diagram showing a specific implementation of the content generation method provided in an embodiment of the present application;

[0047] Figure 8 A schematic diagram of the structure of a content generation device provided in an embodiment of the present application;

[0048] Fig. 9 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] At present, AI technology has developed to a new stage, and it is a general trend to use AI technology to provide users with a better intelligent experience. For example, Google Assistant and Stable Diffusion. Google Assistant is a virtual assistant equipped with artificial intelligence, which can conduct two-way conversations. Users can interact with Google Assistant through natural language / voice to improve the intelligence of terminal devices. Stable Diffusion is a deep learning model that can generate corresponding images from text descriptions. However, both Google Assistant and Stable Diffusion are deployed on terminal devices, mainly relying on local computing power. Due to the lack of terminal computing power, the content output quality of Google Assistant and Stable Diffusion is poor compared to large models.

[0050] Currently, AI models are basically concentrated on super-large models, and the demand for computing power is growing exponentially. However, terminal chips are subject to multiple constraints such as area, power consumption, heat dissipation, and process. The growth rate of computing power is difficult to match the rapidly growing demand for AI computing, making it difficult for products like Google Assistant to provide capabilities that satisfy users.

[0051] Another way to solve the bottleneck of terminal computing power is to use end-cloud collaboration and high-speed network connections to offload part of the load to remote cloud servers. However, this solution also has a problem of being too dependent on the network. In places where the network connection is not very stable, such as subways and elevators, data transmission delays are greatly increased, and there is even the possibility of interruption, which will greatly affect the user experience.

[0052] For example, ChatGPT and Midtravel are both deployed on the cloud and provide content generation services on the cloud, but both ChatGPT and Midtravel have experienced service interruptions, resulting in poor user experience.

[0053] In response to the above problems, an embodiment of the present application provides a content generation method, which deploys a local AIGC small model on the terminal and a remote AIGC large model on the server side. When the communication link between the terminal and the server is normally connected, the remote AIGC large model deployed on the server side is used to generate content to obtain high-quality content; when the remote AIGC large model deployed on the server side responds abnormally, the locally deployed local AIGC small model is used to generate the required content, so that when the network or server is abnormal, the content generation service can continue to be obtained.

[0054] The technical solution of the present application is further described in detail below through the accompanying drawings and embodiments.

[0055] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and devices described herein may be implemented according to an embodiment disclosed in the present application. Figure 1 As shown, the system 100 includes a terminal 110, a server 130, and a network 120 that connects the terminal 110 and the server 130. The terminal 110 can run one or more services or software applications that can execute the content generation method, a local AIGC small model is deployed on the terminal 110, and a remote AIGC large model is deployed on the server 130.

[0056] In some embodiments, server 130 can also provide other services or software applications including non-virtualized environments and virtualized environments. In some embodiments, these services can be provided as web-based services or cloud services, such as provided to users of terminals under a software as a service (SaaS) model.

[0057] exist Figure 1 In the system shown, the server 130 may include one or more components that implement the functions performed by the server 130. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user of the operating terminal 110 may in turn use one or more client applications to interact with the server 130 to utilize the services provided by these components. It should be understood that Figure 1 This is only an example of a system for implementing the various methods described in the embodiments of the present application. There may be other different configurations, and the embodiments of the present application are not limited to this.

[0058] The user can use the content generation client on the terminal 110 to send input data to the server 130. The terminal 110 can provide an interface that enables the user of the terminal 110 to interact with the terminal. The terminal 110 can also output information to the user via the interface. After receiving the input data, the server 130 uses the input data as the input of the remote AIGC large model, responds with high-quality content, and feeds the content back to the terminal 110. The terminal 110 displays the generated content to the user through the content generation client. When the terminal 110 does not receive the content fed back by the server 130 within the preset time, the terminal 110 calls the local AIGC small model to generate content corresponding to the input data, and displays the generated content to the user through the content generation client to ensure that the user continues to receive content generation services.

[0059] The terminal 110 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers or laptop computers), workstation computers, wearable devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple IOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or various mobile operating systems, such as Microsoft Windows Mobile OS, Windows phone, Android, iPhone OS, etc. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays and smart watches, etc.

[0060] The network 120 may be any type of network known to those skilled in the art, and may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, 3G, 4G, 5G, etc. Exemplarily, the one or more networks 120 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0061] Server 130 may include one or more general purpose computers, dedicated server computers (e.g., personal computer servers, UNIX servers, terminal servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement or combination. Server 130 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization, such as one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device. In various embodiments, server 130 may run one or more services or software applications that provide the functionality described below.

[0062] The computing units in the server 130 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 130 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0063] In some embodiments, server 130 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of terminal 110. Server 130 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of terminal 110.

[0064] In some embodiments, the server 130 may be a server of a distributed system or a server combined with a blockchain. The server 130 may also be a cloud server or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0065] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and devices provided according to the embodiments of the present application.

[0066] Figure 2 A flow chart of a content generation method provided in an embodiment of the present application is shown in FIG. Figure 1 The terminal 110 in the embodiment performs the implementation so that the terminal can obtain a high-quality content generation response in a good network environment and can continue to obtain a content generation response in a poor network environment, thereby improving the user experience. Figure 2 As shown, the content generation method provided in the embodiment of the present application at least includes steps S201 to S204.

[0067] In step S201, the input data of the AIGC model is obtained.

[0068] The AIGC model in the embodiment of the present application may be an AIGC model that responds to and outputs any type of content. For example, the content output by the AIGC model may be text, image, video, audio, or other types of content.

[0069] AIGC models that output different types of content can be applied to different task scenarios. For example, AIGC models that output text content (such as ChatGPT model, Pangu ChatGPT model, etc.) can be applied to text creation scenarios, such as news writing, writing in a given format, and style rewriting. For example, the user can enter a description or requirement for the target article, and the model will automatically capture the data and create and output text content that meets the user's requirements according to the instructions described by the user. AIGC models that output image content (such as various AI drawing models that are currently very popular) can be applied to image creation scenarios. For example, the AIGC model can generate image works based on the text description entered by the user. AIGC models that output video content can be applied to video creation scenarios. AIGC models that output audio content can be applied to audio creation scenarios, such as using the AIGC model to generate virtual human voices.

[0070] Taking the AIGC model as the Pangu ChatGPT model as an example, the input and output of the Pangu ChatGPT model are both text. Users can interact with the terminal in a variety of ways, input text content that can express their intentions into the terminal, and the corresponding terminal can also output the content generated by the model to the user in a variety of ways.

[0071] For example, users can interact with the terminal through text. Users directly input text data through the interactive interface provided by the content generation client deployed on the terminal. The content generation client then feeds back the text content generated by the AIGC model to the user in the form of text.

[0072] For another example, the user can interact with the terminal in the form of voice. For example, the user inputs voice content to the terminal through voice, and then the terminal calls the translation engine to translate the voice content into text content. The terminal calls the translation engine to translate the feedback text output by the AIGC model into voice content, and the terminal outputs the voice content to the user (see Figure 3 ).

[0073] Of course, users can choose the way to interact with the terminal according to their own needs.

[0074] Optionally, the translation engine can achieve mutual conversion between voice content and text content by deploying a speech recognition model.

[0075] The input data of the AIGC model is a text prompt word that conforms to the AIGC model paradigm or grammar. The prompt word can be directly input by the user through the interactive interface of the content generation client.

[0076] For example, users can directly input prompt words by interacting with the content generation client deployed on the terminal. For example, Pangu ChatGPT requires a text prompt word as the information input source. The user inputs several prompt words through the content generation client. If the input information does not meet the minimum amount of information required for it to perform the task, it will interact with the user through the content generation client to obtain further information from the user. When the amount of information collected meets the necessary conditions for starting the task, Pangu ChatGPT starts an inference calculation, generates a response, organizes the response according to the user's requirements, and then outputs the organized response to the user.

[0077] However, the method of users directly inputting prompt words requires users to learn the paradigm or grammar of the prompt words of the AIGC model, which requires users to pay a lot of learning and time costs. The quality of the prompt words directly affects the quality of the content generated by the AIGC model. Therefore, due to different levels of learning, the quality of the prompt words input by users will be different, and the quality of the content output by the AIGC model will be different, affecting the user experience.

[0078] In another example, in order to provide users with a consistent user experience, a prompt word generation model can be used to obtain prompt words that conform to the AIGC model grammar or paradigm. For example, the prompt word generation model automatically generates prompt words that conform to the AIGC model grammar or paradigm based on the user's input text. For example, the text input by the user is used as the input of the prompt word generation model, and the prompt word that conforms to the AIGC model grammar or paradigm corresponding to the text input by the user is output.

[0079] By inputting the user's input text into the trained prompt word generation model, prompt words that conform to the grammatical rules of the AIGC model are obtained. Users do not need to learn the grammar or specifications of each AIGC model prompt word, which reduces the user's learning and time costs. Users can directly interact with the terminal according to their daily conversation habits and obtain the response output of the AIGC model that conforms to their intentions.

[0080] It is understandable that the user input text may be text directly input by the user, or may be text translated and output by a translation engine.

[0081] Optionally, the prompt word generation model is a trained text generation network, for example, the text generation network can be an MLP network. In one example, the prompt word generation model can be trained in a supervised training manner, and the training data set includes multiple training samples, each training sample includes a text sentence with a label, and the label is a prompt word corresponding to the text sentence that conforms to the AIGC model grammar or paradigm. The training sample is used as the input of the prompt word generation model, and the predicted prompt word is output. The loss is determined based on the predicted prompt word and the label, and the weight parameters of the prompt word generation model are adjusted based on the loss. The training is iterated until the training completion condition is met, that is, a trained prompt word generation model is obtained. Optionally, the training completion condition can be reaching a preset training round or model convergence, etc.

[0082] The quality of the content generated by the AIGC model based on the prompt words is directly positively correlated with the quality of the prompt words. In order to obtain higher quality generated content, certain rules and techniques are required to obtain higher quality prompt words. For example, the prompt words need to include one or more of the user's current emotional information and preference information.

[0083] In another example, in order to further obtain higher quality prompt words and obtain content output that better meets the user's intention, the generation of prompt words is also related to the context of the user's input text. The context includes one or more of the user's current emotional information and the user's preference information.

[0084] The context information of the input text can be extracted from the user's historical operation records and historical chat records. For example, the user's emotional information, preferences and personal related information are extracted from the operation records (such as web browsing records) and chat records stored in the terminal, and corresponding tags are generated, such as emotional tags and preference tags, etc. These tags constitute the context information of the input text. In another example, the user's historical operation records can be the historical operation records of the previous 24 hours, and the user's historical chat records can also be the historical chat records of the previous 24 hours. In the context information extraction, the closer the distance to the current (the current can be understood as the time point when the user interacts with the content generation client to input data), the greater the weight of the extracted information, and the farther the distance from the current time, the smaller the weight; for example, the weight of the information extracted from the historical operation records and historical chat records 1 hour away from the current is the largest, and the weight of the information extracted from the historical operation records and historical chat records 24 hours away from the current is the smallest, and finally the context information of the input text is obtained by weighting the information extracted from each hour away from the current. In other words, the closer the historical operation records and historical chat records are, the more they can represent the user's current emotions, preferences and personal related information.

[0085] Of course, in some other examples, the time granularity of the user's historical operation records and the user's historical chat records can be larger or smaller. For example, the user's historical operation records and the user's historical chat records can be historical records from the current week, and the historical records from one to seven days from the current are extracted respectively, and then the context information of the input text is obtained by weighting. Or the historical operation records and the user's historical chat records can be historical records from the current hour, and the historical records from 10 to 60 minutes from the current are extracted respectively, and then the context information of the input text is obtained by weighting.

[0086] Optionally, extracting the context information of the input text from the operation records and chat records stored in the terminal can be implemented by means of an AI semantic model. The operation records and chat records stored in the terminal are used as inputs of the AI ​​language model, and the emotion tags and preference tags are output to obtain the context information of the input text.

[0087] Based on the user's input text and the context information of the input text, the prompt word of the AIGC model is obtained. Exemplarily, the input text and context information are used as inputs of the prompt word generation model, and the prompt word is output; wherein the prompt word generation model is trained based on a training data set, and the training data set includes multiple training samples with labels, and the training samples include text sentences and several text words, and the labels include prompt words corresponding to the text sentences and several text words. The training samples are used as inputs of the prompt word generation model, and the predicted prompt words are output. The loss is determined based on the predicted prompt words and labels, and the weight parameters of the prompt word generation model are adjusted based on the loss. The training is iterated in this way until the training completion condition is reached, that is, a trained prompt word generation model is obtained. Optionally, the training completion condition may be reaching a preset training round or model convergence, etc.

[0088] Figure 4 A schematic diagram of the generation process of prompt words in an embodiment of the present application is shown.

[0089] After the prompt words are supplemented with contextual information, they will be able to more accurately reflect the user's intention and be more in line with the grammatical preferences of the AIGC model, so that the AIGC model can provide output that is more in line with the user's intention, realizing "the AIGC model understands you better."

[0090] For example, the output content of the AIGC model is an image, and the AI ​​painting task can be completed according to the prompt word input by the user. The user is very angry at this time, and wants the AIGC model to create a picture that can express the user's mood at this time. The user's input text is "draw a phoenix with fire on a phoenix tree", and the prompt word generated without supplementary context information may be "fire, phoenix, phoenix", and in response to the input prompt word, the picture generated by the AIGC model is a phoenix with fire on a phoenix tree. However, the user may feel that the fire on the phoenix is ​​not strong enough to express his current anger. By extracting the user's historical operation records and / or the user's historical chat records, the user's emotional information is obtained as "anger", and the emotional label is added to the prompt word to obtain the prompt word "anger, fire, phoenix, phoenix" that can express the user's emotional information. In response to the input prompt word, the picture generated by the AIGC model is a phoenix with blazing fire all over the body on the phoenix tree. The user obtains an AI painting that meets his own intentions, and feels that the AIGC model understands him very well, which improves the user experience.

[0091] For another example, the user's input text is "Draw a running Sam", and the prompt word generated without supplementary context information may be "Running, Sam". In response to the input prompt word, the picture generated by the AIGC model may be a running little boy. However, Sam is a pet dog raised by the user, and the real intention of the user is to create a picture of his pet dog running. By extracting the user's historical operation records and / or the user's historical chat records, it is obtained that Sam is actually a pet dog raised by the user, and Sam's appearance features, then Sam's real information is supplemented into the prompt word, and the prompt word "Running, Dog" that can express the user's preference information is obtained. In response to the input prompt word, the picture generated by the AIGC model is a running dog, where the dog's appearance features are consistent with the appearance features of Sam raised by the user, and the user obtains an AI painting that meets his own intentions.

[0092] In step S202, input data is uploaded to a server via a first routing path, so as to generate first content through a remote AIGC large model deployed on the server.

[0093] Optionally, the server can be a cloud server, that is, the remote AIGC large model is deployed in the cloud to provide high-quality content generation services to users.

[0094] When the content generation client obtains a prompt word that can express the user's intention, it preferentially sends the prompt word to the remote AIGC large model deployed in the cloud through the first routing path. The remote AIGC large model generates and outputs content that meets the user's intention in response to the input prompt word.

[0095] In step S203, the first content returned by the server is received.

[0096] After the remote AIGC large model deployed in the cloud generates content that meets the user's intention through inference, the content is fed back to the terminal, and the first content is displayed to the user through the content generation client deployed in the terminal.

[0097] In step S204, when an exception occurs in acquiring the first content through the first routing path, the input data is input into the local AIGC small model deployed locally through the second routing path to obtain the second content.

[0098] The terminal will give priority to inputting the prompt words into the remote AIGC large model deployed on the cloud side, and generate the content required by the user through the remote AIGC large model. When an abnormal situation occurs, the terminal will input the prompt words into the local AIGC small model deployed locally, and use the local AIGC small model deployed locally to generate the required content, so that the content generation service can continue to be obtained when the cloud side responds to the abnormal situation.

[0099] The abnormal situation can be understood as: abnormal terminal content acquisition due to abnormal response of the generative big model deployed at the remote end, such as one or more of abnormal content quality output by the generative big model in response to input data and abnormal time of content output by the generative big model in response to input data.

[0100] For example, due to the large number of parameters (usually reaching hundreds of billions or trillions), the generative large model may experience "intelligence decline" after a training update, such as a decrease in the accuracy of the generated answers, or the generated answers are irrelevant to the questions, etc. For another example, the generative large model is deployed in the cloud to provide content generation cloud services to the public. When there are too many access users, the generative large model may take too long to infer due to insufficient computing power, which in turn causes users to take too long to get responses.

[0101] In another example, the exception may also include an abnormal situation that causes the terminal to fail to receive a response from the remote AIGC large model, or fail to receive a response from the remote AIGC large model within a preset time, such as an abnormality in the communication link between the terminal device and the server, an abnormality in the response output of the first content generation model deployed on the server, and an abnormality in the server. One or more of the following.

[0102] For example, the terminal can detect whether the content receives the content response generated by the remote AIGC large model within a preset time period (for example, the preset time period can be 1 minute) after the prompt word is sent to the cloud side. If it is detected that the content response generated by the remote AIGC large model is not received within the preset time period, the prompt word is input into the locally deployed local AIGC small model, and the locally deployed local AIGC small model is used to generate the required content.

[0103] For another example, the terminal can detect whether the communication link between the terminal and the cloud server is abnormal. For example, when the terminal is in an environment with poor network signal or no network signal, such as an elevator, subway or airplane, the terminal detects that the communication link between the terminal and the cloud server is abnormal, resulting in problems in sending or receiving information. The prompt word is input into the locally deployed local AIGC small model, and the locally deployed local AIGC small model is used to generate the required content.

[0104] For another example, the response output of the remote AIGC large model deployed on the cloud server produces an exception. The terminal receives a message from the cloud side notifying it of the abnormality in the content generation service. The terminal inputs the prompt word to the locally deployed local AIGC small model and uses the locally deployed local AIGC small model to generate the required content.

[0105] For another example, an exception occurs in the cloud server where the remote AIGC large model is deployed. The terminal receives a notification message from the cloud side about the exception, and the terminal inputs the prompt word into the locally deployed local AIGC small model, and uses the locally deployed local AIGC small model to generate the required content.

[0106] Taking the remote AIGC large model as Pangu ChatGPT as an example, the local AIGC small model can be referred to as the local model. Figure 5 A schematic block diagram of content generation is shown. Figure 5 As shown, the terminal deploys two soft routers, including router A and router B. When the terminal receives the prompt word input, both router A and router B will give priority to path a, send the prompt word to Pangu ChatGPT deployed on the cloud side, and obtain high-quality content of the response output of Pangu ChatGPT deployed on the cloud side. When the communication link is abnormal or the response of Pangu ChatGPT on the cloud side is abnormal, router A and router B select path b, and the terminal loads the local model to temporarily replace Pangu ChatGPT on the cloud side to provide timely feedback output to users.

[0107] It is understandable that the parameter magnitude of the remote AIGC large model is greater than that of the local AIGC small model. For example, the parameter magnitude of the remote AIGC large model is in the hundreds of billions to trillions, while the parameter magnitude of the local AIGC small model is in the billions. The storage overhead of the remote AIGC large model is 700G, and the computing power overhead is 100+PTFLOPS, while the storage overhead of the local AIGC small model is 2 to 4G, and the computing power overhead is 10-20TFLOPS.

[0108] Optionally, the local AIGC small model is an AIGC model obtained after the remote AIGC large model has been processed through knowledge distillation, quantization, pruning, etc. The response output of the local AIGC small model is relatively simpler than that of the remote AIGC large model, but the response quality is still within the user's acceptable range.

[0109] In another example, the local AIGC small model and the remote AIGC large model may also be generative models that are trained separately, but the local AIGC small model and the remote AIGC large model respond to the same input data and output content in the same format. For example, the local AIGC small model and the remote AIGC large model respond to the input data and output content in image format or content in text format, etc.

[0110] In another example, an auxiliary small model is also deployed on the terminal device, which includes a multi-layer neural network; when the communication link is abnormal or the Pangu ChatGPT response on the cloud side is abnormal, the terminal uses the prompt word as the input of the local model and the auxiliary small model respectively, and the prompt word is processed by the multi-layer neural network to obtain a first processing result, and the prompt word is processed by the multi-layer intermediate layer neural network of the local model to obtain a second processing result; the output layer of the local model processes the first processing result and the second processing result, and outputs the second content.

[0111] By setting up an auxiliary model on the terminal, the auxiliary model and the local AIGC model work together to achieve response output to meet some customized needs of users. For example, different users may have different preferences. For example, some users may want to interact in modern Chinese. In this case, an auxiliary model can be trained with modern Chinese. After the auxiliary model acts on the local AIGC model, the output layer of the local AIGC model outputs modern Chinese that meets the user's preferences. However, some users may prefer "dragging" and like to interact in ancient Chinese. In this case, an auxiliary model can be trained with ancient Chinese. After the auxiliary model acts on the local AIGC model, the output layer of the local AIGC model outputs modern Chinese that meets the user's preferences.

[0112] It is understandable that the subsidiary mini-model is similar to a patch of traditional software, providing functions including but not limited to bug fixes, feature enhancements, etc.

[0113] The subsidiary model can be called the subsidiary part, and the local AIGC model can be called the main part. The optional subsidiary part only contains the weight information of some intermediate layers, so the volume is generally much smaller than the main part, and it will not significantly increase the computing load of the terminal. A main neural network can load multiple subsidiary neural networks at the same time to achieve different functions or purposes. The relationship between them is as follows: Figure 6 shown.

[0114] After the main and subsidiary neural networks are loaded, the input data is sent to these neural networks at the same time. After each calculation, they are summarized in the output layer. The subsidiary neural network exerts influence on the results of the main network through the weight configuration of the corresponding layer, and finally obtains personalized output content for each user, improving the user experience.

[0115] Figure 7 The overall system block diagram of a specific implementation of the content generation method provided in the embodiment of the present application is shown. Figure 7 As shown, the user interacts with the content generation client on the terminal and inputs chat information by voice. The terminal generates emotion tags and preference tags according to the user's operations and chat records in the local application; the terminal uses the user's input text (the text generated by translating the user's input voice through the translation engine) and the emotion tag and preference tag as the input of the prompt word generation model, outputs prompt words that meet the characteristics of Pangu ChatGPT, sends the prompt words to Pangu ChatGPT on the cloud side via the network, and waits for a response. Pangu ChatGPT on the cloud side generates high-quality content output text in response to the prompt words, and the content output text is translated into voice by the translation engine and output to the user. When the cloud side output data transmission or feedback is abnormal, the route deployed by the terminal will switch to the local model and activate it to provide timely feedback. Compared with the output of ChatGPT on the cloud side, the output of the local model is relatively simple.

[0116] For example, the output content of ChatGPT on the cloud side in response to the prompt word is more detailed and has more words (for example, about 1,000 words), as well as some related network links. The output content of the local model in response to the prompt word is more concise and has fewer words (for example, 100 words), and does not include related network links.

[0117] Based on the same concept as the above-mentioned embodiment of the content generation method, the present application embodiment further provides a content generation device 800, which can be deployed in Figure 1On the terminal 110 shown in the figure, when the network environment is good, high-quality content can be obtained through the remote AIGC large model deployed on the cloud side. When the network signal is abnormal, timely content feedback can still be obtained through the local AIGC small model deployed locally. The content generation device 800 includes a device for implementing Figure 2-7 The units or modules of each step in the content generation method shown.

[0118] Figure 8 This is a schematic diagram of the structure of a content generation device provided in an embodiment of the present application. Figure 8 As shown, the content generation device 800 at least includes an acquisition module 801, a sending module 802, a receiving module 803 and an input module 804, wherein the acquisition module 801 is used to acquire input data of a content generation model; the sending module 802 is used to upload the input data to a server via a first routing path; the receiving module 803 is used to receive a first content returned by the server, the first content is generated by a first content generation model deployed on the server in response to the input data, and the first content generation model is a generative large model; the input module 804 is used to input the input data to a second content generation model deployed locally via a second routing path to obtain a second content when an exception occurs when acquiring the first content via the first routing path, and the exception is related to the first content generation model; wherein the number of parameters of the second content generation model is less than the number of parameters of the first content generation model.

[0119] In another possible implementation, the anomaly includes one or more of an anomaly in content quality of the generative large model output in response to input data and an anomaly in time of content output by the generative large model in response to input data.

[0120] In another example, the anomaly may also include: an anomaly in the communication link between the terminal device and the server, an anomaly in the response output of the first content generation model deployed on the server, and an anomaly in the server.

[0121] Optionally, the server is a cloud server.

[0122] In another possible implementation, an auxiliary small model is also deployed on the terminal device, which includes a multi-layer neural network; the second content generation model includes a multi-layer intermediate layer neural network and an output layer; the input data is input into the second content generation model deployed locally through a second routing path, and a specific implementation of obtaining the second content is: the input data is used as the input of the second content generation model, the input data is processed by the multi-layer neural network to obtain a first processing result, and the input data is processed by the multi-layer intermediate layer neural network to obtain a second processing result; the output layer processes the first processing result and the second processing result, and outputs the second content.

[0123] In another possible implementation, the input data includes a prompt word; the acquisition module 801 is specifically used to: receive the data input by the user; obtain the input text based on the data input by the user; obtain the context information of the input text; and obtain the prompt word based on the input text and the context information.

[0124] In another possible implementation, obtaining context information of the input text includes: obtaining historical operation records and historical chat records of the user; and obtaining context information based on the historical operation records and historical chat records.

[0125] In another possible implementation, the context information includes one or more of the user's emotion information and the user's preference information.

[0126] In another possible implementation, a specific implementation of obtaining a prompt word based on input text and context information is: using the input text and context information as input of a prompt word generation model, and outputting a prompt word; wherein the prompt word generation model is trained based on a training data set, the training data set includes multiple training samples with labels, the training samples include text sentences and several text words, and the labels include prompt words corresponding to the text sentences and the several text words.

[0127] Optionally, the prompt word generation model is a multi-layer perceptron network.

[0128] The content generation device 800 according to the embodiment of the present application may correspond to executing the method described in the embodiment of the present application, and the above and other operations and / or functions of each module in the content generation device 800 are respectively to implement Figure 2-7 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0129] The present application also provides a terminal device, including at least one processor, a memory and a communication interface, wherein the processor is used to execute Figure 2-7 The method described.

[0130] Fig. 9 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application.

[0131] like Fig. 9As shown, the terminal device 900 includes at least one processor 901, a memory 902 and a communication interface 903. The processor 901, the memory 902 and the communication interface 903 are connected in communication, and the communication connection can be realized by wired (such as bus) mode, or by wireless mode. The communication interface 903 is used to send and / or receive data sent by other devices, such as the terminal sends prompt words to the remote AIGC large model deployed on the cloud side, and receives the content of the remote AIGC large model deployed on the cloud side in response to the output; the memory 902 stores computer instructions, and the processor 901 executes the computer instructions to execute the content generation method in the aforementioned method embodiment, so as to realize that when the generative large model deployed on the remote server side responds normally, the generative large model deployed on the server side generates content to obtain high-quality content; when the content generation model deployed on the server side responds abnormally, the locally deployed miniaturized content generation model is used to generate the required content, so that when the generative large model deployed on the server side is abnormal, the content generation service can still be continuously obtained.

[0132] It should be understood that in the embodiment of the present application, the processor 901 may be a central processing unit CPU, and the processor 901 may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0133] The memory 902 may include a read-only memory and a random access memory, and provides instructions and data to the processor 901. The memory 902 may also include a nonvolatile random access memory.

[0134] The memory 902 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0135] It should be understood that the terminal device 900 according to the embodiment of the present application can execute the implementation of the embodiment of the present application. Figure 2-7 The method shown, the detailed description of the implementation of this method can be found above, for the sake of brevity, it will not be repeated here.

[0136] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer instructions are executed by a processor, the above-mentioned method is implemented.

[0137] An embodiment of the present application provides a chip, which includes at least one processor and an interface, wherein the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the method mentioned above.

[0138] An embodiment of the present application provides a computer program or a computer program product, wherein the computer program or the computer program product comprises instructions, and when the instructions are executed, the computer is caused to execute the above-mentioned method.

[0139] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0140] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented by hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0141] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A content generation method, characterized in that: Applied to a terminal device, the method comprises: Obtain input data for the content generation model; Uploading the input data to a server via a first routing path; receiving the first content returned by the server, where the first content is generated by a first content generation model deployed on the server, and the first content generation model is a generative large model; When an exception occurs when acquiring the first content through the first routing path, inputting the input data into a second content generation model deployed locally through a second routing path to obtain second content, and the exception is related to the first content generation model; The number of parameters of the second content generation model is smaller than the number of parameters of the first content generation model.

2. The method according to claim 1, characterized in that The anomaly includes: one or more of an anomaly in content quality output by the first content generation model and an anomaly in the time when the first content generation model outputs the first content in response to the input data.

3. The method according to claim 1 or 2, characterized in that: The server is a cloud server.

4. The method according to any one of claims 1 to 3, characterized in that: The terminal device is also deployed with an auxiliary small model, the auxiliary small model includes a multi-layer neural network; the second content generation model includes a multi-layer intermediate layer neural network and an output layer; The step of inputting the input data to a second content generation model deployed locally through a second routing path to obtain second content includes: The input data is used as input of the second content generation model and the subsidiary small model respectively, the input data is processed by the multi-layer neural network to obtain a first processing result, and the input data is processed by the multi-layer intermediate layer neural network to obtain a second processing result; The output layer processes the first processing result and the second processing result, and outputs the second content.

5. The method according to any one of claims 1 to 4, characterized in that: The input data includes prompt words; The step of obtaining input data of the content generation model includes: Receive data input by the user; Based on the data input by the user, obtaining input text; Obtaining context information of the input text; The prompt word is obtained based on the input text and the context information.

6. The method according to claim 5, characterized in that The obtaining of the context information of the input text includes: Obtain the user's historical operation records and historical chat records; The context information is obtained based on the historical operation records and the historical chat records.

7. A content generating device, characterized in that: Deployed on a terminal device, the device includes: An acquisition module, used to obtain input data of a content generation model; A sending module, used for uploading the input data to a server via a first routing path; A receiving module, configured to receive the first content returned by the server, wherein the first content is generated by a first content generation model deployed on the server, and the first content generation model is a generative large model; An input module, configured to input the input data to a second content generation model deployed locally through a second routing path to obtain second content when an exception occurs in acquiring the first content through a first routing path, and the exception is related to the first content generation model; The number of parameters of the second content generation model is smaller than the number of parameters of the first content generation model.

8. The device according to claim 7, characterized in that The anomaly includes: one or more of an anomaly in content quality output by the first content generation model and an anomaly in the time when the first content generation model outputs the first content in response to the input data.

9. The device according to claim 7 or 8, characterized in that The server is a cloud server.

10. The device according to any one of claims 7 to 9, characterized in that: The terminal device is also deployed with an auxiliary small model, the auxiliary small model includes a multi-layer neural network; the second content generation model includes a multi-layer intermediate layer neural network and an output layer; The step of inputting the input data to a second content generation model deployed locally through a second routing path to obtain second content includes: The input data is used as input of the second content generation model and the subsidiary small model respectively, the input data is processed by the multi-layer neural network to obtain a first processing result, and the input data is processed by the multi-layer intermediate layer neural network to obtain a second processing result; The output layer processes the first processing result and the second processing result, and outputs the second content.

11. The device according to any one of claims 7 to 10, characterized in that: The input data includes prompt words; The acquisition module is specifically used for: Receive data input by the user; Based on the data input by the user, obtaining input text; Obtaining context information of the input text; The prompt word is obtained based on the input text and the context information.

12. The device according to claim 11, characterized in that The obtaining of the context information of the input text includes: Obtain the user's historical operation records and historical chat records; The context information is obtained based on the historical operation records and the historical chat records.

13. A terminal comprising a memory and a processor, characterized in that: Instructions are stored in the memory, and when the instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

14. A chip, characterized in that: The method comprises at least one processor and a communication interface, wherein the processor is configured to execute the method according to any one of claims 1 to 6.

15. A computing device comprising a memory and a processor, characterized in that: Instructions are stored in the memory, and when the instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images