A content generation method and system

Abstract

The content generation method and system provided in the specification, in the content generation process, target prompt data input by a target object and a content template set corresponding to the target object are obtained, the content template set is obtained based on a plurality of historical content generation data of the target object; and the target prompt data is matched with the content template set to determine a target content template from the content template set; in the process of generating target content, the target content template is used as a reference for content generation based on the target prompt data, and the target content is generated. The method and system can adaptively select matching historical data from the historical data of the target object as a reference template, improve the efficiency of content generation, and ensure the personalization and controllability of the generated content.

Description

Technical Field

[0001] This specification relates to the field of image processing technology, and in particular to a content generation method and system. Background Technology

[0002] Significant breakthroughs have been achieved in the field of Artificial Intelligence Generated Content (AIGC), which is now widely used in various creative endeavors. While AIGC technology has achieved success in areas such as creative inspiration and content modification, challenges remain in the personalized and controllable generation process, requiring substantial human experience and time investment. For example, users need to manually adjust prompts to refine the generated content, and they also need to manually select reference images to inform the content generation process.

[0003] In conclusion, there is a need to provide a new content generation method and system that can improve the efficiency of AIGC technology.

[0004] The information in the background section is merely information known only to the inventor and does not imply that such information had entered the public domain before the date of this application, nor does it imply that it could be considered prior art in this disclosure. Summary of the Invention

[0005] This specification provides a content generation method and system that can improve the controllability and personalization of content generated using AIGC technology.

[0006] Firstly, this specification provides a content generation method, comprising the following steps: obtaining target prompt data input by a target object and a content template set corresponding to the target object, wherein the content template set is obtained based on multiple historical content generation data of the target object; matching the target prompt data with the content template set to determine a target content template from the content template set; using the target content template as a reference for content generation to generate target content based on the target prompt data; and outputting the target content.

[0007] In some embodiments, the content template set is updated based on a preset period.

[0008] In some embodiments, the plurality of historical content generation data includes the content template set of the previous period and the content generation data generated from the previous period to the current period.

[0009] In some embodiments, the content template set includes K content templates, the template similarity between the K content templates satisfies a preset condition, and each of the K content templates includes a prompt data template and its corresponding adoption content template, where K is a positive integer.

[0010] In some embodiments, the preset conditions include: the template similarity between the K content templates is less than a preset threshold; and / or in all combinations of the plurality of historical content generation data, the statistical measure of the template similarity between the K content templates is the smallest, the statistical measure of template similarity includes the sum or mean of the template similarity, wherein each of the combinations includes K historical content generation data from the plurality of historical content generation data.

[0011] In some embodiments, the template similarity includes one of the following: the similarity of the prompt data template features corresponding to the prompt data template; the similarity of the adoption content template features corresponding to the adoption content template; and the similarity of the combination of the prompt data template features and the adoption content template features.

[0012] In some embodiments, matching the target prompt data with the content template set to determine the target content template from the content template set includes: matching the target prompt data with K prompt data templates among the K content templates, and using the content template corresponding to the target prompt data template that matches the target prompt data as the target content template.

[0013] In some embodiments, matching the target prompt data with K prompt data templates from the K content templates, and using the content template corresponding to the target prompt data template that matches the target prompt data as the target content template, includes the following steps: matching the target prompt data with each of the K prompt data templates to determine a matching score for each prompt data template; and using the prompt data template with the highest matching score as the target prompt data template, and using the content template corresponding to the target prompt data template as the target content template.

[0014] In some embodiments, matching the target prompt data with each of the K prompt data templates to determine the matching score corresponding to each prompt data template includes: inputting the target prompt data and each prompt data template into a prompt data matching model to obtain the matching score corresponding to each prompt data template, wherein the training data of the prompt data matching model includes sample prompt data and sample prompt data templates of the sample content template set, and the training objective of the prompt data matching model includes constraining the difference between its output predicted matching score and the pre-labeled true matching score to be within a first preset difference threshold.

[0015] In some embodiments, the step of matching the target prompt data with the content template set and determining the target content template from the content template set further includes: matching the target prompt data with the target content template at least to determine the target clue that matches the target prompt data.

[0016] In some embodiments, the step of matching the target prompt data with the target content template at least to determine the target clue matching the target prompt data includes the following steps: matching the target prompt data with the target content template at least to determine the target clue type matching the target prompt data; and determining the target clue content corresponding to the target clue type from the target content template, wherein the target clue includes the target clue type and the target clue content.

[0017] In some embodiments, the step of matching the target cue data with the target content template to determine the target clue type that matches the target cue data includes: inputting the target cue data and the target adoption content template in the target content template into a clue type matching model to obtain the target clue type, wherein the training data of the clue type matching model includes sample cue data and its corresponding sample reference content, the training objective of the clue type matching model includes constraining the difference between its output predicted clue type and the pre-labeled real clue type to be within a second preset difference threshold, and the sample reference content includes the real clue type.

[0018] In some embodiments, the step of matching the target prompt data with the target content template to determine the target clue type that matches the target prompt data includes: inputting the target prompt data, the target adopted content template in the target content template, and the matching score corresponding to the target content template into a clue type matching model to obtain the target clue type. The training data of the clue type matching model includes sample prompt data and its corresponding sample reference content and sample matching score. The training objective of the clue type matching model includes constraining the difference between its output predicted clue type and the pre-labeled real clue type to be within a second preset difference threshold. The sample reference content includes the real clue type.

[0019] In some embodiments, using the target content template as a reference for content generation and generating target content based on the target prompt data includes: inputting the target prompt data and the target adoption content template and target clue content in the target content template into a generation model to obtain the target content, such that the target content is consistent with the target clue content. The training data of the generation model includes sample prompt data and its corresponding sample reference content and sample reference clue content. The training objective of the generation model includes constraining the difference between its output predicted generated content and the pre-labeled real generated content to be within a third preset difference threshold. The real generated content includes the sample reference clue content. The generation model is a diffusion model.

[0020] Secondly, this specification also provides a content generation system, comprising: at least one storage medium storing at least one instruction set for content generation; and at least one processor communicatively connected to the at least one storage medium, wherein, when the content generation system is running, the at least one processor reads the at least one instruction set and executes the method described in any one of the first aspects according to the instructions of the at least one instruction set.

[0021] As can be seen from the above technical solutions, the content generation method and system provided in this specification can adaptively select target content templates with high matching degree with target prompt data from the historical content generation data corresponding to the target object as references for generating target content, thereby improving the efficiency of content generation. Simultaneously, by using historical generated content data that matches the target prompt data as reference content for content generation, the method and system can ensure and improve the controllability of the target content generation process and the personalized effect of the target content.

[0022] The content production methods and other functions of the system provided in this specification will be partially listed in the following description. The figures and examples described below will be readily apparent to those skilled in the art. The inventive aspects of the content generation methods and systems provided in this specification can be fully understood through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1A A schematic diagram illustrating an application scenario of a content generation system provided according to an embodiment of this specification is shown.

[0025] Figure 1B This diagram illustrates a scenario in which content generation is being performed on a target object according to some embodiments of this specification;

[0026] Figure 2 A schematic diagram of the structure of a computing device provided according to an embodiment of this specification is shown;

[0027] Figure 3 A flowchart of a content generation method provided according to an embodiment of this specification is shown;

[0028] Figure 4 A flowchart illustrating the determination of a target content template according to embodiments of this specification is shown; and

[0029] Figure 5 A flowchart illustrating the determination of target clues according to embodiments of this specification is shown. Detailed Implementation

[0030] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0031] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0032] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0033] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0034] In this specification, "X includes at least one of A, B, or C" means that X includes at least A, or X includes at least B, or X includes at least C. That is, X can include only one of A, B, and C, or any combination of A, B, and C, as well as other possible content / elements. The arbitrary combination of A, B, and C can be A, B, C, AB, AC, BC, or ABC.

[0035] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0036] For ease of description, the terms used in this specification are explained as follows:

[0037] AIGC: Artificial Intelligence Generated Content (AIGC) refers to methods of generating content using artificial intelligence (AI) technology. The generated content can include images, text, audio, and video, among others. Prompt: This refers to a prompt word, a form of input guidance, describing the content generation goal. It can be understood as the language used to communicate with the AI ​​model, telling the AI ​​what content to generate. The content generation goal can also be understood as the content the user intends to generate (which can be images, text, audio, video, etc.).

[0038] Text-to-image generation: generating images from text descriptions, such as inputting a prompt describing the content of an image into an AI model to generate an image.

[0039] Image-to-image: Using an image as a reference, another image is generated. For example, a reference image is input into an AI model to generate a new image.

[0040] In the process of using AIGC technology, to improve the controllability and personalization of AIGC-generated content, existing methods mainly include the following, taking the application of content generation methods in image generation scenarios as an example. For example, in the process of text-to-image generation, users can adjust the generated image by adjusting the prompt. Adjustments to the prompt may include, but are not limited to, prompt expansion, style adjustment, and synonym replacement. However, such methods have high barriers to entry and rely heavily on human experience. Although some existing auto-prompt methods can reduce the learning cost, they are still far from scalable. Another example is the technology of generating content from a single reference image, which requires users to input an additional reference image and then generate new AIGC image content based on specific cues from the reference image (such as contours, depth, or segmentation maps). However, this method requires manually specifying the type of cue (e.g., specifying contours makes the generated image more similar to the reference image in terms of contours). This selection itself relies on human experience, and the reference image needs to be manually selected, which inherently involves experience dependence and trial-and-error costs.

[0041] Therefore, this specification provides a new content generation method that can reduce reliance on human experience, improve the efficiency of content generation, and enhance the controllability and personalization of the generated content.

[0042] Before describing the specific embodiments in this specification, the application scenarios of this specification will be introduced as follows:

[0043] The content generation methods provided in this manual can be applied to any content generation scenario. For example, in scenarios requiring image content generation, AIGC technology can use input prompts or reference images to generate new images, such as generating comic images using prompts or line art with the same style using reference images; in scenarios requiring text content generation, AIGC technology can use input article keywords / prompts to generate high-quality text or to modify and polish text; in scenarios requiring audio generation, AIGC technology can use input prompts or reference audio to generate new audio, such as generating corresponding background music using prompts or extracting vocals, accompaniment, and various instruments from reference audio and video to generate new songs; in scenarios requiring video content generation, AIGC technology can use input prompts to generate new videos, such as generating product introduction videos or videos of article readings using prompts.

[0044] The content generation method described in this specification can also be applied to any content generation scenario, which will not be elaborated upon here. For ease of description, this application will use the application of the content generation method in an image generation scenario to describe an image generated by AIGC technology as an example.

[0045] Those skilled in the art should understand that the content generation methods and systems described in this specification, when applied to other use cases, are also within the scope of protection of this specification.

[0046] Figure 1A The diagram illustrates an application scenario of a content generation system 100 according to some embodiments of this specification. The content generation system 100 (hereinafter referred to as system 100) can be applied to content generation in any scenario, such as the generation of image content, text content, audio content, video content, etc., as described above. System 100 may include a server 110 and a client 120. In some embodiments, system 100 may further include a network 130 and a database 140. The application scenario of system 100 may include a target object 150.

[0047] The target object 150 can be a user who needs to generate content, or a user who is currently generating content.

[0048] Network 130 facilitates the exchange of information and / or data. Network 130 can be any type of wired or wireless network, or a combination thereof. For example, network 130 may include cable networks, wired networks, fiber optic networks, telecommunications networks, intranets, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), Bluetooth networks, ZigBee networks, near field communication (NFC) networks, or similar networks. Network 130 may include one or more network access points. For example, network 130 may include wired or wireless network access points, such as base stations and / or Internet switching points. Through this access point, one or more components of client 120, server 110, and database 140 can connect to network 130 to exchange data and / or information, wherein target object 150 can connect to network 130 through client 120. Figure 1A As shown, server 110, client 120, and database 140 can connect to network 130 and transmit information and / or data to each other through network 130. For example, client 120 can obtain services from server 110 through network 130.

[0049] Client 120 can be a device for generating content for target object 150. The content generation method can be executed on client 120. At this time, client 120 can store data or instructions for the content generation method described in this specification, and can execute or be used to execute the data or instructions. Client 120 may include a hardware device with data processing capabilities and the necessary programs required to drive the hardware device. In some embodiments, client 120 may include a mobile device, tablet computer, laptop computer, built-in device of a motor vehicle, or similar content, or any combination thereof. In some embodiments, mobile devices may include smart home devices, smart mobile devices, virtual reality devices, augmented reality devices, or similar devices, or any combination thereof. In some embodiments, smart home devices may include smart TVs, desktop computers, etc., or any combination thereof. In some embodiments, smart mobile devices may include smartphones, personal digital assistants, gaming devices, navigation devices, etc., or any combination thereof. In some embodiments, virtual reality devices or augmented reality devices may include virtual reality headsets, virtual reality glasses, virtual reality patches, augmented reality headsets, augmented reality glasses, augmented reality patches, or similar content, or any combination thereof. For example, virtual reality devices or augmented reality devices may include Google Glass, head-mounted displays, VR, etc. In some embodiments, the built-in devices in a motor vehicle may include an onboard computer, an onboard television, etc.

[0050] Client 120 may have one or more applications (APPs) installed. The APPs provide target object 150 with the ability and interface to interact with the outside world via network 130. APPs include, but are not limited to: web browser APPs, search APPs, chat APPs, shopping APPs, video APPs, financial management APPs, instant messaging tools, email terminals, social media platform software, etc. In some embodiments, client 120 may have a target APP installed. The target APP provides target object 150 with the ability to generate content using AIGC.

[0051] Figure 1B This diagram illustrates a scenario where a target object 150 is generating content according to some embodiments of this specification. The target object 150 is using a painting AI on a client 120 and has selected a text-based image format for content generation. Figure 1B As shown, the target app on client 150 can provide target object 150 with options for image generation (painting AI), audio content generation (audio AI), text content generation (writing AI), and video content generation (video AI), etc. When target object 150 selects painting AI and its text-to-image format, the target app can obtain the prompt input by target object 150. In some embodiments, target object 150 can also trigger a content generation request through the target app. The target app can respond to the content generation request by executing the content generation method described in this specification. The content generation method will be described in detail later.

[0052] Server 110 may be a server that provides various preset services corresponding to client 120. For example, server 110 may be a backend server deployed on client 120 that can support the target APP for content generation. The content generation method described in this specification can be executed on server 110. At this time, server 110 may store data or instructions for executing the content generation method described in this specification, and may execute or be used to execute said data or instructions. Server 110 may include hardware devices with data information processing functions and the necessary programs required to drive the hardware devices to work.

[0053] Database 140 can be used to store data and / or instructions generated by performing the methods described in this specification. Server 110 and client 120 may have access to database 140. Data of target object 150 on client 120 may be stored in the database. For example, Figure 1BWhen the target object 150 generates an image using the client 120, the client 120 can log in to its own account to save the generated image or a satisfactory image from the generated image. This image data can be temporarily stored on the client 120, or it can be transferred by the client 120 to the database 140 for storage. The server 110 and client 120 can access the data or instructions stored in the database 140 via the network 130. The database 140 can be directly connected to the server 110 and client 120. The database 140 can be part of the server 200. The database 140 can include mass storage, removable storage, volatile read-write memory, read-only memory (ROM), or similar content, or any combination thereof. Exemplary mass storage may include non-transitory storage media such as disks, optical discs, and solid-state drives. Exemplary removable storage may include flash drives, floppy disks, optical discs, memory cards, zip disks, and magnetic tapes. Typical volatile read-write memory may include random access memory (RAM). Example RAM may include Dynamic RAM (DRAM), Dual Date Rate Synchronous Dynamic RAM (DDR-SDRAM), Static RAM (SRAM), Thyristor RAM (T-RAM), and Zero Capacitor RAM (Z-RAM), etc. Exemplary ROM may include Mask ROM (MROM), Programmable ROM (PROM), Virtually Programmable ROM (PEROM), Electronically Programmable ROM (EEPROM), Optical Disc ROM (CD-ROM), and Digital Multifunction Disk ROM, etc. Database 140 can be implemented on a cloud platform. For example only, the cloud platform may include private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, inter-cloud, or similar, or any combination thereof. Storage media may include remote, independent storage media, such as the disks, hard drives, and solid-state drives mentioned above. Server 110 / Client 120 can access or update data stored on remote, independent storage media through mobile devices with access permissions. Storage media may include local storage media, such as the data storage hard drive of the local server 140 or the IoT device itself.

[0054] It should be understood that Figure 1A The number of clients 120 and servers 110 shown is merely illustrative. Depending on implementation needs, there can be any number of clients 120 and servers 110.

[0055] It should be noted that the content generation method can be executed entirely on the client 120, entirely on the server 110, or partially on the client 120 and partially on the server 110.

[0056] Figure 2A schematic diagram of a computing device 200 according to an embodiment of this specification is shown. The server 110 and client 120 may have, for example... Figure 2 The structure of the computing device 200 shown.

[0057] like Figure 2 As shown, the computing device 200 includes at least one storage medium 230 and at least one processor 220. In some embodiments, the computing device 200 may further include a communication port 250 and an internal communication bus 210. Additionally, the computing device 200 may also include I / O components 260.

[0058] The internal communication bus 210 can connect different system components, including storage medium 230 and processor 220. I / O component 260 supports input / output between computing device 200 and other components.

[0059] Storage medium 230 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 232, a read-only storage medium (ROM) 234, or a random access storage medium (RAM) 236. Storage medium 230 also includes at least one instruction set stored in the data storage device. The instruction set is computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc., that execute the methods for generating content for target object 150 provided in this specification.

[0060] Communication port 250 is used for data communication between computing device 200 and the outside world. For example, computing device 200 can connect to network 130 through communication port 250. At least one processor 220 is connected to at least one storage medium 230 via an internal communication bus 210. At least one processor 220 is used to execute at least one instruction set. When system 100 is running, at least one processor 220 reads at least one instruction set and executes the content generation method provided in this specification according to the instructions of at least one instruction set.

[0061] Processor 220 can execute all the steps included in the content generation method for target object 150. Processor 220 can be in the form of one or more processors. Processor 220 can issue execution instructions. Processor 220 may include one or more hardware processors, such as microcontrollers, microprocessors, reduced instruction set computers (RISC), application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), central processing units (CPUs), graphics processing units (GPUs), physical processing units (PPUs), microcontroller units, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), advanced RISC machines (ARMs), programmable logic devices (PLDs), any circuit or processor capable of performing one or more functions, or any combination thereof.

[0062] For illustrative purposes only, only one processor 220 is described in this specification for computing device 200. However, it should be noted that computing device 200 may also include multiple processors. Therefore, the operation and / or method steps disclosed in this specification may be executed by one processor as described herein, or they may be executed jointly by multiple processors. For example, if processor 220 of computing device 200 in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 220 (e.g., a first processor executes step A, a second processor executes step B, or the first and second processors jointly execute steps A and B).

[0063] Figure 3 A flowchart of a content generation method 300 according to an embodiment of this specification is shown. As previously described, server 110 and / or client 120 can execute content generation method 300 (hereinafter referred to as method 300). Method 300 can be executed entirely on client 120, entirely on server 110, or partially on client 120 and partially on server 110. Server 110 / client 120 can have, for example, […]. Figure 2 The structure of the computing device 200 is shown. Specifically, the storage medium in the server 110 / client 120 may store at least one set of instructions for content generation. The processor 220 may be communicatively connected to the storage medium 230. The processor 220 may read the instruction set stored in its local storage medium and / or database 140, and then execute method 300 according to the instructions.

[0064] like Figure 3 As shown, method 300 may include:

[0065] S310: Obtain the target prompt data input by the target object 150 and the content template set corresponding to the target object 150.

[0066] Step S310 can be executed by either client 120 or server 110. Client 120 can obtain target prompt data input by target object 150. Target prompt data can be the aforementioned Prompt, which is an input guide and a description of the content generation target by target object 150. Target prompt data can be one or more keywords, or a paragraph or multiple paragraphs of text, etc. The content generation target can be the content that target object 150 intends to generate through the AI ​​model, such as the aforementioned image, text content, audio content, video content, etc. Figure 1B The target object 150 can enter its own description of the image in the content box of the screen description. For example, if the target object 150 wants to generate an image of a cat, the content prompt data in the input box can include keywords such as the cat's breed, eye color, fur color, expression, movement, and image sharpness. As another example, if the target object 150 wants to generate an image of a building, the content prompt data can include keywords such as the building's location, background, colors, lines, and image format. In some embodiments, the target object 150 can also import a description of the image in the content box of the screen description. The source of the imported content can be a document stored on the client 120, a web link, etc.

[0067] When step S310 is executed on client 120, client 120 can directly obtain the target prompt data. When step S310 is executed on server 110, server 110 can obtain the target prompt data from client 120.

[0068] The content template set can be generated based on multiple historical content data of the target object 150. These multiple historical content data can be stored on the client 120 of the target object 150 or on the server 110.

[0069] Multiple historical content generation data sets can be historical data generated by target object 150 during content generation within a historical time window. Each historical content generation data set can include historical prompt data input by target object 150 during the content generation process in a historical time period and its corresponding historical adopted content. After target object 150 inputs one historical prompt data set, it may receive multiple different historical content sets. The historical adopted content can be the historical content ultimately selected or adopted from multiple different historical content sets. For example... Figure 1BThe target object 150 can see multiple different images generated based on the same prompt data in the image display content box. The target object 150 can choose the image that meets its expectations or is most satisfactory as the adopted image. The adopted image and its corresponding prompt data are stored as a data pair in the historical content generation data associated with the target object 150. For example, historical prompt data in one of the target object 150's historical content generation data may include multiple keywords about cats, and the corresponding historical adopted content may be an image of a cat generated from these keywords and adopted by the target object 150. Among the multiple images generated based on the same prompt data, the image adopted by the target object 150 can be considered an image that the target object 150 prefers. For example, the image may be adopted because its layout, style, rendering, object outlines, depth, etc., make it more appealing to the target object 150. Therefore, the adopted image contains the target object 150's preference for the generated image, that is, the target object 150's personalized needs for the generated image. Multiple historical content generation data can serve as a reference for the subsequent content generation of target object 150, thereby ensuring that the newly generated content conforms to the preferences of target object 150 and meets the personalized needs of target object 150.

[0070] In some embodiments, the content template set may be all historical content generation data of the target object 150. In some embodiments, the historical content generation data of the target object 150 may contain data with high similarity. In order to save storage space while meeting the personalized needs of the target object 150, the content template set may be historical content generation data with low similarity from multiple historical content generation data of the target object 150, so as to cover multiple different types of historical content generation data. For example, the content template set may be K historical content generation data selected from multiple historical content generation data. In this case, the content template set may include K content templates. Each of the K content templates may include a prompt data template and a corresponding adoption content template. Historical content generation data selected from multiple historical content generation data can be used as content templates. Historical prompt data included in historical content generation data can be used as prompt data templates. Historical adoption content included in historical content generation data can be used as adoption content templates. When the historical content generation data is used for image generation, the adoption content template of the K content templates may be K images, and the prompt data template of the K content templates may be the prompt data (Prompt) corresponding to each image. Wherein, K is a positive integer. The value of K can be any integer, such as 1, 2, 3, 4, 5, 10, 15, etc. The value of K can be set based on the staff's past experience or obtained through neural network training. The value of K can be fixed or variable. The variation of the value of K can be related to the number of historical content generation data points or the similarity between the historical content generation data points. It should be noted that the process of selecting K historical content generation data points as a content template set from multiple historical content generation data points can be executed in the client 120 or in the server 110. In some embodiments, to improve computational efficiency and ensure the running speed of the client 120, the process of selecting K historical content generation data points as a content template set can be executed in the server 110. For ease of description, the following description will use the example of selecting K historical content generation data points as a content template set in the server 110 as an example.

[0071] Server 110 can determine a content template set based on the similarity between multiple historical content generation data. For example, server 110 can calculate the pairwise similarity between multiple historical content generation data and select K historical content generation data whose similarity meets a preset condition as K content templates. The template similarity between the K content templates meets the preset condition, so that the content generation preferences of the target object 150 can be recorded by storing less content.

[0072] The template similarity among K content templates can be either the similarity of the prompt data template features corresponding to the prompt data template, or the similarity of the adopted content template features corresponding to the adopted content template. Since the prompt data template directly and accurately reflects the information of the content template, and the adopted content template can supplement the target object's preferences for specific content generation, such as image layout, style, rendering, etc., considering both, the template similarity among K content templates can also be the similarity of the combination of prompt data template features and adopted content template features, such as the similarity of the features of the adopted content template and the concatenated features of the corresponding prompt data template. The prompt data template features corresponding to the prompt data template and the adopted content template features corresponding to the adopted content template can be extracted from the content templates using a contrastive language-image pre-training (CLIP) model.

[0073] The preset conditions for template similarity can include that the template similarity between K content templates is less than a preset threshold. This preset threshold can be set based on staff's past screening experience or obtained through neural network training. Another preset condition for template similarity can be that, among all combinations of historical content generation data, the statistical value of the template similarity among the K content templates is minimized. The template similarity statistics include the sum or mean of template similarities. Each combination includes K historical content generation data points.

[0074] For ease of description, we define the number of historical content generated data as M, where M is a positive integer. Server 110 can determine the content template set through the following steps.

[0075] Server 110 can calculate the pairwise similarity between M historical content generation data points, resulting in an M×M similarity matrix. The element in the i-th row and j-th column represents the similarity between the i-th and j-th historical content generation data points. Here, i = 1, 2, ..., M, j = 1, 2, ..., M. The pairwise similarity between historical content generation data points can be the pairwise similarity between the corresponding features of the historical content generation data points. That is, the pairwise similarity between historical content generation data points can be the feature similarity between the pairwise historical content generation data points. The corresponding features of the historical content generation data points can be the features of historical prompt data points corresponding to historical prompt data points, the features of historical adopted content points corresponding to historical adopted content points, or a combination of the features of historical prompt data points and the features of historical adopted content points. The features of historical prompt data points and the features of historical adopted content points can be extracted from the historical content generation data points using a Contrastive Language-Image Pre-training (CLIP) model. The similarity can be determined based on methods such as cosine similarity and Euclidean distance.

[0076] In some embodiments, the server 110 may select K historical content generation data from M historical content generation data, where the similarity between any two pairs is less than a preset threshold, as K content templates.

[0077] In some embodiments, server 110 selects K historical content generation data points with the smallest sum or mean of pairwise similarities as K content templates. In some embodiments, server 110 can determine the K content templates using integer programming. For example, there can be multiple different groups among the M historical content generation data points. Each group includes K historical content generation data points. The combination of the K content templates has the smallest sum or mean of pairwise similarities among all combinations.

[0078] It should be noted that when there is little or no historical content generation data for target object 150, K templates can be selected from the common templates as the content template set for target object 150. The content templates in the common templates can be selected from the content template sets of at least some users on server 110. The content templates in the common templates can be randomly selected, or they can be selected based on low similarity through similarity calculation. Since the content templates in the common templates can come from historical content generation data of different objects, the template similarity between templates in the common templates is inherently low. Server 110 can randomly select K templates from the common templates as the content template set for target object 150. Server 110 can also calculate the template similarity of the templates in the common templates and use the K templates that meet the above template similarity conditions as the content template set for target object 150.

[0079] The content template set can be updated based on a preset period. For example, the content production process of target object 150 during the historical time period from 0 to T yields historical content generation data Con1; the content production process during the historical time period from T to 2T yields historical content generation data Con2; and the content production process during the historical time period from 2T to 3T yields historical content generation data Con3. The content template set at time T is Set1. If the content template set is updated with a preset period of 2T, then the content template set can be updated at time 3T. At each update time point, the multiple historical content generation data used to select the content template set can include the content template set from the previous period and the content generation data generated from the previous period to the current period. For example, when target object 150 updates its content template set at time 3T, Set1, Con2, and Con3 are used as multiple historical content generation data at update time 3T. A content template set Set2, whose similarity meets the above conditions, is obtained from these multiple historical content generation data to replace content template set Set1. When target object 150 generates content during the time period from 3T to 5T, the content template set obtained by server 110 for target object 150 is Set2. If the content template set Set1 in the previous period contains K content templates, when updating the content template set, we can obtain the average template similarity among the K content templates in Set1, and calculate the average similarity between each data point in Con2 and Con3 and the K content templates in Set1. From the calculated average similarity values ​​of Con2 and Con3, we select the values ​​with template similarity less than the average of the K content templates, and the corresponding data can replace some of the content templates with higher similarity among the K content templates. When the target object 150 updates the content template set at time 3T, if the content template set Set1 in the previous period contains KN content templates, that is, the number of content templates in Set1 is less than K, we can also retain the content template set in Set1, and select N content templates from Con2 and Con3 to supplement Set1 to form Set2. At this time, we only need to calculate the template similarity between each data point in Con2 and Con3 and the KN content templates in Set1, and select the N data with smaller similarity values.

[0080] The content template set can be stored on client 120, server 110, or both. When step S310 is executed on client 120 and the content template set is stored on client 120, client 120 can retrieve the content template set locally. When step S310 is executed on client 120 and the content template set is stored on server 110, client 120 can retrieve the content template set from server 110. When step S310 is executed on server 110 and the content template set is stored on client 120, server 110 can retrieve the content template set from client 120. When step S310 is executed on server 110 and the content template set is stored on server 110, server 110 can retrieve the content template set locally.

[0081] Each target object 150 can be associated with the latest content template set. Therefore, when target object 150 generates current content, server 110 / client 120 can directly obtain the associated content template set. Each target object 150 can also be associated with only multiple historical content generation data sets. When target object 150 generates current content, server 110 / client 120 can obtain the content template set in real time. By storing historical content generation data adopted by target object 150, or storing the content template sets associated with target object 150, server 110 / client 120 can record the target object 150's preferences and personalized needs for content generation.

[0082] like Figure 3 As shown, method 300 may further include:

[0083] S330: Match the target prompt data with the content template set, and determine the target content template from the content template set.

[0084] Step S330 can be executed on client 120 or server 110. Server 110 / client 120 can match the target prompt data with K content templates in the content template set to determine the target content template. The higher the matching degree between the target content template and the target prompt data, the higher the reference value for generating new content. For example, when generating content, target object 150 intends to generate an image of a cat. In this case, if there is a content template (image) containing a cat in the content template set associated with target object 150, then the content template containing a cat has higher reference value for generating the image of a cat again compared to other content templates in the content template set that do not contain a cat. The template containing a cat can reflect target object 150's preferences for cat breeds, fur color, outline, etc. When there are images with cats as the main subject and images without cats as the main subject in the content template set associated with target object 150 (such as cats mentioned incidentally in the description of a scene), the image with cats as the main subject may reflect the characteristics of cats in more detail. Therefore, when target object 150 intends to generate another image of a cat, images in the content template set that feature cats provide a more detailed or specific description of the cat compared to images that do not feature cats, thus having higher reference value for generating new content. In this case, images in the content template set that feature cats can be used as target content templates for generating a new image. Figure 4 A flowchart illustrating the determination of a target content template according to embodiments of this specification is shown. Figure 4 As shown, step S330 may include:

[0085] S410: Match the target prompt data with the K prompt data templates among the K content templates, and take the content template corresponding to the prompt data template that matches the target prompt data as the target content template.

[0086] Since the prompt data template of the content template can directly and accurately reflect the content of the template, when determining the target content template, server 110 / client 120 can match the target prompt data with K prompt data templates out of K content templates. The prompt data template that matches the target prompt data is taken as the target prompt data template, and the content template corresponding to the target prompt data template is taken as the target content template. For example, server 110 / client 120 can match the target prompt data with each of the K prompt data templates to obtain a matching score for each prompt data template. The prompt data template with the highest matching score can be taken as the target prompt data template, and the content template corresponding to the target prompt data template is taken as the target content template. The higher the matching score, the higher the degree of overlap and similarity between the prompt data template and the target prompt data. For example, as mentioned above, the matching score of an image with a cat as the main character is > the matching score of an image that contains a cat but not a cat as the main character is > the matching score of an image that does not contain a cat.

[0087] In some embodiments, server 110 / client 120 can calculate K similarities between the target prompt data and K prompt data templates. Server 110 / client 120 can also obtain K matching scores corresponding to the K prompt data templates based on the K similarities. The similarity and matching score can be directly proportional; the higher the similarity, the higher the matching score; the lower the similarity, the lower the matching score.

[0088] In some embodiments, server 110 / client 120 can determine the matching score between the target prompt data and each prompt data template based on a neural network model. For ease of description, we define the neural network model as a prompt data matching model. For example, a pre-trained prompt data matching model can be pre-deployed in server 110 / client 120. Server 110 / client 120 can input the target prompt data and each prompt data template into the prompt data matching model to obtain the matching score corresponding to each prompt data template.

[0089] The prompt data matching model can be trained in the following way. For ease of description, we define the device that trains the prompt data matching model as the training device. The training data of the prompt data matching model may include multiple first training samples. The training device can obtain multiple first training samples and their corresponding first labels, and obtain a first loss based on the multiple first training samples and their corresponding first labels to train the prompt data matching network to obtain the prompt data matching model. Each of the multiple first training samples may include sample prompt data and its corresponding sample content template set. The sample content template set is generated based on the historical content of the sample user corresponding to the sample prompt data. The sample content template set and the sample prompt data correspond to the same sample user. For example, the sample content template set can be obtained based on the aforementioned method of obtaining the content template set, which will not be elaborated here. Each sample content template set may include multiple sample content templates. Each sample content template may include a sample prompt data template. The sample prompt data is different from the multiple sample prompt data templates in its corresponding sample content template set. The first label may include the true matching score between the pre-labeled sample prompt data and each sample content template. For example, the first label may include the content template that matches the sample prompt data in its corresponding first training sample. The matching content template is one of the sample content templates in the set. The first tag can be manually labeled, for example, by a designer labeling the matching score between the sample prompt data and each sample prompt data template. For example, the matching template gets a perfect score, while the other templates get a low score or zero score.

[0090] The training device can sequentially input sample prompt data and each sample prompt data template from its corresponding sample content template into the prompt data matching network, which then outputs the corresponding predicted matching score. The training device can derive a first loss based on the difference between the predicted matching score and the pre-labeled true matching score, and train the prompt data matching network based on this first loss until convergence, thus obtaining the prompt data matching model. The training objective of the prompt data matching model can be to constrain the difference between its output predicted matching score and the pre-labeled true matching score to be less than a first preset difference threshold. The first loss can be the difference between the predicted matching score and the true matching score. The training objective of the prompt data matching model can be to constrain the first loss to be less than the first preset difference threshold. The first preset difference threshold can be a small value to make the predicted matching score and the pre-labeled true matching score tend to be consistent.

[0091] Determining the target content template from the content template set is equivalent to determining the reference template for content generation. Therefore, in method 300, the server 110 / client 120 can adaptively select a target content template from the content template set that matches the target prompt data as the reference template for the target prompt data, avoiding the user's significant manpower investment in selection and thus improving the efficiency of content generation. Simultaneously, the target content template can reflect the preferences of the target object 150, thus satisfying the target object 150's personalized needs, while also ensuring the controllability of the content generation process, avoiding frequent adjustments after content generation due to over-reliance on user personal choices, thereby improving the quality of content generation. As mentioned earlier, the target content template can serve as a reference template for the content generation process. To further meet personalized needs, specific reference content is determined within the reference template, such as... Figure 4 As shown, step S330 may further include:

[0092] S430: At least match the target prompt data with the target content template to determine the target clues that match the target prompt data.

[0093] Server 110 / Client 120 can match target prompt data with target content templates to select target clues from the target content templates. Clues can be understood as precise control over referable content in the reference template when generating content. As mentioned earlier, prompt data can describe the content to be generated. In some embodiments, textual expressions may not be able to fully and accurately depict the details of the content to be generated. Therefore, when generating content, clues contained in the reference template can be used to finely depict the details of the content to be generated. By controlling the clues of the content, the controllability and stability of content generation can be further improved. Clues can include various types of content. For example, taking an image as an example, the types of clues can be the edges of objects in the image, the depth information of the image, the segmentation map of the image, the pose of a character in the image, etc. Among them, the edges of objects in the image can be understood as the edge contours of each object in the image; the depth information of the image can be understood as the distance of each point in the image space relative to the camera; the segmentation map of the image can be understood as distinguishing different types of objects in the image through different gray levels, colors, etc.; the pose of a character in the image can be understood as the action or posture of a character in the image. Target clues can include target clue types and corresponding target clue content. Target clue content can be the content within the target content template under the target clue type. A target clue can be understood as a clue within the target content template that matches the target hint data. In other words, a target clue is a clue that matches the target content that target object 150 wants to generate. That is, target content that retains target clues better meets target object 150's expectations for the target content and is more likely to be adopted by target object 150.

[0094] In some embodiments, all content in the target content template can be used as a reference for content generation based on the target prompt data. In some embodiments, the portion of the target content template that matches the target prompt data can be used as a reference for content generation based on the target prompt data. Figure 5 A flowchart illustrating the determination of target clues according to embodiments of this specification is shown. Figure 5 As shown, step S430 of server 110 / client 120 determining a target clue that matches the target cue data may include the following steps:

[0095] S510: At least match the target prompt data with the target content template to determine the target clue type that matches the target prompt data.

[0096] The target cue type can be any cue type that matches the target cue data from all cue types in the target content template. The target cue type can serve as a reference cue type when generating content from the target cue data. For example, the target cue type can be any one or any combination of the cue types mentioned above. For instance, taking an image as an example, the target cue type can include one or a combination of the following: the edges of objects within the image, the depth information of the image, the segmentation map of the image, the pose of characters in the image, etc.

[0097] In some embodiments, server 110 / client 120 can determine the target clue type that matches the target cue data based on a neural network model. For ease of description, we define the neural network model as a cue type matching model. For example, a pre-trained cue type matching model can be pre-deployed in server 110 / client 120. Server 110 / client 120 can input the target cue data and the target adoption content template in the target content template into the cue type matching model to obtain the target cue type.

[0098] The cue type matching model can be trained in the following way. For ease of description, we define the device that trains the cue type matching model as the training device. The training data of the cue type matching model can include multiple second training samples. The training device can obtain multiple second training samples and their corresponding second labels, and obtain a second loss based on the multiple second training samples and their corresponding second labels to train the cue type matching network to obtain the cue type matching model. Each of the multiple second training samples can include sample cue data and its corresponding sample reference content. Among them, the sample reference content is the content in the sample content template set that best matches the sample cue data. The sample reference content can be obtained from the sample content template set corresponding to the sample user based on the aforementioned method of obtaining the target content template. The sample reference content can also be manually labeled from the sample content template set. The sample reference content can serve as a reference when generating content based on the sample cue data. The second label can be a pre-labeled real cue type. For example, the second label can include the cue type that matches the sample cue data in the sample reference content of its corresponding second training sample. That is, the sample reference content includes the matching cue type. The second label can be manually annotated, for example, by the designer marking the clue types in the sample reference content that match the sample hint data. The types of clues can be preset. The designer can mark the clue types in the sample reference content that match the sample hint data as 1, and mark the other clue types as 0.

[0099] The training device can sequentially input sample cue data and its corresponding sample reference content into the cue type matching network, which then outputs the corresponding predicted cue type. The training device can derive a second loss based on the difference between the predicted cue type and the pre-labeled real cue type, and train the cue type matching network based on this second loss until convergence, thus obtaining the cue type matching model. The training objective of the cue type matching model can be to constrain the difference between its output predicted cue type and the pre-labeled real cue type to be less than a second preset difference threshold. The second loss can be the difference between the predicted cue type and the real cue type. The training objective of the cue type matching model can be to constrain the second loss to be less than a second preset difference threshold. The second preset difference threshold can be a small value to make the predicted cue type and the real cue type tend to be consistent.

[0100] In some embodiments, the server 110 / client 120 can also sequentially input the target prompt data, the target adopted content template in the target content template, and the matching score corresponding to the target content template into a trained clue type matching model to obtain the target clue type. As mentioned earlier, the higher the matching score between the target prompt data and the target content template, the higher the similarity between them. At this time, the matching degree between the target adopted content template and the content that the target object 150 wants to generate based on the target prompt data is also higher. That is, the target adopted content template has more clues that match the content that the target object 150 wants to generate based on the target prompt data, i.e., the target adopted content template has more types of clues that can be used as reference clues for the target object 150 to generate content based on the target prompt data. Therefore, the matching score can be used to control the number of target clue types. For example, with a matching score of 100 as the highest score, a matching score of 95 is equivalent to a matching score of 50, meaning that the target adopted content template with a matching score of 95 can have more reference clue types.

[0101] At this point, each second training sample can also include a matching score corresponding to the sample reference content. The matching score is obtained based on the sample cue data and the sample cue data template corresponding to the sample reference content. The matching score can be obtained using the method described above for obtaining the matching score between the target cue data and the cue data template. Alternatively, the matching score can be obtained through pre-labeling. The matching score indicates the similarity between the sample cue data template and the sample cue data. The higher the matching score, the higher the similarity between the two. The training method for the cue type matching model is similar to that described above and will not be repeated here.

[0102] To further determine the specific content that can be referenced in the target content template when generating content, such as... Figure 5 Step S430 may also include:

[0103] S530: Determine the target clue content corresponding to the target clue type from the target content template.

[0104] Server 110 / Client 120 can determine the target clue content corresponding to the target clue type from the target content template. The target clue content can be the content in the target adoption content template that corresponds to the target clue type. Server 110 / Client 120 can extract the content of the target clue type reflected in the target adoption content template as the target clue content. For example, taking an image as an example, the target clue content can be an image that reflects the content corresponding to the target clue type. If the target adoption content template is an image of a cat, and the target clue type is edge, the target clue content can be an image composed of lines including the outline of the cat's body, eyes, and nose, etc., while the cat's fur color, pupil color, and background are not shown. Server 110 / Client 120 can use algorithms such as Canny edge detection to extract the edge image. As another example, if the target adoption content template is an image of a person against a building background, and the clue type is depth, the target clue content can be a depth map with depth structure. For example, in the obtained depth image, the parts closer to the camera are whiter, and the parts farther away from the camera are darker, thus reflecting depth. However, the target clues did not display details such as people's appearances, clothing, or building colors. The depth of the image extracted by server 110 / client 120 can be estimated using algorithms such as LeReS depth information.

[0105] When there are multiple target cue types, there are also multiple target cue contents. Taking images as an example, target cue types can include depth and pose. In this case, the target cue content can include two images: one showing depth and one showing pose. The target cue content can also include only one image, where both depth and pose are shown simultaneously.

[0106] Back Figure 3 After step S330, method 300 may further include the following steps:

[0107] S350: Using the target content template as a reference for content generation, generate target content based on the target prompt data.

[0108] Step S350 can be executed by either client 120 or server 110. Server 110 / client 120 can generate target content based on target prompt data and can use the target content template as a reference for content generation. As mentioned earlier, the target content template is the content template in the content template set of target object 150 that has the highest matching degree with the target prompt data. Therefore, the target content obtained by server 110 / client 120 using the target content template as a reference for content generation not only conforms to the preferences of target object 150 and meets the personalized needs of target object 150, but also ensures the controllability of the content generation process.

[0109] In some embodiments, using the target content template as a reference can refer to the target adoption content template within the target content template. In some embodiments, using the target content template as a reference can refer to the target lead content within the target adoption content template, ensuring that the content corresponding to the target content under the target lead type is completely or nearly identical to the target lead content. In some embodiments, using the target content template as a reference can refer to both the target adoption content template and the target lead content therein. For example, the target lead content can be used as a primary reference, ensuring that the content corresponding to the target content under the target lead type is completely or nearly identical to the target lead content. Alternatively, the target adoption content template can be used as a secondary reference, ensuring that the content corresponding to the target content under other types besides the target lead type can partially match the content in the target adoption content template.

[0110] In summary, the target content generated based on the target hint data and target content template can include content that is completely identical or completely matched with the content described in the target hint data. The target content can also include content that is identical or nearly identical to the target clue content. Furthermore, the target content can include other content from the target adoption content template besides the target clue content, and must match the target hint data. For ease of description, we will use the target adoption content template and its target clue content as a reference example.

[0111] For example, taking an image as an example, target object 150 intends to generate an image of a cat. The target cue data obtained by server 110 / client 120 may include keywords such as {<watercolor>, <looking forward>, <silver shaded>, <black pupils>, ...}. Server 110 / client 120 obtains the target cue type as edge and the corresponding edge line drawing as target cue content from the target content template. When server 110 / client 120 generates a new cat image based on the target cue data, the edge outline of the new cat and the edge line drawing of the target cue content can be approximately the same or completely the same. Other characteristics of the cat are generated based on the target cue data. Other characteristics may include the cat's black pupils, silver shaded fur, watercolor style, and looking forward expression, etc. However, since the silver shaded fur in the target cue data is a relatively general description of the cat's fur color, and the target cue data does not limit the breed of the cat, server 110 / client 120 can still generate multiple images based on the target cue data and target content template. Furthermore, some of the generated images can reference information such as the cat's fur color and depth from the target content template. However, the outlines of the cats in all the new cat images generated by server 110 / client 120 conform to the preset outline edges, which are the outlines in the target cue content, and the cats all possess the characteristics described in the target cue data.

[0112] In some embodiments, server 110 / client 120 can generate target content based on a neural network. For ease of description, we define the neural network model as a generative model. For example, a pre-trained generative model can be pre-deployed in server 110 / client 120. In some embodiments, server 110 / client 120 can input target hint data and target content template into the generative model to obtain target content, such that the target content matches the target clue content. In some embodiments, server 110 / client 120 can input target hint data and target adoption content template into the generative model to obtain target content, such that the target content matches the target clue content. In some embodiments, server 110 / client 120 can input target hint data, target adoption content template, and target clue content template into the generative model to obtain target content, such that the target content matches the target clue content. For ease of description, we will use the example of server 110 / client 120 inputting target hint data, target adoption content template, and target clue content template into the generative model to obtain target content, such that the target content matches the target clue content, for illustration.

[0113] Generative models can be any type of model that generates content based on target prompt data and reference content. Examples include diffusion models and generative adversarial networks (GANs). For ease of description, we will use diffusion models as an example.

[0114] In some embodiments, the generated model may include a template feature encoding module, a clue feature encoding module, and a diffusion module. The server 110 / client 120 can input the target content template into the template feature encoding module to obtain the output target content template features. The server 110 / client 120 can input the target clue content into the clue feature encoding module to obtain the output target clue content features. The server 110 / client 120 can input the target hint data, the target content template features, and the target clue content features into the diffusion module, and the diffusion module will output the target content.

[0115] The generative model can be trained in the following way. For ease of description, we define the device that trains the generative model as the training device. The training data for the generative model can include multiple third training samples. The training device can obtain multiple third training samples and their corresponding third labels, and obtain a third loss based on the multiple third training samples and their corresponding third labels to train the generative network to obtain the generative model. Each of the multiple third training samples can include sample cue data and its corresponding sample reference content and sample reference clue content. Among them, the sample reference clue content is the clue content corresponding to the sample clue type that best matches the sample cue data in the sample reference content. The sample reference clue content can be obtained from the sample reference content based on the above-mentioned method of obtaining target clue content. The sample reference content can be used as a reference when generating content based on the sample cue data. The sample reference clue content can appear in the content generated based on the sample cue data. The third label can include pre-labeled real generated content generated based on the sample cue data, sample reference content, and sample reference clue content. The third label can be pre-labeled using a manual annotation method. For example, multiple pieces of content can be generated based on sample prompt data, sample reference content, and sample reference clue content. The third tag can be content that matches the sample user's expectations, manually marked from the multiple generated contents, or content that the sample user ultimately adopted from the multiple generated contents. The actually generated content can include sample reference clue content.

[0116] The training device can sequentially input sample cue data, along with their corresponding sample reference content and sample reference clues, into the generator network, which then outputs the corresponding predicted generated content. The training device can derive a third loss based on the difference between the predicted generated content and the pre-labeled real generated content, and train the generator network based on this third loss until convergence, thus obtaining the generator model. The training objective of the generator model can be to constrain the difference between its output predicted generated content and the pre-labeled real generated content to be less than a third preset difference threshold. The third loss can be the difference between the predicted generated content and the real generated content. The training objective of the generator model can be to constrain the third loss to be less than a third preset difference threshold. The third preset difference threshold can be a small value to make the generated content and the real generated content tend to be consistent.

[0117] like Figure 3 As shown, method 300 may further include:

[0118] S370: Output the target content.

[0119] Step S370 can be executed by the client 120 or by the server 110.

[0120] When step S370 is executed by client 120, client 120 can visually display the generated target content. For example, in Figure 1B In this system, when the target content is an image, one or more generated images can be displayed in the image display box. When the target content is audio, it can be played through a player, and so on. In some embodiments, the generated target content can be saved as a file on the client 120. After the target object 150 clicks on the corresponding file, the target content can be displayed.

[0121] When step S370 is executed by server 110, server 110 can send the target content to client 120. After receiving the target content, client 120 can output it to target object 150 in the manner described above.

[0122] In summary, the content generation method 300 and system 100 provided in this specification, when generating target content for target object 150, allow server 110 / client 120 to adaptively select a target content template that matches the target prompt data from the historical content generation data corresponding to target object 150 as a reference template for the target prompt data, thereby improving the efficiency of content generation. Simultaneously, by using historical generated content data that matches the target prompt data as reference content for content generation, method 300 and system 100 can ensure and improve the controllability of the target content generation process and the personalization effect of the target content.

[0123] It should be noted that the content generation data in the embodiments of this specification is not data specific to any particular user and does not reflect relevant information about a particular user. The collection, storage, use, processing, transmission, provision, and disclosure of user-related content data involved in the technical solutions of this disclosure all comply with relevant laws and regulations and do not violate public order and good morals.

[0124] This specification also provides a non-transitory storage medium storing at least one set of executable instructions for performing content generation. When the executable instructions are executed by the processor 220, they instruct the processor 220 to implement the content generation method described in this specification.

[0125] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0126] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0127] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0128] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and aiding in the understanding of a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art may readily identify some of the devices as separate embodiments when reading this specification. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. It is also valid when each secondary embodiment contains fewer than all the features of a single foregoing disclosed embodiment.

[0129] Each patent, patent application, publication of the patent application, and other materials such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated by reference. All contents used for all purposes, except for any history of prosecution documents relating to it, that may be inconsistent with or conflict with this document, or any such history of prosecution documents that may have a limiting effect on the widest extent of the claims, are now or hereafter associated with this document. For example, in the event of any inconsistency or conflict between the description, definition, and / or use of terms associated with any of the included materials and the terms, description, definition, and / or used in connection with this document, the terms used herein shall prevail.

[0130] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A content generation method, comprising: Obtain target prompt data input by the target object and a content template set corresponding to the target object. The content template set is obtained based on multiple historical content generation data of the target object. The historical content generation data includes historical prompt data input by the target object during the content generation process in a historical time, and historical adopted content adopted by the target object from multiple historical content generated based on the historical prompt data. The target prompt data is matched with the content template set to determine the target content template from the content template set; The target content template is used as a reference for content generation, and target content is generated based on the target prompt data. as well as Output the target content.

2. The method as described in claim 1, wherein, The content template set is updated based on a preset cycle.

3. The method as described in claim 2, wherein, The multiple historical content generation data include the content template set of the previous period and the content generation data generated from the previous period to the current period.

4. The method of claim 1, wherein, The content template set includes K content templates, and the template similarity between the K content templates meets a preset condition. Each of the K content templates includes a prompt data template and its corresponding adoption content template, where K is a positive integer.

5. The method of claim 4, wherein, The preset conditions include: The template similarity of the K content templates is less than a preset threshold; and / or Among all combinations of the multiple historical content generation data, the template similarity statistic among the K content templates is the smallest. The template similarity statistic includes the sum or mean of the template similarities. Each of the multiple combinations includes K historical content generation data.

6. The method described in claim 4, wherein, The template similarity includes one of the following: The similarity of the features of the prompt data templates corresponding to the prompt data templates; The similarity of the features of the adopted content templates corresponding to the adopted content templates; as well as The similarity of the combination of the prompt data template features and the adopted content template features.

7. The method of claim 4, wherein, The step of matching the target prompt data with the content template set to determine the target content template from the content template set includes: The target prompt data is matched with K prompt data templates from the K content templates, and the content template corresponding to the target prompt data template that matches the target prompt data is taken as the target content template.

8. The method of claim 7, wherein, The step of matching the target prompt data with K prompt data templates from the K content templates, and using the content template corresponding to the target prompt data template that matches the target prompt data as the target content template, includes: The target prompt data is matched with each of the K prompt data templates to determine a matching score for each prompt data template; and The prompt data template with the highest matching score is used as the target prompt data template, and the content template corresponding to the target prompt data template is used as the target content template.

9. The method of claim 8, wherein, The step of matching the target prompt data with each of the K prompt data templates to determine the matching score corresponding to each prompt data template includes: The target prompt data and each prompt data template are input into the prompt data matching model to obtain the matching score corresponding to each prompt data template. The training data for the prompt data matching model includes sample prompt data and a sample content template set. The training objective of the prompt data matching model includes constraining the difference between its output predicted matching score and the pre-labeled true matching score to be within a first preset difference threshold.

10. The method of claim 8, wherein, The step of matching the target prompt data with the content template set and determining the target content template from the content template set further includes: At least the target prompt data is matched with the target content template to determine the target clues that match the target prompt data.

11. The method of claim 10, wherein, The step of matching the target cue data with the target content template to determine the target clues that match the target cue data includes: At least the target cue data is matched with the target content template to determine the type of target clue that matches the target cue data; and Determine the target clue content corresponding to the target clue type from the target content template. The target clues include the target clue type and the target clue content.

12. The method of claim 11, wherein, The step of matching the target cue data with the target content template to determine the target cue type that matches the target cue data includes: The target prompt data and the target adoption content template in the target content template are input into the clue type matching model to obtain the target clue type. The training data of the clue type matching model includes sample cue data and its corresponding sample reference content. The training objective of the clue type matching model includes constraining the difference between the predicted clue type output by the model and the pre-labeled real clue type to be within a second preset difference threshold. The sample reference content includes the real clue type.

13. The method of claim 11, wherein, The step of matching the target cue data with the target content template to determine the target cue type that matches the target cue data includes: The target prompt data, the target adopted content template in the target content template, and the matching score corresponding to the target content template are input into the clue type matching model to obtain the target clue type. The training data of the clue type matching model includes sample cue data and its corresponding sample reference content and sample matching score. The training objective of the clue type matching model includes constraining the difference between the predicted clue type output and the pre-labeled real clue type to be within a second preset difference threshold. The sample reference content includes the real clue type.

14. The method of claim 11, wherein, The step of using the target content template as a reference for content generation and generating target content based on the target prompt data includes: The target prompt data, the target adoption content template from the target content template, and the target clue content are input into the generation model to obtain the target content, such that the target content matches the target clue content. The training data for the generative model includes sample cue data and its corresponding sample reference content and sample reference clue content. The training objective of the generative model includes constraining the difference between its output predicted generated content and the pre-labeled real generated content to be within a third preset difference threshold. The real generated content includes the sample reference clue content. The generative model is a diffusion model.

15. A content generation system, comprising: At least one storage medium storing at least one instruction set for content generation; as well as At least one processor is communicatively connected to the at least one storage medium. When the content generation system is running, the at least one processor reads the at least one instruction set and executes the method according to any one of claims 1-14 according to the instructions of the at least one instruction set.

Images