A content generation method and device, electronic equipment and storage medium
By using an alternating and collaborative text and image generation model, user-created content is optimized to generate richer and higher-quality text and image results. This solves the problem of simple content and low generation efficiency in existing technologies, and achieves diversification and quality improvement in content creation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING ZITIAO NETWORK TECH CO LTD
- Filing Date
- 2023-11-13
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, user-generated content may be simple and monotonous, and the efficiency and quality of generating other types of content are low.
By using an alternating collaborative approach, text generation and image generation models are employed to generate optimized target text descriptions and images. Based on key text descriptions and target text descriptions, different types of graphic and textual results are generated, representing the varying degrees of importance of the images and text.
It improves the efficiency and quality of content creation, making the generated content richer and more diverse.
Smart Images
Figure CN117523020B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and more specifically, to a content generation method, apparatus, electronic device, and storage medium. Background Technology
[0002] With the development of computer technology, users can create content more conveniently. Users usually have different content creation needs, such as creating pictures and image text, and then they can upload the created content to the platform to share with other users. However, the text or pictures created by users may be relatively simple and monotonous. In related technologies, it is also possible to convert a certain type of content into other types to improve the richness of the content. However, when generating other types of content, the efficiency is low and the quality of the generated content is also poor. Summary of the Invention
[0003] This disclosure provides at least one content generation method, apparatus, electronic device, and storage medium.
[0004] In a first aspect, embodiments of this disclosure provide a content generation method, comprising: acquiring input content to be processed; determining optimized target text description information corresponding to the content to be processed, and determining corresponding key text description information based on the target text description information, wherein the target text description information is generated in an alternating collaborative manner based on a text generation model and an image generation model, representing optimized text description information corresponding to the content to be processed; generating a first type of first image-text result or a second type of second image-text result based on the key text description information and the target text description information, wherein the different types of image-text results represent different degrees of importance of image genres and text genres, and the target images of the image genres included in the different types of image-text results are generated based on the key text description information or the target text description information.
[0005] Secondly, this disclosure also provides a content generation apparatus, comprising: an acquisition module for acquiring input content to be processed; a determination module for determining optimized target text description information corresponding to the content to be processed, and determining corresponding key text description information based on the target text description information, wherein the target text description information is generated based on a text generation model and an image generation model in an alternating collaborative manner; and a generation module for generating a first type of first image-text result or a second type of second image-text result based on the key text description information and the target text description information, wherein the different types of image-text results represent different degrees of importance of image genre and text genre, and the target image of the image genre in the different types of image-text results is generated based on the key text description information or the target text description information.
[0006] Thirdly, an optional implementation of this disclosure also provides an electronic device, including a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is configured to execute the machine-readable instructions stored in the memory, wherein when the machine-readable instructions are executed by the processor, the processor performs the steps in the possible implementation of the first aspect described above.
[0007] Fourthly, an optional implementation of this disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the possible implementations of the first aspect described above.
[0008] In this embodiment of the disclosure, the image and text are optimized through image-text generation interaction, thereby generating richer and higher quality target images and target text description information, improving content creation efficiency, as well as content quality and diversity.
[0009] For a description of the effects of the above-mentioned content generation apparatus, electronic device, and computer-readable storage medium, please refer to the description of the above-mentioned content generation method; it will not be repeated here.
[0010] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure.
[0011] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0012] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.
[0013] Figure 1 A flowchart of a content generation method provided by an embodiment of this disclosure is shown;
[0014] Figure 2 A schematic diagram illustrating the implementation effect of a content generation method provided in an embodiment of this disclosure is shown.
[0015] Figure 3 This diagram illustrates the principle of a text optimization process in the content generation method provided by this disclosure embodiment;
[0016] Figure 4 A schematic diagram illustrating the implementation effect of another content generation method provided in this embodiment of the present disclosure is shown.
[0017] Figure 5 This illustration shows a schematic diagram of another text optimization process in the content generation method provided in this disclosure embodiment;
[0018] Figure 6 A schematic diagram illustrating the implementation effect of another content generation method provided in this embodiment of the present disclosure is shown.
[0019] Figure 7 A flowchart of another content generation method provided by an embodiment of this disclosure is shown;
[0020] Figure 8 A schematic diagram of a content generation apparatus provided in an embodiment of this disclosure is shown;
[0021] Figure 9 A schematic diagram of another content generation apparatus provided in an embodiment of this disclosure is shown;
[0022] Figure 10 A schematic diagram of an electronic device provided in an embodiment of the present disclosure is shown. Detailed Implementation
[0023] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.
[0024] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown herein can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.
[0025] Research has found that users typically have different content creation needs, such as creating images and accompanying text, which they can then upload to the platform to share with other users. However, the text or images created by users may be relatively simple and monotonous. In order to make the content more attractive, how to more effectively generate richer and higher-quality content is an urgent problem to be solved.
[0026] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below should be considered as the inventor's contribution to this disclosure.
[0027] Based on the above research, this disclosure provides a content generation method. Specifically, in one method, input content to be processed is obtained; optimized target text description information corresponding to the content to be processed is determined, and corresponding key text description information is determined based on the target text description information, wherein the target text description information is generated in an alternating collaborative manner based on a text generation model and an image generation model; based on the key text description information and the target text description information, a first type of first image-text result or a second type of second image-text result is generated, wherein the different types of image-text results represent different degrees of importance of image genre and text genre, and the target image of the image genre in the different types of image-text results is generated based on the key text description information or the target text description information. In this way, for the input text or image to be processed, richer and higher quality target images and target text description information for the target images can be generated through image-text generation interaction, thereby improving efficiency and making content creation more diversified.
[0028] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0029] To facilitate understanding of this embodiment, a detailed description of the content generation method disclosed in this disclosure is provided first. The execution subject of the content generation method provided in this disclosure is generally an electronic device with a certain computing capability. This electronic device may include, for example, a terminal device, a server, or other processing devices. The terminal device can be a user equipment (UE), mobile device, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. A personal digital assistant is a handheld electronic device that possesses some functions of a computer. It can be used to manage personal information, browse the internet, send and receive emails, etc., and generally does not have a keyboard; it can also be called a handheld computer. In some possible implementations, this content generation method can be implemented by a processor calling computer-readable instructions stored in memory.
[0030] The content generation method can be executed in any computing-capable entity; for example, it can be executed on a terminal device or a server device. See also Figure 1 The diagram shown is a flowchart of a content generation method provided in an embodiment of this disclosure. The method includes:
[0031] S101: Obtain the input content to be processed, where the content to be processed includes text to be optimized or image to be optimized.
[0032] This disclosure primarily targets content creation scenarios based on machine learning. Users may require expanding, imagining, and enhancing initial content to obtain richer and higher-quality content. Considering the diversity and pluralistic needs of content creation, such as the combination of text and images in content uploaded to the platform to increase traffic, users in this disclosure only need to input the initial text or image to be optimized to obtain the optimized text-image pair.
[0033] S102: Determine the optimized target text description information corresponding to the content to be processed, and determine the corresponding key text description information based on the target text description information. The target text description information is generated based on the text generation model and the image generation model in an alternating collaborative manner.
[0034] When performing step S102, this disclosure provides possible implementation methods:
[0035] S1. Based on the text generation model, generate the first text description information of the first image corresponding to the optimized target text description information, wherein the first image is generated based on the image generation model and the text to be processed, or the first image is the target frame image included in the video to be processed.
[0036] In this embodiment of the disclosure, the content to be processed includes text to be processed or video to be processed. For the case where the user input is text to be optimized, a first image can be generated first, and the first text description information of the first image can be obtained. Then, based on a text generation model, optimized target text description information can be generated. For the case where the user input is an image to be optimized, the first text description information of the image to be optimized can be obtained, and optimized target text description information can be generated based on a text generation model.
[0037] In this embodiment of the disclosure, generating optimized target text description information corresponding to the first text description information of the first image based on a text generation model includes: when the content to be processed is a video to be processed, determining a target frame image based on the keyframes of the video to be processed, or determining a corresponding target frame image based on a preset frame interval; obtaining the first text description information of the target frame image based on an image description model, using the target frame image as input; determining target optimization guidance information; inputting the target optimization guidance information and the first text description information into the text generation model, performing semantic analysis on the first text description information based on the target optimization guidance information, and generating optimized target text description information corresponding to the first text description information.
[0038] In this embodiment of the disclosure, generating optimized target text description information corresponding to the first text description information of the first image based on a text generation model includes: when the content to be processed is text to be processed, generating a first image corresponding to the text to be processed based on an image generation model, using the text to be processed as input; obtaining the first text description information of the first image based on an image description model, using the first image as input; determining target optimization guidance information; inputting the target optimization guidance information and the first text description information into the text generation model, performing semantic analysis on the first text description information according to the target optimization guidance information, and generating optimized target text description information corresponding to the first text description information.
[0039] In this embodiment of the disclosure, determining the target optimization guidance information includes: determining that the target optimization guidance information is a preset optimization guidance statement; or, determining that the content to be processed corresponds to the target style to be optimized, and determining the target optimization guidance information corresponding to the target style based on the target style.
[0040] In this embodiment of the disclosure, determining the target style to be optimized for the content to be processed includes: in response to a selection operation for each preset style, determining the preset style corresponding to the selection operation as the target style to be optimized for the content to be processed; or, extracting the target style to be optimized for the content to be processed based on the content to be processed.
[0041] S2. Determine the corresponding key text description information based on the target text description information.
[0042] S103. Based on the key text description information and the target text description information, generate a first type of first image and text result or a second type of second image and text result. The different types of image and text results represent different levels of importance of image genre and text genre, and the target image of the image genre in the different types of image and text results is generated based on the key text description information or the target text description information.
[0043] In this embodiment of the disclosure, generating a first type of first image-text result based on key text description information and target text description information includes: performing semantic analysis on key text description information based on an image generation model to generate an optimized target image; and obtaining a first type of first image-text result based on the target text description information and the target image.
[0044] In this embodiment of the disclosure, generating a first image-text result of the second type based on key text description information and target text description information includes: obtaining multiple sub-target text description information after segmentation based on target text description information; performing semantic analysis on the multiple sub-target text description information based on an image generation model to generate multiple optimized target images; and obtaining a second image-text result of the second type based on key text description information and multiple target images.
[0045] In this embodiment of the disclosure, based on an image generation model, semantic analysis is performed on the text description information of multiple sub-targets according to the text description information of multiple sub-targets, and multiple optimized target images are generated accordingly. This includes: based on an image generation model, semantic analysis is performed on the text description information of multiple sub-targets and the target style as input, and multiple optimized target images are generated accordingly.
[0046] In this embodiment of the disclosure, based on an image generation model, semantic analysis is performed on the key text description information according to the key text description information to generate an optimized target image, including: based on an image generation model, taking the key text description information and the target style as input, performing semantic analysis on the key text description information and the target style to generate an optimized target image.
[0047] In one possible implementation, in this embodiment of the present disclosure, during free optimization, i.e. when the target optimization guidance information does not include the target style, the target text description information has already been obtained based on the text generation model. Therefore, the target text description information can be input into the image generation model to perform sentence analysis on the target text description information, and an optimized target image can be generated.
[0048] In one possible implementation, under the guidance of optimization towards a specific target style, optimized target text description information is obtained based on the target style. In order to further enhance the target style generation effect of the image, the target style can be substituted into the image generation process again in this embodiment of the disclosure. Specifically, this disclosure provides a possible implementation method, which is based on an image generation model, takes target text description information and target style as input, performs semantic analysis on target text description information and target style to obtain text features, and obtains image features associated with text features based on text features, and generates optimized target images based on image features.
[0049] Furthermore, in this embodiment of the present disclosure, in order to improve the correlation between the text and the target image after generating the optimized target image, textual description information of the target image can be generated based on the target image. Specifically, this disclosure provides a possible implementation method: inputting the target image into an image description model to obtain the second textual description information of the target image; inputting the target optimization guidance information and the second textual description information into a text generation model, performing semantic analysis on the second textual description information, generating optimized second textual description information, and returning the target image and the optimized second target textual description information.
[0050] Furthermore, in this embodiment of the disclosure, there is no limitation on the number of optimized target images and target text description information. It can be a preset target number or based on the target number requirement input by the user. Specifically, based on the text generation model, the first text description information of the first image is generated to correspond to the optimized target number of target text description information; for each target text description information, an optimized target image is generated based on the image generation model and the target text description information.
[0051] In this embodiment of the disclosure, the target text description information is the description information for the target image. The target image and the target text description information are image-text pairs, and the number of them is the same.
[0052] In this embodiment of the disclosure, optimized target text description information and target image are returned. After being returned to the terminal device, the optimized target text description information and target image can be displayed to the user on the terminal device.
[0053] Furthermore, to further improve accuracy, the processes in steps S102-S103 above can be executed repeatedly. Specifically, the process of generating target text description information and target image involves generating optimized target text description information based on the text generation model, generating optimized target image based on the image generation model and target text description information, then further optimizing the text description information of the target image, and generating a further optimized target image based on the further optimized text description information, until a preset number of iterations is reached or other preset conditions are met. This improves accuracy, makes the final target image and target text description information richer and of higher quality, and also enhances the correlation between the target text description information and the target image.
[0054] In this embodiment, the input content to be processed is obtained; the optimized target text description information corresponding to the content to be processed is determined, and the corresponding key text description information is determined based on the target text description information. The target text description information is generated using an alternating collaborative approach based on a text generation model and an image generation model. Based on the key text description information and the target text description information, a first type of first image-text result or a second type of second image-text result is generated. The different types of image-text results represent different levels of importance between the image genre and the text genre, and the target image for the image genre in the different types of image-text results is generated based on the key text description information or the target text description information. Thus, by connecting the text generation model and the image generation model, and alternately generating text and images for the content to be optimized, diversified image-text pair generation is achieved. Furthermore, the input content to be optimized can be optimized to generate optimized target images and target text description information, resulting in richer and higher-quality content, improving content quality, and increasing content creation efficiency.
[0055] In this embodiment of the disclosure, in step S102, the optimized target text description information corresponding to the content to be processed is determined, and the corresponding key text description information is determined based on the target text description information. The target text description information is generated using an alternating collaborative approach based on a text generation model and an image generation model. Accordingly, in step S102, the target text description information can be generated using an alternating collaborative approach based on a text generation model and an image generation model. Possible implementation methods are also provided in this process:
[0056] In one possible embodiment, based on a text generation model, optimized target text description information corresponding to the first text description information of the first image is generated. The first image is generated based on an image generation model and the text to be processed, or the first image is a target frame image included in the video to be processed, comprising:
[0057] 1) When the content to be processed is a video, determine the target frame image based on the keyframes of the video, or determine the corresponding target frame image based on the preset frame interval; based on the image description model, take the target frame image as input to obtain the first text description information of the target frame image.
[0058] Alternatively and / or additionally, when the content to be processed is text to be processed, a first image corresponding to the text to be processed is generated based on an image generation model, with the text to be processed as input; and a first text description information of the first image is obtained based on an image description model, with the first image as input.
[0059] For example, a user may only have a simple image, or an image that only contains the user's basic idea. The user may need to create more detailed and engaging images based on this simple image, along with more vivid text descriptions. In this case, the user can input the simple image, and then use it as the image to be optimized for further refinement.
[0060] In this embodiment of the disclosure, after obtaining the image to be optimized, the image feature vector of the image to be optimized can be extracted based on the image description model, and the image feature vector can be decoded to obtain the first text description information.
[0061] The first text description information generated by the image description model is usually simple and basic. For example, if the image to be optimized is a picture containing a cat, the first text description information obtained by the image description model may be "a cat", which is relatively simple. Therefore, in this embodiment of the disclosure, it is necessary to further optimize and expand the first text description information to obtain richer and more vivid image description information.
[0062] 2) Define the target and optimize the guidance information.
[0063] In this embodiment of the disclosure, determining the target optimization guidance information includes: determining that the target optimization guidance information is preset optimization guidance statement information; or, determining that the content to be processed corresponds to the target style to be optimized, and determining the target optimization guidance information corresponding to the target style based on the target style. Specifically, when optimizing images or text, different optimization directions are also provided, and different optimization directions correspond to different target optimization guidance information. Possible implementation methods are specifically provided as follows:
[0064] One approach: Determine the target optimization guidance information as preset optimization guidance statement information.
[0065] In this embodiment of the disclosure, optimization guidance information can be preset. For example, the preset optimization guidance information is: "Optimize the language of the following text to make it more vivid. You can add imagination and emoticons as appropriate:". In this embodiment of the disclosure, the preset optimization guidance information is more inclined to freely optimize the image or text. It does not specify the optimization direction or style, but only to optimize to produce richer, more vivid and higher quality content.
[0066] Another approach: Determine the target style corresponding to the content to be optimized, and based on the target style, determine the target optimization guidance information corresponding to the target style.
[0067] In this embodiment of the disclosure, a target style can also be determined to guide the image or text to be optimized toward the target style, which can meet the user's personalized needs and provide more choices. The determination of the target style corresponding to the content to be optimized includes: in response to the operation for each preset style, determining the preset style corresponding to the operation as the target style to be optimized for the content to be processed; or, extracting the target style to be optimized for the content to be processed based on the content to be processed.
[0068] For example, in this embodiment of the disclosure, users can be provided with some selectable preset styles, such as a funny style, a warm style, a horror style, etc. When users input content to be optimized, they can select the target style to be optimized. For example, if the user selects the horror style, the image or text can be optimized to the horror style.
[0069] For example, to meet more diverse user needs, it is also possible to support users to input other target styles to be optimized. For instance, when inputting text to be optimized, users can simultaneously input words of the target style they wish to optimize. Similarly, when inputting an image to be optimized, users can simultaneously input words of the target style. Alternatively, the target style can be identified from the image to be optimized through image recognition. This disclosure does not impose any limitations on this aspect.
[0070] Once the target style is determined, the corresponding target optimization guidance information can be determined. Each target style can have corresponding target optimization guidance information set. For example, if the target style is horror, the target optimization guidance information for the horror style is: "Rewrite the following description, you can add some imagination, the content must be scary:".
[0071] 3) Input the target optimization guidance information and the first text description information into the text generation model, perform semantic analysis on the first text description information according to the target optimization guidance information, and generate the optimized target text description information corresponding to the first text description information.
[0072] In this process, the target optimization guidance information and the first description information can be combined and transformed in a certain way, and then input into the text generation model. For example, the combination and transformation method is target optimization guidance information + first text description information. The input text generation model would be: "Optimize the language of the following text, making it more vivid, and you can add imagination and emoticons appropriately: a cat."
[0073] In this embodiment of the disclosure, the text generation model can expand and optimize the relatively simple and basic first text description information according to the target optimization guidance information, thereby generating richer and higher-quality target text description information. The text generation model is, for example, a large language model (LLM), which can be fine-tuned based on internal data samples.
[0074] In another possible embodiment, based on a text generation model, the optimized target text description information corresponding to the first text description information of the first image is generated, including:
[0075] 1) When the content to be optimized is text to be optimized, the first image corresponding to the text to be optimized is generated based on the image generation model, taking the text to be optimized as input.
[0076] In this embodiment of the disclosure, it is also possible to support users to input only the text to be optimized, and to generate a first image of the text to be optimized based on the image generation model. The image generation model is used for text-to-image generation, such as a stable diffusion model, and can also be fine-tuned based on LoRA and internal data samples to make it more suitable for the requirements.
[0077] 2) Based on the image description model, the first image is used as input to obtain the first text description information of the first image.
[0078] 3) Define the target and optimize the guidance information.
[0079] 4) Input the target optimization guidance information and the first text description information into the text generation model. Based on the target optimization guidance information, perform semantic analysis on the first text description information to generate the optimized target text description information corresponding to the first text description information.
[0080] In this embodiment of the disclosure, when the input is text to be optimized, a first image is generated first, and then the first text description information of the first image is generated. The text is then optimized to generate the target text description information. In this way, by generating the image and text alternately, the accuracy can be improved and the possibility of the final optimized text and image deviating too much can be reduced.
[0081] Of course, the text to be optimized can also be optimized directly based on the text generation model, and the target image can be generated based on the optimized text description information. This embodiment does not impose any restrictions on this.
[0082] The following uses specific application scenarios for illustration. For ease of explanation, the application scenarios are described separately as follows:
[0083] Application Scenario 1: Taking the input image to be optimized as the input content, and free optimization without a specific target style as an example, see [link / reference]. Figure 2 The diagram shown illustrates the implementation effect of a content generation method in an embodiment of this disclosure. Figure 2 As shown, the user inputs an image to be optimized, such as a picture of a dish. In this embodiment of the disclosure, when the input is an image to be optimized, an image description needs to be generated and optimized first, and then the optimized image is generated based on the optimized image description.
[0084] Among them, see Figure 3 The diagram shown illustrates the principle of a text optimization process in the content generation method of this disclosure. For the image to be optimized, a first text description information of the image to be optimized is obtained based on an image description model, such as "a plate of stir-fried dishes". The first text description information is relatively simple. Therefore, based on the text generation model, the first text description information and the target optimization guidance information are used to expand and optimize the target text description information to generate more vivid and richer content.
[0085] For example, if the target optimization guidance information is: "Optimize the language of the following text to make it more vivid, and you can add appropriate imagination and emoticons:", then "Optimize the language of the following text to make it more vivid, and you can add appropriate imagination and emoticons: A plate of stir-fried vegetables" will be input into the text generation model. Based on the text generation model, optimized target text description information can be generated. For example, a number of image-text pairs can be pre-set to provide the user with the target text. Figure 2In this example, with a target number of 4, four different target text descriptions can be generated: 1. "A plate of meat and vegetables on the table, with cabbage, garlic slices, and green peppers, served in a small bowl, constitutes a beautiful and delicious dish." 2. "A plate of meat and vegetables on the table. The plate of roasted meat and vegetables is piled high on the table." 3. "A plate of meat and vegetables on the table seems like a container, holding our happiness and harvest. They are the foundation of our lives, allowing us to feel the abundance of food and the beauty of life. Happiness at the table, a dish that lets you feel the beauty of life, full of harvest and enjoyment!" 4. "Let the deliciousness of food be presented in a more relaxed and fun way! The deliciousness of meat and vegetables spread out on the table looks more tempting. Food should be enjoyed in a relaxed and pleasant atmosphere, so that everyone can enjoy the food together, enhance relationships, and add to the happiness!"
[0086] Furthermore, the optimized target text description information can be input into the image generation model, which can generate corresponding optimized target images for each target text description information. For example, Figure 2 As shown, the generated target image has more diverse and richer content compared to the previous image to be optimized.
[0087] It should be noted that the target text description information and target image obtained in this embodiment are already optimized and more vivid and rich in content, and can be directly returned to the user. Furthermore, in order to obtain text description information with a higher degree of relevance to the target image, in this embodiment, the generated target image can be used as input again to repeat the above process. Figure 3 The text optimization process involves first generating simple second text description information based on the image description model, and then optimizing the second text description information of the target image based on the text generation model and the target optimization guidance information to generate the final optimized text description information, which is then returned to the user.
[0088] Thus, in this embodiment of the disclosure, by inputting an image to be optimized, an optimized target image and target text description information can be generated. This allows for free optimization to obtain content of various styles, improving content creation efficiency and enhancing user experience.
[0089] Application Scenario 2: Taking the input content to be optimized as the image to be optimized, and guiding optimization according to a specific target style as an example, see [link / reference]. Figure 4 The diagram shown illustrates the implementation effect of another content generation method in this embodiment of the present disclosure. Figure 4As shown, the user inputs an image to be optimized, such as a picture of a cat. In this embodiment, several different preset styles are provided for the user to choose from, such as a funny style, a heartwarming style, a horror style, and a magical world style. Then, based on the selected target style, the image to be optimized is optimized to generate a target image and target text description information that conform to the target style.
[0090] Among them, see Figure 5 The diagram illustrates another text optimization process in the content generation method of this disclosure. The image to be optimized is input into an image description model to obtain the first text description information of the image, such as "a cat." Then, based on the text generation model, according to the first text description information, target optimization guidance information, and target style, such as... Figure 5 The medium gray squares represent the target style selected by the user. By expanding and optimizing the style, more vivid and richer target text description information is generated.
[0091] In this embodiment of the disclosure, corresponding target optimization guidance information can be set for each style. For example, the horror style is: "Rewrite the following description, but add some imagination to make the content more terrifying:", and the horror-warmth style is: "Rewrite the following description, but add some imagination to make the content more warm:", etc. If the user selects the horror style as the target style, then "Rewrite the following description, but add some imagination to make the content more terrifying: a cat" can be input into the text generation model, and then based on the text generation model, optimized target text description information that conforms to the target style can be generated.
[0092] like Figure 4 As shown, if a humorous style is selected, the generated target text description might be: "A cat looking at a camera suddenly screamed and ran back to its bed to hide because it discovered a mouse in the photo playing television." If a heartwarming style is selected, the generated target text description might be: "A cat looking at a camera seems to see a beautiful world in its mind. Its eyes reveal excitement and a touch of tenderness. It's like witnessing an infinitely beautiful adventure, so it begins to enjoy this moment." If a horror style is selected, the generated target text description might be: "A cat looking at a camera, a mysterious light flashes in its eyes, sending chills down one's spine." If a magical world style is selected, the generated target text description might be: "A cat stares at the camera, stunned, then suddenly transforms into a lively snake, its tail swaying rhythmically, extremely happy. It has a daring idea, so it opens the camera and turns back time. The cat and the snake's bodies can actually intertwine perfectly, they hear a sound, and all their memories are brought back to the past."
[0093] Furthermore, the optimized target text description information can be input into the image generation model, which can then generate corresponding optimized target images for each target text description information. For example, ... Figure 4 As shown, based on the target text description information of different target styles, target images that conform to the target style are generated accordingly, and the image content is more vivid and diverse.
[0094] Thus, in this embodiment of the disclosure, text and image content can be generated according to a specific target style, which can guide the realization of personalized optimization needs and improve the user experience.
[0095] Application Scenario 3: Taking the input content to be optimized as the text to be optimized as an example, see [link / reference]. Figure 6 The diagram shown illustrates the implementation effect of another content generation method in this embodiment of the present disclosure. Figure 6 As shown, the user inputs text to be optimized, for example, the text to be optimized is "my cat". In this embodiment of the disclosure, for the text to be optimized, a first image of the text to be optimized is first generated based on the image generation model. Then, optimized target text description information and target image are generated for the first image. The specific processing operation for the first image is the same as the processing operation for the image to be optimized in the above embodiment, so it will not be described again here.
[0096] like Figure 6 As shown, the text to be optimized is "My cat". Optimization can be performed using different modes, such as free optimization or optimization for a specific target style. Figure 6 Given the input "my cat", a first image is generated. Then, based on the image description model, the first text description information of the first image is obtained. Based on the text generation model, along with target optimization guidance information and the first text description information, or based on the text generation model, along with target optimization guidance information, target style, and the first text description information, an image is generated as follows: Figure 6 The four target text descriptions are as follows: 1. "A cute cat is looking at the camera with focused eyes, instantly winning people's hearts. Its eyes are clear and lively, as if it is gazing into the photographer's mind." 2. "A cat is looking at the camera lens. It sneaks around in the corner every day, trying to see if we've noticed it's thinking." 3. "A cat is fiddling with a camera, but it looks like it's in a good mood! Move the camera away and let the cat have some fun!" 4. "A cat is sitting under a tree, quietly squinting its eyes. It peeks out, looking at the azure sky, as if trying to catch something." Based on the image generation model, the corresponding target image is generated for each target text description.
[0097] Thus, in this embodiment of the disclosure, by connecting the image generation model and the text generation model, image and text optimization can also be performed on the text to be optimized, obtaining optimized target text description information and target image, meeting users' needs for creating various types of content and improving efficiency.
[0098] The following describes another content generation method provided in this disclosure, using a terminal device as the execution subject as an example. See also Figure 7 The diagram shown is a flowchart of another content generation method provided in this disclosure embodiment, the method including:
[0099] S701: Receive input content to be optimized, wherein the content to be optimized includes text to be optimized or image to be optimized.
[0100] S702: Obtain the optimized target image and target text description information corresponding to the content to be optimized. The target text description information is generated based on the text generation model and the first text description information of the first image. The first image is the image to be optimized, or the first image is generated based on the text to be optimized, and the target image is generated based on the image generation model and the target text description information.
[0101] In this embodiment, the method for generating the optimized target image and target text description information of the content to be optimized is the same as the implementation method in the above embodiments, and will not be repeated here.
[0102] For example, if the input text to be optimized is "I have a robot vacuum cleaner", it can be freely optimized to generate various target text descriptions without a specific target style. For example, the generated target text description could be: "We can imagine that using this robot to clean the house will not only make it clean but also make the house more comfortable and cozy." In addition, in this embodiment, a target style instruction can be added to guide optimization in the direction of that target style. For example, if the target style is a story narrative style, the generated target text description could be: "When the robot vacuum cleaner was sweeping, a bird suddenly bumped into it, scaring it so much that it gave up its cleaning work. So, the bird lay on the ground and sighed, 'You robot vacuum cleaner, you never know that I also have a need to sweep the floor, that's too much!'"
[0103] Furthermore, in this embodiment of the present disclosure, a target image with corresponding optimized target text description information can also be generated, and the terminal device can obtain the optimized target image and the target text description information for the target image.
[0104] S703: Displays optimized target text description information and target image.
[0105] In this embodiment of the disclosure, the input content to be optimized is received, and the optimized target image and target text description information corresponding to the content to be optimized can be obtained. Then, the target text description information and target image are displayed. In this way, the optimized target image and target text description information can be obtained quickly for the content to be optimized, which improves the efficiency of content creation. Moreover, through optimization, more diversified, richer and higher quality content can be obtained, thereby improving the quality of content creation.
[0106] This disclosure further provides a content generation method. The method includes: acquiring input content to be optimized, wherein the content to be optimized includes text to be optimized or an image to be optimized; generating optimized target text description information corresponding to first text description information of a first image based on a text generation model, and generating an optimized target image based on an image generation model and the target text description information, wherein the first image is the image to be optimized, or the first image is generated based on the text to be optimized; and returning the optimized target text description information and the target image.
[0107] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.
[0108] Based on the same inventive concept, this disclosure also provides a content generation device corresponding to the content generation method. Since the principle of the device in this disclosure for solving the problem is similar to the content generation method described above in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.
[0109] Reference Figure 8 The diagram shows a content generation apparatus provided in an embodiment of this disclosure. The apparatus includes: an acquisition module 81, used to acquire input content to be processed; a determination module 82, used to determine the optimized target text description information corresponding to the content to be processed, and to determine the corresponding key text description information based on the target text description information, wherein the target text description information is generated based on a text generation model and an image generation model in an alternating collaborative manner; and a generation module 83, used to generate a first type of first image-text result or a second type of second image-text result based on the key text description information and the target text description information, wherein the different types of image-text results represent different levels of importance of image genre and text genre, and the target image of the image genre in the different types of image-text results is generated based on the key text description information or the target text description information.
[0110] In this embodiment of the disclosure, the content to be processed includes text to be processed or video to be processed. The determining module includes: a text description information generation module, used to generate optimized target text description information corresponding to the first text description information of the first image based on a text generation model, wherein the first image is generated based on an image generation model and the text to be processed, or the first image is a target frame image included in the video to be processed; and an information determining module, used to determine the corresponding key text description information based on the target text description information.
[0111] In this embodiment of the disclosure, the text description information generation module includes: an image determination module, used to determine a target frame image based on keyframes of the video to be processed, or to determine a corresponding target frame image based on a preset frame interval, when the content to be processed is a video to be processed; an acquisition module, used to obtain first text description information of the target frame image based on an image description model and taking the target frame image as input; a guidance information determination module, used to determine target optimization guidance information; and an analysis module, used to input the target optimization guidance information and the first text description information into a text generation model, perform semantic analysis on the first text description information based on the target optimization guidance information, and generate optimized target text description information corresponding to the first text description information.
[0112] In this embodiment of the disclosure, the text description information generation module includes: an image generation module, used to generate a first image corresponding to the text to be processed based on an image generation model, taking the text to be processed as input, when the content to be processed is text to be processed; an acquisition module, used to obtain first text description information of the first image based on an image description model, taking the first image as input; a guidance information determination module, used to determine target optimization guidance information; and an analysis module, used to input the target optimization guidance information and the first text description information into the text generation model, perform semantic analysis on the first text description information according to the target optimization guidance information, and generate optimized target text description information corresponding to the first text description information.
[0113] In this embodiment of the disclosure, the guidance information determination module includes: a statement information determination module, used to determine that the target optimization guidance information is a preset optimization guidance statement information; or, a style-based determination module, used to determine the target style to be optimized corresponding to the content to be processed, and to determine the target optimization guidance information corresponding to the target style based on the target style.
[0114] In this embodiment of the disclosure, a style determination module is further included, comprising: a selection module, configured to, in response to a selection operation for each preset style, determine the preset style corresponding to the selection operation as the target style to be optimized for the content to be processed; or, an extraction module, configured to extract the target style to be optimized for the content to be processed based on the content to be processed.
[0115] In this embodiment of the disclosure, the generation module includes: a first optimization module, used to perform semantic analysis on key text description information based on an image generation model and key text description information to generate an optimized target image; and a first image and text result acquisition module, used to obtain a first type of first image and text result based on the target text description information and the target image.
[0116] In this embodiment of the disclosure, the generation module includes: a segmentation module that obtains multiple sub-target text description information after segmentation based on target text description information; a second optimization module that performs semantic analysis on the multiple sub-target text description information based on an image generation model to generate multiple optimized target images; and a second image-text result acquisition module that obtains a second type of second image-text result based on key text description information and multiple target images.
[0117] In this embodiment of the disclosure, the first optimization module includes: a first analysis module, used to perform semantic analysis on the key text description information and target style based on an image generation model, taking key text description information and target style as input, and generate an optimized target image.
[0118] In this embodiment of the disclosure, the second optimization module includes: a second analysis module, used to perform semantic analysis on the multiple sub-target text description information and target style based on the image generation model, taking multiple sub-target text description information and target style as inputs respectively, and generate multiple optimized target images accordingly.
[0119] Reference Figure 9 The diagram shows another content generation apparatus provided in this embodiment of the present disclosure. The apparatus includes: an acquisition module 91, configured to acquire input content to be optimized, wherein the content to be optimized includes text to be optimized or an image to be optimized; a generation module 92, configured to generate optimized target text description information corresponding to the first text description information of the first image based on a text generation model, and generate an optimized target image based on an image generation model and the target text description information, wherein the first image is the image to be optimized, or the first image is generated based on the text to be optimized; and a return module 93, configured to return the optimized target text description information and the target image.
[0120] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.
[0121] This disclosure also provides an electronic device, such as... Figure 10 The diagram shown is a schematic representation of the structure of an electronic device provided in an embodiment of this disclosure, including:
[0122] A processor 101 and a memory 102; the memory 102 stores machine-readable instructions executable by the processor 101, and the processor 101 executes the machine-readable instructions stored in the memory 102. When the machine-readable instructions are executed by the processor 101, the processor 101 performs the following steps:
[0123] Obtain the input content to be optimized, wherein the content to be optimized includes text to be optimized or image to be optimized;
[0124] Based on the text generation model, the first text description information of the first image is generated, corresponding to the optimized target text description information. Based on the image generation model and the target text description information, the optimized target image is generated. The first image is the image to be optimized, or the first image is generated based on the text to be optimized.
[0125] Return the optimized target text description information and the target image.
[0126] Alternatively, processor 101 may perform the following steps:
[0127] Receive input content to be optimized, wherein the content to be optimized includes text to be optimized or image to be optimized;
[0128] Obtain the optimized target image and target text description information corresponding to the content to be optimized, wherein the target text description information is generated based on the text generation model and the first text description information of the first image, the first image is the image to be optimized, or the first image is generated based on the text to be optimized, and the target image is generated based on the image generation model and the target text description information;
[0129] The optimized target text description information and the target image are displayed.
[0130] The aforementioned memory 102 includes a main memory 1021 and an external memory 1022; the main memory 1021, also known as internal memory, is used to temporarily store the computational data in the processor 101, as well as the data exchanged with external memory 1022 such as a hard disk. The processor 101 exchanges data with the external memory 1022 through the main memory 1021.
[0131] The specific execution process of the above instructions can be referred to the steps of the content generation method described in the embodiments of this disclosure, and will not be repeated here.
[0132] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the content generation method described in the above method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.
[0133] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the content generation method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.
[0134] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.
[0135] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.
[0136] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0137] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0138] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0139] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.
Claims
1. A content generation method, characterized in that, include: Obtain the input content to be processed; The optimized target text description information corresponding to the content to be processed is determined, and the corresponding key text description information is determined based on the target text description information. The target text description information is generated based on the text generation model and the image generation model in an alternating collaborative manner. Based on the key text description information and the target text description information, an image-text result is generated. The image-text result includes: a first type of first image-text result and a second type of second image-text result. The different types of image-text results represent different levels of importance for image and text genres. Furthermore, the target image for the image genre in each type of image-text result is generated based on either the key text description information or the target text description information. The generation of the second type of second image-text result based on the key text description information and the target text description information includes: Based on the target text description information, obtain multiple sub-target text description information after segmentation; Based on the image generation model, semantic analysis is performed on the multiple sub-target text description information according to the multiple sub-target text description information to generate the corresponding optimized multiple target images; Based on the key text description information and multiple target images, a second type of second image-text result is obtained.
2. The method according to claim 1, characterized in that, The content to be processed includes text or video to be processed, and determining the optimized target text description information corresponding to the content to be processed includes: Based on the text generation model, the first text description information of the first image is generated to correspond to the optimized target text description information, wherein the first image is generated based on the image generation model and the text to be processed, or the first image is a target frame image included in the video to be processed; Based on the target text description information, determine the corresponding key text description information.
3. The method according to claim 2, characterized in that, The step of generating optimized target text description information corresponding to the first text description information of the first image based on the text generation model includes: When the content to be processed is the video to be processed, the target frame image is determined based on the keyframes of the video to be processed, or the corresponding target frame image is determined based on the preset frame interval. Based on the image description model, the first text description information of the target frame image is obtained by taking the target frame image as input; Define objectives and optimize guiding information; The target optimization guidance information and the first text description information are input into the text generation model. Based on the target optimization guidance information, semantic analysis is performed on the first text description information to generate optimized target text description information corresponding to the first text description information.
4. The method according to claim 2, characterized in that, The step of generating optimized target text description information corresponding to the first text description information of the first image based on the text generation model includes: When the content to be processed is the text to be processed, a first image corresponding to the text to be processed is generated based on the image generation model, using the text to be processed as input. Based on the image description model, the first text description information of the first image is obtained by taking the first image as input; Define objectives and optimize guiding information; The target optimization guidance information and the first text description information are input into the text generation model. Based on the target optimization guidance information, semantic analysis is performed on the first text description information to generate optimized target text description information corresponding to the first text description information.
5. The method according to claim 3 or 4, characterized in that, The determined target optimization guidance information includes: The target optimization guidance information is determined to be the preset optimization guidance statement information; or, Determine the target style to be optimized corresponding to the content to be processed, and determine the target optimization guidance information corresponding to the target style based on the target style.
6. The method according to claim 5, characterized in that, Determining the target style to be optimized for the content to be processed includes: In response to the selection operation for each preset style, the preset style corresponding to the selection operation is determined as the target style to be optimized for the content to be processed; or, Based on the content to be processed, extract the target style that needs to be optimized for the content to be processed.
7. The method according to claim 5, characterized in that, The step of generating a first type of first image-text result based on the key text description information and the target text description information includes: Based on the image generation model, semantic analysis is performed on the key text description information according to the key text description information to generate the optimized target image; Based on the target text description information and the target image, a first type of first image and text result is obtained.
8. The method according to claim 5, characterized in that, The image generation model, based on the image generation model, performs semantic analysis on the multiple sub-target text descriptions according to the multiple sub-target text descriptions, and generates corresponding optimized target images, including: Based on the image generation model, semantic analysis is performed on the multiple sub-target text descriptions and target styles as inputs to generate multiple optimized target images.
9. The method according to claim 7, characterized in that, The step of generating the optimized target image based on the image generation model, by performing semantic analysis on the key text description information according to the key text description information, includes: Based on the image generation model, the key text description information and the target style are used as inputs to perform semantic analysis on the key text description information and the target style, and an optimized target image is generated.
10. A content generation apparatus, characterized in that, include: The acquisition module is used to acquire the input content to be processed; The determination module is used to determine the optimized target text description information corresponding to the content to be processed, and to determine the corresponding key text description information based on the target text description information, wherein the target text description information is generated based on the text generation model and the image generation model in an alternating collaborative manner; The generation module is configured to generate image-text results based on the key text description information and the target text description information. The image-text results include: a first type of first image-text result and a second type of second image-text result. The different types of image-text results represent different levels of importance for image and text genres, and the target image for the image genre in each type of image-text result is generated based on either the key text description information or the target text description information. The generation module is further configured to: Based on the target text description information, obtain multiple sub-target text description information after segmentation; Based on the image generation model, semantic analysis is performed on the multiple sub-target text description information according to the multiple sub-target text description information to generate the corresponding optimized multiple target images; Based on the key text description information and multiple target images, a second type of second image-text result is obtained.
11. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, the processor executing the machine-readable instructions stored in the memory, wherein when the machine-readable instructions are executed by the processor, the processor performs the steps of the method as described in any one of claims 1 to 9.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-9.