Content generation method, electronic equipment, computer readable storage medium and product

Abstract

The invention relates to a content generation method, electronic equipment, a computer readable storage medium and a product, and relates to the technical field of computers. The content generation method comprises the following steps: determining an entity and a background in a first image from a first user; understanding the first image, and determining a first story plot expressed by the first image; according to an entity and a background in the first image, the first story plot is expanded to generate a second story plot; generating one or more second images according to the second story plot; and generating the first multimedia content according to the first image and the one or more second images. The interaction participation enthusiasm of the user can be improved, and the application utilization rate is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a content generation method, an electronic device, a computer-readable storage medium, and a product. Background Art

[0002] With the development of artificial intelligence technology, users can use the functions provided by various applications to send instructions for content generation. Then, the computer automatically generates various types of content such as text, images, and videos based on the instructions. This process can be implemented using machine learning models, such as large language models (LLM), foundation models, and other neural network models, to use the powerful computing power of the model to meet various user needs. Summary of the invention

[0003] According to some embodiments of the present disclosure, a content generation method is provided, including: determining entities and background in a first image from a first user; understanding the first image to determine a first storyline expressed by the first image; expanding the first storyline based on the entities and background in the first image to generate a second storyline; generating one or more second images based on the second storyline; and generating first multimedia content based on the first image and the one or more second images.

[0004] According to some embodiments of the present disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute a method of any embodiment described in the present disclosure based on instructions stored in the memory.

[0005] According to some embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method of any embodiment described in the present disclosure is performed.

[0006] According to some embodiments of the present disclosure, a computer program product is provided. When the computer program product is executed on a computer, the computer is enabled to execute the method of any one of the embodiments described in the present disclosure.

[0007] Other features, aspects and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The following describes embodiments of the present disclosure with reference to the accompanying drawings. It should be understood that the accompanying drawings described below only relate to some embodiments of the present disclosure and do not constitute a limitation to the present disclosure.

[0009] Figure 1A schematic flow chart of a content generation method according to some embodiments of the present disclosure is shown.

[0010] Figure 2 A schematic diagram of a process for generating a second storyline according to some embodiments of the present disclosure is shown.

[0011] Figure 3 A schematic diagram of a process for generating a second image according to some embodiments of the present disclosure is shown.

[0012] Figure 4 A schematic diagram of a process for generating second multimedia content according to some embodiments of the present disclosure is shown.

[0013] Figure 5 A schematic diagram of a shooting interface according to some embodiments of the present disclosure is shown.

[0014] Figure 6 A schematic diagram of first multimedia content according to some embodiments of the present disclosure is shown.

[0015] Figure 7 A schematic diagram of a third image according to some embodiments of the present disclosure is shown.

[0016] Figure 8 A schematic diagram of second multimedia content according to some embodiments of the present disclosure is shown.

[0017] Fig. 9 A schematic structural diagram of a content generating device according to some embodiments of the present disclosure is shown.

[0018] Fig.10 A block diagram of an electronic device according to some embodiments of the present disclosure is shown.

[0019] Fig.11 A block diagram of an electronic device according to some other embodiments of the present disclosure is shown.

[0020] It should be understood that, for the convenience of description, the sizes of various parts shown in the drawings are not necessarily drawn according to the actual proportional relationship. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present disclosure. It should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein.

[0022] It should be understood that the various steps recorded in the method embodiments of the present disclosure can be performed in different orders, and / or performed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown in the execution. The scope of the present disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments should be interpreted as being merely exemplary and do not limit the scope of the present disclosure.

[0023] The term “including” and its variations used in the present disclosure are intended to be open terms that include at least the following elements / features but do not exclude other elements / features, that is, “including but not limited to.” The term “based on” means “at least partly based on.”

[0024] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. Unless otherwise specified, the concepts of "first", "second", etc. are not intended to imply that the objects described in this way must be in a given order in time, space, ranking, or any other manner.

[0025] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0026] The embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings, but the present disclosure is not limited to these specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. In addition, in one or more embodiments, specific features, structures or characteristics can be combined in any suitable manner that will be clear from the present disclosure by a person of ordinary skill in the art.

[0027] Some applications provide users with various interaction methods based on content generation technology. For example, images are generated based on text provided by users, new images are generated based on images provided by users, or videos are generated based on images provided by users. These interaction methods meet users' content generation needs to a certain extent.

[0028] However, these current interactive methods require heavy user participation and are not very intelligent. For example, when generating a new image based on an image provided by a user, the user needs to indicate how to generate the new image. In addition, the content generated by these interactive methods is not rich and interesting. This neither fully utilizes the role of content generation technology nor is conducive to the enthusiasm of users to participate in the interaction.

[0029] Based on this, the present disclosure provides a content generation method. The present disclosure can automatically generate multimedia content with a storyline based on images provided by users. That is, the storyline is automatically generated based on the images provided by users, and the multimedia content is generated based on the storyline. In the process of generating multimedia content, the intelligence of the content generation technology is fully utilized, and the storyline and multimedia content can be automatically generated without the user providing too much information. This not only improves the richness of the generated content, but also improves the user's interactive experience, thereby increasing the user's enthusiasm for participating in the interaction and improving the usage rate of the application.

[0030] Figure 1 FIG. 1 is a flow chart showing a method for generating content according to some embodiments of the present disclosure. Figure 1 As shown, the content generation method includes steps S11 to S19.

[0031] In step S11 , an entity and a background in a first image from a first user are determined.

[0032] A first image of a first user is obtained, and content is generated based on the first image. The first image may be an image of various styles and types, and may be a color image or a grayscale image.

[0033] The entity in the first image refers to the main object in the first image, which is used to reflect the theme or focus of the first image. The entity can be any visual object, including people, objects, etc. The background in the first image refers to the surrounding environment or background of the entity, which is used to reflect the atmosphere of the first image, making the image content richer and more layered.

[0034] The entity in the first image can be determined using a target recognition algorithm. Alternatively, the foreground and background in the first image can be divided using image processing technology, and the entity can be extracted from the foreground. After the entity in the first image is determined, other objects in the first image except the entity are determined as the background.

[0035] By determining the entities and background in the first image, the objects in the first image and the relationships between the objects can be determined. Generating content based on the objects in the first image and the relationships between the objects can improve the relevance of the generated content to the first image, that is, generating content around the first image, thereby improving the accuracy of the generated content.

[0036] In step S13, the first image is understood to determine a first storyline expressed by the first image.

[0037] Understanding the first image includes performing target detection and recognition, scene understanding and semantic analysis on the first image to determine the first storyline expressed by the first image. For example, when the first image shows a girl standing on the roadside with a bus stop sign on the roadside, the first storyline expressed by the first image is that the girl is waiting for the bus.

[0038] The first image is understood and the first storyline expressed by the first image is determined, so that content can be generated according to the storyline expressed by the first image, that is, content is generated around the first storyline, which can improve the accuracy of the generated content.

[0039] In step S15, the first storyline is expanded according to the entity and the background in the first image to generate a second storyline.

[0040] After determining the entities and background included in the first image, and the first storyline expressed by the first image, the first storyline is expanded to generate a second storyline. The second storyline is, for example, generated by continuing, adapting, or supplementing the first storyline, and this process is based on the entities and background in the first image. For example, the first image shows a girl standing on the side of the road, and there is a sign of a bus stop on the side of the road. The subject of the first image is the girl, and the background is the sign of the bus stop. The first storyline is that the girl is waiting for the bus. Based on the girl and the sign of the bus stop, the generated second storyline can be that the girl is sitting on a seat on the bus and listening to music.

[0041] Based on the entities and background in the first image, the first storyline is extended so that the characters or objects appearing in the generated second storyline are associated with the entities and background in the first image, thereby enabling the second storyline to be a reasonable extension of the first storyline.

[0042] In step S17, one or more second images are generated according to the second storyline.

[0043] Before generating the second image, information such as the number and style of the second image may be determined first.

[0044] In some embodiments, the number of second images generated may be determined according to the richness of the second storyline. The richer the second storyline, the more second images generated; the less rich the second storyline, the fewer second images generated.

[0045] In some embodiments, the style of the generated second image can be determined according to the story style of the second storyline. For example, when the story style of the second storyline is warm and touching, the style of the second image can also be warm and touching by setting the color, light and shadow, composition, etc. of the second image.

[0046] Since the second storyline is derived from the expansion of the first storyline, the one or more second images generated according to the second storyline are associated with the first image. Thus, according to the first image provided by the first user, one or more second images associated with the first image are generated, and the one or more second images correspond to the second storyline, so one or more second images with a storyline are generated for the first image of the first user.

[0047] In step S19, first multimedia content is generated according to the first image and one or more second images.

[0048] After one or more second images with storylines are generated for the first image, first multimedia content can be generated by combining the first image and the one or more second images.

[0049] When the second storyline is a continuation of the first storyline, for example, the first image and one or more second images are sequentially displayed to generate the first multimedia content, so that the first multimedia content corresponds to the first storyline and the second storyline representing the subsequent development of the first storyline.

[0050] The second storyline is an expansion, adaptation, supplement, etc. of the first storyline. For example, the first image and one or more second images are cross-displayed to generate the first multimedia content, that is, the storyline corresponding to the first multimedia content is determined according to the relationship between the first storyline and the second storyline.

[0051] In addition to the first image and one or more second images, the first multimedia content may also have other forms of content besides images, such as text, audio, etc. The text and audio may be determined based on the second storyline, so that the various forms of content in the first multimedia content are unified.

[0052] The first multimedia content is generated based on the first image of the first user to generate the second storyline, and then generated according to the second storyline. Therefore, the first multimedia content is multimedia content with a corresponding storyline, and the first multimedia content is richer in content and more interesting.

[0053] In the above embodiment, based on the first image of the first user, the first multimedia content with the corresponding storyline can be automatically generated. This content generation method can automatically generate a storyline based on an image, and then automatically generate multimedia content based on the storyline. Therefore, this content generation method is more intelligent, and the generated content is richer, which helps to improve the enthusiasm of user interaction, thereby improving the usage rate of the application.

[0054] Figure 2 FIG. 4 is a flow chart showing a process of generating a second storyline according to some embodiments of the present disclosure. Figure 2 As shown, generating the second storyline includes steps S21 to S23.

[0055] In step S21, extended information is determined based on the entity and background in the first image and the first storyline, wherein the extended information is used to represent at least one of change information of the entity and background in the first image and association information of the entity and background in the first image.

[0056] The extended information is used to generate the second storyline. That is, in addition to generating the second storyline, starting from the first image, the extended information is determined by analyzing the entity and background in the first image and the first storyline, and then the second storyline is generated based on the extended information.

[0057] The change information of the entity and background in the first image is determined by changing the state, action, environment, etc. of the entity and background in the first image. For example, when the entity in the first image includes a person, the change information of the person may be the change information of the person's action or expression.

[0058] The change information of the entity and background in the first image can be determined based on the entity or the background itself, or can be determined based on the relationship between the entity and the background. For example, when the entity in the first image includes a person, the change information of the person is the change information of the action. That is, the change information of the entity can be determined based on the entity itself. When the entity in the first image includes a person, the background includes the environment, and the environment is raining, the change information of the expression of the person is determined to be sad. When the entity in the first image includes a person, the background includes the environment, and the environment is a clear sky, the change information of the expression of the person is determined to be happy. That is, the change information of the entity can be determined based on the entity and the background. Similarly, the change information of the background can be determined based on the background itself, or it can be determined based on the entity and the background.

[0059] The association information of the entity and the background in the first image is determined by associating the entity and the background in the first image. For example, when the entity in the first image includes potato chips, the association information of potato chips may be convenience stores, because convenience stores often sell potato chips.

[0060] The association information of the entity and the background in the first image can also be determined based on the entity or the background itself or based on the relationship between the entity and the background. For example, the entity in the first image includes potato chips, and the association information of potato chips is convenience store. That is, the association information of the entity can be determined based on the entity itself. When the entity in the first image includes potato chips and the background includes a computer playing a video, the association information of potato chips is juice. That is, the association of the entity can be determined based on the entity and the background. Similarly, the association information of the background can be determined based on the background itself or based on the entity and the background.

[0061] The extended information includes at least one of the change information of the entity and background in the first image and the association information of the entity and background in the first image. After the extended information is determined, the direction and elements of the extension of the first storyline are determined, so that the second storyline can be generated.

[0062] In step S23, the first storyline is expanded using the extended information to generate a second storyline.

[0063] After the extended information is determined, the second storyline may be generated by integrating the change information of the entity and the background in the first image and the association information of the entity and the background in the first image included in the extended information.

[0064] Since emotional tone is an important component of the storyline, in some embodiments, the emotion contained in the first image may be determined first, and then the second storyline may be generated in combination with the emotion contained in the first image. For example, according to the entities and background in the first image, the first storyline is expanded to generate the second storyline, which includes: determining the emotion contained in the first image according to the entities and background in the first image; and expanding the first storyline according to the emotion contained in the first image, the entities and background in the first image, to generate the second storyline.

[0065] The emotion contained in the first image affects the expansion result of the first storyline. For example, when the emotion contained in the first image is sad, the emotional tone of the generated second storyline is also sad. In this way, the first multimedia content generated based on the second storyline matches the first image, which can increase the probability of user satisfaction with the generated content.

[0066] The emotion contained in the first image may be determined based on the entity and background in the first image. For example, when the entity in the first image includes a person, the emotion of the person may be determined based on the person's actions, expressions, etc. At the same time, the emotion contained in the first image is determined by combining the person's emotion with the environment, atmosphere, etc. of the background in the first image.

[0067] When expanding the first storyline based on the emotions contained in the first image, it can be done in the following manner: determine the change information of the entities and background in the first image based on the emotions; expand the first storyline based on the emotions, the change information of the entities and background in the first image to generate a second storyline.

[0068] That is, according to the emotion contained in the first image, the change information of the entity and the background in the first image is determined. For example, when the entity in the first image includes a person and the emotion contained in the first image is sadness, the change information of the person may be the change information of the expression of the person, such as the expression of the person changes to tears. Meanwhile, the change information of the background may be the change information of the atmosphere, such as the color tone of the background becomes dark.

[0069] In some embodiments, determining change information of entities and background in the first image based on emotions includes: in response to the entities in the first image including characters, determining change information of at least one of the character's actions and expressions based on the emotions; or in response to the entities in the first image including objects, determining change information of the object's shape based on the emotions; or determining change information of at least one of the environment and atmosphere of the background in the first image based on the emotions.

[0070] A character can be a person or an anthropomorphic object. The form of an object can be the appearance or display method of the object. The environment and atmosphere can be controlled by setting the objects in the background and the color of the background.

[0071] In addition to determining the change information of the entity and the background in the first image according to the emotion contained in the first image, the association information of the entity and the background in the first image can also be determined according to the emotion contained in the first image. That is, according to the emotion, the association information of the entity and the background in the first image is determined; according to the emotion, the association information of the entity and the background in the first image is expanded to generate the second storyline.

[0072] That is, the extended information is determined based on the emotion contained in the first image, and then the first storyline is extended based on the extended information to generate the second storyline. Determining the extended information based on the emotion can improve the consistency of the change information of the entity and background of the first image and the association information of the entity and background of the first image in the extended information, and reduce the risk of confusion in the content of the extended information caused by inconsistent change directions of different entities and backgrounds in the first image and inconsistent association directions of different entities and backgrounds in the first image.

[0073] After the second storyline is generated, one or more second images are generated according to the second storyline. Since the second storyline is often embodied or occurs in one or more story scenes, the second images can be generated based on the story scenes of the second storyline. Figure 3 FIG. 4 is a flow chart showing a process of generating a second image according to some embodiments of the present disclosure. Figure 3 As shown, generating the second image includes steps S31 to S33.

[0074] In step S31, one or more story scenes and a story plot corresponding to each of the one or more story scenes are determined according to the second story plot.

[0075] When determining one or more story scenes and the story plot corresponding to each of the one or more story scenes according to the second story plot, the second story plot may be determined after being sorted out and summarized. In this way, the key story plot in the second story plot is determined through one or more story scenes, and the key story plot is the story plot corresponding to the story scene. The second image is generated according to the key story plot, which can improve the completeness and accuracy of the generated second image for reflecting the second story plot.

[0076] In addition, when generating multiple second images, the style of the second images can be unified. The unified style here means that the style is the same, or the transition of the style is smooth. The style can usually be reflected by emotions. Since the second image is generated based on the first image, the style of the second image can be determined according to the emotions contained in the first image. Alternatively, since the second image corresponds to the second storyline, the style of the second image can also be determined according to the emotions contained in the second storyline. That is, according to the emotions contained in at least one of the second storyline and the first image, the style of one or more second images is determined; according to the style of one or more second images, one or more second images are generated. The style of the second image can be reflected, for example, by setting the filter, composition, color, light and shadow of the second image.

[0077] In step S33, for each story scene corresponding to the story plot, a second image corresponding to the story scene is generated.

[0078] For each story scene, a second image corresponding to the story scene is generated. The second image generated in this way can accurately reflect the second story plot.

[0079] After generating one or more second images that can accurately reflect the second storyline and have a reasonable style, the first multimedia content can be generated by combining with the first image. In addition to the first image and one or more second images, the first multimedia content can also include content in the form of text, audio, etc. For example, subtitles can be generated for each image in the first multimedia content. This can not only enrich the first multimedia content, but also enhance the expressiveness of the first multimedia content with the corresponding storyline and enhance its fun.

[0080] In some embodiments, generating first multimedia content based on a first image and one or more second images includes: determining subtitles for each of the first image and the one or more second images, wherein the subtitles for each image are generated based on a storyline corresponding to the image; generating first multimedia content based on the first image and the one or more second images, wherein each image in the first multimedia content has subtitles.

[0081] The first multimedia content may be presented in the form of an image or a video. That is, the first multimedia content is an image, wherein the first multimedia content includes multiple images that can be switched for display, or an image formed by splicing multiple images; or the first multimedia content is a video.

[0082] The multiple images that can be switched for display are equivalent to a collection of images. The switching display of the multiple images can respond to user triggers or conditional triggers, such as switching to the next image after the current image is displayed for a specified period of time. Stitching multiple images means cropping or scaling the multiple images to form a single image.

[0083] In some embodiments, the first multimedia content includes multiple images that can be switched for display, and an image in the multiple images can be an image formed by splicing multiple images. In other words, the first multimedia content can include multiple images that are formed by splicing multiple images and can be switched for display.

[0084] Whether the first multimedia content is presented in the form of an image or a video can be determined based on the emotional intensity of the second plot. That is, in response to the emotional intensity of the second plot being greater than the first specified threshold, the first multimedia content is a video; in response to the emotional intensity of the second plot being not greater than the first specified threshold, the first multimedia content is an image. For example, when the second plot is similar to a micro-movie, the emotional intensity is high, and it is more appropriate to present it in the form of a video. When the second plot is similar to a travelogue, the emotional intensity is low, and it is more appropriate to present it in the form of an image.

[0085] Furthermore, when the first multimedia content is an image, it may be determined whether the first multimedia content includes a plurality of images that can be displayed switchably or an image formed by splicing a plurality of images.

[0086] Since stitching multiple images involves the display effect of the image, whether to stitch multiple images can be determined based on the image ratio of the image. In some embodiments, in response to the image ratio of the first image and the one or more second images being within a specified range, the first multimedia content includes an image formed by stitching multiple images.

[0087] Alternatively, whether to stitch multiple images may be determined based on whether the image may be cropped. When determining whether the image may be cropped, it is usually determined based on the degree of influence of cropping on the entity in the image. When the degree of influence of cropping on the entity in the image is large, for example, when the main information of the entity cannot be displayed, the image is not cropped, and the first multimedia content is formed by switching the displayed image. When the degree of influence of cropping on the entity in the image is small, the image may be collected to form the first multimedia content by stitching multiple images together.

[0088] In some embodiments, generating the first multimedia content based on the first image and one or more second images includes: determining the influence of a specified cropping ratio on entities in the first image and the one or more second images; in response to the influence being less than a second specified threshold, determining an image formed by splicing the first image and the one or more second images as the first multimedia content; in response to the influence being not less than the second specified threshold, determining the switchable first image and the one or more second images as the first multimedia content.

[0089] Determining the display form of the image in the first multimedia content according to the influence of cropping on the entity in the image can reduce the influence of cropping on the expressiveness of the first multimedia content and improve the expressive effect of the first multimedia content.

[0090] The above embodiment describes generating the first multimedia content with the corresponding storyline according to the first image of the first user. In some embodiments, after generating the first multimedia content, content generation may continue based on the first multimedia content. Figure 4 FIG. 2 shows a flow chart of generating second multimedia content according to some embodiments of the present disclosure. Figure 4 As shown, generating the second multimedia content includes steps S41 to S45.

[0091] Similar to the generation process of the first multimedia content mentioned above, the second multimedia content is generated by first generating a third storyline, generating one or more fourth images based on the third storyline, and finally combining the first multimedia content, the third image, and the one or more fourth images to generate the second multimedia content.

[0092] In step S41 , a third storyline is generated according to the third image from the second user and the first multimedia content.

[0093] The second user may be a user other than the first user, or may be the first user. That is, any user may participate in content generation based on the first multimedia content. For example, the second user may be invited by the first user. After receiving the first multimedia content shared by the first user, the second user determines the third image to further generate content based on the first multimedia content.

[0094] The third image of the second user is obtained in a manner similar to the manner of obtaining the first image of the first user, and can be obtained by the second user taking an image or selecting an image from an image library.

[0095] Similar to the above-mentioned generation of the second storyline, which is based on the entities and background in the first image, and the first storyline expressed by the first image is expanded, when generating the third storyline, it includes: determining the entities and background in the third image from the second user; understanding the third image and determining the fourth storyline expressed by the third image; based on the entities and background in the third image, the entities and background in the first multimedia content, the fourth storyline and at least one of the second storyline are expanded to generate the third storyline. The third storyline generated in this way is equivalent to expanding the second storyline based on the third image. Here, expansion refers to continuation, supplementation, adaptation, etc.

[0096] In step S43, one or more fourth images are generated according to the third storyline.

[0097] Similar to the above-mentioned generation of one or more second images according to the second storyline, when generating one or more fourth images according to the third storyline, the number and style of the generated fourth images may be determined first, or the number of the generated fourth images may be determined by determining the number of story scenes in the third storyline. No further details will be given here.

[0098] In step S45 , second multimedia content is generated according to the first multimedia content, the third image, and one or more fourth images.

[0099] When generating the second multimedia content, the first multimedia content, the third image and one or more fourth images may be integrated, for example, the second multimedia content includes the first multimedia content, the third image and one or more fourth images. Alternatively, the first multimedia content may be adaptively modified according to the third image and one or more fourth images, and then the modified first multimedia content, the third image and one or more fourth images may be combined to generate the second multimedia content.

[0100] Similar to the first multimedia content, the second multimedia content may be a video or an image. In some embodiments, in order to improve the generation efficiency and the consistency between the first multimedia content and the second multimedia content, the second multimedia content may be in the same format as the first multimedia.

[0101] After generating the second multimedia content, content generation may be performed again based on the image of the third user according to actual conditions. For example, the termination condition of content generation may be that the content volume of the first multimedia content reaches a first threshold, or the number of participating users reaches a second threshold, etc.

[0102] The content generation method disclosed in the present invention can not only generate the first multimedia content with a storyline based on the first image of the first user, but also continue to generate content based on the first multimedia content, that is, generate the third multimedia content by combining the third image of the second user and the first multimedia content. The content generation method disclosed in the present invention not only has a high richness of content generation, but also has social attributes, which can promote multiple users to participate in interaction, thereby promoting the usage rate of the application.

[0103] The content generation method disclosed in the present invention can be implemented in a specified interaction mode. That is, the first image of the first user can be obtained in response to the first user participating in the specified interaction mode. For example, in response to the first user triggering the shooting control on the shooting interface, the first image shot by the first user is obtained; or in response to the first user triggering the image selection control on the shooting interface, the first image selected by the first user from the image library is obtained. In the specified interaction mode, the shooting interface can be displayed to the user to obtain the first image from the user through the shooting control or the image selection control. The shooting interface may also include other information besides the shooting control and the image selection control, such as the name of the specified interaction mode, the gameplay, etc. Of course, the image selection control can also be displayed on other interfaces besides the shooting interface.

[0104] The shooting interface is equivalent to the entrance for the first user to participate in the specified interaction method. After acquiring the first image of the first user, the specified interaction method will generate the first multimedia content by implementing the content generation method of the present disclosure. In some embodiments, in response to the generation of the first multimedia content, the first multimedia content is displayed on the preview interface. In this way, after the first user participates in the specified interaction method, he can view the first multimedia content through the preview interface, and can subsequently save or share the first multimedia content to achieve the dissemination of the first multimedia content.

[0105] When generating the first multimedia content, the first storyline may be expanded according to the entity and background in the first image to generate the second storyline, then one or more second images may be generated according to the second storyline, and finally the first multimedia content may be generated.

[0106] Combine the following Figures 5 to 8 An application example of the present disclosure is described.

[0107] Figure 5 Schematic diagram of a shooting interface according to some embodiments of the present disclosure is shown. Figure 5 As shown, the shooting interface 5 includes a shooting control 51 and an image selection control 52, which are used to obtain a first image of the first user. At this time, the shooting interface has obtained a first image 53 from the first user, and the first image 53 includes entities 531, 532, and a background, and the background includes 533. The first image 53 shows a girl 531 standing under a stop sign 532 of bus No. 1.

[0108] By understanding the first image 53, it is determined that the first storyline expressed by the first image is that the characters are waiting for the bus. According to the entities in the first image 53: the girl 531 and the station logo 532, and the sunny weather indicated by 533 in the background, the first storyline is expanded to generate a second storyline. For example, the extended information of the station logo 532 is the bus. The extended information of the girl 531 may be that the girl 531 is listening to music. The extended information of 533 in the background may be a vibrant flower bush on the roadside. In addition, combined with the background and the first storyline, the emotion contained in the first image 53 may be happy.

[0109] In this way, the first storyline is expanded by combining the extended information and the emotion to generate the second storyline. The second storyline is, for example, a girl waiting for a bus. After a while, the girl waits for the bus and gets on the bus. While the bus is running, the girl takes a picture of the flowers on the roadside and wants to share it with her friends.

[0110] In this way, a plurality of second images are generated based on the second storyline, and finally the first multimedia content is generated. Figure 6 FIG. 4 is a schematic diagram showing a first multimedia content according to some embodiments of the present disclosure. Figure 6 As shown, the first multimedia content includes an image formed by splicing multiple images, and subtitles are displayed on each image.

[0111] like Figure 6 As shown, the interface 6 displaying the first multimedia content may further include an extended control 61 for acquiring a third image of the second user and generating the second multimedia content based on the third image and the first multimedia content.

[0112] After triggering the extended control 61 , the second user can determine the third image by shooting or selecting an image from an image library. Figure 7 FIG. 4 shows a schematic diagram of a third image according to some embodiments of the present disclosure. Figure 7As shown, the third image in the interface 7 shows an outdoor scene, including entities 71, 72, and a background, which includes, for example, 73. The first image 73 shows two girls 71 preparing to have a picnic at a table 72 under a tree 73.

[0113] Similar to the generation of the second plot, the third plot is generated based on the entity and background in the third image and the entity and background in the first multimedia content. The third plot is, for example, a girl waiting for a bus. After a while, the girl got on the bus. While the bus was driving, the girl took a photo of the flowers on the roadside and wanted to share it with her friends. After receiving the message, the friend suggested going out for a picnic. So the two girls found a place for a picnic and spent a happy time.

[0114] Figure 8 A schematic diagram of second multimedia content according to some embodiments of the present disclosure is shown. The second multimedia content generated according to the third image and the first multimedia content includes an image formed by splicing multiple images, and subtitles are displayed on each image. Figure 8 The second multimedia content shown is only a partial example. The complete second multimedia content may include Figure 6 and Figure 8 In addition, the interface for displaying the second multimedia content may also include an expansion control 61 for generating content again based on the second multimedia content.

[0115] Fig. 9 FIG. 2 shows a schematic diagram of the structure of a content generation device according to some embodiments of the present disclosure. Fig. 9 As shown, the content generating device 9 includes modules 91-95.

[0116] The first determination module 91 is configured to determine an entity and a background in a first image from a first user.

[0117] The second determination module 92 is configured to understand the first image and determine a first storyline expressed by the first image.

[0118] The expansion module 93 is configured to expand the first storyline according to the entity and the background in the first image to generate a second storyline.

[0119] The first generating module 94 is configured to generate one or more second images according to the second storyline.

[0120] The second generating module 95 is configured to generate first multimedia content according to the first image and one or more second images.

[0121] In some embodiments, the extension module 93 is configured to determine extended information based on the entities and background in the first image and the first storyline, wherein the extended information is used to represent at least one of change information of the entities and background in the first image and association information of the entities and background in the first image; and to expand the first storyline using the extended information to generate a second storyline.

[0122] In some embodiments, the expansion module 93 is configured to determine the emotions contained in the first image based on the entities and background in the first image; and expand the first storyline based on the emotions contained in the first image, the entities and background in the first image to generate a second storyline.

[0123] In some embodiments, the expansion module 93 is configured to determine the change information of entities and background in the first image according to the emotion; and expand the first storyline according to the emotion, the change information of entities and background in the first image to generate a second storyline.

[0124] In some embodiments, the expansion module 93 is configured to, in response to the entity in the first image including a character, determine change information of at least one of the character's action and expression according to the emotion; or, in response to the entity in the first image including an object, determine change information of the object's shape according to the emotion; or determine change information of at least one of the environment and atmosphere of the background in the first image according to the emotion.

[0125] In some embodiments, the expansion module 93 is configured to determine the association information between the entity and the background in the first image according to the emotion; and expand the first storyline according to the emotion, the association information between the entity and the background in the first image to generate a second storyline.

[0126] In some embodiments, the first generating module 94 is configured to determine one or more story scenes and a story plot corresponding to each of the one or more story scenes based on the second story plot; and generate a second image corresponding to the story scene for each story plot corresponding to the story scene.

[0127] In some embodiments, the first generation module 94 is configured to determine the style of one or more second images according to the second storyline and the emotions contained in at least one of the first images; and generate one or more second images according to the style of the one or more second images.

[0128] In some embodiments, the second generation module 95 is configured to determine subtitles for each of the first image and one or more second images, wherein the subtitles for each image are generated based on a storyline corresponding to the image; and generate first multimedia content based on the first image and one or more second images, wherein each image in the first multimedia content has subtitles.

[0129] In some embodiments, the first multimedia content is an image, wherein the first multimedia content includes a plurality of images that can be switched for display, or an image formed by splicing a plurality of images; or the first multimedia content is a video.

[0130] In some embodiments, in response to the emotional intensity of the second storyline being greater than a first specified threshold, the first multimedia content is a video; in response to the emotional intensity of the second storyline being not greater than the first specified threshold, the first multimedia content is an image.

[0131] In some embodiments, in response to an image ratio between the first image and the one or more second images being within a specified range, the first multimedia content includes an image formed by stitching the multiple images.

[0132] In some embodiments, the second generation module 95 is configured to determine the influence of a specified cropping ratio on entities in a first image and one or more second images; in response to the influence being less than a second specified threshold, determining an image formed by splicing the first image and one or more second images as first multimedia content; in response to the influence being not less than the second specified threshold, determining the switchably displayed first image and one or more second images as first multimedia content.

[0133] In some embodiments, the content generating device 9 is also configured to obtain a first image captured by the first user in response to the first user triggering a shooting control on the shooting interface; or to obtain a first image selected by the first user from the image library in response to the first user triggering an image selection control on the shooting interface.

[0134] In some embodiments, the content generating device 9 is further configured to display the first multimedia content on the preview interface in response to the generation of the first multimedia content.

[0135] In some embodiments, the content generating device 9 is also configured to generate a third storyline based on a third image from a second user and the first multimedia content; generate one or more fourth images based on the third storyline; and generate second multimedia content based on the first multimedia content, the third image, and one or more fourth images.

[0136] In some embodiments, the content generating device 9 is further configured to determine entities and background in a third image from a second user; understand the third image to determine a fourth storyline expressed by the third image; and expand the fourth storyline and at least one of the second storyline based on the entities and background in the third image, the entities and background in the first multimedia content to generate a third storyline.

[0137] In the above embodiment, based on the first image of the first user, the first multimedia content with the corresponding storyline can be automatically generated. This content generation method can automatically generate a storyline based on an image, and then automatically generate multimedia content based on the storyline. Therefore, this content generation method is more intelligent, and the generated content is richer, which helps to improve the enthusiasm of user interaction, thereby improving the usage rate of the application.

[0138] Fig.10 1 shows a block diagram of an electronic device according to some embodiments of the present disclosure. Fig.10 As shown, the electronic device 10 includes a memory 101; and a processor 102 coupled to the memory 101, and the processor 102 is configured to execute a method as in any of the above embodiments based on instructions stored in the memory 101. Thus, the first multimedia content with a corresponding storyline can be automatically generated based on the first image of the first user. This content generation method can automatically generate a storyline based on an image, and then automatically generate multimedia content based on the storyline. Therefore, this content generation method is more intelligent, and the generated content is richer, which helps to improve the enthusiasm of user interaction, thereby improving the usage rate of the application.

[0139] The memory 101 is used to store one or more computer-readable instructions. The memory 101 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory, including but not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), flash memory. The memory 101 may store, for example, an operating system, an application, a boot loader (BootLoader), a database, and other programs, and may also store various applications and various data.

[0140] The processor 102 is used to run computer-readable instructions to implement the method described in any of the above embodiments. The specific implementation of each step of the method can be referred to the above embodiments, and the repeated parts are not repeated here.

[0141] The processor 102 may be configured to execute Figures 1 to 8The processor 102 may be embodied as various processing devices, such as a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The central processing unit (CPU) may be an X86 or ARM architecture, etc.

[0142] The processor 102 and the memory 101 may communicate with each other directly or indirectly. For example, the processor 102 and the memory 101 may communicate with each other via a network. The network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The processor 102 and the memory 101 may also communicate with each other via a system bus, which is not limited in the present disclosure.

[0143] It should be noted that Fig.10 The components of the electronic device 10 shown are only exemplary and non-limiting. The electronic device 10 may also have other components according to actual application requirements. The processor 102 may control other components in the electronic device 10 to perform desired functions.

[0144] The electronic device 10 may be implemented by software, firmware and / or hardware, and may be integrated into a device installed with relevant application programs.

[0145] Fig.11 A block diagram of an electronic device according to some other embodiments of the present disclosure is shown.

[0146] Fig.11 The electronic device 11 shown may be a computer system with a dedicated hardware structure, which can execute corresponding functions when a relevant application program is installed.

[0147] Electronic devices include, but are not limited to, mobile terminals such as smart phones, laptops, personal digital assistants (PDA), tablet personal computers (Tablet PC), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable devices, etc., and fixed terminals such as digital televisions, desktop computers, etc.

[0148] like Fig.11As shown, a central processing unit (CPU) 111 performs various processes according to a program stored in a read-only memory (ROM) 112 or a program loaded from a storage section 118 to a random access memory (RAM) 113. In the RAM 113, data required when the CPU 111 performs various processes, etc., is stored as needed. The central processing unit is merely exemplary, and it may also be other types of processors, such as the various processors described above. The ROM 112, the RAM 113, and the storage section 118 may be various forms of computer-readable storage media. It should be noted that although Fig.11 ROM 112, RAM 113 and storage section 118 are shown separately in FIG. 1 , but one or more of them may be combined or located in the same or different memory or storage modules.

[0149] The CPU 111, the ROM 112, and the RAM 113 are connected to each other via a bus 114. To the bus 114, an input / output interface 115 is also connected.

[0150] The following components are connected to the input / output interface 115: an input portion 116, such as a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output portion 117, including a display, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage portion 118, including a hard disk, a magnetic tape, etc.; and a communication portion 119, including a network interface card such as a LAN card, a modem, etc. The communication portion 119 allows communication processing to be performed via a network such as the Internet. It is easy to understand that although Fig.11 Parts of the electronic device 11 are shown to communicate via the bus 114, but they may also communicate via a network or other means, wherein the network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network.

[0151] A drive 1110 is also connected to the input / output interface 115 as needed. A removable medium 1111 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 1110 as needed so that a computer program read therefrom is installed into the storage section 118 as needed.

[0152] When the series of processing described above is implemented by software, the program constituting the software can be installed from a network such as the Internet or a storage medium such as the removable medium 1111 .

[0153] According to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, and when the computer program product is run on a computer, the computer implements the method described in any of the aforementioned embodiments. Thus, the first multimedia content with the corresponding storyline can be automatically generated based on the first image of the first user. This content generation method can automatically generate a storyline based on an image, and then automatically generate multimedia content based on the storyline. Therefore, this content generation method is more intelligent, and the richness of the generated content is higher, which helps to improve the enthusiasm of user interaction, thereby improving the usage rate of the application. The computer program product includes a computer instruction carried on a computer-readable medium, including a program code for executing the method shown in the flowchart. In such an embodiment, the computer instruction can be downloaded and installed from the network through the communication part 119, or installed from the storage part 118, or installed from the ROM 112. When the computer program is executed by the CPU 111, the method of the embodiment of the present disclosure is executed.

[0154] It should be noted that, in the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, apparatus, or device or for use in conjunction with an instruction execution system, apparatus, or device.

[0155] The computer readable medium may be a computer readable storage medium, or a computer readable signal medium, or any combination of the two.

[0156] Computer-readable storage media include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. Computer instructions are stored on the computer-readable storage medium, and when the instructions are executed by the processor, the method described in any of the aforementioned embodiments is implemented. Thus, the first multimedia content with the corresponding storyline can be automatically generated based on the first image of the first user. This content generation method can automatically generate a storyline based on an image, and then automatically generate multimedia content based on the storyline, so this content generation method is more intelligent, and the generated content is more rich, which helps to improve the enthusiasm of user interaction, thereby improving the usage rate of the application.

[0157] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer readable program codes. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0158] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0159] In some embodiments, a computer program is further provided, comprising: instructions, which, when executed by a processor, cause the processor to execute the method described in any of the above embodiments. For example, the instructions may be embodied as computer program codes.

[0160] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In situations involving a remote computer, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0161] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0162] The functions described above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0163] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A content generation method, comprising: determining an entity and a background in a first image from a first user; Understanding the first image to determine a first storyline expressed by the first image; Expanding the first storyline according to the entity and the background in the first image to generate a second storyline; generating one or more second images based on the second storyline; First multimedia content is generated based on the first image and the one or more second images.

2. The content generation method according to claim 1, wherein: The step of extending the first storyline according to the entity and the background in the first image to generate a second storyline includes: Determine extended information according to the entity and the background in the first image and the first storyline, wherein the extended information is used to represent at least one of change information of the entity and the background in the first image and association information of the entity and the background in the first image; The first storyline is expanded using the extended information to generate a second storyline.

3. The content generation method according to claim 1, wherein: The step of extending the first storyline according to the entity and the background in the first image to generate a second storyline includes: Determining an emotion contained in the first image according to the entity and the background in the first image; The first storyline is expanded according to the emotion contained in the first image, the entity in the first image and the background to generate a second storyline.

4. The content generation method according to claim 3, wherein: The step of expanding the first storyline according to the emotion contained in the first image, the entity in the first image, and the background to generate a second storyline includes: Determining change information of the entity and the background in the first image according to the emotion; The first storyline is expanded according to the emotion, the entity in the first image, and the change information of the background to generate a second storyline.

5. The content generation method according to claim 4, wherein: The determining, according to the emotion, change information of the entity and the background in the first image includes: In response to the entity in the first image including a character, determining change information of at least one of an action and an expression of the character according to the emotion; or In response to the entity in the first image including an object, determining change information of the shape of the object according to the emotion; or Determine change information of at least one of the environment and atmosphere of the background in the first image according to the emotion.

6. The content generation method according to claim 3, wherein: The step of expanding the first storyline according to the emotion contained in the first image, the entity in the first image, and the background to generate a second storyline includes: Determining association information between the entity and the background in the first image according to the emotion; The first storyline is expanded according to the emotion, the entity in the first image, and the associated information of the background to generate a second storyline.

7. The content generation method according to any one of claims 1 to 6, wherein: Generating one or more second images according to the second storyline comprises: Determining, according to the second storyline, one or more story scenes and a storyline corresponding to each of the one or more story scenes; For the storyline corresponding to each story scene, a second image corresponding to the story scene is generated.

8. The content generation method according to any one of claims 1 to 6, wherein: Generating one or more second images according to the second storyline comprises: Determining the style of one or more second images according to the emotions contained in at least one of the second storyline and the first image; The one or more second images are generated according to the style of the one or more second images.

9. The content generation method according to any one of claims 1 to 6, wherein: Generating first multimedia content according to the first image and the one or more second images includes: determining a caption for each of the first image and the one or more second images, wherein the caption for each image is generated based on a storyline corresponding to the image; First multimedia content is generated according to the first image and the one or more second images, wherein each image in the first multimedia content has a subtitle.

10. The content generation method according to any one of claims 1 to 6, wherein: The first multimedia content is an image, wherein the first multimedia content includes a plurality of images that can be switched for display, or an image formed by splicing a plurality of images; or The first multimedia content is a video.

11. The content generation method according to claim 10, wherein: In response to the emotional intensity of the second storyline being greater than a first specified threshold, the first multimedia content is a video; In response to the emotional intensity of the second storyline being not greater than the first specified threshold, the first multimedia content is an image.

12. The content generation method according to claim 10, wherein: In response to an image ratio between the first image and the one or more second images being within a specified range, the first multimedia content includes an image formed by splicing a plurality of images.

13. The content generation method according to claim 10, wherein: Generating first multimedia content according to the first image and the one or more second images includes: Determining an effect of a specified crop ratio on entities in the first image and the one or more second images; In response to the influence being less than a second specified threshold, determining an image formed by splicing the first image and the one or more second images as the first multimedia content; In response to the influence degree being not less than the second specified threshold, the first image and the one or more second images that can be switchably displayed are determined as the first multimedia content.

14. The content generation method according to any one of claims 1 to 6, further comprising: In response to the first user triggering a shooting control on a shooting interface, acquiring the first image shot by the first user; or In response to the first user triggering an image selection control on the shooting interface, acquiring the first image selected by the first user from an image library.

15. The content generation method according to any one of claims 1 to 6, further comprising: In response to the generation of the first multimedia content, the first multimedia content is displayed on a preview interface.

16. The content generation method according to any one of claims 1 to 6, further comprising: generating a third storyline based on a third image from a second user and the first multimedia content; generating one or more fourth images according to the third storyline; Second multimedia content is generated based on the first multimedia content, the third image, and the one or more fourth images.

17. The content generation method according to claim 16, wherein: The generating a third storyline according to the third image from the second user and the first multimedia content comprises: determining entities and background in a third image from a second user; Understanding the third image to determine a fourth storyline expressed by the third image; At least one of the fourth storyline and the second storyline is expanded according to the entity and the background in the third image and the entity and the background in the first multimedia content to generate a third storyline.

18. An electronic device, comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the content generating method according to any one of claims 1 to 17 based on instructions stored in the memory.

19. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the processor implements the content generation method according to any one of claims 1 to 17. 20 . A computer program product, when the computer program product is executed on a computer, enables the computer to implement the content generation method according to any one of claims 1 to 17.

Images