Content generation method, electronic device, computer-readable storage medium, and product

Abstract

The present disclosure relates to a content generation method, electronic device, computer-readable storage medium, and product, and relates to the field of computer technology. The content generation method includes: determining entities and background in a first image from a first user; understanding the first image to determine a first storyline expressed by the first image; expanding the first storyline based on the entities and background in the first image to generate a second storyline; generating one or more second images based on the second storyline; and generating first multimedia content based on the first image and the one or more second images. The present disclosure can enhance user engagement and improve application usage.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a content generation method, an electronic device, a computer-readable storage medium, and a product. Background Art

[0002] With the development of artificial intelligence technology, users can utilize the functions provided by various applications to send content generation instructions. Computers then automatically generate various types of content, such as text, images, and videos, based on these instructions. This process can be implemented using machine learning models, such as large language models (LLMs) and foundation models, and other neural network models, leveraging the powerful computing power of these models to meet various user needs. Summary of the Invention

[0003] According to some embodiments of the present disclosure, a content generation method is provided, including: determining entities and background in a first image from a first user; understanding the first image to determine a first storyline expressed by the first image; expanding the first storyline based on the entities and background in the first image to generate a second storyline; generating one or more second images based on the second storyline; and generating first multimedia content based on the first image and the one or more second images.

[0004] According to some embodiments of the present disclosure, an electronic device is provided, including: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the method of any embodiment described in the present disclosure based on instructions stored in the memory.

[0005] According to some embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method of any embodiment described in the present disclosure is performed.

[0006] According to some embodiments of the present disclosure, a computer program product is provided. When the computer program product is run on a computer, the computer is caused to perform the method of any one of the embodiments described in the present disclosure.

[0007] Other features, aspects and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The following describes embodiments of the present disclosure with reference to the accompanying drawings. It should be understood that the drawings described below only relate to some embodiments of the present disclosure and do not limit the present disclosure.

[0009] Figure 1A flowchart of a content generation method according to some embodiments of the present disclosure is shown.

[0010] Figure 2 A schematic diagram of a process for generating a second storyline according to some embodiments of the present disclosure is shown.

[0011] Figure 3 A schematic diagram of a process for generating a second image according to some embodiments of the present disclosure is shown.

[0012] Figure 4 A schematic diagram of a process for generating second multimedia content according to some embodiments of the present disclosure is shown.

[0013] Figure 5 A schematic diagram of a shooting interface according to some embodiments of the present disclosure is shown.

[0014] Figure 6 A schematic diagram illustrating first multimedia content according to some embodiments of the present disclosure is shown.

[0015] Figure 7 A schematic diagram of a third image according to some embodiments of the present disclosure is shown.

[0016] Figure 8 A schematic diagram illustrating second multimedia content according to some embodiments of the present disclosure is shown.

[0017] Figure 9 A schematic structural diagram of a content generation device according to some embodiments of the present disclosure is shown.

[0018] Figure 10 A block diagram of an electronic device according to some embodiments of the present disclosure is shown.

[0019] Figure 11 A block diagram of an electronic device according to some other embodiments of the present disclosure is shown.

[0020] It should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not necessarily drawn according to the actual proportional relationship. DETAILED DESCRIPTION

[0021] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. It should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein.

[0022] It should be understood that the various steps described in the method embodiments of the present disclosure can be performed in different orders and / or performed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments should be interpreted as being merely exemplary and do not limit the scope of the present disclosure.

[0023] The term “including” and its variations used in the present disclosure are open terms that include at least the following elements / features but do not exclude other elements / features, that is, “including but not limited to.” The term “based on” means “at least in part based on.”

[0024] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules, or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules, or units. Unless otherwise specified, concepts such as "first" and "second" are not intended to imply that the objects described in such a manner must be in a given order in time, space, ranking, or any other manner.

[0025] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0026] The following detailed description of the embodiments of the present disclosure is provided in conjunction with the accompanying drawings, but the present disclosure is not limited to these specific embodiments. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. In addition, in one or more embodiments, specific features, structures, or characteristics may be combined in any suitable manner that will be apparent to those skilled in the art from this disclosure.

[0027] Some applications use content generation technology to provide users with various interactive methods. For example, they can generate images based on user-provided text, generate new images based on user-provided images, or generate videos based on user-provided images. These interactive methods meet users' content generation needs to a certain extent.

[0028] However, these current interactive methods require significant user involvement and lack intelligence. For example, when generating a new image based on a user-provided image, the user must instruct them on how to generate the new image. Furthermore, the content generated by these interactive methods is relatively low in richness and lacks interest. This neither fully utilizes the power of content generation technology nor encourages user engagement.

[0029] Based on this, the present disclosure provides a content generation method. This method can automatically generate multimedia content with a storyline based on user-provided images. Specifically, the storyline is automatically generated based on the user-provided images, and the multimedia content is generated based on the storyline. The multimedia content generation process fully utilizes the intelligence of content generation technology, automatically generating the storyline and multimedia content without requiring the user to provide excessive information. This not only increases the richness of the generated content but also enhances the user's interactive experience, thereby increasing user engagement and application usage.

[0030] Figure 1 FIG. 1 shows a flow chart of a content generation method according to some embodiments of the present disclosure. Figure 1 As shown, the content generation method includes steps S11 to S19.

[0031] In step S11 , an entity and a background in a first image from a first user are determined.

[0032] A first image of a first user is obtained, and content is generated based on the first image. The first image can be an image of various styles and types, and can be a color image or a grayscale image.

[0033] The entity in the first image refers to the primary subject within the first image, representing the main theme or key point of the first image. The entity can be any visual object, including people and objects. The background in the first image refers to the surrounding environment or background surrounding the entity, creating a richer and more layered atmosphere.

[0034] The entity in the first image can be identified using an object recognition algorithm. Alternatively, image processing techniques can be used to separate the foreground and background in the first image and extract the entity from the foreground. After the entity in the first image is identified, other objects in the first image other than the entity are identified as the background.

[0035] By determining the entities and background in the first image, the objects in the first image and the relationships between the objects can be determined. Generating content based on the objects in the first image and the relationships between the objects can improve the relevance of the generated content to the first image, that is, generating content around the first image, thereby improving the accuracy of the generated content.

[0036] In step S13 , the first image is understood to determine the first storyline expressed by the first image.

[0037] Understanding the first image includes performing object detection and recognition, scene understanding, and semantic analysis on the first image to determine the first plot expressed by the first image. For example, if the first image shows a girl standing on the side of the road with a bus stop sign, the first plot expressed by the first image is that the girl is waiting for the bus.

[0038] The first image is understood and the first storyline expressed by the first image is determined, so that content can be generated according to the storyline expressed by the first image, that is, content is generated around the first storyline, which can improve the accuracy of the generated content.

[0039] In step S15 , the first storyline is expanded according to the entity and background in the first image to generate a second storyline.

[0040] After determining the entities and background included in the first image, as well as the first storyline expressed by the first image, the first storyline is expanded to generate a second storyline. The second storyline can be generated, for example, by continuing, adapting, or supplementing the first storyline, based on the entities and background in the first image. For example, if the first image shows a girl standing on the side of the road with a bus stop sign, the subject of the first image is the girl, and the background is the bus stop sign. The first storyline is about the girl waiting for a bus. Based on the girl and the bus stop sign, the generated second storyline can be about the girl sitting on a bus seat listening to music.

[0041] Based on the entities and background in the first image, the first storyline is expanded so that the characters or objects appearing in the generated second storyline are associated with the entities and background in the first image, thereby enabling the second storyline to be a reasonable extension of the first storyline.

[0042] In step S17 , one or more second images are generated according to the second storyline.

[0043] Before generating the second image, information such as the number and style of the second image may be determined first.

[0044] In some embodiments, the number of generated second images may be determined based on the richness of the second plot. The richer the second plot, the more second images are generated; the less rich the second plot, the fewer second images are generated.

[0045] In some embodiments, the style of the generated second image can be determined based on the story style of the second plot. For example, if the story style of the second plot is warm and touching, the style of the second image can also be warm and touching by setting the color, lighting, and composition of the second image.

[0046] Because the second storyline is an extension of the first storyline, the one or more second images generated based on the second storyline are associated with the first image. Thus, based on the first image provided by the first user, one or more second images associated with the first image are generated, and these one or more second images correspond to the second storyline, thus generating one or more second images with a storyline for the first user's first image.

[0047] In step S19 , first multimedia content is generated according to the first image and one or more second images.

[0048] After one or more second images with storylines are generated for the first image, first multimedia content can be generated by combining the first image and the one or more second images.

[0049] When the second storyline is a continuation of the first storyline, for example, the first image and one or more second images are displayed sequentially to generate first multimedia content, so that the first multimedia content corresponds to the first storyline and the second storyline representing the subsequent development of the first storyline.

[0050] The second storyline is an expansion, adaptation, supplement, etc. of the first storyline. For example, the first image and one or more second images are cross-displayed to generate the first multimedia content, that is, the storyline corresponding to the first multimedia content is determined based on the relationship between the first storyline and the second storyline.

[0051] In addition to the first image and one or more second images, the first multimedia content may also include other forms of content besides images, such as text, audio, etc. The text and audio may be determined based on the second storyline, so that the various forms of content in the first multimedia content are unified.

[0052] The first multimedia content is generated based on the first image of the first user to generate the second storyline, and then generated based on the second storyline. Therefore, the first multimedia content is multimedia content with a corresponding storyline, and the first multimedia content is richer and more interesting.

[0053] In the above embodiment, first multimedia content with a corresponding storyline can be automatically generated based on the first image of the first user. This content generation method can automatically generate a storyline based on the image, and then automatically generate multimedia content based on the storyline. Therefore, this content generation method is more intelligent and generates richer content, which helps to increase user interaction enthusiasm and thus improve application usage.

[0054] Figure 2 FIG. 4 shows a flow chart of generating a second storyline according to some embodiments of the present disclosure. Figure 2 As shown, generating the second storyline includes steps S21 to S23.

[0055] In step S21, extended information is determined based on the entity and background in the first image and the first storyline, wherein the extended information is used to represent at least one of change information of the entity and background in the first image and association information of the entity and background in the first image.

[0056] The extended information is used to generate a second storyline. That is, in addition to generating the second storyline, starting from the first image, the extended information is determined by analyzing the entities and background in the first image and the first storyline, and then the second storyline is generated based on the extended information.

[0057] The change information of the entity and background in the first image is determined by changing the state, action, environment, etc. of the entity and background in the first image. For example, when the entity in the first image includes a person, the change information of the person may be change information of the person's action or expression.

[0058] The change information of the entity and background in the first image can be determined based on the entity or background itself, or can be determined based on the relationship between the entity and the background. For example, when the entity in the first image includes a person, the change information of the person is the change information of the action. That is, the change information of the entity can be determined based on the entity itself. When the entity in the first image includes a person, the background includes the environment, and the environment is raining, the change information of the person's expression is determined to be sad. When the entity in the first image includes a person, the background includes the environment, and the environment is a clear sky, the change information of the person's expression is determined to be happy. That is, the change information of the entity can be determined based on the entity and the background. Similarly, the change information of the background can be determined based on the background itself, or it can be determined based on the entity and the background.

[0059] The association information between the entity and the background in the first image is determined by associating the entity and the background in the first image. For example, if the entity in the first image includes potato chips, the associated information for potato chips may be convenience stores, as convenience stores often sell potato chips.

[0060] The association information between the entity and the background in the first image can also be determined based on the entity or background itself, or based on the relationship between the entity and the background. For example, if the entity in the first image includes potato chips, the associated information for potato chips is convenience store. That is, the association information of the entity can be determined based on the entity itself. If the entity in the first image includes potato chips and the background includes a computer playing a video, the associated information for potato chips is juice. That is, the association of the entity can be determined based on the entity and the background. Similarly, the association information of the background can be determined based on the background itself, or based on the entity and the background.

[0061] The extended information includes at least one of the change information of the entity and background in the first image and the association information of the entity and background in the first image. After determining the extended information, the direction and elements of the expansion of the first storyline are also determined, so that the second storyline can be generated.

[0062] In step S23, the first storyline is expanded using the extended information to generate a second storyline.

[0063] After the extended information is determined, a second storyline may be generated by integrating the change information of the entity and background in the first image and the association information of the entity and background in the first image included in the extended information.

[0064] Because emotional tone is an important component of a storyline, in some embodiments, the emotion contained in the first image can be first determined, and then the second storyline can be generated based on the emotion contained in the first image. For example, expanding the first storyline to generate the second storyline based on the entities and background in the first image may include: determining the emotion contained in the first image based on the entities and background in the first image; and expanding the first storyline to generate the second storyline based on the emotion contained in the first image, the entities and background in the first image.

[0065] The emotion contained in the first image influences the expansion of the first storyline. For example, if the first image contains a sad emotion, the emotional tone of the generated second storyline will also be sad. This way, the first multimedia content subsequently generated based on the second storyline will match the first image, increasing the likelihood of user satisfaction with the generated content.

[0066] The emotion contained in the first image can be determined based on the entities and background in the first image. For example, if the entity in the first image includes a person, the person's emotion can be determined based on the person's actions, expressions, etc. The emotion contained in the first image can also be determined by combining the person's emotion with the environment and atmosphere of the background in the first image.

[0067] When expanding the first storyline based on the emotions contained in the first image, it can be done in the following manner: determine the change information of the entities and background in the first image based on the emotions; expand the first storyline based on the emotions, the change information of the entities and background in the first image to generate a second storyline.

[0068] That is, based on the emotion contained in the first image, change information about the entities and background in the first image is determined. For example, if the entities in the first image include people and the emotion contained in the first image is sadness, the change information about the people can include changes in their facial expressions, such as the person's expression becoming tearful. Meanwhile, the change information about the background can include changes in the atmosphere, such as the background becoming darker.

[0069] In some embodiments, determining the change information of the entities and background in the first image based on the emotion includes: in response to the entity in the first image including a character, determining the change information of at least one of the character's actions and expressions based on the emotion; or in response to the entity in the first image including an object, determining the change information of the object's shape based on the emotion; or determining the change information of at least one of the environment and atmosphere of the background in the first image based on the emotion.

[0070] A character can be a person or an anthropomorphic object. An object's form can be its appearance or display. The environment and atmosphere can be controlled by setting objects in the background and the background color.

[0071] In addition to determining changes in entities and background within the first image based on the emotion contained within the first image, the association information between the entities and background within the first image can also be determined based on the emotion contained within the first image. Specifically, the association information between the entities and background within the first image is determined based on the emotion; and based on the emotion and the association information between the entities and background within the first image, the first storyline is expanded to generate a second storyline.

[0072] In other words, extended information is determined based on the emotion contained in the first image, and the first storyline is then expanded based on the extended information to generate a second storyline. Determining extended information based on emotion can improve the consistency of the extended information regarding changes in the entities and background of the first image, as well as the associations between the entities and background of the first image. This reduces the risk of content confusion in the extended information due to inconsistent changes in the directions of change between different entities and the background in the first image, or inconsistent associations between different entities and the background in the first image.

[0073] After the second storyline is generated, one or more second images are generated according to the second storyline. Since the second storyline is often embodied or occurs in one or more story scenes, the second images can be generated based on the story scenes of the second storyline. Figure 3 FIG. 5 shows a flow chart of generating a second image according to some embodiments of the present disclosure. Figure 3 As shown, generating the second image includes steps S31 to S33.

[0074] In step S31 , one or more story scenes and a story plot corresponding to each of the one or more story scenes are determined according to the second story plot.

[0075] When determining one or more story scenes and the story plot corresponding to each of the one or more story scenes based on the second storyline, the determination can be made after sorting and summarizing the second storyline. Thus, by determining the key story plots in the second storyline based on the one or more story scenes, the key story plots being the story plots corresponding to the story scenes, and generating the second image based on the key story plots, the generated second image can be used to enhance the completeness and accuracy of the second storyline.

[0076] In addition, when generating multiple second images, the style of the second images can be unified. The unified style here refers to the same style, or a smooth transition of style. Style can usually be reflected by emotions. Since the second image is generated based on the first image, the style of the second image can be determined based on the emotions contained in the first image. Alternatively, since the second image corresponds to the second storyline, the style of the second image can also be determined based on the emotions contained in the second storyline. That is, based on the emotions contained in at least one of the second storyline and the first image, the style of one or more second images is determined; based on the style of one or more second images, one or more second images are generated. The style of the second image can be reflected, for example, by setting the filter, composition, color, light and shadow, etc. of the second image.

[0077] In step S33 , for each story scene corresponding to the story plot, a second image corresponding to the story scene is generated.

[0078] For each story scene, a second image corresponding to the story scene is generated. The second image generated in this way can accurately reflect the second story plot.

[0079] After generating one or more second images that accurately represent the second storyline and have a reasonable style, they can be combined with the first image to generate the first multimedia content. In addition to the first image and one or more second images, the first multimedia content can also include content in the form of text, audio, etc. For example, subtitles can be generated for each image in the first multimedia content. This not only enriches the first multimedia content, but also enhances the expressiveness of the corresponding storyline and increases its interest.

[0080] In some embodiments, generating first multimedia content based on a first image and one or more second images includes: determining subtitles for each of the first image and the one or more second images, wherein the subtitles for each image are generated based on a storyline corresponding to the image; and generating first multimedia content based on the first image and the one or more second images, wherein each image in the first multimedia content has subtitles.

[0081] The first multimedia content may be presented in the form of an image or a video. That is, the first multimedia content is an image, wherein the first multimedia content includes multiple images that can be switched for display, or an image formed by splicing multiple images; or the first multimedia content is a video.

[0082] The multiple images that can be switched for display are equivalent to a collection of images. The switching of these images can be triggered by the user or by a condition, such as switching to the next image after displaying the current image for a specified period of time. Stitching multiple images means cropping or scaling multiple images to form a single image.

[0083] In some embodiments, the first multimedia content includes multiple images that can be switched for display, and an image in the multiple images can be an image formed by stitching the multiple images. In other words, the first multimedia content can include multiple images that can be switched for display formed by stitching the multiple images.

[0084] Whether the first multimedia content should be presented as an image or video can be determined based on the emotional intensity of the second plot. Specifically, if the emotional intensity of the second plot exceeds a first specified threshold, the first multimedia content is presented as a video; if the emotional intensity of the second plot does not exceed the first specified threshold, the first multimedia content is presented as an image. For example, if the second plot resembles a short film, the emotional intensity is high, making a video presentation more appropriate. On the other hand, if the second plot resembles a travelogue, the emotional intensity is low, making an image presentation more appropriate.

[0085] Furthermore, when the first multimedia content is an image, it may be determined whether the first multimedia content includes a plurality of images that can be switched for display, or includes an image formed by splicing a plurality of images.

[0086] Because stitching multiple images affects the display quality of the images, determining whether to stitch the multiple images may be based on their image ratios. In some embodiments, in response to the image ratio of the first image and one or more second images being within a specified range, the first multimedia content includes an image formed by stitching the multiple images.

[0087] Alternatively, whether to stitch multiple images together can be determined based on whether the image can be cropped. The determination of whether to crop an image is typically based on the impact of cropping on entities within the image. If cropping significantly impacts the entity within the image, such as when the primary information of the entity cannot be displayed, the image is not cropped, and the first multimedia content is formed by switching the displayed images. If cropping does not significantly impact the entity within the image, the image can be captured and stitched together to form the first multimedia content.

[0088] In some embodiments, generating the first multimedia content based on the first image and one or more second images includes: determining the influence of a specified cropping ratio on entities in the first image and the one or more second images; in response to the influence being less than a second specified threshold, determining the image formed by splicing the first image and the one or more second images as the first multimedia content; in response to the influence being not less than the second specified threshold, determining the switchably displayed first image and the one or more second images as the first multimedia content.

[0089] Determining the display form of the image in the first multimedia content based on the degree of impact of cropping on entities in the image can reduce the impact of cropping on the expressiveness of the first multimedia content and improve the presentation effect of the first multimedia content.

[0090] The above embodiment describes generating first multimedia content with a corresponding storyline based on the first image of the first user. In some embodiments, after generating the first multimedia content, further content generation may be performed based on the first multimedia content. Figure 4 FIG. 5 shows a flow chart of generating second multimedia content according to some embodiments of the present disclosure. Figure 4 As shown, generating the second multimedia content includes steps S41 to S45.

[0091] Similar to the generation process of the first multimedia content described above, the second multimedia content is generated by first generating a third storyline, generating one or more fourth images based on the third storyline, and finally combining the first multimedia content, the third image, and the one or more fourth images to generate the second multimedia content.

[0092] In step S41 , a third storyline is generated based on the third image from the second user and the first multimedia content.

[0093] The second user can be a user other than the first user, or the first user. In other words, any user can participate in content generation based on the first multimedia content. For example, the second user may be invited by the first user. After receiving the first multimedia content shared by the first user, the second user determines a third image to further generate content based on the first multimedia content.

[0094] The third image of the second user is obtained in a manner similar to the manner of obtaining the first image of the first user, and can be obtained by the second user taking an image or selecting an image from an image library.

[0095] Similar to the aforementioned generation of the second storyline, which expands the first storyline expressed by the first image based on the entities and background in the first image, the generation of the third storyline includes: determining the entities and background in the third image from the second user; understanding the third image to determine the fourth storyline expressed by the third image; and expanding the fourth storyline and at least one of the second storyline based on the entities and background in the third image and the entities and background in the first multimedia content to generate the third storyline. The third storyline generated in this way is equivalent to expanding the second storyline based on the third image. Here, expansion refers to continuation, supplementation, adaptation, etc.

[0096] In step S43, one or more fourth images are generated according to the third storyline.

[0097] Similar to the aforementioned generation of one or more second images based on the second storyline, when generating one or more fourth images based on the third storyline, the number and style of the generated fourth images can be determined first, or the number of fourth images can be determined by determining the number of story scenes in the third storyline. This will not be further elaborated here.

[0098] In step S45 , second multimedia content is generated according to the first multimedia content, the third image, and one or more fourth images.

[0099] When generating the second multimedia content, the first multimedia content, the third image, and one or more fourth images may be integrated. For example, the second multimedia content includes the first multimedia content, the third image, and one or more fourth images. Alternatively, the first multimedia content may be adaptively modified based on the third image and one or more fourth images, and then the modified first multimedia content, the third image, and one or more fourth images may be combined to generate the second multimedia content.

[0100] Similar to the first multimedia content, the second multimedia content may be a video or an image. In some embodiments, in order to improve generation efficiency and consistency between the first and second multimedia contents, the second multimedia content may be in the same format as the first multimedia.

[0101] After generating the second multimedia content, content generation may be performed again based on the image of the third user, depending on the actual situation. For example, the termination condition for content generation may be that the amount of the first multimedia content reaches a first threshold, or the number of participating users reaches a second threshold.

[0102] The disclosed content generation method not only generates first multimedia content with a storyline based on a first user's first image, but also continues to generate content based on the first multimedia content, namely, combining a third image of a second user with the first multimedia content to generate third multimedia content. The disclosed content generation method not only generates rich content but also has social attributes, promoting multiple users to participate in interaction, thereby increasing application usage.

[0103] The content generation method disclosed in the present invention can be implemented in a specified interactive mode. That is, the first image of the first user can be obtained in response to the first user participating in the specified interactive mode. For example, in response to the first user triggering the shooting control on the shooting interface, the first image shot by the first user is obtained; or in response to the first user triggering the image selection control on the shooting interface, the first image selected by the first user from the image library is obtained. In the specified interactive mode, the shooting interface can be displayed to the user to obtain the first image from the user through the shooting control or the image selection control. The shooting interface may also include other information in addition to the shooting control and the image selection control, such as the name of the specified interactive mode, the gameplay, etc. Of course, the image selection control can also be displayed on other interfaces besides the shooting interface.

[0104] The capture interface serves as the entry point for the first user to participate in the designated interaction method. After acquiring the first image of the first user, the designated interaction method generates first multimedia content by implementing the content generation method disclosed herein. In some embodiments, in response to the generation of the first multimedia content, the first multimedia content is displayed on the preview interface. Thus, after participating in the designated interaction method, the first user can view the first multimedia content through the preview interface and subsequently save or share the first multimedia content, thereby disseminating the first multimedia content.

[0105] When generating the first multimedia content, the first storyline may be expanded based on the entity and background in the first image to generate a second storyline, and then one or more second images may be generated based on the second storyline to finally generate the first multimedia content.

[0106] The following combination Figures 5 to 8 An application example of the present disclosure is described.

[0107] Figure 5 Schematic diagram of a shooting interface according to some embodiments of the present disclosure is shown. Figure 5 As shown, the capture interface 5 includes a capture control 51 and an image selection control 52, which are used to capture a first image of a first user. At this point, the capture interface has captured a first image 53 from the first user. First image 53 includes entities 531 and 532, as well as a background, such as 533. First image 53 shows a girl 531 standing under a stop sign 532 for bus route 1.

[0108] By understanding first image 53, it is determined that the first storyline expressed in the first image is a person waiting for a bus. Based on the entities in first image 53: girl 531 and station sign 532, as well as the sunny weather indicated by background 533, the first storyline is expanded to generate a second storyline. For example, the expanded information of station sign 532 is bus. The expanded information of girl 531 may be listening to music. The expanded information of background 533 may be a vibrant roadside flowerbed. Furthermore, combined with the background and the first storyline, the emotion contained in first image 53 is likely to be happy.

[0109] By combining the extended information and emotions, the first storyline is expanded to generate a second storyline. For example, the second storyline involves a girl waiting for a bus. After a while, the girl finally gets on the bus. While the bus is driving, the girl takes a photo of a roadside flower bush and wants to share it with her friends.

[0110] In this way, multiple second images are generated based on the second storyline, and finally the first multimedia content is generated. Figure 6 FIG. 1 shows a schematic diagram of first multimedia content according to some embodiments of the present disclosure. Figure 6 As shown, the first multimedia content includes an image formed by splicing multiple images, and subtitles are displayed on each image.

[0111] like Figure 6 As shown, the interface 6 displaying the first multimedia content may further include an extended control 61 for acquiring a third image of the second user and generating second multimedia content based on the third image and the first multimedia content.

[0112] After triggering the extended control 61 , the second user can determine the third image by photographing or selecting an image from an image library. Figure 7 FIG. 5 shows a schematic diagram of a third image according to some embodiments of the present disclosure. Figure 7As shown, the third image in the interface 7 shows an outdoor scene, including entities 71 and 72, and a background, which includes, for example, 73. The first image 73 shows two girls 71 preparing to have a picnic at a table 72 under a tree 73.

[0113] Similar to the generation of the second storyline described above, a third storyline is generated based on the entities and background in the third image and the entities and background in the first multimedia content. For example, the third storyline involves a girl waiting for a bus. After a while, the girl finally gets on the bus. While the bus is driving, the girl takes a photo of a roadside flower bush and wants to share it with a friend. After receiving the photo, the friend suggests a picnic. The two girls find a picnic spot and have a great time.

[0114] Figure 8 A schematic diagram of second multimedia content according to some embodiments of the present disclosure is shown. The second multimedia content generated based on the third image and the first multimedia content includes an image formed by splicing multiple images, and subtitles are displayed on each image. Figure 8 The following are only partial examples of the second multimedia content. The complete second multimedia content may include: Figure 6 and Figure 8 In addition, the interface displaying the second multimedia content may also include an expansion control 61 for regenerating content based on the second multimedia content.

[0115] Figure 9 FIG. 1 shows a schematic diagram of the structure of a content generation device according to some embodiments of the present disclosure. Figure 9 As shown, the content generating device 9 includes modules 91 - 95 .

[0116] The first determining module 91 is configured to determine an entity and a background in a first image from a first user.

[0117] The second determination module 92 is configured to understand the first image and determine the first storyline expressed by the first image.

[0118] The expansion module 93 is configured to expand the first storyline according to the entity and background in the first image to generate a second storyline.

[0119] The first generating module 94 is configured to generate one or more second images according to the second storyline.

[0120] The second generating module 95 is configured to generate first multimedia content according to the first image and one or more second images.

[0121] In some embodiments, the extension module 93 is configured to determine extended information based on the entities and background in the first image and the first storyline, wherein the extended information is used to represent at least one of the change information of the entities and background in the first image and the association information of the entities and background in the first image; and use the extended information to expand the first storyline to generate a second storyline.

[0122] In some embodiments, the expansion module 93 is configured to determine the emotions contained in the first image based on the entities and background in the first image; and expand the first storyline based on the emotions contained in the first image, the entities and background in the first image to generate a second storyline.

[0123] In some embodiments, the expansion module 93 is configured to determine the change information of entities and background in the first image according to the emotion; and expand the first storyline according to the emotion, the change information of entities and background in the first image to generate a second storyline.

[0124] In some embodiments, the expansion module 93 is configured to, in response to the entity in the first image including a character, determine change information of at least one of the character's actions and expressions based on the emotion; or, in response to the entity in the first image including an object, determine change information of the object's shape based on the emotion; or, in response to the entity in the first image including an object, determine change information of at least one of the environment and atmosphere of the background in the first image based on the emotion.

[0125] In some embodiments, the expansion module 93 is configured to determine the association information between the entity and the background in the first image according to the emotion; and expand the first storyline according to the emotion, the association information between the entity and the background in the first image to generate a second storyline.

[0126] In some embodiments, the first generation module 94 is configured to determine one or more story scenes and a storyline corresponding to each of the one or more story scenes based on the second storyline; and generate a second image corresponding to the story scene for the storyline corresponding to each story scene.

[0127] In some embodiments, the first generating module 94 is configured to determine the style of one or more second images according to the second storyline and the emotions contained in at least one of the first images; and generate one or more second images according to the style of the one or more second images.

[0128] In some embodiments, the second generation module 95 is configured to determine subtitles for each of the first image and one or more second images, wherein the subtitles for each image are generated based on a storyline corresponding to the image; and generate first multimedia content based on the first image and the one or more second images, wherein each image in the first multimedia content has subtitles.

[0129] In some embodiments, the first multimedia content is an image, wherein the first multimedia content includes multiple images that can be switched for display, or an image formed by splicing multiple images; or the first multimedia content is a video.

[0130] In some embodiments, in response to the emotional intensity of the second storyline being greater than a first specified threshold, the first multimedia content is a video; in response to the emotional intensity of the second storyline being not greater than the first specified threshold, the first multimedia content is an image.

[0131] In some embodiments, in response to an image ratio between the first image and the one or more second images being within a specified range, the first multimedia content includes an image formed by stitching the multiple images.

[0132] In some embodiments, the second generation module 95 is configured to determine the influence of the specified cropping ratio on the entities in the first image and one or more second images; in response to the influence being less than a second specified threshold, the image formed by splicing the first image and the one or more second images is determined as the first multimedia content; in response to the influence being not less than the second specified threshold, the switchable first image and the one or more second images are determined as the first multimedia content.

[0133] In some embodiments, the content generating device 9 is also configured to obtain a first image captured by the first user in response to the first user triggering a shooting control on the shooting interface; or to obtain a first image selected by the first user from the image library in response to the first user triggering an image selection control on the shooting interface.

[0134] In some embodiments, the content generating device 9 is further configured to display the first multimedia content on the preview interface in response to the generation of the first multimedia content.

[0135] In some embodiments, the content generating device 9 is further configured to generate a third storyline based on a third image from a second user and the first multimedia content; generate one or more fourth images based on the third storyline; and generate second multimedia content based on the first multimedia content, the third image, and one or more fourth images.

[0136] In some embodiments, the content generating device 9 is further configured to determine entities and background in a third image from a second user; understand the third image and determine a fourth storyline expressed by the third image; and expand the fourth storyline and at least one of the second storyline based on the entities and background in the third image, the entities and background in the first multimedia content to generate a third storyline.

[0137] In the above embodiment, first multimedia content with a corresponding storyline can be automatically generated based on the first image of the first user. This content generation method can automatically generate a storyline based on the image, and then automatically generate multimedia content based on the storyline. Therefore, this content generation method is more intelligent and generates richer content, which helps to increase user interaction enthusiasm and thus improve application usage.

[0138] Figure 10 1 shows a block diagram of an electronic device according to some embodiments of the present disclosure. Figure 10 As shown, the electronic device 10 includes a memory 101; and a processor 102 coupled to the memory 101, wherein the processor 102 is configured to execute a method as described in any of the aforementioned embodiments based on instructions stored in the memory 101. This enables automatic generation of first multimedia content with a corresponding storyline based on a first image of a first user. This content generation method can automatically generate a storyline based on an image, and then automatically generate multimedia content based on the storyline. Therefore, this content generation method is more intelligent and generates richer content, which helps to increase user interaction enthusiasm and thus improve application usage.

[0139] Memory 101 is used to store one or more computer-readable instructions. Memory 101 may include any combination of various forms of computer-readable storage media, such as volatile and / or non-volatile memory, including but not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Memory 101 may store, for example, an operating system, applications, a boot loader, databases, and other programs, as well as various applications and data.

[0140] The processor 102 is configured to execute computer-readable instructions to implement the method described in any of the aforementioned embodiments. The specific implementation of each step of the method can be found in the aforementioned embodiments, and the repeated parts are not repeated here.

[0141] The processor 102 may be configured to execute Figures 1 to 8The processor 102 may be embodied as various processing devices, such as a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The central processing unit (CPU) may be of X86 or ARM architecture, etc.

[0142] The processor 102 and the memory 101 can communicate with each other directly or indirectly. For example, the processor 102 and the memory 101 can communicate with each other via a network. The network can include a wireless network, a wired network, and / or any combination of wireless networks and wired networks. The processor 102 and the memory 101 can also communicate with each other via a system bus, which is not limited in this disclosure.

[0143] It should be noted that Figure 10 The components of the electronic device 10 shown are merely exemplary and non-limiting. The electronic device 10 may further include other components according to actual application requirements. The processor 102 may control other components in the electronic device 10 to perform desired functions.

[0144] The electronic device 10 may be implemented by software, firmware and / or hardware, and may be integrated into a device installed with relevant application programs.

[0145] Figure 11 A block diagram of an electronic device according to some other embodiments of the present disclosure is shown.

[0146] Figure 11 The electronic device 11 shown may be a computer system with a dedicated hardware structure, which can execute corresponding functions when a relevant application program is installed.

[0147] Electronic devices include, but are not limited to, mobile terminals such as smartphones, laptops, personal digital assistants (PDAs), tablet personal computers (Tablet PCs), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable devices, etc., as well as fixed terminals such as digital televisions and desktop computers, etc.

[0148] like Figure 11As shown, the central processing unit (CPU) 111 executes various processes according to the program stored in the read-only memory (ROM) 112 or the program loaded from the storage part 118 to the random access memory (RAM) 113. In the RAM 113, data required when the CPU 111 executes various processes is stored as needed. The central processing unit is only an example, and it can also be other types of processors, such as the various processors described above. The ROM 112, RAM 113 and the storage part 118 can be various forms of computer-readable storage media. It should be noted that although Figure 11 ROM 112, RAM 113 and storage portion 118 are shown separately in FIG, but one or more of them may be combined or located in the same or different memory or storage modules.

[0149] The CPU 111, the ROM 112, and the RAM 113 are connected to one another via a bus 114. To the bus 114, an input / output interface 115 is also connected.

[0150] The following components are connected to the input / output interface 115: an input portion 116 such as a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output portion 117 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage portion 118 including a hard disk, a magnetic tape, etc.; and a communication portion 119 including a network interface card such as a LAN card, a modem, etc. The communication portion 119 allows communication processing to be performed via a network such as the Internet. It is easy to understand that although Figure 11 The parts of the electronic device 11 are shown to communicate via the bus 114, but they can also communicate via a network or other means, where the network can include a wireless network, a wired network, and / or any combination of wireless networks and wired networks.

[0151] A drive 1110 is also connected to the input / output interface 115 as needed. A removable medium 1111 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 1110 as needed so that a computer program read therefrom is installed in the storage section 118 as needed.

[0152] When the series of processes described above is implemented by software, the program constituting the software can be installed from a network such as the Internet or a storage medium such as the removable medium 1111 .

[0153] According to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product that, when executed on a computer, causes the computer to implement the method described in any of the aforementioned embodiments. This enables automatic generation of first multimedia content with a corresponding storyline based on a first image of a first user. This content generation method can automatically generate a storyline based on an image, and then automatically generate multimedia content based on the storyline. Therefore, this content generation method is more intelligent and generates richer content, helping to increase user interaction enthusiasm and thus improve application usage. The computer program product includes computer instructions carried on a computer-readable medium, containing program code for executing the method illustrated in the flowchart. In such embodiments, the computer instructions can be downloaded and installed from a network via the communication unit 119, or installed from the storage unit 118, or installed from the ROM 112. When the computer program is executed by the CPU 111, the method of the embodiments of the present disclosure is performed.

[0154] It should be noted that, in the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, apparatus, or device or for use in conjunction with an instruction execution system, apparatus, or device.

[0155] The computer readable medium may be a computer readable storage medium, or a computer readable signal medium, or any combination of the two.

[0156] Computer-readable storage media include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method described in any of the aforementioned embodiments. This enables automatic generation of first multimedia content with a corresponding storyline based on a first image of a first user. This content generation method can automatically generate a storyline based on an image, and then automatically generate multimedia content based on the storyline. Therefore, this content generation method is more intelligent and generates richer content, helping to increase user interaction enthusiasm and thus improve application usage.

[0157] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0158] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0159] In some embodiments, a computer program is further provided, comprising: instructions, which, when executed by a processor, cause the processor to perform the method described in any of the aforementioned embodiments. For example, the instructions may be embodied as computer program codes.

[0160] In embodiments of the present disclosure, computer program code for performing the operations of the present disclosure can be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In situations involving a remote computer, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0161] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0162] The functions described above may be performed at least in part by one or more hardware logic components. For example, and without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0163] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art will appreciate that the above examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Those skilled in the art will appreciate that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A content generation method, comprising: determining an entity and a background in a first image from a first user; Understanding the first image and determining a first storyline expressed by the first image; Expanding the first storyline based on the entity and the background in the first image to generate a second storyline includes: determining an emotion contained in the first image based on the entity and the background in the first image, and expanding the first storyline based on the emotion contained in the first image, the entity and the background in the first image to generate a second storyline; generating one or more second images based on the second storyline; First multimedia content is generated based on the first image and the one or more second images.

2. The content generation method according to claim 1, wherein: The step of expanding the first storyline according to the entity and the background in the first image to generate a second storyline includes: Determining extended information based on the entity and the background in the first image and the first storyline, wherein the extended information is used to represent at least one of change information of the entity and the background in the first image and association information of the entity and the background in the first image; The first storyline is expanded using the extended information to generate a second storyline.

3. The content generation method according to claim 1, wherein: The step of expanding the first storyline according to the emotion contained in the first image, the entity in the first image, and the background to generate a second storyline includes: determining, based on the emotion, change information of the entity and the background in the first image; The first storyline is expanded according to the emotion, the entity in the first image, and the change information of the background to generate a second storyline.

4. The content generation method according to claim 3, wherein: Determining, based on the emotion, change information of the entity and the background in the first image includes: In response to the entity in the first image including a character, determining change information of at least one of an action and an expression of the character according to the emotion; or In response to the entity in the first image including an object, determining information about a change in the shape of the object according to the emotion; or Determine change information of at least one of the environment and atmosphere of the background in the first image according to the emotion.

5. The content generation method according to claim 1, wherein: The step of expanding the first storyline according to the emotion contained in the first image, the entity in the first image, and the background to generate a second storyline includes: determining, based on the emotion, association information between the entity and the background in the first image; The first storyline is expanded according to the emotion, the entity in the first image, and the associated information of the background to generate a second storyline.

6. The content generation method according to any one of claims 1 to 5, wherein: Generating one or more second images according to the second storyline includes: determining, based on the second storyline, one or more story scenes and a storyline corresponding to each of the one or more story scenes; For the storyline corresponding to each story scene, a second image corresponding to the story scene is generated.

7. The content generation method according to any one of claims 1 to 5, wherein: Generating one or more second images according to the second storyline includes: determining a style of one or more second images based on emotions contained in at least one of the second storyline and the first image; The one or more second images are generated according to the style of the one or more second images.

8. The content generation method according to any one of claims 1 to 5, wherein: Generating first multimedia content according to the first image and the one or more second images includes: determining a subtitle for each of the first image and the one or more second images, wherein the subtitle for each image is generated based on a storyline corresponding to the image; First multimedia content is generated according to the first image and the one or more second images, wherein each image in the first multimedia content has a subtitle.

9. The content generation method according to any one of claims 1 to 5, wherein: The first multimedia content is an image, wherein the first multimedia content includes a plurality of images that can be switched for display, or an image formed by splicing a plurality of images; or The first multimedia content is a video.

10. The content generation method according to claim 9, wherein: In response to the emotional intensity of the second storyline being greater than a first specified threshold, the first multimedia content is a video; In response to the emotional intensity of the second storyline being no greater than the first specified threshold, the first multimedia content is an image.

11. The content generation method according to claim 9, wherein: In response to an image ratio between the first image and the one or more second images being within a specified range, the first multimedia content includes an image formed by splicing a plurality of images.

12. The content generation method according to claim 9, wherein: Generating first multimedia content according to the first image and the one or more second images includes: determining an effect of a specified cropping ratio on entities in the first image and the one or more second images; In response to the influence being less than a second specified threshold, determining an image formed by splicing the first image and the one or more second images as the first multimedia content; In response to the influence degree being not less than the second specified threshold, the first image and the one or more second images that can be switchably displayed are determined as the first multimedia content.

13. The content generation method according to any one of claims 1 to 5, further comprising: In response to the first user triggering a shooting control on a shooting interface, acquiring the first image shot by the first user; or In response to the first user triggering the image selection control on the shooting interface, the first image selected by the first user from the image library is acquired.

14. The content generation method according to any one of claims 1 to 5, further comprising: In response to the generation of the first multimedia content, the first multimedia content is displayed on a preview interface.

15. The content generation method according to any one of claims 1 to 5, further comprising: generating a third storyline based on a third image from a second user and the first multimedia content; generating one or more fourth images based on the third storyline; Second multimedia content is generated based on the first multimedia content, the third image, and the one or more fourth images.

16. The content generation method according to claim 15, wherein: Generating a third storyline according to the third image from the second user and the first multimedia content includes: determining an entity and a background in a third image from a second user; understanding the third image and determining a fourth storyline expressed by the third image; At least one of the fourth storyline and the second storyline is expanded according to the entity and the background in the third image and the entity and the background in the first multimedia content to generate a third storyline.

17. An electronic device comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the content generating method according to any one of claims 1 to 16 based on instructions stored in the memory. 18 . A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the processor is enabled to implement the content generation method according to claim 1 .

19. A computer program product, which, when executed on a computer, enables the computer to implement the content generation method according to any one of claims 1 to 16.

Images