Automatically generate visual content

Abstract

The present disclosure provides a method and apparatus for automatically generating visual content. A text sentence may be obtained. Structured data corresponding to the text sentence may be generated. A template may be selected from a template library based on the structured data. Configuration information of the template may be generated using at least the structured data. A material may be selected from a material library based on the configuration information. Visual content may be generated by filling the template with the selected material.

Description

Background Art

[0001] Corresponding visual content can be created based on text content to enhance intuitiveness, readability, fun, etc. Visual content is a visual presentation of text content, which can be in the form of images, animations, etc. For example, storyboards are a common visual content. Generally, visual content can be combined with associated audio to generate multimedia content. For example, a dynamic picture book is an exemplary multimedia content that presents stories in the form of images, animations, sounds, etc. It can also be called an animated picture book, a visual storybook, a dynamic audio book, an animated picture book, etc. The dynamic picture book may include a series of storyboards, which are visual presentations of multiple sentences in a text story. The dynamic picture book may also include a series of audios generated based on the sentences in the text story, which are played together with the corresponding storyboards. Compared with text stories, dynamic picture books can simplify the understanding of text stories and express the story content more vividly through visual aids. Summary of the invention

[0002] This summary is provided to introduce a set of concepts that will be further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] Embodiments of the present disclosure provide a method and apparatus for automatically generating visual content. A text sentence may be obtained. Structured data corresponding to the text sentence may be generated. A template may be selected from a template library based on the structured data. Configuration information of the template may be generated using at least the structured data. A material may be selected from a material library based on the configuration information. Visual content may be generated by filling the template with the selected material.

[0004] It should be noted that one or more of the above aspects include the features specifically pointed out in the following detailed description and claims. The following description and drawings set forth in detail certain illustrative features of the one or more aspects. These features are merely indicative of the various ways in which the principles of various aspects may be implemented, and the present disclosure is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The disclosed aspects will be described below in conjunction with the accompanying drawings, which are provided to illustrate rather than limit the disclosed aspects.

[0006] Figure 1 An exemplary process for automatically generating multimedia content according to an embodiment is shown.

[0007] Figure 2An example of visual content automatically generated based on text content according to an embodiment is shown.

[0008] Figure 3 An exemplary process for automatically generating visual content according to an embodiment is shown.

[0009] Figure 4 An exemplary template grid is shown in accordance with an embodiment.

[0010] Figure 5 Exemplary physical dimensions are shown according to embodiments.

[0011] Figure 6 Exemplary entity locations are shown according to an embodiment.

[0012] Figure 7 An exemplary depth of field is shown according to an embodiment.

[0013] Figure 8 An exemplary depth of field is shown according to an embodiment.

[0014] Fig. 9 A visual representation of an exemplary customized template is shown in accordance with an embodiment.

[0015] Fig.10 An exemplary entity image library according to an embodiment is shown.

[0016] Fig.11 An exemplary entity component image library is shown in accordance with an embodiment.

[0017] Fig.12 An exemplary background image library according to an embodiment is shown.

[0018] Fig.13 A flow chart of an exemplary method for automatically generating visual content according to an embodiment is shown.

[0019] Fig.14 An exemplary apparatus for automatically generating visual content according to an embodiment is shown.

[0020] Fig.15 An exemplary apparatus for automatically generating visual content according to an embodiment is shown. DETAILED DESCRIPTION

[0021] The present disclosure will now be discussed with reference to various exemplary embodiments. It should be understood that the discussion of these embodiments is only for enabling those skilled in the art to better understand and thereby implement the embodiments of the present disclosure, and does not teach any limitation on the scope of the present disclosure.

[0022] Visual content is usually created manually. For example, for a given text content, it can first be composed manually, frame by frame. The composition of each frame can include artificially designed physical size, physical position, depth of field, etc. Then, according to the composition, materials can be added to the picture manually, including, for example, physical images, background images, animations, etc., in order to form the final visual content. Each link in the above-mentioned visual content creation process is completed manually, so it will be time-consuming and inefficient.

[0023] The embodiment of the present disclosure proposes to utilize the template library and material library prepared in advance to automatically generate visual content based on text content, and then generate multimedia content. In this article, visual content can include various types of visual presentation, for example, storyboard sequence, image sequence, dynamic image sequence, animation sequence, etc. The template library includes multiple candidate templates prepared in advance. In this article, template can refer to a set of presentation rules to be adopted in the picture of visual content. The material library includes multiple pre-prepared materials. In this article, material can refer to the visual elements contained in the picture of visual content. In the process of automatically generating visual content, matching templates can be selected from the template library based on text content, and materials can be selected from the material library according to the selected template and filled into the template to form visual content. Through the embodiment of the present disclosure, visual content can be created quickly and efficiently in an automatic manner. Even for users who lack painting or animation skills, visual content can be easily created. The created visual content can be further combined with the audio corresponding to the text content to form multimedia content.

[0024] The embodiments of the present disclosure can be applied to various scenarios of generating multimedia content based on text content, for example, automatically generating dynamic picture books based on text stories, automatically generating multimedia educational courseware based on text teaching materials, automatically generating multimedia news based on text news, automatically generating animated advertisements based on text advertising words, etc. The following discussion mainly takes the automatic generation of dynamic picture books based on the text story as an example to illustrate the exemplary process of automatically generating visual content and multimedia content according to the embodiments of the present disclosure. However, it should be understood that the embodiments of the present disclosure can also be applied to any other scenarios through similar processes and methods.

[0025] Figure 1An exemplary process 100 for automatically generating multimedia content according to an embodiment is shown. Process 100 can be executed to generate corresponding multimedia content for a text sentence in text content. The text content can be a collection of multiple text sentences, such as a text story. Each text sentence can be a semantically complete statement identified or extracted from the text content, and the statement is suitable for presentation through a picture. The text sentence may include language elements for describing at least one of, for example, time, place, person, animal, object, action, etc. It should be understood that in this article, the text sentence may have the same or different division method as the grammatical sentence. Grammatical sentences are usually grammatically divided by sentence-ending punctuation such as a period, while a text sentence in this article may be a grammatical sentence, a part of a grammatical sentence, a combination of more than one grammatical sentence, and the like.

[0026] Assume that process 100 is performed to generate multimedia content for text sentences 110 in text content.

[0027] At 120, the visual content automatic generation process according to the present disclosure embodiment may be performed based on the text sentence 110 to generate visual content 130 corresponding to the text sentence 110. The visual content 130 may include a screen corresponding to the text sentence 110, which may include an image, an animation, and the like.

[0028] At 140, text-to-speech synthesis may be performed on the text sentence 110 to generate audio 150 corresponding to the text sentence 110. Any speech synthesis technique for generating speech from text may be employed at 140.

[0029] The multimedia content 160 corresponding to the text sentence 110 can be generated by combining the visual content 130 and the audio 150. In the process of generating the multimedia content 160, the visual content 130 and the audio 150 can be temporally aligned. For example, the same or corresponding timestamp t can be set for the visual content 130 corresponding to the text sentence 110 and the audio 150 corresponding to the text sentence 110. Thus, when generating the multimedia content 160, the visual content 130 and the audio 150 can be aligned based on the timestamp t. Accordingly, the corresponding audio 150 can be played during the presentation of the visual content 130. It should be understood that the temporal alignment between the visual content and the audio can have a wide range of meanings, for example, the duration of the visual content falls within the duration of the audio, the duration of the audio falls within the duration of the visual content, the start time of the visual content is a predetermined amount of time earlier than the start time of the audio, the start time of the audio is a predetermined amount of time earlier than the start time of the visual content, and the like.

[0030] It should be understood that process 100 can be performed for each text sentence in the text content. By performing process 100 for all text sentences in the text content, a series of multimedia content corresponding to the entire text content can be obtained. Thus, when the text content is, for example, a text story, a dynamic picture book corresponding to the text story can be obtained.

[0031] Figure 2 An example of visual content automatically generated based on text content according to an embodiment is shown.

[0032] Assume that the text content is a text story and includes text sentence 212 "There is half a watermelon on the table", text sentence 214 "Tom said: Let's eat it", text sentence 216 "Mary said: How can we eat it without a knife?", etc. According to the automatic generation process of visual content proposed in the embodiment of the present disclosure, visual contents 222, 224 and 226 corresponding to the text sentences 212, 214 and 216 can be automatically generated. Each visual content can also be called a storyboard.

[0033] Visual content 222 includes: an image about a table, which corresponds to “table” in text sentence 212 ; and an image about half a watermelon, which corresponds to “half a watermelon” in text sentence 212 .

[0034] Visual content 224 includes: a boy image corresponding to "Tom" in text sentence 214; a dialog image in which the words of the character Tom in text sentence 214 are displayed; and a background image.

[0035] Visual content 226 includes: a girl image corresponding to "Mary" in text sentence 216; a dialog image in which the words of the character Mary in text sentence 216 are displayed; and a background image.

[0036] The visual contents 222 , 224 , and 226 present the text sentences 212 , 214 , and 216 in the text contents in a visual manner.

[0037] It should be understood that although not Figure 2 As shown in FIG. 2 , audios corresponding to text sentences 212, 214, and 216 may also be generated. The audio corresponding to text sentence 212 may be combined with visual content 222 to form a first multimedia content, the audio corresponding to text sentence 214 may be combined with visual content 224 to form a second multimedia content, and the audio corresponding to text sentence 216 may be combined with visual content 226 to form a third multimedia content. The first multimedia content, the second multimedia content, and the third multimedia content may together constitute a dynamic picture book.

[0038] Figure 3An exemplary process 300 for automatically generating visual content is shown in accordance with an embodiment.

[0039] According to process 300, a text sentence 310 may be obtained. For example, text sentence 310 may be extracted from text content.

[0040] According to the process 300, the text sentence 310 can be analyzed to generate structured data 320 corresponding to the text sentence 310. The structured data 320 may include various language information identified and extracted from the text sentence 310, such as one or more of scene, weather, time, number of entities, entity information of each entity, etc. The scene may refer to the place described in the text sentence 310, such as grassland, street, park, etc. The weather may include, for example, rain, sunny, windy, etc. The time may include, for example, daytime, night, etc. The text sentence 310 may also describe entities, such as people, anthropomorphic animals, primitive animals, objects, etc. The number of entities may refer to the number of entities included in the text sentence 310. The entity information may refer to the description of the entity in the text sentence 310. The entity information of an entity may include at least one of the following: entity type, entity name, entity gender, entity age, entity emotion, entity color, entity action, etc. The entity type may refer to the type to which the entity belongs, such as people, anthropomorphic animals, primitive animals, objects, etc. The entity name may refer to the name of the entity in the text sentence 310, for example, Tom, Mary, Grandpa, teacher, policeman, etc. The entity gender may include, for example, male, female, etc. It should be understood that the entity gender may also be inferred from the entity name. For example, if the entity name of an entity is "Tom", then since the name is usually a male name, it can be inferred that the gender of the entity is male. The entity age may include adults, children, etc. It should be understood that the entity age may also be inferred from the entity name. For example, if the entity name of an entity is "Grandpa", it can be inferred that the age of the entity is an adult. The entity emotion may be a description of the emotion of the entity, for example, happy, sad, etc. The entity color may refer to the color described in association with the entity, for example, from the sentence "Mary is wearing a red skirt", it can be detected that the entity color of the entity "Mary" is "red". The entity action may refer to the action performed by the entity, for example, walking, jumping, running, talking, etc. It should be understood that the information included in the structured data 320 listed above is only exemplary, and more or less information may be included in the structured data 320 according to specific application scenarios and requirements. In one embodiment, the structured data 320 may be extracted from the text sentence 310 by any pre-trained natural language processing model. The natural language processing model may take a text sentence as input and output structured data corresponding to the text sentence.

[0041] Taking the text sentence "It was raining on the grassland, Grandpa looked at Mary, and Mary smiled happily" as an example, the extracted structured data can be represented as: {<scene="grassland">, <weather="raining">, <time="default">, <number of entities="2">, entity 1<type="person", name="Grandpa", gender="male", age="adult", emotion="default", color="default", action="default">, entity 2<type="person", name="Mary", gender="female", age="child", emotion="happy", color="default", action="smile">}. It should be understood that "default" in the above structured data can indicate that no information that explicitly corresponds to the item is extracted from the text sentence, so that any pre-defined default information can be used.

[0042] According to process 300, a template 340 can be selected from a template library 330 based on structured data 320. The template library 330 may include a plurality of pre-prepared candidate templates. Each candidate template may include a set of rendering rules to be adopted in the picture of the visual content. The set of rendering rules may include a composition configuration defined by the candidate template. The composition configuration may specify, for example, the rendering and arrangement rules of entities. For example, the composition configuration may include at least one of the following: entity position, entity size, entity orientation, entity order, depth of field type, framing position, etc. The plurality of candidate templates in the template library 330 may correspond to different composition configurations, respectively, so as to be applicable to different picture rendering requirements. The rendering rules defined by a template may be used as tags or keywords to mark the template.

[0043] In one embodiment, the selection of the template 340 may include first extracting composition reference information from the structured data 320. The composition reference information may refer to information that can be referenced in the process of determining the composition configuration of the template or information associated with the composition configuration. For example, the composition reference information may include at least one of the following: the number of entities, the type of entity, the age of the entity, the action of the entity, etc. Specific composition reference information or a combination thereof may correspond to the composition configuration of a specific template. Then, the composition reference information may be used to select a template 340 from the template library 330, which template 340 may have a composition configuration that matches the composition reference information extracted from the structured data 320. For example, a matching template 340 may be retrieved from the template library 330 by comparing or matching the composition reference information with tags or keywords of the candidate templates.

[0044] For ease of understanding, Figure 3, an exemplary visual presentation of template 340 is shown in FIG. 340 , which visually displays a portion of the presentation rules defined in template 340, such as a portion of the composition configuration. Block 342 represents the composition area defined by template 340, and blocks 344 and 346 represent two entities involved in template 340. The sizes of blocks 344 and 346 correspond to the sizes of the two entities. As shown in the figure, the size of block 344 is larger than the size of block 346, which may indicate that the entity corresponding to block 344 is an adult, while the entity corresponding to block 345 is a child. The positions of blocks 344 and 346 in block 342 correspond to the positions of the two entities in the composition area. In addition, template 340 may also define other presentation rules that are not suitable for visualization, such as the direction, order, etc., in which the entity corresponding to block 344 and the entity corresponding to block 346 will appear in the picture. In addition, although not shown, module 340 may also define presentation rules about depth of field type, framing position, etc.

[0045] In one embodiment, optionally, the template library 330 may also include customized candidate templates. The customized candidate template may be, for example, a template corresponding to a specific entity action. Taking the entity action "speaking" as an example, a corresponding customized template may be prepared in advance, and the customized template defines a composition configuration adapted to presenting a scene of an entity speaking, for example, the entity is located in the left third of the scene, and a speech box is set in the upper part of the right third of the scene, etc. In this case, the selection of the template 340 may include first extracting the entity action from the structured data 320, and then selecting a customized template corresponding to the entity action from the template library 330.

[0046] According to process 300, after template 340 is selected, configuration information 350 of the template 340 can be generated. The configuration information of the template can widely include the presentation scheme defined by the template for a specific text sentence. The configuration information 350 can be generated at least using structured data 320. For example, the configuration information 350 can include information about structured data 320 to guide the selection and filling of subsequent materials. The process of generating configuration information 350 using structured data 320 can also be regarded as a process of initializing the template 340 using information in the text sentence 310. Optionally, the configuration information 350 can also include composition configuration information for describing the composition configuration of the template 340. It should be understood that since the composition configuration is pre-defined for the template and is not specific to a certain text sentence, the composition configuration information may not be included in the configuration information 350, so that the composition configuration information 350 specifically contains information related to the text sentence.

[0047] According to process 300 , asset 370 may be selected from asset library 360 based on configuration information 350 .

[0048] The material library 360 may include a plurality of candidate materials prepared in advance. The material may be, for example, an image, an animation, etc. that can be presented in the screen of the visual content. The candidate materials in the material library 360 may be static materials or dynamic materials. Static materials may be, for example, images, etc., while dynamic materials may be, for example, animations, etc. Optionally, the material library 360 may include a foreground image library. The foreground image library may include static images associated with entities. For example, the foreground image library may include an entity image library and / or an entity component image library. The entity image library is a collection of images corresponding to various entities. The entity component image library is a collection of images corresponding to different components of various types of entities. One or more entity component images may be used to form an entity image. That is, an image of an entity may be composed of images of multiple components of the entity. For example, an image of a person may be composed of a facial image, a head shape image, a body image, a clothing image, a shoe image, etc. Optionally, the material library 360 may include a background image library. The background image library may include static images associated with scenes, weather, time, etc., for example, images about grasslands, images about streets, images about rain, images about night, etc. Optionally, the material library 360 may include an animation library. The animation library may include a plurality of predefined animations as dynamic materials. The animation may include, for example, zooming, shaking, flashing, left fade-in, right fade-out, etc. Each animation is defined by animation parameters. The animation parameters may include at least one of the following: position, time, direction, frequency, etc.

[0049] As described above, the configuration information 350 may include structured data information, and therefore, the corresponding material 370 may be selected from the material library 360 based at least on the entries in the structured data information. For example, for the scene "grassland" in the structured data information, an image about grassland may be selected from the background image library in the material library 360. For example, for the entity <type="person", name="Mary", gender="female", age="child", emotion="happy", color="default", action="laugh"> in the structured data information, an image about a little girl may be selected from the foreground image library in the material library 360, and an animation for the entity action "laugh" may be selected from the animation library in the material library 360.

[0050] According to the process 300, visual content 380 may be generated by filling the template 340 with the selected material 370. In one embodiment, the material 370 may be filled into the template 340 according to the composition configuration of the template 340. Taking the entity position in the composition configuration as an example, assuming that entity position 1 and entity position 2 are respectively specified for entity 1 and entity 2 in the composition configuration, the material corresponding to entity 1 may be filled into entity position 1, and the material corresponding to entity 2 may be filled into entity position 2. In addition, it should be understood that multiple materials may be stacked in the template in, for example, different layers to form a final visual content screen.

[0051] For ease of understanding, Figure 3 An exemplary presentation screen of visual content 380 is shown in , wherein block 382 defines the presentation area of ​​the screen. Assuming the text sentence is "It is raining on the grassland, Grandpa looks at Mary, and Mary smiles happily", the screen includes the background image "grassland" stacked with the background image "raining", character entity 384 "Grandpa", character entity 386 "Mary", a smiling face expressing the emotion "happy" and action "smile" of character entity 386, and so on. Although not shown, the screen may also include animations for presenting the entity action "smile", such as shaking, etc. Figure 3 As can be seen in FIG, the addition of materials in visual content 380 follows the composition configuration of template 340. For example, character entity 384 has the size specified by block 344 and is filled to the position specified by block 344, character entity 386 has the size specified by block 346 and is filled to the position specified by block 346, and so on.

[0052] It should be understood that Figure 3 The process 300 is merely exemplary, and the process 300 may be modified in any manner according to specific application scenarios and requirements.

[0053] Various aspects of the patterning configuration of the template according to embodiments of the present disclosure will be further discussed below.

[0054] As mentioned above, the composition configuration of the template can specify, for example, physical position, physical size, framing position, etc. In order to facilitate the specification of these parameters related to position or size, various types of grids can be defined in the template. The grid is implemented by dividing the composition area of ​​the template using grid lines, and the grid and / or grid lines can define the position or size, etc. in a manner similar to coordinates. Figure 4 An exemplary template grid is shown according to an embodiment. Figure 4A variety of different types of grid divisions are provided in the visual presentation of the template. In one aspect, the composition area of ​​the template can be divided into multiple horizontal grids by horizontal grid lines. For example, in view 412, the composition area is evenly divided into 9 grids by 8 horizontal grid lines. In view 414, the composition area is evenly divided into 3 grids by 2 horizontal grid lines. It should be understood that the disclosed embodiment can also use any other number of horizontal grid lines to divide the composition area into any other number of horizontal grids. In one aspect, the composition area of ​​the template can be divided into multiple vertical grids by vertical grid lines. For example, in view 422, the composition area is evenly divided into 2 grids by 1 vertical grid line. In view 424, the composition area is evenly divided into 3 grids by 2 vertical grid lines. In view 426, the composition area is evenly divided into 4 grids by 3 vertical grid lines. In view 428, the composition area is evenly divided into 5 grids by 4 vertical grid lines. It should be understood that the embodiments of the present disclosure may also use any other number of vertical grid lines to divide the composition area into any other number of vertical grids. In one aspect, the composition area of ​​the template may be divided into a plurality of cross grids by cross grid lines. For example, in view 432, the composition area is evenly divided into 4 grids by 2 cross grid lines. In view 434, the composition area is evenly divided into 9 grids by 4 cross grid lines. It should be understood that the embodiments of the present disclosure may also use any other number of cross grid lines to divide the composition area into any other number of cross grids. In addition, it should be understood that the composition area of ​​a template may also be divided into grids by a combination of the above-mentioned different methods. For example, the methods of view 412 and view 424 may be combined so as to divide the composition area of ​​a template into 9 grids in the horizontal direction and, at the same time, divide the composition area of ​​the template into 3 grids in the vertical direction.

[0055] The composition configuration of the template may specify entity size. Entity size may be associated with one or more of, for example, entity type, entity name, entity age, etc. For entities of the same entity type, different relative sizes may be defined based on entity name, entity age, etc. For example, if an entity is of type "person" and named "grandfather", the entity may have an "adult" size. If an entity is of type "person" and named "child", the entity may have a "child" size, which is smaller than the "adult" size. For example, if an entity is of type "primitive animal" and named "tiger", and another entity is of type "primitive animal" and named "pig", the size of the primitive animal "tiger" may be larger than the size of the primitive animal "pig". In addition, different relative sizes may also be defined between entities of different entity types. For example, if the entity type is "anthropomorphic animal", the entity name is "cat", and the entity age is "adult", the size of the anthropomorphic animal "cat" may be defined with reference to the size of "person", for example, the size of the anthropomorphic animal "cat" may correspond to the size of "child" in the entity type "person". For example, if the entity type is "primitive animal" and the entity name is "cat", the size of the primitive animal "cat" can be defined with reference to the size of the anthropomorphic animal "cat", for example, the size of the primitive animal "cat" can be half the size of the anthropomorphic animal "cat".

[0056] Figure 5 Exemplary physical dimensions according to embodiments are shown. Figure 5 The sizes defined for different entities are shown in the example where the presentation area of ​​the template is divided into 9 horizontal grids. It should be understood that the entity sizes can also be defined in any other grid division method. Entity 502 is an adult, and its size can be defined as 8 horizontal grid heights. Entity 504 is a child, and its size can be defined as 7 horizontal grid heights, which is smaller than the size of an adult. Entity 506 is an anthropomorphic adult animal, and its size can be defined as 7 horizontal grid heights, which is equal to the size of a child. Entity 508 is an anthropomorphic infant animal, and its size can be defined as 6 horizontal grid heights, which is smaller than the anthropomorphic adult animal. Entities 510, 512, 514, and 516 are the original animals "elephant", "tiger", "pig", and "rabbit", respectively. Different horizontal grid heights can be defined for these entities according to the relative sizes of these original animals. Entity 518 is an object "table", and its size can be defined as 4 horizontal grid heights, which is smaller than the size of a child. Entity 520 is an object "football", and its size can be defined as 2 horizontal grid heights. It should be understood that although Figure 5 In the , the height of the entity is defined by the number of horizontal grids and thus the entity size, but the width of the entity can also be defined by the number of vertical grids and thus the entity size. In addition, Figure 5All entity sizes in the template are exemplary only, and sizes for different entities may be defined in any other manner in the template.

[0057] The composition configuration of the template may specify the entity position. The entity position may be associated with one or more of, for example, the number of entities, the age of the entity, etc. For example, different numbers of entities may determine different horizontal positions of the entities in the composition area, different ages of entities may determine different vertical positions of the entities in the composition area, etc. In one embodiment, the entity position may be defined using horizontal grid lines and / or vertical grid lines.

[0058] Figure 6 : shows an exemplary entity position according to an embodiment. Assume that two entities 612 and 614 are included in the composition area 610, wherein the entity 612 may be an adult and the entity 614 may be a child. In the case where the composition area is divided into an exemplary 9 horizontal grids, the upper edge of the entity 612 may be located at the 1st horizontal grid line and the lower edge may be located at the 8th horizontal grid line, that is, the horizontal position of the entity 612 is between the upper 1 / 9 part of the composition area and the lower 1 / 9 part of the composition area. The upper edge of the entity 614 may be located at the 2nd horizontal grid line and the lower edge may be located at the 8th horizontal grid line, that is, the horizontal position of the entity 612 is between the upper 2 / 9 part of the composition area and the lower 1 / 9 part of the composition area. In addition, in the case where the composition area is divided into an exemplary 3 vertical grids, the midline of the entity 612 may be located on the 1st vertical grid line on the left, that is, the vertical position of the entity 612 is at the left 1 / 3 of the composition area. The center line of entity 614 may be located on the first vertical grid line on the right side, that is, the vertical position of entity 614 is at the right 1 / 3 of the composition area.

[0059] The composition configuration of the template can specify the entity orientation. In this article, the entity orientation can refer to the direction of the face of the entity, for example, half face to the left, half face to the left, half face to the right, right face, front face, etc. The entity orientation can be associated with, for example, the number of entities. For example, when there is only one entity, the entity can be specified to have a predetermined entity orientation, such as half face to the left. When there are two entities, the entity orientation of the left entity can be specified as, for example, half face to the left, while the entity orientation of the right entity can be specified as, for example, half face to the right. When there are three entities, the entity orientation of the two entities on the left can be specified as, for example, half face to the left, while the entity orientation of the right entity can be specified as, for example, half face to the right. It should be understood that the above entity orientations are all exemplary, and any other type of entity orientation can be defined in the template for different numbers of entities.

[0060] The composition configuration of the template may specify the entity order. In this article, the entity order may refer to the time order in which two or more entities are presented in the picture. The entity order may be associated with, for example, the order in which different entities appear in a text sentence. For example, assuming the text sentence is "Grandpa stood by the river, and Mary came over", in which the entity "Grandpa" is first described, and then the entity "Mary" is described. Therefore, in the composition configuration, it may be specified that the entity "Grandpa" is presented first, and then the entity "Mary" is presented.

[0061] The composition configuration of the template may specify the depth type and the framing position. The depth type may include different framing methods, such as full body, half body, close-up, long shot, etc. Different types of depth may be achieved by defining framing frames of different positions or sizes in the composition area of ​​the template. The depth type may be associated with at least one of, for example, the number of entities, the action of the entity, etc. For example, if the number of entities is 1, the depth type "half body" may be adopted. If the number of entities is 2, the depth type "full body" may be adopted. If the number of entities is 1 and the action of the entity is "walking", the depth type "full body" may be adopted. If the number of entities is 3 and the action of the entity is "kicking a ball", the depth type "long shot" may be adopted, etc. Different depth types determine different framing positions. The framing position may broadly refer to the position or size of the framing frame, which may be associated with, for example, at least one of the depth type, the size of the entity, etc. For example, if the depth type is "half body", the position of the framing frame may be in the area of ​​the upper body of the entity, and the size of the framing frame may be suitable for enclosing the upper body of the entity. For example, if the depth of field type is "full body", the position of the framing box may be in the area of ​​the full body of the entity, and the size of the framing box may be adapted to encompass the entire entity.

[0062] Figure 7 An exemplary depth of field is shown according to an embodiment. Figure 7It is intended to illustrate an exemplary framing method for an entity type of a person, an anthropomorphic animal, etc. As an example, an entity 712 is defined in a composition area 710 of a template, and the entity 712 can be, for example, an adult. It can be determined that the depth of field type "half body" is adopted in the template, and a framing frame 714 is defined in the composition area 710. An exemplary picture 716 is generated based on the template, in which the entity is shown in a half-body depth of field. As an example, an entity 722 is defined in a composition area 720 of a template, and the entity 722 can be, for example, a child, whose size is smaller than that of the entity 712. It can be determined that the depth of field type "half body" is adopted in the template, and a framing frame 724 is defined in the composition area 720. It can be seen that the position of the framing frame 724 is slightly lower than that of the framing frame 714 so as to adapt to the size of the entity 722. An exemplary picture 726 is generated based on the template, in which the entity is shown in a half-body depth of field. As an example, two entities 732 and 734 are defined in a composition area 730 of a template, and the entities 732 and 734 may be, for example, adults. It can be determined that the depth of field type "full body" is adopted in the template, and a framing frame 736 is defined in the composition area 730. It can be seen that the size of the framing frame 730 is substantially equal to the size of the composition area 730, so as to be able to completely surround both entities 732 and 724. An exemplary picture 738 is generated based on the template, in which two entities are shown in a full body depth of field.

[0063] Figure 8 An exemplary depth of field is shown according to an embodiment. Figure 8It is intended to illustrate an exemplary framing method for an entity type of an object, a primitive animal, etc. As an example, an entity 812 is defined in a composition area 810 of a template, and the entity 812 can be, for example, an object. It can be determined that the depth of field type "close-up" is adopted in the template, and a framing frame 814 is defined in the composition area 810. The position of the framing frame 814 is in the middle and lower part of the composition area 810, and the size of the framing frame can surround the entity 812. In one embodiment, a predetermined blank area can be reserved between the framing frame 814 and the entity 812. An exemplary screen 816 is generated based on the template, in which the entity is shown with a close-up depth of field. It can be seen that in the screen 816, corresponding to the predetermined blank area reserved between the framing frame 814 and the entity 812, a predetermined blank area is also reserved between the entity and the border of the presentation area of ​​the screen 816 to achieve a better visual effect. It should be understood that the entity may have various shapes and sizes. In this case, the viewfinder may preferentially reserve sufficient predetermined blank areas in the dimension direction of the entity with a larger size. As an example, an entity 822 is defined in a composition area 820 of a template, and the longitudinal dimension of the entity 822 is larger than the transverse dimension. A viewfinder 824 is defined in the composition area 820. The viewfinder 824 may reserve a predetermined blank area with reference to the longitudinal dimension of the entity 822. An exemplary screen 826 is generated based on the template, wherein, corresponding to the predetermined blank area reserved between the viewfinder 824 and the entity 822, a predetermined blank area is also reserved between the entity and the border of the presentation area of ​​the screen 826. As an example, an entity 832 is defined in a composition area 830 of a template, and the transverse dimension of the entity 832 is larger than the longitudinal dimension. A viewfinder 834 is defined in the composition area 830. The viewfinder 834 may reserve a predetermined blank area with reference to the transverse dimension of the entity 832. The exemplary screen 836 is generated based on the template, wherein, corresponding to the predetermined blank area reserved between the viewfinder 834 and the entity 832 , a predetermined blank area is also reserved between the entity and the border of the presentation area of ​​the screen 836 .

[0064] It should be understood that the depth of field type and framing position specified for the entity in the template are also applied to the background image. For example, assuming that the background image is tiled in the composition area of ​​the template, only the part of the background image that falls within the area surrounded by the framing frame will be presented in the final visual content.

[0065] As mentioned above, the template library may include customized candidate templates. The customized candidate templates may be, for example, templates corresponding to specific entity actions. Fig. 9 A visual representation of an exemplary customized template is shown in accordance with an embodiment. Fig. 9Take the customized template corresponding to the entity action "speaking" as an example. It should be understood that the embodiments of the present disclosure may also include customized templates for any other entity actions. As an example, assuming that a text sentence involves an entity speaking, an entity 912, a speech box 914, etc. may be defined in the composition area 910, wherein the speech spoken by the entity 912 may be presented in the speech box 914. The entity 912 may be exemplarily located on the vertical midline of the template, and the entity is oriented to the left half face. The speech box 914 may be exemplarily located at the upper right of the entity 912. In addition, the framing type "half body" and the framing box 916 are also defined in the composition area 910. As an example, assuming that a text sentence involves a conversation between two entities. An entity 922, a speech box 924, etc. may be defined in the composition area 920 of the template for the entity that speaks first. The entity 922 may be exemplarily located at the left 1 / 3 of the template, and the entity is oriented to the left half face. The speech box 924 may be exemplarily located at the upper right of the entity 922. In addition, the framing type "half body" and the framing frame 926 are also defined in the composition area 920. An entity 932, a speech frame 934, etc. can be defined in the composition area 930 of the template for the entity that speaks later. The entity 932 can be exemplarily located at the right 1 / 3 of the template, and the entity is oriented to the right half of the face. The speech frame 934 can be exemplarily located at the upper left of the entity 932. In addition, the framing type "half body" and the framing frame 936 are also defined in the composition area 930.

[0066] Various aspects of the material library according to embodiments of the present disclosure will be further discussed below.

[0067] The material library may include multiple pre-prepared candidate materials. These candidate materials may be prepared based on various possible template configuration information. The template configuration information may include various information such as scene, weather, time, number of entities, entity type, entity name, entity gender, entity age, entity emotion, entity color, entity action, entity position, entity size, entity orientation, entity order, depth of field type, framing position, etc., so that materials can be prepared for each template configuration information or a combination of one or more template configuration information. The candidate material can be artificially created in advance or extracted from any type of image source.

[0068] The material library may include a foreground image library associated with an entity. The foreground image library may include, for example, an entity image library, an entity component image library, and the like. Fig.10An exemplary entity image library according to an embodiment is shown. The entity image library may be a collection of images of various entities. For example, the entity image library may include a group of images 1010 about people, a group of images 1020 about anthropomorphic animals, a group of images 1030 about primitive animals, a group of images 1040 about objects, etc. Each entity image in the entity image library may be accompanied by description information, including, for example, name, gender, age, emotion, size, etc. The description information of the entity image may be used to compare and match with the template configuration information so that the corresponding entity image may be selected based on the template configuration information. Fig.11 An exemplary entity component image library according to an embodiment is shown. An entity component image library is a collection of images corresponding to different components of various types of entities. One or more entity component images can be used to form an entity image. Fig.11 , multiple entity component images of components related to a person are shown in the figure taking the entity type "person" as an example. For example, the entity component image library may include a group of images 1110 about faces, a group of images 1120 about head shapes, a group of images 1130 about clothes, a group of images 1140 about shoes, a group of images 1150 about bodies, etc. Each entity component image in the entity component image library may be accompanied by description information, including, for example, name, color, size, gender, etc. The description information of the entity component image may be used to compare and match with the template configuration information, so that the corresponding entity component images may be selected based on the template configuration information and combined into an entity image. For example, assuming that the template configuration information describes a variety of information about the entity "Mary", such as "child", "female", "red skirt", "short hair", "red shoes", etc., then a facial image 1112 about a girl can be selected from the group of images 1110, a head shape image 1122 about short hair can be selected from the group of images 1120, a clothing image 1132 about a red skirt can be selected from the group of images 1130, a shoe image 1142 about red shoes can be selected from the group of images 1140, and a body image 1152 about a little girl can be selected from the group of images 1150. By combining the selected facial image 1112, head shape image 1122, clothing image 1132, shoe image 1142, and body image 1152, an entity image 1160 corresponding to the entity "Mary" can be formed. It should be understood that the embodiments of the present disclosure are not limited to Fig.10 and Fig.11 The entity images and entity component images shown may also include any other entity images and entity component images.

[0069] The material library may include a background image library. The background image library may include images associated with, for example, scenes, weather, time, etc. Fig.12An exemplary background image library according to an embodiment is shown. For example, background image 1202 is about the scene "home", background image 1204 is about the scene "classroom", background image 1206 is about the scene "home" and time "night", background image 1208 is about the time "night", background images 1210 and 1212 are about the scene "forest", background image 1214 is about the scene "street", background image 1210 is about the scene "park, and so on. Each background image in the background image library can be accompanied by description information, including, for example, scene, weather, time, etc. The description information of the background image can be used to compare and match with the template configuration information, so that the corresponding background image can be selected based on the template configuration information. It should be understood that the embodiments of the present disclosure are not limited to Fig.12 The background image shown may also include any other background images.

[0070] The material library may include an animation library. The animation library may include multiple predefined animations. Each animation may be defined by animation parameters such as position, time, direction, frequency, etc. The position parameter may indicate whether the animation causes displacement. If displacement is caused, the position parameter further includes the starting position and the ending position of the displacement. For example, for the animation "left fade-in", the position parameter may define, for example, the starting position is the left edge of the composition area, the ending position is the center line of the composition area, etc. The time parameter may indicate the starting time and the ending time of the animation. For example, the starting time may be the moment of presenting the picture, the moment of playing the audio, the predetermined time after the moment of playing the audio, the predetermined time before the moment of playing the audio, etc. The ending time may be, for example, the moment of completing the audio playback, the predetermined time before or after the moment of completing the audio playback, etc. The direction parameter may indicate the direction of the displacement caused by the animation, for example, to the left, to the right, to the lower right, etc. The frequency parameter may refer to the frequency of presenting the animation. For example, for the animation "flash", the frequency parameter may specify, for example, the number of flashes per second, etc. In one embodiment, when the word describing the entity action is, for example, a literal verb, an animation with a larger action displacement may be randomly selected, and when the word describing the entity action is, for example, a linking verb, an animation with a smaller action displacement may be randomly selected.

[0071] It should be understood that the animation library may also include some customized animations corresponding to predetermined entity actions. These predetermined entity actions may be some high-frequency actions, such as talking, walking, jumping, running, laughing, etc. For example, for the entity action "jump", the customized animation may be "flash", and its animation parameters may indicate the starting position and the ending position of the displacement, the direction is upward and downward, the frequency is 2 times per second, etc. When the template configuration information includes a predetermined entity action, the customized animation corresponding to the predetermined entity action may be directly selected from the animation library.

[0072] Fig.13 A flow chart of an exemplary method 1300 for automatically generating visual content according to an embodiment is shown.

[0073] At 1310, a textual statement can be obtained.

[0074] At 1320 , structured data corresponding to the text sentence may be generated.

[0075] At 1330, a template may be selected from a template library based on the structured data.

[0076] At 1340, configuration information of the template may be generated using at least the structured data.

[0077] At 1350 , a material may be selected from a material library based on the configuration information.

[0078] At 1360 , visual content may be generated by populating the template with the selected assets.

[0079] In one embodiment, the structured data may include at least one of the following: scene, weather, time, number of entities, and entity information of each entity. The entity information may include at least one of the following: entity type, entity name, entity gender, entity age, entity emotion, entity color, and entity action. The entity may include at least one of the following: person, anthropomorphic animal, primitive animal, and object.

[0080] In one embodiment, the selecting a template may include: extracting composition reference information from the structured data; and selecting the template from the template library using the composition reference information, wherein the template has a composition configuration that matches the composition reference information. The composition reference information may include at least one of the following: the number of entities, the type of entity, the age of an entity, and an entity action. The composition configuration may include at least one of the following: an entity position, an entity size, an entity orientation, an entity order, a depth of field type, and a framing position. The configuration information of the template may also include composition configuration information about the composition configuration.

[0081] In one implementation, the selecting a template may include: extracting an entity action from the structured data; and selecting a customized template corresponding to the entity action from the template library.

[0082] In one embodiment, the material library may include a foreground image library and a background image library. The foreground image library may include at least one of the following: an entity image library; and an entity component image library, wherein one or more entity component images are used to form an entity image.

[0083] In one embodiment, the material library may include an animation library, the animation library includes a plurality of predefined animations, and the animation parameters of each animation include at least one of the following: position, time, direction, and frequency. The plurality of predefined animations may include: customized animations corresponding to predetermined entity actions.

[0084] In one implementation, the visual content may be a storyboard.

[0085] In one implementation, method 1300 may further include: generating audio corresponding to the text sentence; and generating multimedia content by combining the visual content and the audio.

[0086] It should be understood that method 1300 may also include any steps / processes for automatically generating visual content according to the above-mentioned embodiments of the present disclosure.

[0087] Fig.14 An exemplary apparatus 1400 for automatically generating visual content according to an embodiment is shown.

[0088] The device 1400 may include: a text sentence acquisition module 1410, used to obtain a text sentence; a structured data generation module 1420, used to generate structured data corresponding to the text sentence; a template selection module 1430, used to select a template from a template library based on the structured data; a configuration information generation module 1440, used to generate configuration information of the template using at least the structured data; a material selection module 1450, used to select material from a material library based on the configuration information; and a visual content generation module 1460, used to generate visual content by filling the template with the selected material.

[0089] In one embodiment, the structured data may include at least one of the following: scene, weather, time, number of entities, and entity information of each entity. The entity information may include at least one of the following: entity type, entity name, entity gender, entity age, entity emotion, entity color, and entity action.

[0090] In one embodiment, the template selection module 1430 may be used to: extract composition reference information from the structured data; and select the template from the template library using the composition reference information, wherein the template has a composition configuration matching the composition reference information. The composition reference information may include at least one of the following: entity quantity, entity type, entity age, and entity action. The composition configuration may include at least one of the following: entity position, entity size, entity orientation, entity order, depth of field type, and framing position.

[0091] In one implementation, the template selection module 1430 may be used to: extract entity actions from the structured data; and select a customized template corresponding to the entity actions from the template library.

[0092] In one embodiment, the material library may include a foreground image library, a background image library, and an animation library. The foreground image library may include at least one of the following: an entity image library; and an entity component image library, wherein one or more entity component images are used to form an entity image. The animation library may include a plurality of predefined animations, and the animation parameters of each animation include at least one of the following: position, time, direction, and frequency. The plurality of predefined animations may include: customized animations corresponding to predetermined entity actions.

[0093] In addition, the apparatus 1400 may further include any other modules configured to automatically generate visual content according to the above-mentioned embodiments of the present disclosure.

[0094] Fig.15 An exemplary apparatus 1500 for automatically generating visual content according to an embodiment is shown.

[0095] The device 1500 may include at least one processor 1510 and a memory 1520 storing computer executable instructions. When executing the computer executable instructions, the at least one processor 1510 may: obtain a text sentence; generate structured data corresponding to the text sentence; select a template from a template library based on the structured data; generate configuration information of the template using at least the structured data; select a material from a material library based on the configuration information; and generate visual content by filling the template with the selected material. The at least one processor 1510 may also be configured to perform any operation of the method for automatically generating visual content according to the above-mentioned embodiments of the present disclosure.

[0096] The embodiments of the present disclosure may be implemented in a non-transitory computer-readable medium. The non-transitory computer-readable medium may include instructions, which, when executed, cause one or more processors to perform any operation of the method for automatically generating visual content according to the above embodiments of the present disclosure.

[0097] It should be understood that all operations in the method described above are merely exemplary, and the present disclosure is not limited to any operation in the method or the order of these operations, but should cover all other equivalent transformations under the same or similar concept.

[0098] It should also be understood that all modules in the above described device can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. In addition, any module in these modules can be further divided into submodules or combined together functionally.

[0099] Processors have been described in conjunction with various devices and methods. These processors can be implemented using electronic hardware, computer software or any combination thereof. Whether these processors are implemented as hardware or software will depend on specific application and the overall design constraints imposed on the system. As an example, the processor provided in the present disclosure, any part of the processor or any combination of processors can be implemented as a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a state machine, a gate logic, a discrete hardware circuit, and other suitable processing components configured to perform the various functions described in the present disclosure. The function of the processor provided in the present disclosure, any part of the processor or any combination of processors can be implemented as software performed by a microprocessor, a microcontroller, a DSP or other suitable platforms.

[0100] Software should be broadly considered to mean instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, processes, functions, etc. Software can reside in a computer-readable medium. A computer-readable medium can include, for example, a memory, which can be, for example, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic stripe), an optical disk, a smart card, a flash memory device, a random access memory (RAM), a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a register, or a removable disk. Although the memory is shown as being separated from the processor in the various aspects provided in the present disclosure, the memory can also be located inside the processor, such as a cache or a register.

[0101] The above description is provided to enable any person skilled in the art to implement the various aspects described herein. Various modifications to these aspects are apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents of the elements of the various aspects described in this disclosure that are known or to be known to those skilled in the art will be covered by the claims.

Claims

1. A method for automatically generating visual content, comprising: Get the text sentence; generating structured data corresponding to the text sentence; Based on the structured data, select a template from a template library; generating configuration information of the template using at least the structured data; Based on the configuration information, selecting a material from a material library; as well as generating visual content by filling the template with the selected material, The selection of the template includes at least one of the following: extracting composition reference information from the structured data, and selecting the template from the template library using the composition reference information, wherein the template has a composition configuration matching the composition reference information; or extracting entity actions from the structured data, and selecting a customized template corresponding to the entity actions from the template library, Wherein, the visual content includes a visual representation of the textual content.

2. The method of claim 1, wherein: The structured data includes at least one of the following: scene, weather, time, number of entities, and entity information of each entity, and The entity information includes at least one of the following: entity type, entity name, entity gender, entity age, entity emotion, entity color, and entity action.

3. The method of claim 2, wherein: The entity includes at least one of: a person, an anthropomorphic animal, a primitive animal, and an object.

4. The method of claim 1, wherein: The composition reference information includes at least one of the following: entity quantity, entity type, entity age, and entity action, and The composition configuration includes at least one of the following: physical position, physical size, physical orientation, physical order, depth of field type, and framing position.

5. The method of claim 1, wherein: The configuration information of the template also includes composition configuration information about the composition configuration.

6. The method of claim 1, wherein: The material library includes a foreground image library and a background image library, and The foreground image library includes at least one of: an entity image library; and an entity component image library, wherein one or more entity component images are used to form an entity image.

7. The method of claim 1, wherein: The material library includes an animation library, and the animation library includes a plurality of predefined animations, and The animation parameters of each animation include at least one of the following: position, time, direction, and frequency.

8. The method of claim 7, wherein: The plurality of predefined animations include customized animations corresponding to predetermined entity actions.

9. The method of claim 1, wherein: The visual content is a storyboard.

10. The method of claim 1, further comprising: generating audio corresponding to the text sentence; as well as Multimedia content is generated by combining the visual content and the audio.

11. An apparatus for automatically generating visual content, comprising: A text sentence obtaining module, used for obtaining text sentences; A structured data generation module, used to generate structured data corresponding to the text sentence; A template selection module, used for selecting a template from a template library based on the structured data; A configuration information generating module, configured to generate configuration information of the template using at least the structured data; A material selection module, used to select a material from a material library based on the configuration information; as well as a visual content generation module, configured to generate visual content by filling the template with the selected material, The template selection module is configured to perform at least one of the following: extracting composition reference information from the structured data, and selecting the template from the template library using the composition reference information, wherein the template has a composition configuration matching the composition reference information; or extracting entity actions from the structured data, and selecting a customized template corresponding to the entity actions from the template library, Wherein, the visual content includes a visual representation of the textual content.

12. The device of claim 11, wherein: The structured data includes at least one of the following: scene, weather, time, number of entities, and entity information of each entity, and The entity information includes at least one of the following: entity type, entity name, entity gender, entity age, entity emotion, entity color, and entity action.

13. The device of claim 11, wherein: The composition reference information includes at least one of the following: entity quantity, entity type, entity age, and entity action, and The composition configuration includes at least one of the following: physical position, physical size, physical orientation, physical order, depth of field type, and framing position.

14. The device of claim 11, wherein: The material library includes a foreground image library, a background image library and an animation library. The foreground image library includes at least one of: an entity image library; and an entity component image library, wherein one or more entity component images are used to form an entity image, and The animation library includes a plurality of predefined animations, and the animation parameters of each animation include at least one of the following: position, time, direction, and frequency.

15. The device of claim 14, wherein: The plurality of predefined animations include customized animations corresponding to predetermined entity actions.

16. An apparatus for automatically generating visual content, comprising: at least one processor; as well as a memory storing computer executable instructions that, when executed, cause the at least one processor to: Get the text sentence, generating structured data corresponding to the text sentence, Based on the structured data, a template is selected from a template library, generating configuration information of the template using at least the structured data, Based on the configuration information, selecting a material from a material library, and generating visual content by filling the template with the selected material, The selection of the template includes at least one of the following: extracting composition reference information from the structured data, and selecting the template from the template library using the composition reference information, wherein the template has a composition configuration matching the composition reference information; or extracting entity actions from the structured data, and selecting a customized template corresponding to the entity actions from the template library, Wherein, the visual content includes a visual representation of the textual content.

Images