Multimedia content generation method, apparatus, device, and storage medium

Abstract

This application discloses a method, apparatus, device, and storage medium for generating multimedia content. The method involves converting rich text data into a predetermined data structure; classifying the data structure according to data type to obtain different types of sub-data structures; generating content elements in a target multimedia format corresponding to each type of sub-data structure; and combining the content elements to obtain the created content in the target multimedia format. This application can directly generate created content in a target multimedia format based on rich text data. This target multimedia format can be video, live content, presentation slides, etc., making it easier for multimedia designers to directly use rich text data to create multimedia content, greatly facilitating their work. This application can break down the barriers between rich text and other multimedia content formats, enabling format conversion between rich text data and other multimedia content formats.

Description

Technical Field

[0001] This application relates to the field of multimedia data processing technology, and more specifically, to a method, apparatus, device and storage medium for generating multimedia content. Background Technology

[0002] In the process of multimedia content creation, multimedia designers can use a variety of mature software to create content. For example, in video creation, designers can first obtain a plain text data, such as a speech or product introduction document, and then use video creation software to generate video content based on the plain text data. During the process, designers may need to adjust the format and other attributes of the text data in the software to ensure the quality of the generated video.

[0003] However, in actual copywriting, copywriters frequently use various rich text editors to generate rich text documents. Rich text documents, compared to plain text data, inherently possess a data structure and format. Currently, existing video creation software does not support video content creation based on rich text, significantly impacting the video content creation capabilities of multimedia designers. Summary of the Invention

[0004] In view of the above problems, this application is made to provide a multimedia content generation method, apparatus, device, and storage medium to enable the creation of multimedia content directly based on rich text, thereby meeting the creative needs of multimedia designers. The specific solution is as follows:

[0005] Firstly, a method for generating multimedia content is provided, including:

[0006] Obtain rich text data for generating multimedia content;

[0007] Convert the rich text data into a defined data structure;

[0008] The data structure is classified according to data type to obtain different types of sub-data structures;

[0009] Generate content elements in the target multimedia format corresponding to each type of sub-data structure;

[0010] By combining the generated content elements, the created content in the target multimedia format is obtained.

[0011] Preferably, the set data structure is a JSON object.

[0012] Preferably, the process of converting the rich text data into the JSON object includes:

[0013] Obtain the Hypertext Markup Language (HTML) content of the rich text data;

[0014] The HTML content is parsed into a DOM tree;

[0015] Traverse the DOM tree and extract the content of each element, then save the extracted content of each element into a created JSON object.

[0016] Preferably, the process of saving the extracted element content into the created JSON object includes:

[0017] Determine the field type corresponding to each extracted element;

[0018] The content of each element is used as the value of the corresponding type field, and the content of each element and its corresponding type field are saved in the created JSON object as a tree-level node.

[0019] Preferably, the field types corresponding to the content of each element include any one or more of the following:

[0020] Heading type, paragraph type, image type.

[0021] Preferably, the process of classifying the data structure according to data type to obtain different types of sub-data structures includes:

[0022] Read each tree-level node in the JSON object to obtain the fields contained in each tree-level node and the element content that serves as the field value;

[0023] Based on the different field types, the read element content is divided into fields to obtain the element content of each field type, which serves as the sub-data structure for each field type.

[0024] Preferably, the target multimedia format includes any one or more of the following:

[0025] Videos, live stream content, and presentations.

[0026] Preferably, when the target multimedia format is video;

[0027] The process of generating video content elements corresponding to each type of sub-data structure includes:

[0028] For a sub-data structure of type heading, generate a video title element that matches the sub-data structure;

[0029] For a sub-data structure of type paragraph, retrieve video clips that match the text semantics of the sub-data structure;

[0030] For the sub-data structure of image type, it is used as a floating image element embedded in the video.

[0031] Secondly, a multimedia content generation device is provided, comprising:

[0032] The rich text data acquisition unit is used to acquire rich text data for generating multimedia content;

[0033] A data conversion unit is used to convert the rich text data into a set data structure;

[0034] The sub-data structure partitioning unit is used to classify the data structure according to data type to obtain different types of sub-data structures;

[0035] The content element generation unit is used to generate content elements in the target multimedia format corresponding to each type of sub-data structure.

[0036] The content element combination unit is used to combine the generated content elements to obtain the created content in the target multimedia format.

[0037] Thirdly, a multimedia content generation device is provided, including: a memory and a processor;

[0038] The memory is used to store programs;

[0039] The processor is used to execute the program to implement the various steps of the multimedia content generation method as described above.

[0040] Fourthly, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the various steps of the multimedia content generation method as described above.

[0041] Using the above technical solution, this application first converts the acquired rich text data into a predetermined data structure. While retaining the element data and structural information of the rich text data, it also converts it into a unified format data structure for convenient subsequent processing. Furthermore, since rich text data can contain different types of data, such as titles, paragraphs, and images, the converted data structure can also include different types of sub-data structures. Moreover, different types of sub-data structures are displayed differently in the final generated target multimedia format content. Therefore, in this embodiment, the data structure can be further classified according to data type to obtain different types of sub-data structures. Then, content elements in the target multimedia format corresponding to each type of sub-data structure are generated. The generated content elements are combined to obtain the target multimedia format content. The solution provided in this embodiment can directly generate target multimedia format content based on rich text data. This target multimedia format can be various, such as video, live content, presentation slides (PPT), etc., making it more convenient for multimedia designers to directly use rich text data to create multimedia content, greatly facilitating their work.

[0042] By adopting the above-mentioned solution provided in this application, the creative barriers between rich text and other multimedia content formats can be broken down, and the form conversion between rich text data and other multimedia content formats can be realized. Attached Figure Description

[0043] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0044] Figure 1 A flowchart illustrating a multimedia content generation method provided in an embodiment of this application;

[0045] Figure 2 This example illustrates the process of converting rich text data into a JSON object.

[0046] Figure 3 This example illustrates a partial display interface of a video created using rich text.

[0047] Figure 4 This example illustrates a partial display interface of another video creation-based platform;

[0048] Figure 5 This example illustrates how to generate a corresponding DOM tree based on rich text data.

[0049] Figure 6 This is a schematic diagram of a multimedia content generation device provided in an embodiment of this application;

[0050] Figure 7 This is a schematic diagram of the structure of a multimedia content generation device provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] This application provides a multimedia content generation scheme based on rich text data, which can generate multimedia content in other formats based on the provided rich text data, breaking down the creative barriers between rich text and other multimedia content formats, realizing the form conversion between rich text data and other multimedia content formats, and making the work of multimedia designers more convenient.

[0053] The solution proposed in this application can be implemented based on a terminal with data processing capabilities, such as a mobile phone, computer, server, cloud, etc. In one optional scenario, the multimedia content generation method proposed in this application can be applied to a terminal browser.

[0054] Next, combined Figure 1 The multimedia content generation method of this application may include the following steps:

[0055] Step S100: Obtain rich text data for generating multimedia content.

[0056] Specifically, rich text data can contain one or more types of data, such as titles, quotes, paragraphs, images, etc. Figure 2 The text box at the top shows an example of rich text data, including the title: "Sports Brand Sales Continue to Recover," the image: "US Stock Market Closing.png," and the paragraph: "Sales of sports brands continued to improve in the second quarter of this year. On August 3, sports brand xx announced its financial results for the second quarter and first half of 2023. As xx's most important strategic market, Greater China contributed 766 million euros in revenue in the second quarter of this year, a year-on-year increase of 16.4%." This shows that... Figure 2 The example rich text data is a news article related to finance.

[0057] In this step, you can obtain rich text data edited by copywriters using a rich text editor, or you can obtain rich text data from other sources. The rich text data obtained in this step is the data specified by the user for generating multimedia content. The specific content of the rich text data obtained will vary depending on the domain of the multimedia content to be created, and will be determined based on the user's actual usage scenario.

[0058] Step S110: Convert the rich text data into a set data structure.

[0059] Specifically, a data structure refers to a collection of data elements that have one or more specific relationships with each other; that is, a collection of data elements with a "structure." Here, "structure" refers to the relationships between the data elements. In this step, by converting rich text data into a defined data structure, both the data elements themselves and their structural information are preserved. Furthermore, converting to a unified data structure facilitates further processing of the structured data in subsequent steps.

[0060] The data structure specified in this step can be any existing data structure, such as JSON objects, Extensible Markup Language (XML), etc. Alternatively, it can be any data structure you design yourself or that may emerge in the future.

[0061] Taking a JSON object as the data structure as an example, Figure 2 This example illustrates how to convert rich text data into a JSON object. Figure 2 The example text in the lower middle text box is the converted JSON object. The element data in rich text data is stored as field values.

[0062] Step S120: Classify the data structure according to data type to obtain different types of sub-data structures.

[0063] Specifically, as mentioned earlier, rich text data contains various data types, and after being converted into a data structure, it can also yield various sub-data structures of different types, such as... Figure 2The JSON object shown has the following characteristics: For the title in the rich text data, the corresponding description field after conversion to JSON is "heading", with a value of "Sports brand sales continue to recover"; for the image in the rich text data, the corresponding description field after conversion to JSON is "image", with a value of "US stock market closing.png"; and for the paragraph in the rich text data, the corresponding description field after conversion to JSON is "paragraph", with a value of "Sports brand sales continued to improve in the second quarter of this year. On August 3, sports brand xx announced its financial report for the second quarter and first half of 2023. As xx's most important strategic market, Greater China contributed 766 million euros in revenue in the second quarter of this year, a year-on-year increase of 16.4%."

[0064] As shown above, the transformed data structure can also contain several different types of data. Therefore, the data structure can be classified according to different data types to obtain different types of sub-data structures. Figure 2 The example includes three types of sub-data structures: heading field, image field, and paragraph field.

[0065] Step S130: Generate content elements in the target multimedia format corresponding to each type of sub-data structure.

[0066] Specifically, different types of sub-data structures can serve as different content elements when generating content in the target multimedia format.

[0067] The target multimedia format can be of various types, such as video, live content, and PowerPoint presentations.

[0068] Taking video as the target multimedia format as an example, for the image type sub-data structure, it can be directly embedded as a floating image into the generated video. For the paragraph type sub-data structure, video segments that semantically match the text of that sub-data structure can be retrieved. For the heading type sub-data structure, video title elements that match that sub-data structure can be generated. Figure 2 Taking the example of rich text data, where the heading type substructure data is "Sports brand sales continue to recover", a video title element is generated, as shown below. Figure 3 As shown. For the sub-data structure "US stock market closing.png" of type image, a floating image is generated and embedded in the video, as shown. Figure 4 As shown.

[0069] The process of generating content elements in the target multimedia format corresponding to each type of sub-data structure in this step can be achieved by using a pre-configured algorithm for generating content elements in the target multimedia format corresponding to each data type, based on each type of sub-data structure. For example, when the target multimedia format is video, for a heading type sub-data structure, the heading type sub-data structure can be filled into the video title template according to a pre-configured video title template to obtain the generated video title element. For a paragraph type sub-data structure, natural language processing capabilities can be used to determine the text semantics corresponding to the sub-data structure, and then video segments matching the text semantics can be retrieved from the video resource library, or a multimodal large model can be used to generate video segments matching the text semantics. For an image type sub-data structure, it can be used as a floating image element and then directly embedded into the video.

[0070] Of course, this embodiment only takes the target multimedia format in video form as an example to introduce the optional generation methods of content elements corresponding to each type of sub-data structure. For other multimedia formats such as live video content, PPT, etc., matching algorithms can also be used to generate corresponding content elements, which will not be listed here.

[0071] Step S140: Combine the generated content elements to obtain the created content in the target multimedia format.

[0072] Specifically, the previous step has generated the various content elements of the target multimedia format. In this step, the various content elements generated in the previous step can be combined according to certain rules to obtain the created content of the target multimedia format. Figure 3 and Figure 4 Using video as an example, this document demonstrates a portion of the display interface of the combined video.

[0073] Understandably, during the process of combining various content elements, it can also respond to user editing operations, such as adjusting the display position, size, and duration of floating images and video titles, and then adjust the content created in the target multimedia format according to the user's editing operations to obtain the final content.

[0074] The multimedia content generation method provided in this application first converts the acquired rich text data into a predetermined data structure. While retaining the element data and structural information of the rich text data, it also converts it into a unified format data structure for convenient subsequent processing. Furthermore, since rich text data can contain different types of data, such as titles, paragraphs, and images, the converted data structure can also include different types of sub-data structures. Moreover, different types of sub-data structures are displayed differently in the final generated target multimedia format content. Therefore, this embodiment can further classify the data structure according to data type to obtain different types of sub-data structures, and then generate content elements in the target multimedia format corresponding to each type of sub-data structure. The generated content elements are then combined to obtain the target multimedia format content. Using the solution provided in this embodiment, target multimedia format content can be generated directly from rich text data. This target multimedia format can be various, such as video, live content, presentation slides (PPT), etc., making it more convenient for multimedia designers to directly use rich text data to create multimedia content, greatly facilitating their work.

[0075] By adopting the above-mentioned solution provided in this application, the creative barriers between rich text and other multimedia content formats can be broken down, and the form conversion between rich text data and other multimedia content formats can be realized.

[0076] In some embodiments of this application, taking a JSON object as an example, an optional implementation of converting runeboard data into a JSON object is described, which may include the following steps:

[0077] S1. Obtain the Hypertext Markup Language (HTML) content of the rich text data.

[0078] S2. Parse the HTML content into a DOM tree.

[0079] Reference Figure 5 It provides an example diagram of parsing HTML content from rich text data into a DOM tree.

[0080] S3. Traverse the DOM tree and extract the content of each element, and save the extracted content of each element into the created JSON object.

[0081] Specifically, the DOM tree stores the content of each element in the HTML as nodes. In this step, the nodes of the DOM tree are traversed to extract the content of each element. The extracted content is then saved into a created JSON object.

[0082] In one optional implementation, the process of saving the extracted element content into a created JSON object may specifically include:

[0083] First, determine the field type corresponding to each extracted element content. Then, use each element content as the value under the corresponding type field, and save each element content and its corresponding type field as a tree-level node into the created JSON object.

[0084] The content of each element extracted from the DOM tree may belong to the same or different types. For example, some may be headings, some may be images, some may be paragraphs, some may be quotes, etc. The DOM tree already records the field type to which each element's content belongs, such as... Figure 5 The DOM tree records field types such as "heading", "paragraph", and "quote". Therefore, the content of each element can be used as the value of the corresponding field type, and the combination of "field:field value" can be used as a tree-level node and saved into a JSON tree-level object.

[0085] Based on the above embodiments, step S120, the process of classifying data structures according to data types to obtain different types of sub-data structures, may include:

[0086] Read each tree-level node in the JSON object to obtain the fields contained in each tree-level node and the element content that serves as the value of the field.

[0087] Based on the different field types, the read element content is divided into fields to obtain the element content of each field type, which serves as the sub-data structure for each field type.

[0088] As can be seen from the above embodiments, this application converts rich text data into a standard data structure, classifies the data structure according to data type, and then outputs it as content elements in the target multimedia format (such as video format content elements) according to different categories. The content elements are then combined into the creation content in the target multimedia format, realizing the conversion of different creation formats and multimedia forms. This breaks the traditional content encapsulation format and makes it easier for multimedia designers to create multimedia content in other formats based on rich text.

[0089] The multimedia content generation apparatus provided in the embodiments of this application is described below. The multimedia content generation apparatus described below and the multimedia content generation method described above can be referred to in correspondence.

[0090] See Figure 6 , Figure 6This is a schematic diagram of the structure of a multimedia content generation device disclosed in an embodiment of this application.

[0091] like Figure 6 As shown, the device may include:

[0092] Rich text data acquisition unit 11 is used to acquire rich text data for generating multimedia content;

[0093] Data conversion unit 12 is used to convert the rich text data into a set data structure;

[0094] The sub-data structure partitioning unit 13 is used to classify the data structure according to the data type to obtain different types of sub-data structures;

[0095] Content element generation unit 14 is used to generate content elements in the target multimedia format corresponding to each type of sub-data structure;

[0096] The content element combination unit 15 is used to combine the generated content elements to obtain the created content in the target multimedia format.

[0097] Optionally, the data structure set above can be a JSON object or other types of data structures.

[0098] Taking a JSON object as an example, the process by which the data conversion unit converts the rich text data into the JSON object includes:

[0099] Obtain the Hypertext Markup Language (HTML) content of the rich text data;

[0100] The HTML content is parsed into a DOM tree;

[0101] Traverse the DOM tree and extract the content of each element, then save the extracted content of each element into a created JSON object.

[0102] Optionally, the process by which the data transformation unit saves the extracted element content into the created JSON object includes:

[0103] Determine the field type corresponding to each extracted element;

[0104] The content of each element is used as the value of the corresponding type field, and the content of each element and its corresponding type field are saved in the created JSON object as a tree-level node.

[0105] Optionally, the field types corresponding to the content of each of the above elements include any one or more of the following:

[0106] Heading type, paragraph type, image type, etc.

[0107] Optionally, taking the data structure of a JSON object as an example, the process by which the above-mentioned sub-data structure partitioning unit classifies the data structure according to data type to obtain different types of sub-data structures includes:

[0108] Read each tree-level node in the JSON object to obtain the fields contained in each tree-level node and the element content that serves as the field value;

[0109] Based on the different field types, the read element content is divided into fields to obtain the element content of each field type, which serves as the sub-data structure for each field type.

[0110] Optionally, the target multimedia format includes any one or more of the following:

[0111] Videos, live stream content, presentations, etc.

[0112] Taking video as an example of the target multimedia format, the process by which the content element generation unit generates video content elements corresponding to each type of sub-data structure includes:

[0113] For a sub-data structure of type heading, generate a video title element that matches the sub-data structure;

[0114] For a sub-data structure of type paragraph, retrieve video clips that match the text semantics of the sub-data structure;

[0115] For the sub-data structure of image type, it is used as a floating image element embedded in the video.

[0116] The multimedia content generation apparatus provided in this application embodiment can be applied to multimedia content generation devices, such as terminal devices equipped with a browser. Optionally, Figure 7 A hardware block diagram of a multimedia content generation device is shown, with reference to... Figure 7 The hardware structure of a multimedia content generation device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0117] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;

[0118] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0119] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;

[0120] The memory stores a program, which the processor can call. The program is used for:

[0121] Obtain rich text data for generating multimedia content;

[0122] Convert the rich text data into a defined data structure;

[0123] The data structure is classified according to data type to obtain different types of sub-data structures;

[0124] Generate content elements in the target multimedia format corresponding to each type of sub-data structure;

[0125] By combining the generated content elements, the created content in the target multimedia format is obtained.

[0126] Optionally, the refined and extended functions of the program can be found in the description above.

[0127] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used for:

[0128] Obtain rich text data for generating multimedia content;

[0129] Convert the rich text data into a defined data structure;

[0130] The data structure is classified according to data type to obtain different types of sub-data structures;

[0131] Generate content elements in the target multimedia format corresponding to each type of sub-data structure;

[0132] By combining the generated content elements, the created content in the target multimedia format is obtained.

[0133] Optionally, the refined and extended functions of the program can be found in the description above.

[0134] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0135] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0136] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of generating multimedia content, characterized by, The method is used to generate multimedia content in other formats based on rich text data, including video. The method includes: Obtain rich text data for generating multimedia content; Obtain the Hypertext Markup Language (HTML) content of the rich text data; parse the HTML content into a DOM tree; traverse the DOM tree and extract the content of each element from it, determine the field type corresponding to each extracted element content, and the field types corresponding to each element content include: heading type, paragraph type, and image type; use each element content as the value under the corresponding type field, and save each element content and its corresponding type field as a tree-level node into the created JSON object; The content in the JSON object is categorized according to data type to obtain different types of sub-data structures. The types of the sub-data structures include: heading type, paragraph type, and image type. For a sub-data structure of type "heading", generate a title element based on the video format that matches the sub-data structure; for a sub-data structure of type "paragraph", retrieve a video clip that matches the text semantics of the sub-data structure or generate a video clip that matches the text semantics of the sub-data structure; for a sub-data structure of type "image", use it as a floating image element embedded in the video; use the generated video title element, video clip, and floating image element as content elements based on the video format. By combining the generated content elements, a video with creative content is obtained.

2. The method according to claim 1, characterized in that, The process of classifying the content of the JSON object according to data type to obtain different types of sub-data structures includes: Read each tree-level node in the JSON object to obtain the fields contained in each tree-level node and the element content that serves as the field value; Based on the different field types, the read element content is divided into fields to obtain the element content of each field type, which serves as the sub-data structure for each field type.

3. A multimedia content generation device, characterized in that, The apparatus is used to generate multimedia content in other formats based on rich text data, the multimedia content including video formats, and the apparatus includes: The rich text data acquisition unit is used to acquire rich text data for generating multimedia content; The data conversion unit is used to obtain the Hypertext Markup Language (HTML) content of the rich text data; parse the HTML content into a DOM tree; traverse the DOM tree and extract the content of each element from it, determine the field type corresponding to each extracted element content, and the field types corresponding to each element content include: heading type, paragraph type, and image type; use each element content as the value under the corresponding type field, and save each element content and its corresponding type field in the form of a tree-level node into the created JSON object; The sub-data structure partitioning unit is used to classify the content in the JSON object according to the data type to obtain different types of sub-data structures. The types of the sub-data structures include: heading type, paragraph type, and image type. The content element generation unit is used to generate a video-format-based title element that matches the sub-data structure of the heading type; for the paragraph type sub-data structure, retrieve a video segment that matches the text semantics of the sub-data structure or generate a video segment that matches the text semantics of the sub-data structure; for the image type sub-data structure, use it as a floating image element embedded in the video; and use the generated video title element, video segment, and floating image element as video-format-based content elements. The content element combination unit is used to combine the generated content elements to obtain a video with creative content.

4. A multimedia content generation device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the multimedia content generation method as described in any one of claims 1 to 2.

5. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the multimedia content generation method as described in any one of claims 1 to 2.

Images