Method, device and storage medium for aligning text with media material

Abstract

A method, apparatus and storage medium for aligning text with media assets are provided. A set of anchor points in the text is determined; the following operations i) to v) are repeatedly performed until all anchor points are removed from the set of anchor points or all media assets are removed from the set of media assets: i) ordering anchor points in the set of anchor points, ii) selecting a target anchor point from the set of anchor points based on the ordered anchor points in the set, iii) determining a media asset from the set of media assets that matches a text segment starting from the target anchor point, iv) removing the target anchor point, and v) removing the media asset that matches the text segment starting from the target anchor point; and aligning the text segment with the corresponding matched media asset.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to the field of computer technology, and more specifically, to a method, apparatus, and computer-readable storage medium for aligning text with media material. Background Technology

[0002] Short video production is a skill with a certain barrier to entry, often requiring extensive training to master. The process typically involves writing the script, collecting relevant media materials, matching the media materials with the text, and finally, rendering. In recent years, with the rise of deep learning algorithms, many algorithms can solve the problems of script writing and collecting relevant media materials. Summary of the Invention

[0003] In a first aspect, a method for aligning text with media material is provided. The method includes: determining a set of anchor points in the text based on the grammatical structure of the text, each anchor point being the start position of a text segment; repeating operations i) through v) until all anchor points are removed from the set of anchor points or all media material is removed from a set of media material: i) sorting the anchor points in the set based on the text segments starting from the anchor points; ii) selecting a target anchor point from the set of anchor points based on the sorted anchor points; iii) determining media material from the set of media material that matches the text segments starting from the target anchor point; iv) removing the target anchor point from the set of anchor points; and v) removing media material from the set of media material that matches the text segments starting from the target anchor point; and aligning the text segments with the corresponding matching media material.

[0004] In a second aspect, an apparatus for aligning text with media material is provided. The apparatus includes one or more processors; and a memory for storing one or more programs executable by the one or more processors to cause the apparatus to perform operations including: determining a set of anchor points in the text based on the grammatical structure of the text, each of the anchor points being the starting position of a text segment of the text; repeating operations i) to v) until all anchor points are removed from the set of anchor points or all media material is removed from a set of media material: i) sorting the anchor points in the set of anchor points based on text segments starting from the anchor points; ii) selecting a target anchor point from the set of anchor points based on the sorted anchor points in the set; iii) determining media material from the set of media material that matches a text segment starting from the target anchor point; iv) removing the target anchor point from the set of anchor points; and v) removing media material from the set of media material that matches the text segment starting from the target anchor point; and aligning the text segment with the corresponding matching media material.

[0005] In a third aspect, a non-transitory computer-readable storage medium is provided to store instructions, wherein the instructions are executable by a processor to perform operations, the operations including: determining a set of anchor points in the text based on the grammatical structure of the text, each anchor point being the starting position of a text segment of the text; repeating the following operations i) to v) until all anchor points are removed from the set of anchor points or all media material is removed from a set of media material: i) sorting the anchor points in the set of anchor points based on the text segments starting from the anchor points; ii) selecting a target anchor point from the set of anchor points based on the sorted anchor points in the set; iii) determining media material from the set of media material that matches the text segments starting from the target anchor point; iv) removing the target anchor point from the set of anchor points; and v) removing media material from the set of media material that matches the text segments starting from the target anchor point; and aligning the text segments with the corresponding matching media material. Attached Figure Description

[0006] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements:

[0007] Figure 1 This is an example system architecture 100 in which an embodiment of a method for aligning text with media material according to some embodiments of the present disclosure may be applied;

[0008] Figure 2 This is a flowchart of a method for aligning text with media material according to some embodiments of the present disclosure;

[0009] Figure 3 This is a schematic diagram illustrating the implementation of sorting anchor points in an anchor point set according to some embodiments of this disclosure;

[0010] Figure 4 This is a schematic diagram illustrating the implementation of sorting anchor points in an anchor point set according to some embodiments of this disclosure;

[0011] Figure 5 This is a schematic diagram illustrating the implementation of determining the relevance of anchor points to candidate media materials according to some embodiments of this disclosure;

[0012] Figure 6 This is a schematic diagram illustrating the implementation of sorting anchor points in an anchor point set according to some embodiments of this disclosure;

[0013] Figure 7 This is a schematic diagram illustrating the implementation of sorting anchor points in an anchor point set according to some embodiments of this disclosure;

[0014] Figure 8 This is a schematic diagram illustrating the implementation of sorting anchor points in an anchor point set according to some embodiments of this disclosure;

[0015] Figure 9 This is a schematic diagram illustrating the implementation of candidate media material for determining anchor points according to some embodiments of this disclosure;

[0016] Figure 10 This is a schematic diagram illustrating the implementation of selecting a target anchor point from a set of anchor points according to some embodiments of this disclosure;

[0017] Figure 11 This is a schematic diagram illustrating the implementation of selecting a target anchor point from a set of anchor points according to some embodiments of this disclosure;

[0018] Figure 12 Methods for aligning text with media material according to some other embodiments of this disclosure are shown;

[0019] Figure 13 A schematic diagram of the timeline of anchor points according to some embodiments of the present disclosure is shown;

[0020] Figure 14 A method for controlling an interface to generate video according to some embodiments of the present disclosure is shown;

[0021] Figure 15 Examples of a first interface and a second interface according to some embodiments of this disclosure are shown; and

[0022] Figure 16 This is a schematic diagram of a device used to align text with media materials. Detailed Implementation

[0023] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only components relevant to the relevant disclosure are shown in the accompanying drawings.

[0024] It should also be noted that the embodiments and features described in this disclosure can be combined with each other without conflict. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] In related technologies, semantic vectors for images or videos are calculated and stored in a media asset collection. The semantic vector for input text is calculated and compared with the semantic vectors stored in the media asset collection to return matching images or videos. Alternatively, keywords from the text are compared with keywords from the media assets to perform a video search.

[0026] In related techniques, keywords or feature vectors are not combined. When considering the alignment of individual media clips and text, only the features of the current clip or text are considered to obtain a locally optimal solution, without considering a globally optimal solution from the perspective of the media library or the entire text. Furthermore, the alignment between clips and text is not precise enough, and it cannot be guaranteed that the starting position of the clip can be accurate to a specific word in the text. Some methods also have low video generation efficiency.

[0027] Figure 1 An example system architecture 100 is shown, in which embodiments of a method for aligning text with media material according to some embodiments of this disclosure can be applied.

[0028] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, network 104, and first server 105. Network 104 serves as a medium for providing communication links between terminal devices 101, 102, and 103 and first server 105. Network 104 may include various types of connections (e.g., wired or wireless communication links, or fiber optic cables).

[0029] Users can use terminal devices 101, 102, and 103 to interact with the first server 105 via network 104 to receive or send messages, etc. Various communication client applications (e.g., video generation applications and web browser applications) can be installed on terminal devices 101, 102, and 103. Terminal devices 101, 102, and 103 can perform corresponding processing in response to user operations on buttons on the first or second interface.

[0030] Terminal devices 101, 102, and 103 can be hardware or software. When they are hardware, terminal devices 101, 102, and 103 can be various electronic devices, including but not limited to smartphones, tablet computers, laptops, desktop computers, etc. When they are software, terminal devices 101, 102, and 103 can be installed in the electronic devices listed above. A terminal device can be implemented as multiple software blocks or multiple software modules (e.g., software or software modules for providing distributed services), or as a single software block or a single software module, which will not be specifically defined herein.

[0031] The first server 105 may be a server that provides various services, such as a backend server that supports the alignment of text with media materials. The backend server may receive requests or web addresses sent by terminal devices 101, 102, and 103, perform corresponding processing on the requests or web addresses, and return the processing results (e.g., video) to the terminal devices.

[0032] It should be noted that the method for aligning text with media material provided by some embodiments of this disclosure is performed by terminal devices 101, 102, and 103 or server 105. Accordingly, means for aligning text with media material are typically provided in terminal devices 101, 102, and 103 or server 105.

[0033] It should be noted that a server can be either hardware or software. When implemented as hardware, a server can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When implemented as software, a server can be implemented as multiple software blocks or modules (e.g., software or software modules for providing distributed services), or as a single software block or single software module, which will not be specifically defined here.

[0034] It should be understood that Figure 1 The number of terminal devices, networks, and first servers listed is merely illustrative. Any number of terminal devices, networks, and first servers can be provided based on actual needs.

[0035] Figure 2 This is a flowchart of a method for aligning text with media material according to some embodiments of the present disclosure. The method includes steps 201 to 203.

[0036] Step 201 includes: determining a set of anchor points in the text based on the text's grammatical structure, where each anchor point is the starting position of a text segment.

[0037] In some embodiments, text is retrieved from a webpage. The webpage includes images and text. In some embodiments, text is input by the user through a video-generated application interface.

[0038] In some embodiments, after obtaining the text, a search is performed on the Internet, on a local computer, or online or offline to obtain a set of media materials matching the text. In some embodiments, the search is performed based on keyword comparison, semantic processing, etc. In some embodiments, the search is performed based on any existing algorithm that can be used to obtain the set of media materials.

[0039] In some embodiments, the grammatical structure of the text includes paragraphs, sentences, and words. In some embodiments, a text segment is a word or sentence. In some embodiments, step 201 includes: determining the start position of a word or sentence as an anchor in a set of anchors.

[0040] Step 202 includes repeating the following operations i) through v) until all anchors are removed from the anchor set or all media material is removed from the media material set: i) sorting the anchors in the anchor set based on the text segments starting from the anchors; ii) selecting a target anchor from the anchor set based on the sorted anchors in the set; iii) determining media material from the media material set that matches the text segments starting from the target anchor; iv) removing the target anchor from the anchor set; and v) removing media material from the media material set that matches the text segments starting from the target anchor.

[0041] In some embodiments, media material includes images or video clips.

[0042] In some embodiments, operation i) includes step 301. Step 301 includes: sorting the anchors in the anchor set based on the type of the anchors, wherein the anchor types include word anchors as the beginning of a word and sentence anchors as the beginning of a sentence. In some embodiments, sentence anchors are sorted above word anchors.

[0043] In some embodiments, step 301 includes, for example, Figure 4 Steps 401 and 402 are shown. Step 401 involves determining that both anchor points are sentence anchor points. Step 402 involves ranking the two anchor points based on their relevance to candidate media materials.

[0044] In some embodiments, step 402 includes, for example: Figure 5 Steps 501 to 504 are shown. Step 501 includes: determining the source of the candidate media material. Step 502 includes: determining the source score of the candidate media material's source based on a preset correspondence between scores and sources. Step 503 includes: determining the confidence score of the candidate media material. Step 504 includes: determining the product of the source score and the confidence score as the degree of relevance between the anchor point and the candidate media material.

[0045] In some embodiments, a list of source scores for different sources is pre-stored. In some embodiments, the first source, including the text and media material to be aligned, has a score of 1, the second source has a score of 0.75, the third source has a score of 0.25.

[0046] It should be noted that the source score can be set as needed.

[0047] In some embodiments, the first source is a webpage, and the text and media material to be aligned are obtained from a given webpage. In some embodiments, a confidence score is determined based on the distance between the text to be aligned and the media material, said distance being the number of paragraphs between the text to be aligned and the media material in the webpage. In some embodiments, the confidence score is determined by formula 3 / (2 + distance).

[0048] In some embodiments, the media material includes an object, image recognition is performed on the media material to identify the object, and a confidence score is the confidence level of the recognition result. In some embodiments, the object is a face, the entire face of a person is identified from the first media material, and only a portion of the face of a person is identified from the second media material, and a first confidence score of the first media material is higher than a second confidence score of the second media material. In some embodiments, the first confidence score is 0.99, and the second confidence score is 0.8.

[0049] It should be noted that confidence scores can be set as needed, or determined using other appropriate methods for determining confidence scores for media materials.

[0050] In some embodiments, step 301 includes steps 601 and 602. Step 601 includes determining that both anchor points are word anchor points. Step 602 includes ranking the two anchor points based on their scores. In some embodiments, the scores of the two anchor points are calculated based on the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm.

[0051] In some embodiments, step 601 includes step 701. Step 701 includes determining that both anchors are word anchors and have a given score. In some embodiments, step 602 includes step 702. Step 702 includes ranking the two anchors based on the degree of relevance between each of the two anchors and the candidate media material for that anchor. The above method can be used to determine the degree of relevance.

[0052] In some embodiments, step 701 includes step 801, and step 702 includes step 802. Step 801 includes determining that both anchor points are word anchor points with a given score, and that the two anchor points have the same degree of relevance to the candidate media material. Step 802 includes ranking the two anchor points based on their positions in the text. In some embodiments, the anchor point in the preceding position is ranked higher than the anchor point in the following position.

[0053] In some embodiments, two identical words begin at two different anchor points. For example, in the text “Tom likes playing basketball, Peter also likes playing basketball,” the first word starting at the first anchor point and the second word starting at the second anchor point are both “basketball,” and both have a given score and the same degree of relevance to the media material. In this case, the positions of the first and second words are compared to rank them, with the former “basketball” ranking higher than the latter.

[0054] In some embodiments, such as Figure 9 Steps 901 to 904, as shown, determine candidate media materials for the anchor point. Step 901 includes: in response to determining that a first candidate media material in a given source includes a set of media materials and a text fragment starting from the anchor point, determining the first candidate media material as candidate media material for the anchor point. Step 902 includes: in response to determining that the text fragment starting from the anchor point includes a person's name, determining a second candidate media material including that person from the set of media materials as candidate media material for the anchor point. Step 903 includes: in response to determining that the text fragment starting from the anchor point includes a location's name, determining a third candidate media material including that location from the set of media materials as candidate media material for the anchor point. Step 904 includes: in response to determining that the text fragment starting from the anchor point is relevant to the topic, determining a fourth candidate media material relevant to the topic from the set of media materials as candidate media material for the anchor point.

[0055] In some implementations, the subject is finance. Step 904 includes: in response to determining that the text fragment starting from the anchor is a fragment of a financial article, determining media material related to stocks from the media material set. In some implementations, the subject is an online encyclopedia. In some implementations, the online encyclopedia is Wikipedia or Baidu Encyclopedia. Step 904 includes: in response to determining that the text fragment starting from the anchor is included in the online encyclopedia, determining media material included in the online encyclopedia from the media material set.

[0056] In some embodiments, operation ii) includes steps 1001 and 1002. Step 1001 includes determining the degree of relevance of the anchor point to each of one or more candidate media materials. Step 1002 includes determining the anchor point with the highest degree of relevance as the target anchor point. The method for determining the degree of relevance has been described above and will not be repeated here.

[0057] In some embodiments, operation ii) includes: selecting the highest-ranked anchor point in the set of anchor points as the target anchor point.

[0058] In some embodiments, operation ii) includes steps 1101 to 1103. Step 1101 includes determining that matching media material has been determined for the anchor point adjacent to the highest-ranked anchor point. Step 1102 includes removing the highest-ranked anchor point from the anchor point set. Step 1103 includes determining the second-ranked anchor point as the target anchor point.

[0059] In some embodiments, media material that matches a sentence anchor is identified and removed from the media material set; then, the word anchor adjacent to the sentence anchor is identified as the highest-ranking anchor in the media material set; the highest-ranking word anchor adjacent to the sentence anchor is removed from the anchor set; and the second-ranking word anchor is identified as the target anchor.

[0060] Step 203 includes aligning the text fragment with the corresponding matching media material.

[0061] In some embodiments, the start time of the text fragment is aligned with the start time of the matching media material. In some embodiments, audio corresponding to the text fragment is generated, the playback time of anchor points in the audio of the entire text is determined, and the start time of the matching media material is aligned with the playback time of anchor points in the audio of the entire text. After alignment, the audio and media material are combined to generate a video.

[0062] By aligning text with media materials, the efficiency and automation of short video production have been improved. Typically, the alignment quality of the final short video is comparable to that of human editors. Furthermore, due to a unique anchor selection algorithm, the final video is often more appealing to viewers than one edited manually.

[0063] Figure 12 A method for aligning text with media material according to some other embodiments of the present disclosure is shown. The method includes steps 1201 to 1205.

[0064] Steps 1201 and 1202 are the same as steps 201 to 202. This description can be found in the description of steps 201 to 202.

[0065] Step 1203 includes: generating audio for the text segment.

[0066] In some embodiments, an online speech generation platform is used to generate audio for the text segment. In some embodiments, an audio generation application installed on a local computer is used to generate audio for the text segment. In some embodiments, the online speech generation platform is any available online platform.

[0067] In some embodiments, audio is generated for each text segment, and each audio segment has a playback duration. For example... Figure 13 As shown, this is the timeline for generating anchor points. Figure 13 In this context, each anchor point corresponds to the start of an audio segment. In some embodiments, for a second audio segment of a second text segment, the playback time of the second anchor point, which serves as the start of the second text segment, is determined by the first playback duration of the first audio segment preceding the second anchor point. The duration for playing a word in the second audio segment is calculated by dividing the second duration by the number of words in the second audio segment, and then the playback time of a specific word in the second audio segment is the sum of the first duration and the duration of the words preceding the specific word in the second text segment.

[0068] Step 1204 includes setting the start time of each audio clip to the start time of the corresponding matching media clip.

[0069] In some embodiments, the duration for playing a single frame is set, for example, 4 seconds. In some embodiments, the original duration for playing video clips is maintained.

[0070] Step 1205 includes: in response to determining that there is no matching media material to be played within the audio time interval, determining supplementary media material from the set of media materials based on the supplementary score between the supplementary media material and the text segment to be played within the time interval.

[0071] In some embodiments, the supplementary score is determined by the semantic relevance between the source of the text segment to be played within the time interval, the supplementary media material, and the text segment to be played within the time interval, as well as the quality of the supplementary media material. In some embodiments, the semantic relevance between the supplementary media material and the text segment to be played within the time interval is determined by semantic comparison using an erine (enhanced representation via knowledge integration) model. In some embodiments, the quality of the supplementary media material includes aspect ratio, resolution, etc. In some implementations, the supplementary score is determined by a weighted sum of ernie_score, source_score, quality_score, and penalty_score, for example, by the formula: 0.4×ernie_score + 0.45×source_score + 0.05×quality_score - penalty_score, where ernie_score is the score determined by the ernie model, source_score is determined by a preset table of scores and sources, quality_score is determined by a preset table of scores and quality parameters, and penalty_score is determined by a preset table of penalty scores and conditions. In some embodiments, the scenarios include: a first scenario where media material and text fragments are included in a given source, but the media material has a long distance to the text fragments (i.e., the number of paragraphs is too large, such as 10 or more); a second scenario where the score calculated by the Ernie model is too low (such as only 10% or 20% of the average score); or a third scenario where objects identified in the media material are not included in the text fragments. In some embodiments, the maximum score for each of ernie_score, source_score, quality_score, and penalty_score is set to a given value, such as 1.

[0072] It should be noted that any score or relevance level in this paper can be set as needed.

[0073] In some cases, a black screen occurs during video playback. To avoid black screens, in some embodiments, the time interval in which the black screen occurs is located, and the playback duration of all images within that time interval is extended to fill the black screen without moving the position of media clips matching the anchor point. In some embodiments, if there are no images in the interval with the black screen, the black screen is filled by extending the duration of playing images in the interval preceding the black screen interval to minimize the impact on the media clips. In some embodiments, if there are no images before or in the interval with the black screen, the black screen is filled by extending the duration of playing all images or by playing a video clip more than once. In some embodiments, a playback threshold is set. In some embodiments, in response to determining that the black screen is filled or there is no black screen, the media clip with the longest playback duration is found, a portion of the longest playback duration exceeding the playback threshold is removed, and the duration used for playing images is extended such that the total duration used for playing all matching media clips remains unchanged.

[0074] Figure 14 A method for controlling the interface to generate video is shown.

[0075] The method includes steps 1401 to 1404.

[0076] Step 1401 includes displaying a first interface with an input field. Step 1402 includes entering a web address into the input field.

[0077] In some embodiments, the web address is a URL (Uniform Resource Locator).

[0078] Step 1403 includes: in response to inputting a web address into an input box, displaying a second interface with multiple selectable options for setting video parameters. Step 1404 includes: in response to selecting an option in the second interface, performing operations including: retrieving text from a webpage based on the web address; generating a video based on parameters, text fragments of the text, and corresponding matching media material; and playing the video on the first interface. Figure 15Examples of a first interface and a second interface according to some embodiments of the present disclosure are shown. As shown in the first interface, when a URL is entered into an input box, a second interface with multiple input boxes is displayed in response to the input URL. As shown in the second interface, the input boxes are related to a text-to-speech platform, video length, and resolution. The value in each input box is selectable (e.g., by touching each input box, an options menu is displayed, and options can be selected by touching items in the menu). In some embodiments, by entering the selected parameters (e.g., by touching a start button), a webpage is accessed based on a URL, an article comprising text and media material is obtained from the webpage, a search is performed based on the text, a set of media material is obtained by the search, multiple anchors in the text are identified and sorted, matching media material is identified and aligned with each anchor, audio of the text is generated, and video is generated by combining the aligned anchors with the matching media material.

[0079] In some embodiments, a video is generated using default values ​​by entering a URL in a first interface without proceeding to a second interface. In other embodiments, a third interface is displayed for setting scores for different sources, confidence scores for different types of images or videos, or other scores to be used in video generation. These interfaces help users control the computer-generated video.

[0080] like Figure 16 As shown, the device includes one or more processors 1601, a memory 1602, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be mounted on a common motherboard or otherwise as needed. The processors can process instructions executed within the device, including instructions stored in or on memory, to display graphical information of a GUI on an external input / output device (e.g., a display device coupled to an interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple storage devices if desired. Similarly, multiple devices can be connected, and the devices provide some of the necessary operations, such as serving as a server array, a group of blade servers, or a multiprocessor system. Figure 16 In this example, processor 1601 is used.

[0081] The memory 1602 is a non-transitory computer-readable storage medium provided in this disclosure. The memory stores instructions executable by at least one processor, causing the at least one processor to perform a method for aligning text with media material according to some embodiments of the disclosure. The non-transitory computer-readable storage medium of this disclosure stores computer instructions for causing a computer to perform a method for aligning text with media material according to some embodiments of the disclosure.

[0082] The memory 1602, serving as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions / modules corresponding to a method for aligning text with media material according to some embodiments of the disclosure. The processor 901 executes the non-transitory software programs, instructions, and modules stored in the memory 902 to perform various functional applications and data processing of the server, i.e., to implement the method for aligning text with media material according to some embodiments of the disclosure.

[0083] Memory 1602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one required functional application; and the data storage area may store data created using means of a method for aligning text with media material according to some embodiments of the disclosure. Furthermore, memory 902 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 902 may optionally include memory remotely disposed relative to processor 901, and such remote memory may be connected to means of a method for aligning text with media material according to some embodiments of the disclosure. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0084] The apparatus for performing a method for aligning text with media material according to some embodiments of the present disclosure may further include an input device 1603 and an output device 1604. The processor 1601, memory 1602, input device 1603, and output device 1604 may be connected via a bus or other means. Figure 16 In this example, a connection via a bus will be used.

[0085] Input device 1603 can receive input numerical or character information and generate key signal inputs related to the user settings and functional control of the device used for learning knowledge representation methods. Such devices include touchscreens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, joysticks, and other input devices. Output device 904 may include display devices, auxiliary lighting devices (e.g., LEDs), haptic feedback devices (e.g., vibration motors), etc. Display devices may include, but are not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touchscreen.

[0086] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order. No limitation is made here as long as the desired results of the technical solutions disclosed in this disclosure can be achieved.

[0087] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure will be included within the scope of protection of this disclosure.

Claims

1. A method for aligning text with media material, the method comprising: Based on the grammatical structure of the text, a set of anchor points in the text is determined, where each anchor point is the starting position of a text segment of the text; Repeat operations i) to v) until all anchors have been removed from the anchor set or all media have been removed from the media set: i) Sort the anchors in the anchor set based on the text segments starting from the anchor point. ii) Based on the sorted anchor points in the set, select a target anchor point from the set of anchor points. iii) Identify media materials from the media material set that match the text segment starting from the target anchor point. iv) Remove the target anchor point from the set of anchor points. v) Remove media materials from the media material set that match the text segment starting from the target anchor; and Align the text fragment with the corresponding matching media material; The sorting of anchors in the anchor set based on text segments starting from the anchors includes: sorting the anchors in the anchor set based on the type of the anchors, wherein the type of anchors includes word anchors as the beginning of a word and sentence anchors as the beginning of a sentence, wherein the sentence anchors are sorted above the word anchors.

2. The method according to claim 1, wherein, Determining the set of anchor points in the text based on its grammatical structure includes: identifying the starting position of a word or sentence as an anchor point in the set of anchor points.

3. The method according to claim 1, wherein, Sort the anchors in the anchor set based on the text fragments starting from the anchors, including: determining that two anchors are sentence anchors, and sorting the two anchors based on the relevance of the two anchors to the candidate media materials of the two anchors.

4. The method according to claim 3, wherein, The media materials in the media material set are obtained from one or more sources, and the relevance of each of the two anchor points to the candidate media materials of the two anchor points is determined by the following steps: determining the source of the candidate media material, determining the source score of the source of the candidate media material based on a preset correspondence between the score and the source, determining the confidence score of the candidate media material, and determining the relevance of the anchor point to the candidate media material by multiplying the source score and the confidence score.

5. The method according to claim 1, wherein, Sort the anchors in the anchor set based on the text segment starting from the anchor point, which further includes: determining that both anchors are word anchors, and sorting the two anchors based on their scores, wherein the scores of the two anchors are calculated based on the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm.

6. The method according to claim 5, wherein, Sort the anchors in the anchor set based on the text fragment starting from the anchor point, further comprising: determining that both anchors are word anchors with a given score, and sorting the two anchors based on the degree of relevance between each of the two anchors and the candidate media material of each anchor.

7. The method according to claim 6, wherein, The sorting of anchors in the anchor set based on the text segment starting from the anchor point further includes: determining that the two anchors are both word anchors with a given score, and that the two anchors have the same degree of relevance to the candidate media material; and sorting the two anchors based on their positions in the text, wherein the anchor located in the earlier position is ranked higher than the anchor located in the later position after the earlier position.

8. The method according to claim 6, wherein, The media materials in the media material set are obtained from one or more sources, and the candidate media materials for the anchor are determined by the following: In response to determining that a first candidate media material from the set of media materials is included in a given source and the text segment starting from the anchor point, the first candidate media material is determined as the candidate media material for the anchor point; In response to determining that the text segment starting from the anchor point includes a person's name, a second candidate media material including the person is determined from the media material set as the candidate media material for the anchor point; In response to determining that the text segment starting from the anchor point includes the name of a location, a third candidate media material including the location is determined from the media material set as the candidate media material for the anchor point; and In response to determining that the text fragment starting from the anchor point is relevant to the topic, a fourth candidate media material relevant to the topic is determined from the media material set as the candidate media material for the anchor point.

9. The method according to claim 1, wherein, The media materials in the media material set are obtained from one or more sources, and the media materials identified from the media material set that match the text segment starting from the target anchor include: Determine one or more candidate media materials for the text segment starting from the target anchor point, and determine from the one or more candidate media materials the media material that matches the text segment starting from the target anchor point, wherein the candidate media material for the target anchor point is determined by the following: In response to determining that a first candidate media material from the media material set and the text segment starting from the target anchor are included in a given source, the first candidate media material is determined as the candidate media material for the target anchor. In response to determining that the text segment starting from the target anchor includes a person's name, a second candidate media material including the person is determined from the media material set as the candidate media material for the target anchor; In response to determining that the text segment starting from the target anchor point includes the name of a location, a third candidate media material including the location is determined from the media material set as the candidate media material for the target anchor point; and In response to determining that the text fragment starting from the anchor point is relevant to the topic, a fourth candidate media material relevant to the topic is determined from the media material set as the candidate media material for the target anchor point.

10. The method according to claim 9, wherein, Determining the media material that matches the text segment starting from the target anchor from the one or more candidate media materials includes: determining the degree of relevance between the target anchor and each of the one or more candidate media materials; and determining the candidate media material with the highest degree of relevance as the media material that matches the text segment starting from the target anchor.

11. The method according to claim 9, wherein, Determining the relevance of the target anchor to each of the one or more candidate media materials includes: determining the relevance of the target anchor to each of the one or more candidate media materials based on the source of each candidate media material and the confidence level of each candidate media material.

12. The method according to claim 1, wherein, Selecting a target anchor point from the set of anchor points based on the sorted anchor points in the set includes: selecting the highest-ranked anchor point in the set of anchor points as the target anchor point.

13. The method according to claim 12, wherein, Selecting a target anchor point from the set of anchor points based on the sorted anchor points in the set includes: determining that media material has been determined for the anchor point adjacent to the highest sorted anchor point, removing the highest sorted anchor point from the set of anchor points, and determining the second sorted anchor point as the target anchor point.

14. The method according to claim 1, wherein, Aligning the text fragment with a corresponding matching media material includes: generating audio of the text fragment, aligning the audio with the corresponding matching media material, and the method further includes generating video based on the audio and the corresponding matching media material.

15. The method according to claim 14, wherein, Aligning the text segment with a corresponding matching media material includes: setting the start time of each of the audio segments to the start time of the corresponding matching media material; and in response to determining that there is no matching media material to be played within the time interval of the audio segments, determining the supplementary media material from the set of media materials based on a supplementary score between the supplementary media material and the text segment to be played within the time interval, wherein the supplementary score is determined by the source of the text segment to be played within the time interval, the semantic relevance between the supplementary media material and the text segment to be played within the time interval, and the quality of the supplementary media material.

16. The method according to claim 1, wherein, The method further includes: displaying a first interface with an input box; inputting a web address into the input box; and, in response to inputting the web address into the input box, displaying a second interface with multiple selectable options for setting parameters of the video.

17. The method according to claim 16, wherein, The method further includes: in response to determining the option in the second interface, performing operations including: retrieving the text from a webpage based on the web address; generating the video based on the parameters, the text fragment of the text, and the corresponding matching media material; and playing the video.

18. An apparatus comprising: processor; as well as A memory storing instructions that, when executed by the processor, cause the processor to perform operations, the operations including: The set of anchor points in the text is determined based on the grammatical structure of the text, and each anchor point is the starting position of a text segment of the text; Repeat operations i) to v) until all anchors have been removed from the anchor set or all media have been removed from the media set: i) Sort the anchors in the anchor set based on the text segments starting from the anchor point. ii) Based on the sorted anchor points in the set, select a target anchor point from the set of anchor points. iii) Identify media materials from the media material set that match the text segment starting from the target anchor point. iv) Remove the target anchor point from the set of anchor points. v) Remove media materials from the media material set that match the text segment starting from the target anchor point; and Align the text fragment with the corresponding matching media material; The sorting of anchors in the anchor set based on text segments starting from the anchors includes: sorting the anchors in the anchor set based on the type of the anchors, wherein the type of anchors includes word anchors as the beginning of a word and sentence anchors as the beginning of a sentence, wherein the sentence anchors are sorted above the word anchors.

19. A non-transitory machine-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations, the operations including: The set of anchor points in the text is determined based on the grammatical structure of the text, and each anchor point is the starting position of a text segment of the text; Repeat operations i) to v) until all anchors have been removed from the anchor set or all media have been removed from the media set: i) Sort the anchors in the anchor set based on the text segments starting from the anchor point. ii) Based on the sorted anchor points in the set, select a target anchor point from the set of anchor points. iii) Identify media materials from the media material set that match the text segment starting from the target anchor point. iv) Remove the target anchor point from the set of anchor points. v) Remove media materials from the media material set that match the text segment starting from the target anchor point; as well as Align the text fragment with the corresponding matching media material; The sorting of anchors in the anchor set based on text segments starting from the anchors includes: sorting the anchors in the anchor set based on the type of the anchors, wherein the type of anchors includes word anchors as the beginning of a word and sentence anchors as the beginning of a sentence, wherein the sentence anchors are sorted above the word anchors.

Images