Multimedia Processing Method, Apparatus, Device, and Medium

Abstract

The present disclosure discloses a multimedia processing method, apparatus, device and medium, the multimedia processing method including: displaying initial text content corresponding to a target multimedia content; and playing the related multimedia content in response to a trigger operation of the related multimedia content, where the related multimedia content includes a fragment of the target multimedia content that is related to the target text content, and the target text content is extracted from the initial text content.

Description

Technical Field

[0001] This application was filed with the China Patent Office on June 8, 2021, claiming the priority of a Chinese patent application with the application number 202110637048.3, and all its contents are incorporated herein by reference.

[0002] This disclosure relates to the technical field of conference identification, for example, to multimedia processing methods, devices, apparatuses, and media.

Background Art

[0003] With the continuous development of multimedia technology, online communication has become increasingly used in daily life and office life because of its excellent performance in communication efficiency and information retention.

[0004] In some related products, the process of online communication can be recorded to generate a multimedia file, and the content can be checked again after the online communication ends. However, when the communication time is long, it takes time to play the multimedia file to check the content, which is inefficient for obtaining important information from the communication.

Summary of the Invention

Means for Solving the Problems

[0005] This disclosure provides a multimedia processing method, device, apparatus, and medium.

[0006] This disclosure provides a multimedia processing method, the method comprising: displaying initial text content corresponding to target multimedia content; and responding to a trigger operation of related multimedia content to play the related multimedia content. Here, the related multimedia content includes fragments related to the target text content among the target multimedia content, and the target text content is extracted from the initial text content.

[0007] The present disclosure further provides a multimedia processing apparatus, the apparatus comprising: a text display module configured to display initial text content corresponding to target multimedia content; and a multimedia playback module configured to play the related multimedia content in response to a trigger operation on the related multimedia content. Here, the related multimedia content includes fragments related to the target text content among the target multimedia content, and the target text content is extracted from the initial text content.

[0008] The present disclosure further provides an electronic device, the electronic device comprising a processor and a memory storing executable instructions of the processor, the processor being configured to read the executable instructions from the memory and execute the instructions to implement the above multimedia processing method.

[0009] The present disclosure further provides a computer-readable storage medium, the storage medium storing a computer program, the computer program being used to execute the above multimedia processing method.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, the present disclosure can be implemented in various forms, and these embodiments are used for the purpose of understanding the present disclosure. The accompanying drawings and embodiments of the present disclosure are for illustrative purposes only.

[0012] It should be understood that the multiple steps described in the method embodiments of the present disclosure may be executed in a different order and / or in parallel. Furthermore, method embodiments can add steps and / or omit the steps shown. The scope of the present disclosure is not particularly limited in this regard.

[0013] The term "comprising" and its variants used herein are open-ended, that is, it means "including but not limited to". The term "based on" means "based at least in part on". The term "an embodiment" means "at least one embodiment", the term "another embodiment" means "at least one other embodiment", and the term "some embodiments" means "at least some embodiments". Related definitions of other terms are shown in the following description.

[0014] Concepts such as "first", "second", etc. mentioned in the present disclosure are used to distinguish different devices, modules or units, and are not used to define the order or interdependence of the functions executed by these devices, modules or units.

[0015] It should be noted that the modifications of "one" and "a plurality" mentioned in this disclosure are illustrative rather than limiting, and those skilled in the art should understand them as "one or a plurality" unless otherwise explicitly stated in the context.

[0016] The names of messages or information that interact between multiple devices in the embodiments of this disclosure are used only for the purpose of description and are not intended to limit the scope of those messages or information.

[0017] FIG. 1 is a schematic flowchart of a multimedia processing method provided by an embodiment of this disclosure. The method can be executed by a multimedia processing device, which can be implemented using software and / or hardware and may generally be integrated into an electronic device. As shown in FIG. 1, the method includes the following steps.

[0018] Step 101: Display initial text content corresponding to target multimedia content.

[0019] The target multimedia content may be multimedia content that records the communication process between multiple users, such as multimedia content for recording a meeting. In this embodiment, the format of the target multimedia content is not particularly limited. For example, the format of the target multimedia content may be audio and / or video. The initial text content may be the text content after identifying and processing the target multimedia content, such as the text content obtained by voice-identifying the target media content.

[0020] In the embodiment of this disclosure, the terminal device can obtain the initial text content corresponding to the target multimedia content and display the initial text content in the text area of the display interface. Optionally, a multimedia area for displaying the above target multimedia content may be further set in the display interface.

[0021] Step 102, in response to a trigger operation on related multimedia content, play the related multimedia content, where the related multimedia content includes a fragment related to the target text content among the target multimedia content, and the target text content is extracted from the initial text content.

[0022] The target text content may be key content generated by extraction and processing based on the initial text content. For example, when the initial text content is meeting subtitles, the target text content may be the main content of the meeting extracted from the meeting subtitles, that is, the meeting summary. The related multimedia content may be a multimedia fragment related to the target text content among the target multimedia content. For example, when the target text content is the meeting summary, the related multimedia content may be the multimedia summary, such as the meeting summary in audio format and / or video format.

[0023] In an embodiment of the present disclosure, the terminal device can detect the user's trigger operation through a built-in detection procedure or algorithm. After receiving the trigger operation on the related multimedia content by the user, obtain the pre-generated related multimedia content and play the related multimedia content. For example, the related multimedia content may be played in a preset area of the display interface where the initial text content is located, or a new display interface may be opened and the related multimedia content may be played in the new display interface, and the new display interface may be floatingly displayed on the display interface where the initial text content is located.

[0024] The triggering operation of the associated multimedia content may be a triggering operation of a setting button or setting information set in the display interface of the initial text content. In this embodiment, the triggering operation method of the associated multimedia content is not limited. For example, it may be a click operation and / or a hover operation. The above setting button and setting information are set according to the actual situation. For example, the setting button may be one button set around the target text content.

[0025] Exemplarily, FIG. 2 is a schematic diagram of multimedia playback provided by an embodiment of the present disclosure. For example, as shown in FIG. 2, taking a meeting scenario as an example, the initial text content is meeting subtitles, the target text content is a meeting summary, the target multimedia content is a meeting video, and the associated multimedia content is a multimedia summary. FIG. 2 shows a first display interface 10, in which a meeting summary is displayed in a first area 11 of the first display interface 10, a meeting video is displayed at the top of the first area 11, meeting subtitles are displayed in a second area 12, and a playback timeline of the meeting video or meeting audio may be displayed at the bottom of the first display interface 10. The second display interface 20 may be floatingly displayed on the upper surface of the first display interface 10. When the second display interface 20 is displayed, the first display interface 10 may be displayed dimly to emphasize the second display interface 20. A button 13 may be set in the first display interface 10. When the user triggers the button 13, the multimedia summary is played in a first area 21 of the second display interface 20, and a playback timeline of the multimedia summary is displayed in the second display interface 20. The setting position of the above button 13 is only an example.

[0026] The multimedia processing solution provided by the embodiments of the present disclosure includes the steps of displaying initial text content corresponding to target multimedia content, and playing related multimedia content in response to a trigger operation on the related multimedia content, where the related multimedia content includes fragments related to the target text content among the target multimedia content, and the target text content is extracted from the initial text content. By the above technical solution, target text content and corresponding related multimedia content are generated based on the initial text content. When the user executes a trigger operation, the related multimedia content can be played. When the user views the related multimedia content, partial content (such as important content) of the initial text content can be quickly understood, saving time, helping the user to understand the initial text content, and thus improving the user experience effect.

[0027] In some embodiments, the step of displaying the initial text content is to display the initial text content in response to a trigger operation on the list page, where summary information of a plurality of multimedia contents is displayed on the list page. The list page may be a page including a plurality of list items, each list item corresponding to one multimedia content. For example, the list item may display summary information of the multimedia content. In addition to the summary information of the multimedia content, the list item may also display the number or recording date of the multimedia content, etc., which is not particularly limited in this embodiment. The user realizes the trigger operation on the list page by selecting the list item corresponding to the target multimedia content on the list page, and the terminal device displays the initial text content corresponding to the target multimedia content in response to the trigger operation.

[0028] In some embodiments, the target text content includes at least two consecutive text fragments extracted from the initial text content. Optionally, in response to a trigger operation of the associated multimedia content, the step of playing the associated multimedia content is to jump and play, according to the order in the target multimedia content, the multimedia fragments corresponding to the associated time zones of each consecutive text fragment among the target multimedia content for the trigger operation of the associated multimedia content, for the associated time zones of at least two consecutive text fragments in the target text content.

[0029] The above target text content includes at least two consecutive text fragments. The consecutive text fragments may be text fragments extracted from the initial text content. Here, there is a correlation before and after between the text fragments extracted by being continuously characterized in this way. That is, in this way, the target text content is regarded as an integral whole, and the multiple text fragments included therein are regarded as consecutive. For example, when the target text content is a meeting summary, the meeting summary may include multiple summaries, and here the meeting summary is processed as a whole subsequently.

[0030] The associated time zone of the consecutive text fragment refers to the time zone in the target multimedia content of the multimedia fragment corresponding to the consecutive text fragment, and the associated time zone may include a start time and an end time.

[0031] In an embodiment of the present disclosure, after receiving a trigger operation for related multimedia content, the terminal device plays the related multimedia content. During the playback process, the multimedia fragments corresponding to the related time ends of each consecutive text fragment in the target multimedia content are jumped and played in the order in the target multimedia content of at least two consecutive text fragments. That is, a part of the related multimedia content corresponding to at least two consecutive text fragments in the target multimedia content may be played sequentially, and thus the related multimedia content may be played. In this way, since the fragment corresponding to the target text content in the target multimedia content can be directly jumped and played on the display interface of the target multimedia content, the user can quickly understand the focus of the meeting and save time.

[0032] In some embodiments, the multimedia processing method may further include the step of determining related multimedia content based on the target text content. Optionally, the step of determining related multimedia content based on the target text content includes generating related multimedia content based on the related time period of the target text content, where the related time period of the target text content is used to characterize the time period in the target multimedia content of the audio information corresponding to the target text content.

[0033] The target text content is extracted from the initial text content but is not the same as the initial text content. Therefore, the relevant time period of the target text content refers to the time period in the target multimedia content of the audio information of the text corresponding to the target text content in the above initial text content. In this embodiment, the relevant multimedia content can be generated based on the relevant time period of the target text content. Here, the relevant multimedia content may be a separate multimedia file. The relevant multimedia content may be generated by a terminal device or a server and is not particularly limited in this embodiment.

[0034] Optionally, the step of generating the relevant multimedia content based on the relevant time period of the target text content includes, when the number of relevant time periods is plural, splicing a plurality of multimedia fragments corresponding to each of the plurality of relevant time periods according to the order in the target multimedia content of the plurality of relevant time periods to generate the relevant multimedia content. After cutting out the multimedia fragments from the target multimedia content based on each relevant time period, the relevant multimedia content can be obtained by splicing the cut-out plurality of multimedia fragments according to the order in the target multimedia content of the plurality of relevant time periods.

[0035] Optionally, the step of generating the relevant multimedia content based on the relevant time period of the target text content includes adjusting the relevant time period of the target text content according to the statement completeness of the relevant text of the target text content, and generating the relevant multimedia content based on the adjusted relevant time period. Here, the relevant text is the text corresponding to the target text content in the initial text content.

[0036] The target text content is obtained by processing the initial text content, and the initial text content is obtained by voice recognition of the voice information in the target multimedia content. Therefore, the related text of the target text content refers to the text corresponding to the target text content in the initial text content. For example, when the target text content is a meeting summary, the related text of the meeting summary may be the subtitle text corresponding to the meeting summary. The related text of the target text content corresponds to the above-mentioned related time period.

[0037] Since the related text of the target text content may be incomplete, in the embodiments of the present disclosure, the terminal device detects the related text of the target text content according to a voice activity detection algorithm (Voice Activity Detection, VAD) and / or a semantic detection algorithm, and extends the related time period of the target text content according to statement completeness and consistency. Here, the extension process refers to, for example, shifting the start time corresponding to the related time period forward and / or shifting the end time corresponding to the related time period backward. Then, related multimedia content is generated based on the related time period after the extension process. The related time period after the extension process is called the target time period, and according to the order in the target multimedia content of multiple target time periods, multiple multimedia fragments corresponding to each of the multiple target time periods are spliced to generate related multimedia content. In this way, by extending the related time period to obtain the target time period, context information can be provided based on ensuring the completeness of the target content corresponding to the cut multimedia fragment.

[0038] Exemplarily, when the target text content is a meeting summary, the relevant time range of one meeting summary is 10S to 20S. When it is determined by detection that the 10S time point is in the middle of a complete sentence or paragraph, it is extended forward by 5S. That is, the finally determined time range after the extension process is 5S to 20S.

[0039] Optionally, during the generation process of the multimedia summary, when splicing multimedia fragments, the voice and screen are processed to add fade-in and fade-out effects, avoiding the abruptness that may be caused by fragment cutting during subsequent playback, and improving the browsing effect of the user.

[0040] In the above method, according to the relevant text of the target text content, the relevant multimedia content containing important information can be generated by condensing through splicing, or, considering the integrity of the statement, the above relevant multimedia content can be generated according to the relevant time range after the extension process, ensuring the integrity of important information. The user can quickly understand the focus by viewing the relevant multimedia content, which contributes to the user's understanding of the focus.

[0041] In some embodiments, the relevant multimedia content is a multimedia summary. The multimedia summary includes at least two sub-multimedia summaries. Each sub-multimedia summary corresponds to one summary type, and the number of summary types is at least two. Optionally, in response to the trigger operation of the relevant multimedia content, the step of playing the relevant multimedia content includes determining the target summary type corresponding to the trigger operation in response to the trigger operation of the relevant multimedia content, obtaining the target sub-multimedia summary corresponding to the target summary type and playing the target sub-multimedia summary, or obtaining the multimedia summary and playing the multimedia summary in the time range in the multimedia summary based on the target summary type.

[0042] In this embodiment, when the target text content is a text summary, there are multiple summary types. The summary types in the embodiments of the present disclosure may include topics, meeting items, discussions, conclusions, ToDo tasks, etc. Each summary type may include an unlimited number of text summaries. In the embodiments of the present disclosure, a corresponding sub-multimedia summary is generated for each summary type, that is, the multimedia summary may include a plurality of sub-multimedia summaries, and the plurality of sub-multimedia summaries respectively correspond to a plurality of summary types. The generation method of the sub-multimedia summary is the same as the generation method of the above complete multimedia summary, and will not be repeated in this specification.

[0043] In the embodiments of the present disclosure, after receiving a trigger operation of related multimedia content by the user, the trigger operation is used to play a sub-multimedia summary of one summary type. First, determine the target summary type corresponding to the playback trigger operation, obtain the target sub-multimedia summary corresponding to the target summary type, and play the target sub-multimedia summary through a summary playback interface. Alternatively, after determining the target summary type, obtain the complete multimedia summary, and play it from the corresponding time period, that is, the time period of the multimedia summary of the target sub-multimedia summary in the multimedia summary, according to the target summary type.

[0044] Exemplarily, for example, as shown in FIG. 2, a button 13 for allowing the user to play only the sub-multimedia summary corresponding to the summary type may be provided near each summary type on the first display interface 10.

[0045] In the above method, the related multimedia content is a multimedia summary, and corresponding sub-multimedia summaries can be generated for different summary types, supporting the user to play only the sub-multimedia summary for one summary type, improving the display flexibility of the multimedia summary, further facilitating the user to focus on and understand the key points of the meeting, and ultimately improving the user experience effect.

[0046] In some embodiments, the target text content is a target text summary, the target text summary includes summaries of at least one summary type, and the multimedia processing method may further include associating and displaying a label of the summary type corresponding to the target text summary on the playback timeline of the related multimedia content. Optionally, associating and displaying a label of the summary type corresponding to the target text summary on the playback timeline of the related multimedia content includes displaying a label of the summary type corresponding to the target text summary at a relevant time corresponding to the target text summary on the playback timeline of the related multimedia content. Here, the relevant time corresponding to the target text summary is any time within the relevant time range of the target text summary.

[0047] When the target text content is the target text summary, the target text summary may include one or more summaries of summary types. In an embodiment of the present disclosure, for related multimedia content, first determine the summary type of each summary included in the corresponding target text summary, and on the playback timeline of the related multimedia content, display the label of the corresponding summary type at the related time point or related time interval corresponding to the target text summary. The label of the above summary type is used to characterize the summary type. In this embodiment, the characterization format of the summary type is not limited, and for example, text, subtitles or graphic expressions may be used. The related time point corresponding to the target text summary may be set according to the actual situation. For example, the initial time point, the intermediate time point or the end time point of the related time zone of the target text summary may be set as the related time point, or a certain time zone from the initial time point, a certain time zone from a certain time point between the initial time point and the end time point, or a certain time zone from a certain time point between the initial time point and the end time point to the end time point may be set as the related time interval.

[0048] In the above method, when the target text content is the target text summary, the terminal device may display the labels of the summary types of different summaries to present the corresponding summary types to the user, which conforms to the actual needs of the user and can improve the efficiency of the user's understanding of the meeting focus.

[0049] In some embodiments, the multimedia processing method may further include the step of sequentially highlighting the target text content corresponding to the playback progress of the related multimedia content during the playback process of the related multimedia content.

[0050] In an embodiment of the present disclosure, the terminal device displays target text content, and during the playback process of related multimedia content, it sequentially determines related time periods corresponding to the playback progress of the related multimedia content, and then determines text content corresponding to the playback progress in the target text content, and the text content corresponding to the playback progress may be highlighted. In an embodiment of the present disclosure, the highlighting method is not particularly limited, and any possible display method that can be distinguished from other meeting summaries may be used. For example, it may include at least one of highlighting, bold display, and underlining, but is not limited thereto.

[0051] Exemplarily, referring to FIG. 2, a meeting summary may be displayed in the second area 22 in the second display interface 20. As shown in FIG. 2, as can be seen from the playback timeline, the playback progress of the multimedia summary is in the middle, and the meeting summary with the summary type being a meeting item corresponds to the playback progress and is highlighted by underlining.

[0052] In the above method, based on playing the related multimedia content, the target text content is displayed, and target text content-related interactive display is supported during the playback process of the related multimedia content, so that the user can intuitively understand the relationship between the target text content and the related multimedia content, and the interactive experience effect of the user can be improved.

[0053] In some embodiments, the multimedia processing method may further include receiving a download operation of related multimedia content by the user and downloading and storing the related multimedia content.

[0054] The client can detect the user's operation, receive the download operation of relevant multimedia content by the user, download the relevant multimedia content, and store the file of the relevant multimedia content. Here, the download operation may be an operation in which the user triggers a setting button or setting information on the display interface of the relevant multimedia content.

[0055] For example, referring to FIG. 2, when the user clicks the "Download Multimedia" button at the lower right corner of the second display 20, the multimedia summary can be downloaded and stored. Further, referring to FIG. 2, when the user clicks "Document Export" on the summary playback interface, the user can export the text file of the meeting summary for use.

[0056] In the above method, since the relevant multimedia content supports the user to download its file, the user can use the file according to actual needs, and the user experience effect can be improved.

[0057] FIG. 3 is a schematic flowchart of another multimedia processing method provided by an embodiment of the present disclosure. This embodiment optimizes the above multimedia processing method based on the above embodiment. As shown in FIG. 3, the method includes the following steps.

[0058] Step 201, display initial text content.

[0059] The initial text content corresponds to the target multimedia content.

[0060] Step 202, determine relevant multimedia content based on the target text content.

[0061] The related multimedia content includes fragments related to the target text content among the target multimedia content, and the target text content is extracted from the initial text content.

[0062] Step 202 may be executed before or after step 201 and is not particularly limited in this embodiment.

[0063] Optionally, the step of determining the related multimedia content based on the target text content includes generating the related multimedia content based on the related time period of the target text content, where the related time period of the target text content is used to characterize the time period in the target multimedia content of the voice information corresponding to the target text content.

[0064] Optionally, generating the related multimedia content based on the related time period of the target text content includes, when there are multiple related time periods, splicing multiple multimedia fragments corresponding to each of the multiple related time periods according to the order in the target multimedia content according to the multiple related time periods to generate the related multimedia content.

[0065] Optionally, generating the related multimedia content based on the related time period of the target text content includes adjusting the related time period of the target text content according to the statement completeness of the related text of the target text content, and generating the related multimedia content based on the adjusted related time period. Here, the related text is the text corresponding to the target text content in the initial text content.

[0066] Step 203, in response to the trigger operation of the related multimedia content, plays the related multimedia content.

[0067] The target text content includes at least two consecutive text fragments extracted from the initial text content.

[0068] Optionally, playing the related multimedia content in response to a trigger operation of the related multimedia content includes jumping and playing, in accordance with the order in the target multimedia content of the related time zones of at least two consecutive text fragments in the target text content, the multimedia fragments corresponding to the related time zones of each of the at least two consecutive text fragments in the target multimedia content, in response to the trigger operation of the related multimedia content.

[0069] In some embodiments, the related multimedia content is a multimedia summary, the multimedia summary includes at least two sub-multimedia summaries, each sub-multimedia summary corresponds to one summary type, and the number of summary types is at least two.

[0070] Optionally, playing the related multimedia content in response to a trigger operation of the related multimedia content includes determining the target summary type corresponding to the trigger operation, obtaining and playing the target sub-multimedia summary corresponding to the target summary type, or obtaining the multimedia summary and playing the multimedia summary in the time zone in the multimedia summary based on the target summary type, in response to the trigger operation of the related multimedia content.

[0071] After step 203, step 204 and / or step 205 may be executed, and it is not particularly limited in this embodiment.

[0072] In step 204, during the playback process of the related multimedia content, the target text content corresponding to the playback progress of the related multimedia content is sequentially highlighted.

[0073] Step 205: Receive a download operation of related multimedia content by the user, and download and store the related multimedia content.

[0074] In this embodiment, the target text content is a target text summary, the target text summary includes at least one summary of a summary type, and the multimedia processing method is to relevantly display a label of the summary type corresponding to the target text summary on the playback timeline of the related multimedia content. Optionally, relevantly displaying a label of the summary type corresponding to the target text summary on the playback timeline of the related multimedia content includes displaying a label of the summary type corresponding to the target text summary at a relevant time corresponding to the target text summary on the playback timeline of the related multimedia content. Optionally, the relevant time corresponding to the target text summary is any time within the relevant time period of the target text summary.

[0075] The multimedia processing solution provided by the embodiments of the present disclosure displays initial text content, determines related multimedia content based on target text content extracted from the initial text content, plays the related multimedia content in response to a trigger operation of the related multimedia content, sequentially highlights the target text content corresponding to the playback progress of the related multimedia content during the playback process of the related multimedia content, receives a download operation of the related multimedia content by the user, and downloads and stores the related multimedia content. By the above technical solution, target text content and corresponding related multimedia content are generated based on the initial text content, the related multimedia content can be played when the user performs a trigger operation, the user can watch the related multimedia content and quickly understand the focus of the initial text content, saving time, helping the user to understand the initial text content, and thus improving the user experience effect. The target text content is highlighted according to the playback progress of the related multimedia content, and the user can intuitively understand the relationship between the target text content and the related multimedia content, improving the user's interactive experience effect.

[0076] FIG. 4 is a schematic structural diagram of a multimedia processing device provided by an embodiment of the present disclosure. The device is implemented by software and / or hardware and may generally be integrated into an electronic device. As shown in FIG. 4, the device includes A text display module 301 configured to display initial text content corresponding to target multimedia content, and a multimedia playback module 302 configured to play back the related multimedia content in response to a trigger operation of the related multimedia content, wherein the related multimedia content includes fragments related to target text content among the target multimedia content, and the target text content is extracted from the initial text content.

[0077] Optionally, the target text content includes at least two consecutive text fragments extracted from the initial text content.

[0078] Optionally, the multimedia playback module 302 is configured to in response to a trigger operation of the related multimedia content, jump and play multimedia fragments corresponding to the related time zones of each consecutive text fragment among the target multimedia content in the order in which they appear in the target multimedia content for the related time zones of the at least two consecutive text fragments among the target text content.

[0079] Optionally, the apparatus further comprises a related multimedia generation module configured to determine the related multimedia content based on the target text content.

[0080] Optionally, the related multimedia generation module is configured to generate related multimedia content based on the related time zone of the target text content, where the related time zone of the target text content is used to characterize the time zone in the target multimedia content of the audio information corresponding to the target text content.

[0081] Optionally, the related multimedia generation module is configured to: when the number of the related time periods is plural, splice a plurality of multimedia fragments corresponding to each of the plurality of related time periods according to the order in the target multimedia content of the plurality of related time periods to generate the related multimedia content.

[0082] Optionally, the related multimedia generation module is configured to: adjust the related time period of the target text content based on the statement completeness of the related text of the target text content, and generate the related multimedia content based on the adjusted related time period.

[0083] Optionally, the related text is the text corresponding to the target text content in the initial text content.

[0084] Optionally, the related multimedia content is a multimedia summary, the multimedia summary includes at least two sub-multimedia summaries, each of the sub-multimedia summaries corresponds to one summary type, and the number of the summary types is at least two.

[0085] Optionally, the multimedia playback module 302 is configured to: in response to a trigger operation on the related multimedia content, determine a target summary type corresponding to the trigger operation, obtain a target sub-multimedia summary corresponding to the target summary type and play the target sub-multimedia summary, or obtain the multimedia summary and play the multimedia summary based on the time period of the target summary type in the multimedia summary.

[0086] Optionally, the target text content is a target text summary, the target text summary includes summaries of at least one summary type, and the apparatus further comprises a labeling module, configured to relevantly display a label of the summary type corresponding to the target text summary on the playback timeline of the related multimedia content.

[0087] Optionally, the labeling module is configured to display a label of the summary type corresponding to the target text summary at a relevant time corresponding to the target text summary on the playback timeline of the related multimedia content.

[0088] Optionally, the relevant time corresponding to the target text summary is any time during the relevant time period of the target text summary.

[0089] Optionally, the apparatus further comprises a highlighting module, configured to sequentially highlight the target text content corresponding to the playback progress of the related multimedia content during the playback process of the related multimedia content.

[0090] Optionally, the apparatus further comprises a summary download module, configured to receive a download operation of the related multimedia content by a user and download and store the related multimedia content.

[0091] Optionally, the text display module 301 is configured to display the initial text content in response to a trigger operation of a list page, and display summary information of a plurality of multimedia contents on the list page.

[0092] The multimedia processing apparatus provided by an embodiment of the present disclosure can execute the multimedia processing method provided by any embodiment of the present disclosure, and has a functional module and an effect corresponding to the method.

[0093] Embodiments of the present disclosure provide a computer program product including a computer program / instructions, and when the computer program / instructions are executed by a processor, the multimedia processing method provided by any embodiment of the present disclosure is realized.

[0094] FIG. 5 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. Hereinafter, reference is made to FIG. 5 showing a schematic structural diagram of an electronic device 400 suitable for realizing the embodiment of the present disclosure. The electronic device 400 in the embodiments of the present disclosure includes, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), in-vehicle terminals (such as in-vehicle navigation terminals), and digital TVs (Televisions, TVs), desktop computers, and the like. The electronic device 400 shown in FIG. 5 is merely an example and does not limit the functions and usage ranges of the embodiments of the present disclosure in any way.

[0095] As shown in FIG. 5, the electronic device 400 includes a processing device (such as a central processor, a graphics processor, etc.) 401 that can execute various appropriate operations and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. The RAM 403 further stores a plurality of programs and data necessary for the operation of the electronic device 400. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0096] Typically, the I / O interface 405 is connected to input devices 406 such as touchscreens, touch pads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, output devices 407 such as liquid crystal displays (LCDs), speakers, vibrators, storage devices 408 such as magnetic tapes, hard disks, and communication devices 409. The communication device 409 enables the electronic device 400 to communicate with other devices wirelessly or wiredly to exchange data. FIG. 5 illustrates the electronic device 400 equipped with various devices, but it is not necessary to implement or include all the illustrated devices. Alternatively, more or fewer devices may be implemented or included.

[0097] According to an embodiment of the present disclosure, the process described above with reference to the flowchart may be implemented as a computer software program. For example, an embodiment of the present disclosure provides a computer program product including a computer program carried on a non-transitory computer-readable medium, where the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When this computer program is executed by the processing device 401, the above functions defined in the multimedia processing method of the embodiment of the present disclosure are realized.

[0098] Note that the computer-readable medium described in the present disclosure may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. Examples of computer-readable storage media include electrical connections having one or more conductors, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above, but are not limited thereto. In the present disclosure, the computer-readable storage medium may be any tangible medium that includes or stores a program, and this program may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier, and carry computer-readable program code. Such a propagated data signal includes, but is not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium that transmits, propagates, or transfers a program used by or in combination with an instruction execution system, apparatus, or device. The program code included in the computer-readable medium may be transferred by any suitable medium such as a wire, an optical fiber cable, a radio frequency (RF), or any suitable combination of the above, but is not limited thereto.

[0099] In some embodiments, the client and the server may communicate using any currently known or future-developed network protocol, such as the HyperText Transfer Protocol (HTTP), and may be interconnected with digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("Local Area Network, LAN"), wide area networks ("Wide Area Network, WAN"), Internet networks (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), and any currently known or future-developed network.

[0100] The computer-readable medium may be included in the electronic device or may be separate and not incorporated into the electronic device.

[0101] The computer-readable medium stores one or more programs, and when the one or more programs are executed by the electronic device, the electronic device is caused to perform a step of displaying initial text content corresponding to target multimedia content, and a step of playing the related multimedia content in response to a trigger operation of the related multimedia content, where the related multimedia content includes a fragment related to target text content among the target multimedia content, and the target text content is extracted from the initial text content.

[0102] Computer program code for performing the operations of the present disclosure can be written in one or more programming languages or combinations thereof, including, but not limited to, object-oriented programming languages (such as Java, Smalltalk, C++), and conventional procedural programming languages (such as the "C" language or similar programming languages). The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, such as a LAN or WAN, or may be connected to an external computer (e.g., connected via the Internet using an Internet service provider).

[0103] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or part of code that includes one or more executable instructions for implementing the specified logical function. Also, it should be noted that in some alternative implementations, the functions shown in the boxes may occur in a different order than that shown in the accompanying drawings. For example, two boxes shown consecutively may actually be executed substantially in parallel, or depending on the related functions, may be executed in the reverse order. Also, each box in the block diagram and / or flowchart, and combinations of boxes in the block diagram and / or flowchart, may be implemented in a dedicated hardware-based system for performing the specified function or operation, or may be implemented in a combination of dedicated hardware and computer instructions.

[0104] The units described in the embodiments of the present disclosure may be implemented by software or by hardware. Here, the name of the unit does not constitute a limitation of the unit itself in a given situation.

[0105] The functions described above in this specification may be executed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and the like.

[0106] In the context of the present disclosure, a machine-readable medium may be a tangible medium that includes or stores a program used by or in combination with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium includes, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Examples of machine-readable storage media include electrical connections based on one or more wires, portable device disks, hard disks, RAM, ROM, EPROM, or flash memory, optical fibers, CD-ROMs, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0107] According to one or more embodiments of the present disclosure, the present disclosure provides a multimedia processing method, displaying initial text content corresponding to target multimedia content; and playing the related multimedia content in response to a trigger operation of the related multimedia content. Here, the related multimedia content includes fragments related to the target text content among the target multimedia content, and the target text content is extracted from the initial text content.

[0108] According to one or more embodiments of the present disclosure, in the multimedia processing method provided by the present disclosure, the target text content includes at least two consecutive text fragments extracted from the initial text content.

[0109] According to one or more embodiments of the present disclosure, in the multimedia processing method provided by the present disclosure, in response to a trigger operation on the related multimedia content, the step of playing the related multimedia content is In response to a trigger operation on the related multimedia content, jump to and play multimedia fragments corresponding to the related time zones of each consecutive text fragment in the target multimedia content in the order of the target multimedia content in the related time zones of the at least two consecutive text fragments in the target text content.

[0110] According to one or more embodiments of the present disclosure, in the multimedia processing method provided by the present disclosure, the method further includes a step of determining the related multimedia content based on the target text content.

[0111] According to one or more embodiments of the present disclosure, in the multimedia processing method provided by the present disclosure, the step of determining the related multimedia content based on the target text content is generating related multimedia content based on a related time period of the target text content, wherein the related time period of the target text content is used to characterize a time period in the target multimedia content of the voice information corresponding to the target text content.

[0112] According to one or more embodiments of the present disclosure, in the multimedia processing method provided by the present disclosure, the step of generating related multimedia content based on a related time period of the target text content includes when the number of the related time periods is plural, splicing a plurality of multimedia fragments corresponding to each of the plurality of related time periods according to an order in the target multimedia content of the plurality of related time periods to generate the related multimedia content.

[0113] According to one or more embodiments of the present disclosure, in the multimedia processing method provided by the present disclosure, the step of generating related multimedia content based on a related time period of the target text content includes adjusting the related time period of the target text content based on a statement completeness of the related text of the target text content, and generating the related multimedia content based on the adjusted related time period.

[0114] According to one or more embodiments of the present disclosure, in the multimedia processing method provided by the present disclosure, the related text is text corresponding to the target text content in the initial text content.

[0115] According to one or more embodiments of the present disclosure, in the multimedia processing method provided by the present disclosure, the related multimedia content is a multimedia summary, the multimedia summary includes at least two sub-multimedia summaries, each of the sub-multimedia summaries corresponds to one summary type, and the number of the summary types is at least two.

[0116] According to one or more embodiments of the present disclosure, in the multimedia processing method provided by the present disclosure, in response to a trigger operation on related multimedia content, the step of playing the related multimedia content includes determining a target summary type corresponding to the trigger operation in response to the trigger operation on the related multimedia content, obtaining a target sub-multimedia summary corresponding to the target summary type and playing the target sub-multimedia summary, or obtaining the multimedia summary and playing the multimedia summary based on the time period of the target summary type in the multimedia summary.

[0117] According to one or more embodiments of the present disclosure, in the multimedia processing method provided by the present disclosure, the target text content is a target text summary, the target text summary includes a summary of at least one summary type, the method further includes displaying in association a label of the summary type corresponding to the target text summary on the playback timeline of the related multimedia content.

[0118] According to one or more embodiments of the present disclosure, in the multimedia processing method provided by the present disclosure, the step of displaying in association a label of the summary type corresponding to the target text summary on the playback timeline of the related multimedia content includes Including displaying a label of a summary type corresponding to the target text summary at a relevant time corresponding to the target text summary on the playback timeline of the related multimedia content.

[0119] According to one or more embodiments of the present disclosure, in the multimedia processing method provided by the present disclosure, the relevant time corresponding to the target text summary is any time within the relevant time period of the target text summary.

[0120] According to one or more embodiments of the present disclosure, in the multimedia processing method provided by the present disclosure, the method Further includes sequentially highlighting target text content corresponding to the playback progress of the related multimedia content during the playback process of the related multimedia content.

[0121] According to one or more embodiments of the present disclosure, in the multimedia processing method provided by the present disclosure, the method Further includes receiving a download operation on the related multimedia content by a user and downloading and storing the related multimedia content.

[0122] According to one or more embodiments of the present disclosure, in the multimedia processing method provided by the present disclosure, the step of displaying the initial text content Includes displaying the initial text content in response to a trigger operation on a list page, where summary information of a plurality of multimedia contents is displayed on the list page.

[0123] According to one or more embodiments of the present disclosure, the present disclosure provides a multimedia processing device A text display module configured to display initial text content corresponding to target multimedia content A multimedia playback module configured to play the associated multimedia content in response to a trigger operation on the associated multimedia content. Here, the associated multimedia content includes fragments related to the target text content among the target multimedia content, and the target text content is extracted from the initial text content.

[0124] According to one or more embodiments of the present disclosure, in the multimedia processing apparatus provided by the present disclosure, the target text content includes at least two consecutive text fragments extracted from the initial text content.

[0125] According to one or more embodiments of the present disclosure, in the multimedia processing apparatus provided by the present disclosure, the multimedia playback module In response to a trigger operation on the associated multimedia content, jump and play the multimedia fragments corresponding to the associated time zones of each consecutive text fragment in the target multimedia content in the order in which they appear in the target multimedia content in the associated time zones of the at least two consecutive text fragments in the target text content.

[0126] According to one or more embodiments of the present disclosure, in the multimedia processing apparatus provided by the present disclosure, the apparatus further includes an associated multimedia generation module configured to determine the associated multimedia content based on the target text content.

[0127] According to one or more embodiments of the present disclosure, in the multimedia processing apparatus provided by the present disclosure, the associated multimedia generation module configured to generate associated multimedia content based on an associated time period of the target text content, where the associated time period of the target text content is used to characterize a time period in the target multimedia content of the voice information corresponding to the target text content.

[0128] According to one or more embodiments of the present disclosure, in the multimedia processing apparatus provided by the present disclosure, the associated multimedia generation module when the number of the associated time periods is plural, is configured to splice a plurality of multimedia fragments corresponding to each of the plurality of associated time periods according to an order in the target multimedia content of the plurality of associated time periods to generate the associated multimedia content.

[0129] According to one or more embodiments of the present disclosure, in the multimedia processing apparatus provided by the present disclosure, the associated multimedia generation module adjusts the associated time period of the target text content based on a statement completeness of the associated text of the target text content, and is configured to generate the associated multimedia content based on the adjusted associated time period.

[0130] According to one or more embodiments of the present disclosure, in the multimedia processing apparatus provided by the present disclosure, the associated text is the text corresponding to the target text content in the initial text content.

[0131] According to one or more embodiments of the present disclosure, in the multimedia processing apparatus provided by the present disclosure, the associated multimedia content is a multimedia summary, the multimedia summary includes at least two sub-multimedia summaries, each of the sub-multimedia summaries corresponds to one summary type, and the number of the summary types is at least two.

[0132] According to one or more embodiments of the present disclosure, in the multimedia processing apparatus provided by the present disclosure, the multimedia playback module in response to a trigger operation on related multimedia content, determines a target summary type corresponding to the trigger operation, obtains a target sub-multimedia summary corresponding to the target summary type and plays the target sub-multimedia summary, or is configured to obtain the multimedia summary and play the multimedia summary based on the time period of the target summary type in the multimedia summary.

[0133] According to one or more embodiments of the present disclosure, in the multimedia processing apparatus provided by the present disclosure, the target text content is a target text summary, and the target text summary includes summaries of at least one summary type, the apparatus further includes a labeling module, and is configured to relevantly display a label of the summary type corresponding to the target text summary on the playback timeline of the related multimedia content.

[0134] According to one or more embodiments of the present disclosure, in the multimedia processing apparatus provided by the present disclosure, the labeling module is configured to display a label of the summary type corresponding to the target text summary at a related time point corresponding to the target text summary on the playback timeline of the related multimedia content.

[0135] According to one or more embodiments of the present disclosure, in the multimedia processing apparatus provided by the present disclosure, the related time point corresponding to the target text summary is any time point within the related time period of the target text summary.

[0136] According to one or more embodiments of the present disclosure, in the multimedia processing apparatus provided by the present disclosure, the apparatus further comprises a highlighting module, configured to sequentially highlight target text content corresponding to the playback progress of the related multimedia content during the playback process of the related multimedia content.

[0137] According to one or more embodiments of the present disclosure, in the multimedia processing apparatus provided by the present disclosure, the apparatus further comprises a summary download module, configured to receive a download operation on the related multimedia content by a user and download and store the related multimedia content.

[0138] According to one or more embodiments of the present disclosure, in the multimedia processing apparatus provided by the present disclosure, the text display module is configured to display the initial text content in response to a trigger operation on a list page, where summary information of a plurality of multimedia contents is displayed on the list page.

[0139] According to one or more embodiments of the present disclosure, the present disclosure provides an electronic device, comprising a processor, and a memory storing instructions executable by the processor, wherein the processor is configured to read the executable instructions from the memory and execute the instructions to implement any one of the multimedia processing methods provided by the present disclosure.

[0140] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium storing a computer program, and the computer program is used to implement any one of the multimedia processing methods provided by the present disclosure.

[0141] Furthermore, although each operation is depicted using a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or sequentially. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although some implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Specific features described in the context of a single embodiment may also be implemented in combination within a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately, or in any suitable subcombination, in a plurality of embodiments.

Claims

1. displaying initial text content corresponding to target multimedia content; and responding to a trigger operation on related multimedia content by playing the related multimedia content, wherein the related multimedia content includes fragments related to target text content among the target multimedia content, and the target text content is extracted from the initial text content, wherein the related multimedia content is a multimedia summary, the multimedia summary includes at least two sub-multimedia summaries, each sub-multimedia summary corresponds to one summary type, and the number of the summary types is at least two, responding to a trigger operation on the related multimedia content by playing the related multimedia content includes determining a target summary type corresponding to the trigger operation in response to the trigger operation on the related multimedia content; obtaining a target sub-multimedia summary corresponding to the target summary type and playing the target sub-multimedia summary, or obtaining the multimedia summary and playing the multimedia summary based on a time period of the target summary type in the multimedia summary; A multimedia processing method characterized by the above.

2. The method according to claim 1, wherein the target text content includes at least two consecutive text fragments extracted from the initial text content.

3. responding to a trigger operation on the related multimedia content by playing the related multimedia content includes in response to a trigger operation on the related multimedia content, jumping and playing multimedia fragments corresponding to related time periods of each consecutive text fragment among the target multimedia content in the order of the target multimedia content in the related time periods of the at least two consecutive text fragments among the target text content; The method according to claim 2, characterized in that it comprises.

4. The method according to claim 1, further comprising the step of determining the relevant multimedia content based on the target text content.

5. The step of determining the relevant multimedia content based on the target text content includes generating the relevant multimedia content based on a relevant time period of the target text content, and the relevant time period of the target text content is used to characterize a time period in the target multimedia content of the voice information corresponding to the target text content. The method according to claim 4, wherein

6. The step of generating the relevant multimedia content based on the relevant time period of the target text content includes, when the number of the relevant time periods is plural, splicing a plurality of multimedia fragments corresponding to each of the plurality of relevant time periods according to an order in the target multimedia content of the plurality of relevant time periods, and generating the relevant multimedia content. The method according to claim 5, wherein

7. The step of generating the relevant multimedia content based on the relevant time period of the target text content includes adjusting the relevant time period of the target text content based on a statement completeness of the relevant text of the target text content, and generating the relevant multimedia content based on the adjusted relevant time period. The method according to claim 5, wherein

8. The relevant text is text corresponding to the target text content in the initial text content. The method according to claim 7, wherein

9. The target text content is a target text summary, the target text summary includes summaries of at least one summary type, The method further includes the step of relevantly displaying a label of the summary type corresponding to the target text summary on a playback timeline of the relevant multimedia content. The method according to claim 1, wherein

10. The step of associatively displaying a label of a summary type corresponding to the target text summary on the playback timeline of the related multimedia content is including displaying a label of a summary type corresponding to the target text summary at a related time corresponding to the target text summary on the playback timeline of the related multimedia content, and the method according to claim 9, characterized in that. **Claim 11** The related time corresponding to the target text summary is one time point within the related time zone of the target text summary, and the method according to claim 10, characterized in that. **Claim 12** The method according to claim 1, further including the step of sequentially highlighting target text content corresponding to the playback progress of the related multimedia content during the playback of the related multimedia content. **Claim 13** The method according to claim 1, further including the step of receiving a download operation on the related multimedia content by the user and downloading and storing the related multimedia content. **Claim 14** The step of displaying the initial text content is including displaying the initial text content in response to a trigger operation on the list page, and summary information of a plurality of multimedia contents is displayed on the list page, and the method according to claim 1, characterized in that. **Claim 15** A text display module configured to display initial text content corresponding to target multimedia content, and a multimedia playback module configured to play the related multimedia content in response to a trigger operation on the related multimedia content, and comprising the related multimedia content includes a fragment related to the target text content among the target multimedia content, and the target text content is extracted from the initial text content, the related multimedia content is a multimedia summary, the multimedia summary includes at least two sub-multimedia summaries, each sub-multimedia summary corresponds to one summary type, and the number of the summary types is at least two, the multimedia playback module further Determining a target summary type corresponding to the trigger operation in response to a trigger operation on the associated multimedia content; Obtaining a target sub-multimedia summary corresponding to the target summary type and playing the target sub-multimedia summary, or obtaining the multimedia summary and playing the multimedia summary based on a time period of the target summary type in the multimedia summary; configured to perform; A multimedia processing apparatus characterized by the above.

16. A processor; A memory configured to store instructions executable by the processor, The processor is configured to read the executable instructions from the memory and execute the instructions to implement the multimedia processing method according to any one of claims 1 to 14. An electronic device.

17. A computer-readable storage medium storing a computer program, wherein the computer program is used to implement the multimedia processing method according to any one of claims 1 to 14. A computer-readable storage medium.

Images