Method, device and equipment for processing real-time multimedia interactive content

Abstract

The embodiment of the invention provides a real-time multimedia interaction content processing method, device and equipment. The method comprises the following steps: acquiring real-time multimedia interaction detail content; according to the detail content, generating respective summary content of a plurality of issues, the summary content of at least one issue including text summary content and at least one frame of video image, and the video image being shared content in the real-time multimedia interaction process; and generating an interaction summary including summary contents of the topics. Therefore, the generated interaction summary can comprise the image related to the issue, the content form of the interaction summary is enriched, in addition, the user can obtain the original real-time multimedia interaction image from the interaction summary, and the efficiency of quickly obtaining the real-time multimedia interaction information by the user is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer and Internet technology, and in particular, to a method, device and equipment for processing real-time multimedia interactive content. Background Art

[0002] Real-time multimedia interaction is the real-time interaction between different users through the network in the form of audio, video, etc. During the real-time multimedia interaction, online communication can be carried out on multiple topics.

[0003] In order to enable users participating in real-time multimedia interaction to better understand the content of communication, corresponding subtitles can be generated for the content of real-time multimedia interaction. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, device and equipment for processing real-time multimedia interactive content.

[0005] In a first aspect, an embodiment of the present disclosure provides a content sharing method, comprising: obtaining detailed content of real-time multimedia interaction; generating summary content of multiple topics respectively based on the detailed content, wherein the summary content of at least one topic includes text summary content and at least one frame of image, wherein the image is the shared content during the real-time multimedia interaction; generating an interaction summary including the summary content of each topic.

[0006] In a second aspect, an embodiment of the present disclosure provides a method for processing real-time multimedia interactive content, comprising: based on a display instruction, displaying an interaction summary of the real-time multimedia interaction, wherein the interaction summary is generated based on the detailed content of the real-time multimedia interaction, including a summary theme and summary content corresponding to multiple topics respectively; wherein the summary content of at least one topic includes text summary content and key frame images.

[0007] In a third aspect, an embodiment of the present disclosure provides a device for processing real-time multimedia interactive content, comprising: an acquisition unit, used to acquire detailed content of real-time multimedia interaction; a first generation unit, used to generate summary content of multiple topics based on the detailed content, wherein the summary content of at least one topic includes text summary content and at least one frame of image, wherein the image is shared content during the real-time multimedia interaction process; a second generation unit, used to generate an interaction summary including the summary content of each topic.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a device for processing real-time multimedia interactive content, comprising: a display unit, for displaying an interaction summary of real-time multimedia interaction based on a display instruction, wherein the interaction summary is generated based on the detailed content of the real-time multimedia interaction, and comprises a summary theme and summary contents corresponding to a plurality of topics respectively; wherein the summary contents of at least one topic comprise text summary contents and key frame images.

[0009] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0010] The memory stores computer-executable instructions;

[0011] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method described in the first aspect, the second aspect, and various possible designs of the first aspect and the second aspect.

[0012] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the method described in the first aspect, the second aspect, and various possible designs of the first aspect and the second aspect is implemented.

[0013] In a seventh aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the methods described in the first aspect, the second aspect, and various possible designs of the first aspect and the second aspect.

[0014] The method, device and apparatus for processing real-time multimedia interactive content provided in this embodiment obtains the detailed content of real-time multimedia interaction; generates summary content of multiple topics according to the detailed content, wherein the summary content of at least one topic includes text summary content and at least one frame of image, wherein the image is the shared content in the real-time multimedia interaction process; and generates an interactive summary including the summary content of each topic. Thus, the generated interactive summary can include images related to the topic, enriching the content form of the interactive summary. In addition, the user can also obtain the original real-time multimedia interactive image in the above interactive summary, which helps the user to quickly obtain the information of the real-time multimedia interaction efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0016] Figure 1 A flowchart of a method for processing real-time multimedia interactive content provided by an embodiment of the present disclosure Figure 1 ;

[0017] Figure 2 A flowchart of a method for processing real-time multimedia interactive content provided by an embodiment of the present disclosure Figure 2 ;

[0018] Figure 3 A flowchart of a method for processing real-time multimedia interactive content provided by an embodiment of the present disclosure Figure 3 ;

[0019] Figure 4 A flowchart of a method for processing real-time multimedia interactive content provided by an embodiment of the present disclosure;

[0020] Figure 5 This is a schematic diagram of an application scenario;

[0021] Figure 6 This is a schematic diagram of an application scenario;

[0022] Figure 7 A structural block diagram of a device for processing real-time multimedia interactive content provided by an embodiment of the present disclosure;

[0023] Figure 8 A structural block diagram of a device for processing real-time multimedia interactive content provided by an embodiment of the present disclosure;

[0024] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0026] During real-time multimedia interaction, the information interaction between multiple participating users generates detailed content such as audio, video images, etc. The speech in real-time multimedia interaction can be converted into text and form subtitles.

[0027] An interaction summary may also be generated based on the above detailed content so that the user can quickly understand the main content of the real-time multimedia interaction. The above interaction summary may include one or more topics, each of which may have corresponding summary content. In addition, the interaction summary may also include to-do tasks, etc.

[0028] Currently, the interactive summary only includes summary content in text form, which is relatively simple and is not conducive to obtaining detailed content of real-time multimedia interaction from the interactive summary.

[0029] The method for processing real-time multimedia interactive content provided by the present disclosure generates an interactive summary including text summary content, timestamp and video image according to the detailed content of the real-time multimedia interaction, thereby enriching the content form of the interactive summary and facilitating users to browse the detailed content of the real-time multimedia interaction from the interactive summary.

[0030] Please refer to Figure 1 , Figure 1 Schematic diagram of the process of processing real-time multimedia interactive content provided by the present disclosure Figure 1 ;like Figure 1 As shown, the method comprises the following steps:

[0031] S101: Obtain detailed content of real-time multimedia interaction.

[0032] In this embodiment, the execution subject of the method for processing real-time multimedia interactive content (such as an application client or a server) can obtain detailed content of the real-time multimedia interaction.

[0033] The above-mentioned real-time multimedia interaction includes audio and video conferencing, live broadcasts, etc.

[0034] The above details include subtitles converted from real-time audio streams and each frame of video images in the video stream. It is understandable that each frame of video images includes a timestamp, and the subtitles may include text content generated by the voice of each participating user with a timestamp.

[0035] S102: Generate summary content of each of the plurality of topics according to the detailed content, wherein the summary content of at least one topic includes text summary content and at least one frame of image, wherein the image is shared content in the real-time multimedia interaction process.

[0036] The execution entity may perform semantic analysis on the detailed content and determine at least one topic according to the result of the semantic analysis.

[0037] In some examples, semantic analysis can be performed on the subtitles, and one or more topics can be determined based on the results of the semantic analysis. For example, the subtitles can be divided into different topics based on the semantic relevance. Each topic can include text content of the voices of one or more participating users. For each topic, the topic title and text summary content can be extracted based on the text content of the topic. In addition, based on the time information corresponding to the topic, at least one frame of image matching the above time information is determined from the user shared content of the real-time multimedia interactive process as the associated image of the topic.

[0038] In one example, in real-time multimedia interaction, a sharing user uses a client to share a presentation (PowerPoint, PPT) with other participating users' clients. The PPT may include multiple slides, and the PPT may be the shared content of real-time multimedia interaction. According to the text content of the voice of the sharing user and other participating users during the PPT presentation, topics A and B are parsed. If the start time of topic A is determined to be 4 minutes and 30 seconds and the end time is 8 minutes and 50 seconds according to the time information of topic A, one or more slides (for example, slides 3 to 5) displayed from 4 minutes and 30 seconds to 8 minutes and 50 seconds will be used as the associated images of topic A.

[0039] If the start time of Topic B is determined to be 9 minutes and 15 seconds and the end time is 11 minutes and 50 seconds based on the time information of Topic B, one or more slides (for example, the 6th to 8th slides) displayed from 9 minutes and 15 seconds to 11 minutes and 50 seconds will be used as associated images of Topic B.

[0040] In some implementations, the above step S102 includes: inputting the subtitles in the detailed content into a pre-trained content generation model, and processing the subtitles by the content generation model to obtain text summary content corresponding to at least one topic.

[0041] In these embodiments, subtitles may be input into a content generation model, and the subtitles include the start timestamp of each speech segment of each speaker. That is, the subtitles are composed of texts corresponding to multiple speech segments, and the text corresponding to each speech segment has a start timestamp. The content generation model may perform semantic analysis on the subtitles to determine the relevant subtitles (i.e., text content) of each of the multiple topics. And generate a text summary of the topic based on the relevant subtitles.

[0042] The content generation model may also output the association relationship between the text summary content and the related subtitle content. Links may also be established for one or more topics, where the links are used to link the text summary content of each topic to the related subtitle content.

[0043] S103: Generate an interactive summary including summary contents of each topic.

[0044] The summary contents of multiple topics can be combined to obtain an interactive summary. For example, the summary contents corresponding to each topic can be combined in the order in which the topics are started to obtain an interactive summary.

[0045] In this embodiment, detailed content of real-time multimedia interaction is obtained; summary content of multiple topics is generated according to the detailed content, wherein the summary content of at least one topic includes text summary content and at least one frame of image, wherein the image is shared content in the real-time multimedia interaction process; an interaction summary including the summary content of each topic is generated, so that the generated interaction summary can include images related to at least one topic, enriching the content form of the interaction summary. In addition, the user can also obtain the original real-time multimedia interaction image in the above interaction summary, which helps the user to quickly obtain the information of the real-time multimedia interaction efficiently.

[0046] Please refer to Figure 2 , Figure 2 Schematic diagram of the process of processing real-time multimedia interactive content provided by the present disclosure Figure 2 .like Figure 2 As shown, the method comprises the following steps:

[0047] S201: Obtain detailed content of real-time multimedia interaction.

[0048] S202: Generate summary content of each of the multiple topics according to the detailed content, wherein the summary content of at least one topic includes text summary content and at least one frame of image, wherein the image is shared content in the real-time multimedia interaction process.

[0049] The specific implementation of steps S201 to S202 can refer to Figure 1 The description of steps S101 to S102 in the illustrated embodiment will not be repeated here.

[0050] S203: In response to not detecting the preset theme of the real-time multimedia interaction, generating a summary theme according to the detailed content.

[0051] In this embodiment, the execution subject (client or server) of the method for processing real-time multimedia interactive content can detect whether the real-time multimedia interaction has a preset theme. If there is a preset theme, the preset theme is used as the summary theme of the interaction summary. The preset theme can be set by the user in advance for the real-time multimedia interaction. If the execution subject does not detect the preset theme, the subtitles in the real-time multimedia interaction details can be processed for text content to obtain the summary theme.

[0052] In one example, the subtitles in the above detailed content may be input into a content generation model, and the content generation model may output a summary topic.

[0053] For example, in some optional implementations, before the real-time multimedia interaction begins, the participant manually adds the discussion topic of the real-time multimedia interaction. In these optional implementations, the discussion topic manually added by the participant can be used as the summary topic of the generated summary.

[0054] In other optional implementations, the participants participating in the real-time multimedia interaction do not manually add discussion topics for the real-time multimedia interaction. In these optional implementations, summary topics can be generated based on the interactive content that actually occurs during the real-time multimedia interaction process.

[0055] S204: Generate an interactive summary including the summary theme and summary content of each topic.

[0056] The execution entity may set the summary theme at the summary title position, and integrate the summary content of each topic according to the start time of each topic to obtain an interactive summary.

[0057] Alternatively or additionally, the generated interactive summary can be a shareable cloud document. In this case, the summary topic can be used as the document title of the cloud document, and the generated interactive summary can be used as the document content of the cloud document.

[0058] In this embodiment, it is described that when no preset topic of real-time multimedia interaction is detected, the content of the summary topic is generated according to the detailed content. Since the summary topic is generated based on the detailed content of real-time multimedia interaction, the detailed content of real-time multimedia interaction can be summarized, which is conducive to improving the efficiency of identifying the real-time multimedia interaction topic through the summary topic, and can reduce the operating cost of manually editing the title. In addition, the management of real-time multimedia interaction content can be optimized, and it is convenient for users to review different real-time multimedia interaction content.

[0059] Please refer to Figure 3 , Figure 3 Schematic diagram of the process of processing real-time multimedia interactive content provided by the present disclosure Figure 3 .like Figure 3 As shown, the method includes the following steps:

[0060] S301: Obtain detailed content of real-time multimedia interaction.

[0061] The specific implementation of step S301 can refer to Figure 1 The description of step S101 of the illustrated embodiment is not repeated here.

[0062] S302: Determine multiple topics based on the detailed content.

[0063] In one example, the execution subject (client or server) of the real-time multimedia interactive content processing method can determine multiple topics based on the subtitles in the detailed content. Specifically, semantic relevance analysis can be performed based on the subtitles, and multiple topics can be determined based on the semantic relevance analysis results.

[0064] In one example, the execution subject can determine multiple topics based on the combination of subtitles and video data of real-time multimedia interaction. For example, if it is detected that the shared content is switched based on the video image, and the semantic correlation between the subtitle content corresponding to the shared content after the switch and the subtitle content corresponding to the shared content before the switch is less than a preset correlation threshold, it is determined that the subtitles before and after the shared content is switched belong to different topics. By analogy, multiple topics can be determined based on the above detailed content.

[0065] S303: For at least one topic among the multiple topics, generate at least one first-level sub-topic of the at least one topic and text summary contents corresponding to the sub-topics at each level.

[0066] Each of the above multiple topics can be regarded as a first-level topic in the interaction summary. Each first-level topic can be further semantically analyzed to parse out at least one first-level sub-topic, such as a second-level sub-topic belonging to the first-level topic, a third-level sub-topic belonging to the second-level sub-topic, etc.

[0067] Specifically, for each first-level topic, the details corresponding to the first-level topic can be analyzed for second-level sub-topics, that is, the above-mentioned details can be analyzed for secondary semantic relevance to obtain at least one second-level sub-topic. The third-level sub-topic analysis can be performed on the partial details corresponding to each second-level sub-topic to obtain at least one third-level sub-topic.

[0068] In one example, identification rules for identifying second-level sub-topics and third-level sub-topics under a first-level topic can be preset, for example, the identification rules include details of which content can be second-level sub-topics and third-level sub-topics. Then, the second-level sub-topics and third-level sub-topics can be determined through these identification rules.

[0069] In one example, the execution entity may input the subtitles in the detailed content into a content generation model, and the content generation model may identify multiple first-level topics and at least one first-level sub-topic under each first-level topic from the subtitles.

[0070] For each level of topic, a text summary of the topic can be generated. For example, for a level one topic, a text summary corresponding to the level one topic can be generated based on the subtitle content of the level one topic. For a level two subtopic, a text summary corresponding to the level two subtopic can be generated based on the subtitle content of the level two subtopic. For a level three subtopic, a text summary corresponding to the level three subtopic can be generated based on the subtitle content of the level three subtopic.

[0071] In some optional implementations, it can be determined whether to generate subtopics of the topic at the current level based on the time information associated with the topic at the current level. For example, if a first-level topic corresponds to the xth to yth minute of real-time multimedia interaction, and (yx) is greater than a preset time threshold, multiple second-level subtopics can be generated for the first-level topic.

[0072] In some other optional implementations, it is also possible to determine whether to generate subtopics of the topic at the current level based on the amount of information in the original subtitles associated with the topic at the current level (e.g., the number of text characters contained and / or the amount of shared content). For example, if the number of words in the original subtitles corresponding to a first-level topic exceeds a preset word count threshold, multiple second-level subtopics can be generated for the first-level topic.

[0073] In some examples, for a first-level topic with at least one first-level sub-topic, associated images corresponding to each level of sub-topics under the first-level topic can be determined, and the associated images corresponding to each sub-topic can be associated with each sub-topic. The associated images of any sub-topic can include images obtained from shared content during the sub-topic.

[0074] For example, for the last level sub-topic, one or more frames of associated images corresponding to the sub-topic can be determined from the video data of real-time multimedia interaction. The one or more frames of associated images can be used as images associated with the last level sub-topic in the interaction summary. For the previous level sub-topic of the last level sub-topic, multiple frames of associated images corresponding to the sub-topic can be determined from the video data of real-time multimedia interaction, and the image obtained by removing the associated image of the last level sub-topic from the multiple frames of associated images is used as the associated image of the previous level sub-topic, and so on.

[0075] S304: Generate an interactive summary including summary content of each topic.

[0076] For example, the summary contents corresponding to each topic are integrated before and after the start of the topic to obtain an interactive summary.

[0077] In this embodiment, it is described that for at least one topic, one or more sub-topics of the topic are generated, as well as summary contents corresponding to each sub-topic. Therefore, the generated interactive summary includes multiple topics and summary contents of each topic. By generating summary contents of multiple topics for detailed contents of real-time multimedia interaction, the layering and visual expression of the interactive summary are enhanced, and the readability and interactivity of the interactive summary are improved. It can intuitively meet the user's demand for the detailed contents of real-time multimedia interaction presented in the interactive summary.

[0078] In some implementations of this embodiment, the method further includes:

[0079] Based on the time information of the relevant subtitle content of each topic in the subtitle, the time information corresponding to the topic and the subtopic is generated.

[0080] Specifically, for each first-level topic, the timestamp of the first-level topic can be determined based on the time information of the first relevant subtitle in the relevant subtitles of the first-level topic. For example, the time indicated by the time information of the first relevant subtitle is used as the timestamp of the first-level topic. The timestamp can be obtained by taking the start time of the real-time multimedia interaction as the time zero.

[0081] For any first-level sub-topic belonging to a first-level topic, the relevant subtitles of the sub-topic can be determined, and the first subtitle in the relevant subtitles can be determined, and the time indicated by the time information of the first subtitle is used as the timestamp of the sub-topic.

[0082] In one example, for each level of sub-topic, each associated image in the summary content of the sub-topic of that level may carry a timestamp corresponding to the associated image.

[0083] In these implementations, by generating timestamps for topics at all levels, it is helpful for users to understand the distribution information of each topic in real-time multimedia interaction according to the interaction summary, which can further meet the refinement requirements of the interaction summary.

[0084] exist Figure 1 to Figure 3 In some implementations of the method for processing real-time multimedia interactive content shown, the method further includes:

[0085] The association relationship between the summary content of each topic and the relevant subtitles is established respectively, so that the relevant subtitles can be called and displayed through the summary content.

[0086] In one example, an association relationship between the summary content of each topic and related subtitles can be established. The related subtitles of each topic can be one or more subtitles used to generate the above summary content. The above related subtitles can include text content converted from the voice of one or more participating users.

[0087] In one example, for each topic, the association relationship between the topic itself and the relevant subtitles can be established so that the relevant subtitles of the topic can be displayed through the topic call.

[0088] Specifically, a link may be set for the above topic or the text summary content of the topic, and the link points to the relevant subtitles of the topic in the subtitles of the real-time multimedia interaction.

[0089] In these implementations, by establishing a correlation between the summary content of the topic and the relevant subtitles, it is convenient for users to search for relevant subtitles in the interaction summary, which helps users view the details of real-time multimedia interaction through the interaction summary.

[0090] Please refer to Figure 4 , Figure 4 The following is a flow chart of a method for processing real-time multimedia interactive content provided by the present disclosure. Figure 4 As shown, the method includes the following steps:

[0091] S401: Based on the display instruction, an interaction summary of the real-time multimedia interaction is displayed. The interaction summary is generated based on the detailed content of the real-time multimedia interaction, including a summary theme and summary contents corresponding to multiple topics respectively; wherein the summary contents of at least one topic include text summary contents and key frame images.

[0092] The executor of the method for processing real-time multimedia interactive content (for example, an application client running in a terminal device) can receive a display instruction instructing to display a real-time multimedia interaction summary, for example, the display instruction can be received during the real-time multimedia interaction process, or the display instruction can be received after the real-time multimedia interaction ends.

[0093] In one example, after the real-time multimedia interaction ends, the user can click the real-time multimedia interaction logo in the history record list to trigger the execution entity to display the record of the real-time multimedia interaction; the user can click the selection control of the interaction summary in the above record to issue a display instruction to the above execution entity.

[0094] After receiving the display instruction, the execution subject may display an interaction summary of the real-time multimedia interaction.

[0095] The above-mentioned interactive summary may include a summary topic and summary contents of multiple topics, wherein the summary contents of at least one topic include text summary contents and key frame images.

[0096] In some implementations, at least one topic includes at least one level of sub-topic, and each level of sub-topic has a text summary content of the sub-topic and / or an associated image.

[0097] Combine the following Figure 5Provide explanation. Figure 5 The middle display interface 51 displays an interaction summary of real-time multimedia interaction. The interaction summary includes a summary theme "Theme A". The theme A can be a theme preset by the user for real-time multimedia interaction, or a theme generated according to the detailed content of the real-time multimedia interaction.

[0098] According to the detailed content analysis of the real-time multimedia interaction, the content of the real-time multimedia interaction includes: 00:00 to 5 minutes and 49 seconds of the real-time multimedia interaction is the beginning stage of the real-time multimedia interaction. The beginning stage may include opening remarks, background, and introduction of participating users. At 5 minutes and 49 seconds, it enters Topic 1, and at 11 minutes and 05 seconds, it enters Topic 2.

[0099] Topic 1 (52) may include second-level sub-topics, such as Figure 5 Topic 1.1 and Topic 1.2 in the second-level sub-topic "Topic 1.1" corresponds to the text summary content "Text Summary Content 1", and the text summary content corresponding to the second-level sub-topic "Topic 1.2" is "Text Summary Content 2". Topic 1 can include key frame images. For example, Figure 5 Image 1 and Image 2 in . Image 1 may be a key frame image corresponding to the second-level sub-topic “Topic 1.1”. Image 2 may be a key frame image corresponding to the second-level sub-topic “Topic 1.2”.

[0100] Topic 2 (52) may include second-level sub-topics, such as Figure 5 The second-level subtopic "Topic 2.1" corresponds to "Text Summary Content 3", and the second-level subtopic "Topic 2.2" corresponds to "Text Summary Content 4". The summary content corresponding to Topic 2 includes the text summary content.

[0101] The above-mentioned image 1 and image 2 may be shared content in the real-time multimedia interaction process.

[0102] In this embodiment, in the interactive summary of the real-time multimedia interaction displayed, the summary content of at least one topic includes text summary content and key frame images. The key frame images are generated based on the shared content of the real-time multimedia interaction, so that the interactive summary includes video image frames related to at least one topic, enriching the content form of the interactive summary. In addition, it helps users to quickly obtain the original information of the real-time multimedia interaction.

[0103] In some embodiments, in response to a first instruction executed on any one of the at least one topic, an image associated with the topic is displayed in the summary content of the topic, and the associated image is used to represent content associated with the topic in the real-time multimedia interaction.

[0104] The first instruction may be an instruction for displaying an image associated with any of the above topics. After receiving the first instruction, one or more images associated with the topic may be displayed in the summary content of the topic.

[0105] In one example, each of at least one topic may be associated with multiple images, and the multiple images may display different image contents. For example, the multiple images may be obtained by recording the shared content during the real-time multimedia interaction process. The multiple images of each of the at least one topic may have a key frame image, for example, the first frame image of the multiple images is used as the key frame image. An expansion display control for displaying all associated image frames may be displayed under the key frame image of the topic, such as Figure 5 In the example, an expansion display control "Expand All" 53 is displayed below the image 1. In response to receiving a trigger operation performed on the expansion display control 53, multiple related images are displayed in the summary content display area of ​​the second-level sub-topic "Topic 1.1". The trigger operation performed on the expansion display control 53 may be the first instruction for "Topic 1.1".

[0106] The associated image of the topic mentioned above may be an image frame recorded from the shared content of the real-time multimedia interaction during the duration of the topic in the real-time multimedia interaction.

[0107] In these embodiments, in the interaction summary, the above-mentioned first instruction for the topic is used to display the related image of the topic in the summary content of the topic, so that the user can obtain more detailed content of the real-time multimedia interaction through the interaction summary, which is beneficial to the efficiency of the user in obtaining the detailed content of the real-time multimedia interaction.

[0108] In some embodiments, the method further comprises:

[0109] In response to a trigger operation performed on a text summary content of any topic, a subtitle page is displayed, and subtitle content associated with the triggered topic is located in the subtitle page, wherein the subtitle content is generated based on speech in real-time multimedia interaction.

[0110] Combine the following Figure 6 To explain, Figure 6 is a schematic diagram of an application scenario. Figure 6 As shown, the interactive summary display interface 51 of the real-time multimedia interactive interaction can display the interactive summary of topic A (for details, please refer to Figure 5 The sub-topics at each level in the above interactive summary topic can be associated with their own subtitle content. For example Figure 5Sub-topic 1.1 is associated with subtitle content 1. The user performs a trigger operation such as clicking or touching on "text summary content 1" to display subtitle page 61 and locate subtitle content 1 (62) associated with sub-topic 1.1 in the subtitle page. Subtitle content 1 (62) can be highlighted in subtitle page 61. In addition, subtitle page 61 can also display multiple subtitle contents after subtitle content 1, such as subtitle content 2 obtained by the voice of user 2, subtitle content 3 obtained by the voice of user 3, and subtitle content 4 obtained by the voice of user 1.

[0111] In the above implementation, by triggering the text summary content in the interaction summary, the subtitle content is displayed and located at the associated subtitle content of the topic corresponding to the text summary content, so that the viewing of the subtitle content details related to the topic can be initiated in the interaction summary, which provides convenience for users to obtain real-time multimedia interaction details and improves the efficiency of users in obtaining real-time multimedia interaction information.

[0112] Corresponding to the above Figure 1 The method for processing real-time multimedia interactive content of the embodiment, Figure 7 This is a structural block diagram of a method and apparatus for processing real-time multimedia interactive content provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 7 The device 70 includes: an acquisition unit 701, a first generation unit 702, and a second generation unit 703.

[0113] The acquisition unit 701 is used to acquire the detailed content of the real-time multimedia interaction;

[0114] A first generating unit 702 is used to generate summary contents of each of the plurality of topics according to the detailed contents, wherein the summary contents of at least one topic include text summary contents and at least one frame of image, wherein the image is the shared content in the real-time multimedia interaction process;

[0115] The second generating unit 703 is used to generate an interactive summary including summary contents of each topic.

[0116] In some embodiments, the first generating unit 702 is further configured to:

[0117] Identify multiple topics based on detailed content;

[0118] For at least one topic among the multiple topics, at least one first-level sub-topic of the at least one topic and text summary contents corresponding to the sub-topics at each level are generated.

[0119] In some embodiments, the first generating unit 702 is further configured to:

[0120] Based on the time information of the relevant subtitle content of each topic in the subtitle, the time information corresponding to the topic and the subtopic is generated.

[0121] In some embodiments, the apparatus 70 further includes a third generating unit (not shown in the figure), and the third generating unit is configured to:

[0122] In response to not detecting the preset topic of the real-time multimedia interaction, generating a summary topic according to the detailed content; and the second generating unit 703 is further used for:

[0123] Generates an interactive summary that includes summary topics and summary content for each topic.

[0124] In some embodiments, the first generating unit 702 is further configured to:

[0125] The subtitles in the detailed content are input into a pre-trained content generation model, and the content generation model processes the subtitles to output text summary content corresponding to at least one topic, and / or a summary theme of the interactive summary.

[0126] In some embodiments, the apparatus 70 further includes a topic-related image determination unit (not shown in the figure), which is used to:

[0127] According to the timestamp corresponding to at least one topic, at least one frame of image associated with each topic in at least one topic is determined from the video data of the real-time multimedia interaction; wherein,

[0128] The at least one frame image includes a key frame image and a non-key frame image. The key frame image is used for display in association with the text summary content of the topic; the non-key frame image is displayed based on the user's image detail viewing instruction.

[0129] In some embodiments, the device 70 further includes an association relationship establishing unit (not shown in the figure), which is used to establish an association relationship between the summary content of each topic and the relevant subtitles, so as to call and display the relevant subtitles through the summary content.

[0130] Corresponding to the above Figure 4 The method for processing real-time multimedia interactive content of the embodiment, Figure 8 This is a structural block diagram of a real-time multimedia interactive content processing device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 8 , the device 80 includes a display unit 801.

[0131] The display unit 801 is used to display an interaction summary of the real-time multimedia interaction based on the display instruction. The interaction summary is generated based on the detailed content of the real-time multimedia interaction, including the summary theme and summary contents corresponding to multiple topics respectively; wherein the summary contents of at least one topic include text summary contents and key frame images.

[0132] In some embodiments, the display unit 801 is further configured to:

[0133] In response to a first instruction executed on any one of the at least one topic, an image frame associated with the topic is displayed in the summary content of the topic, the image frame being used to represent the content associated with the topic in the real-time multimedia interaction.

[0134] In some embodiments, the display unit 801 is further configured to:

[0135] In response to a trigger operation performed on a text summary content of any topic, a subtitle page is displayed, and subtitle content associated with the triggered topic is located in the subtitle page, wherein the subtitle content is generated based on speech in real-time multimedia interaction.

[0136] In some embodiments, at least one topic includes at least one level of sub-topic, and each level of sub-topic has a text summary content of the sub-topic and / or an associated image.

[0137] In order to implement the above embodiment, the embodiment of the present disclosure also provides an electronic device.

[0138] refer to Fig. 9 , which shows a schematic diagram of the structure of an electronic device 900 suitable for implementing the embodiment of the present disclosure, and the electronic device 900 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0139] like Fig. 9As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0140] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0141] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0142] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0143] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0144] The computer-readable medium carries one or more programs (computer-executable instructions). When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0145] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0148] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0149] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0150] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0151] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for processing real-time multimedia interactive content, comprising: Get detailed content of real-time multimedia interactions; Generating summary contents of each of the plurality of topics according to the detailed contents, wherein the summary contents of at least one topic include text summary contents and at least one frame of image, wherein the image is the shared content in the real-time multimedia interaction process; Generate an interactive summary that includes summary content for each topic.

2. The method according to claim 1, characterized in that Generating summary contents of respective multiple topics according to the detailed contents includes: Determining the plurality of topics based on the detailed content; For at least one topic among the multiple topics, at least one first-level sub-topic of the at least one topic and text summary contents corresponding to the sub-topics at each level are generated.

3. The method according to claim 2, characterized in that The step of generating at least one first-level sub-topic for at least one of the plurality of topics further comprises: Based on the time information of the relevant subtitle content of each topic in the subtitle, the time information corresponding to the topic and the subtopic is generated.

4. The method according to any one of claim 1, characterized in that Also includes: In response to not detecting the preset topic of the real-time multimedia interaction, generating a summary topic according to the detailed content; as well as The generating of an interactive summary including summary content of each topic includes: The interactive summary is generated including the summary topic and summary content of each topic.

5. The method according to any one of claims 1 to 4, characterized in that: Generating summary contents of respective multiple topics according to the detailed contents includes: The subtitles in the detailed content are input into a pre-trained content generation model, and the content generation model processes the subtitles to output text summary content corresponding to each of the at least one topic and / or a summary theme of the interactive summary.

6. The method according to any one of claims 1 to 4, characterized in that: Also includes: According to the timestamp corresponding to at least one topic, at least one frame of image associated with each topic in the at least one topic is determined from the video data of the real-time multimedia interaction; wherein, The at least one frame image includes a key frame image and a non-key frame image. The key frame image is used for display in association with the text summary content of the topic; and the non-key frame image is displayed based on the user's image detail viewing instruction.

7. The method according to any one of claims 1 to 4, characterized in that: Also includes: An association relationship between the summary content of each topic and the relevant subtitles is established respectively, so that the relevant subtitles can be called and displayed through the summary content.

8. A method for processing real-time multimedia interactive content, comprising: Based on the display instruction, an interaction summary of the real-time multimedia interaction is displayed, wherein the interaction summary is generated based on the detailed content of the real-time multimedia interaction, including a summary theme and summary contents corresponding to a plurality of topics respectively; wherein the summary contents of at least one topic include text summary contents and key frame images.

9. The method according to claim 8, characterized in that Also includes: In response to a first instruction executed on any one of the at least one topic, an image frame associated with the topic is displayed in the summary content of the topic, wherein the image frame is used to represent the content associated with the topic in the real-time multimedia interaction.

10. The method according to claim 8, characterized in that Also includes: In response to a trigger operation performed on the text summary content of any topic, a subtitle page is displayed, and subtitle content associated with the triggered topic is located in the subtitle page, wherein the subtitle content is generated based on the voice in the real-time multimedia interaction.

11. The method according to any one of claims 8 to 10, characterized in that: At least one topic includes at least one level of sub-topic, and each level of sub-topic has a text summary content of the sub-topic and / or an associated image.

12. A real-time multimedia interactive content processing device, comprising: An acquisition unit, used to acquire detailed content of real-time multimedia interaction; A first generating unit, configured to generate summary contents of respective multiple topics according to the detailed contents, wherein the summary contents of at least one topic include text summary contents and at least one frame of image, wherein the image is the shared content in the real-time multimedia interaction process; The second generating unit is used to generate an interactive summary including summary contents of each topic.

13. A real-time multimedia interactive content processing device, comprising: A display unit is used to display an interaction summary of real-time multimedia interaction based on a display instruction, wherein the interaction summary is generated based on the detailed content of the real-time multimedia interaction, including a summary theme and summary contents corresponding to a plurality of topics respectively; wherein the summary contents of at least one topic include text summary contents and key frame images.

14. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 11.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 11 is implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images