Conference real-time chasing and backtracking method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a conference real-time chasing and backtracking method and system, electronic equipment and a storage medium. The method comprises the steps of collecting multi-modal conference content in real time, determining text content, synchronously monitoring user behaviors of participants in real time, and determining attention states of the participants; dividing the multi-modal conference content collected in real time into a plurality of atomic blocks with indexes according to time, and scoring the importance degree of the conference; judging whether the participants are disjointed or not based on the attention state, and if the participants are judged to be disjointed, determining disjointing time points of the participants; determining a disjunction time difference based on the disjunction time point and the current progress conference time, and determining an atomic block which needs to be pursued and backtracked by the conference participant by using an index; and after the participant pursues the atomic block needing meeting pursuing and backtracking, switching back to the current meeting live broadcast stream. According to the embodiment of the invention, a multi-modal data synchronization and real-time catch-up mechanism and intelligent abstract generation and dynamic pushing are realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent voice, and more particularly to a method, system, electronic device, and storage medium for real-time conference follow-up and recap. Background Technology

[0002] With the increasing popularity of remote conferencing and hybrid work environments, more and more users are participating in online meetings. Due to unstable networks, late arrivals, disconnections, or prolonged periods of inactivity, participants may miss crucial information or fall behind in discussions. These problems are typically addressed by watching pre-recorded videos, listening to chat logs, or asking other participants. However, all of these methods carry the risk of information gaps, inefficiency, and missing key information. Traditional replay mechanisms cannot guarantee real-time synchronization and fail to meet participants' needs for quickly catching up on the meeting's progress. While some technologies attempt to introduce accelerated playback or rapid summaries, they often neglect the actual context and emotional delivery, leading to missing information or difficulties in comprehension.

[0003] To further enhance the real-time meeting experience, existing technologies will also use: 1. Addressing Key Content Hints and Transcription in Video Conferencing: By real-time transcription and highlighting of key information, this feature solves the problems of information loss and participants' inability to access crucial information during meetings. Specifically, it combines audio streams with in-video conferencing prompts. Speakers input prompts for important content via the client, and the system extracts and transcribes the audio stream accordingly. The transcribed text is displayed as subtitles in real-time, with key information highlighted based on the instructions. This allows participants to quickly capture and understand important content during the live meeting. Real-time transcription and key content hints ensure that no participant misses crucial information, while also facilitating real-time participation and comprehension for participants with hearing impairments.

[0004] 2. For digital meeting data processing: Speech feature vectors are generated by extracting speech features from the audio stream (such as pause duration and stressed words). Temporal analysis methods are used to identify tone changes, and tone label sequences are generated based on these changes. A recurrent neural network model and a business negotiation tone and emotion mapping model are used to analyze meeting topic transitions, emotional fluctuations, and atmosphere changes in real time. These analysis results are used to trigger meeting process phase transition markers, helping to identify key decision points and controversial topics, thereby improving the understanding of meeting speech data and sentiment analysis.

[0005] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology: 1. The system's reliance on active input from speakers or participants for highlighting and transcribing important content in video conferences means it cannot automatically identify and mark all crucial information. Without pre-defined marking criteria, the system's automation level is limited, potentially missing key points or subject to subjective bias. Furthermore, while the solution emphasizes real-time transcription and captioning, participants who miss important content still need to watch replays or manually review the meeting minutes to fill the information gap. Therefore, although the system can display real-time captions, it fails to achieve intelligent "information catch-up," leaving a certain information delay for those who have fallen behind.

[0006] 2. The digital meeting data processing solution cannot effectively address the issue of real-time catching up and reviewing missed meeting content. This solution focuses more on sentiment analysis and tone change detection than on real-time catching up on missed meeting content. It does not optimize for information supplementation or catching up mechanisms when participants miss content, nor can it automatically review and supplement missing content. Furthermore, sentiment and tone analysis prioritizes the perception of emotional fluctuations and does not provide a real-time mechanism for supplementing or retrieving missing key information, thus failing to meet the need for real-time catching up on missed content. Summary of the Invention

[0007] In order to at least address the problem of poor real-time meeting backtracking performance in existing technologies.

[0008] In a first aspect, embodiments of the present invention provide a method for real-time meeting catch-up and backtracking, including: The system collects multimodal meeting content, including voice, images, and user interaction data, in real time, determines the text content of the multimodal meeting content, and simultaneously monitors the user behavior of the participants in real time to determine the attention status of the participants. The real-time collected multimodal conference content is divided into multiple indexed atomic blocks according to time. Each atomic block includes audio, images, user interaction data, and text content from a portion of the conference. The conference importance of each atomic block is scored based on the emotional intensity of the audio and keyword density. Based on the attention state, it is determined whether the participants are out of touch. If it is determined that the participants are out of touch, the time point of the participants' out-of-touching is determined. Based on the disconnection time point and the current meeting time, determine the disconnection time difference, and use the disconnection time difference and the index to determine the atomic blocks that the participants need to catch up on in the meeting; After the participants have completed catching up on the atomic blocks that require meeting backtracking, switch back to the current meeting live stream.

[0009] Secondly, embodiments of the present invention provide a real-time meeting follow-up and recap system, comprising: The content acquisition module is used to acquire multimodal meeting content, including voice, images and user interaction data, in real time, determine the text content of the multimodal meeting content, and simultaneously monitor the user behavior of the participants in real time to determine the attention status of the participants. The atomic block segmentation module is used to divide the real-time acquired multimodal conference content into multiple indexed atomic blocks according to time. The atomic blocks include some conference audio, images, user interaction data and text content. The conference importance of the multiple atomic blocks is scored based on the emotional intensity of the audio and keyword density. The disconnection determination module is used to determine whether the participants are disconnected based on their attention state. If the participants are found to be disconnected, the module determines the time point of their disconnection. The catch-up and backtracking module is used to determine the time difference of the disconnection based on the disconnection time point and the current meeting time, and to determine the atomic blocks that the participants need to catch up and backtrack on the meeting by using the time difference of the disconnection and the index. The switching module is used to switch back to the current live conference stream after the participants have finished catching up on the atomic blocks that require meeting backtracking.

[0010] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the real-time meeting catch-up and rewind method of any embodiment of the present invention.

[0011] Fourthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the real-time meeting catch-up and backtracking method of any embodiment of the present invention.

[0012] Fifthly, embodiments of the present invention provide a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of the real-time meeting catch-up and backtracking method of any embodiment of the present invention.

[0013] The beneficial effects of this invention are as follows: it achieves multimodal data synchronization and real-time catch-up mechanism, realizes intelligent summary generation and dynamic push, ensuring the continuity of meeting content without affecting other participants. It effectively reduces information gaps for participants during the meeting. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a real-time meeting catch-up and backtracking method provided in an embodiment of the present invention; Figure 2 This is a system module framework architecture diagram of a real-time meeting catch-up and backtracking method provided in an embodiment of the present invention; Figure 3 This is a flowchart of a real-time meeting catch-up and backtracking method provided by an embodiment of the present invention; Figure 4 This is a wireframe diagram of the device-side participant interface of a real-time meeting catch-up and retrospection method provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a real-time meeting tracking and replay system provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of an embodiment of an electronic device for real-time conference tracking and rewinding, provided as an example of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] like Figure 1 The diagram shown is a flowchart of a real-time meeting catch-up and backtracking method according to an embodiment of the present invention, which includes the following steps: S11: Real-time acquisition of multimodal meeting content including voice, images and user interaction data, determination of the text content of the multimodal meeting content, and synchronous real-time monitoring of the user behavior of the participants to determine the attention state of the participants; S12: Divide the real-time collected multimodal conference content into multiple indexed atomic blocks according to time. The atomic blocks include some conference audio, images, user interaction data and text content. The conference importance of the multiple atomic blocks is scored based on the emotional intensity of the audio and the keyword density. S13: Based on the attention state, determine whether the participants are out of touch. If it is determined that the participants are out of touch, then determine the time point of the participants' out-of-touching. S14: Based on the disconnection time point and the current meeting time, determine the disconnection time difference, and use the disconnection time difference and the index to determine the atomic blocks that the participants need to catch up on in the meeting; S15: After the participants have finished catching up on the atomic blocks that require meeting backtracking, switch back to the current meeting live stream.

[0018] In existing remote conferencing technologies, when participants miss important information due to network issues, lateness, or prolonged inattention, the common solutions are watching pre-recorded sessions, replaying historical chat logs, or asking colleagues. However, these methods fail to provide real-time, interactive information supplementation solutions that integrate voice, video, and emotional feedback. Users often need to spend significant time catching up on missed information, and information loss is unavoidable. Furthermore, due to the limitations of these replay functions, users need to spend considerable time reviewing the replay after the meeting to understand the missed content, which also impacts meeting efficiency.

[0019] In response to the above scenario, this method identifies the shortcomings of existing technical approaches: 1. If attendees are late or rejoin the meeting and miss some information, they can usually check the meeting chat log, have a colleague repeat it, or watch the recorded video at a faster speed. However, this presents new problems: the chat lacks audio and video context, manual repetition may interrupt the meeting, and the recorded video cannot be accessed immediately; it can only be watched after the meeting has ended.

[0020] 2. If a participant disconnects, they can usually listen silently after reconnecting. However, this does not immediately reveal if they missed key points. They need to interrupt to ask or replay the audio themselves.

[0021] 3. If attendees are distracted or daydreaming, attention-boosting plugins can be used. However, these plugins can only provide hints and cannot provide the meeting content that attendees are missing.

[0022] Therefore, considering the shortcomings of existing technologies for real-time conferencing, this method addresses the aforementioned issues through multimodal data fusion and a real-time dynamic backtracking mechanism.

[0023] In short, this method uses multimodal information collection and analysis to segment and score meeting content, and provides an instant "make-up" function for participants who are late, disconnected, or distracted. The overall architecture is as follows: Figure 2 As shown.

[0024] In step S11, during the real-time meeting, the audio and video acquisition module collects voice, image, and user interaction data in real time. For example, in a remote meeting, multiple participants are located in different regions, and each participant has a display device in front of them, such as a monitor with a camera and microphone. Alternatively, for ease of use, participants can also use smart mobile terminals to participate in the meeting. The meeting system of this method is configured in the application and provides services to participants through the aforementioned devices. Participants can view the meeting content, and the meeting system of this method can also collect the voice spoken by the speakers, the real-time behavior of each participant, and the PPT content presented by the speakers.

[0025] Regarding audio, ASR (Automatic Speech Recognition) can be used to convert the speakers' voices into text. Furthermore, since multiple people will speak at the meeting, voiceprints or usernames of each participant can be obtained beforehand, assigning a unique username ID to each audio segment for easier post-meeting use. Simultaneously, emotion recognition can be performed to determine the emotional intensity of each speaker.

[0026] Regarding video, key slide content can be identified using OCR (Optical Character Recognition). Considering the accuracy of OCR, image preprocessing of the PPT slides improves image quality. Furthermore, given that the meeting content may involve Chinese, English, or other languages, language packs can be pre-configured to further enhance OCR accuracy. To identify key content in slides, highlight buttons can be provided for attendees, allowing the speaker or other attendees to click, or the system can automatically identify key content based on the duration of screen time.

[0027] For the content of the aforementioned meetings, in addition to processing it through the steps listed above, this method also considers the behavior of the participants. For example, it uses a camera to capture the eye movements of the participants to determine whether their attention is focused on the meeting playback page. Alternatively, random mouse clicks indicate that the participant's attention is not focused and they are distracted. Therefore, this method combines user behavior (such as eye movements and mouse clicks) to identify the participant's attention level.

[0028] For step S12, the content (audio and video) of the aforementioned meeting is divided into multiple atomic blocks according to a preset time period (e.g., 2-5 seconds) and then vectorized. Each atomic block contains key audio, image, and text information from a specific part of the meeting (this information can be obtained through the methods listed above), and is scored based on factors such as the emotional intensity of the audio and keyword density (the frequency of the same keyword appearing throughout the meeting). In this way, atomic blocks of the meeting are continuously obtained as the meeting is captured in real time.

[0029] For step S13, in addition to dividing the meeting content into real-time atomic blocks, this method also pays attention to the participants and detects whether they are late to join the meeting, disconnect, or lose focus.

[0030] As one implementation method, when a user's behavior of joining late or disconnecting is detected, it is determined that the participant's attention state is lacking, thereby determining that the participant is out of touch, and the time point when the participant joins late or reconnects after disconnection is determined as the time point when the participant is out of touch.

[0031] When a user's inattentive behavior is detected, it is determined that the participant's attention state is missing, and thus the participant is out of touch. Based on the time point at which the participant's inattentiveness is detected, the time point of the participant's out of touch is determined.

[0032] In this implementation, the system determines whether participants are missing (not watching, or watching but not paying attention) part of the meeting content by monitoring whether they are late to the meeting, disconnect, or become distracted. In other words, if a participant is detected to be late or disconnected, the system will determine whether to trigger a catch-up replay function by monitoring the time difference between their arrival and the meeting. If a participant has not performed any action for an extended period (e.g., becoming distracted), the system will trigger attention monitoring and provide a catch-up function to help them quickly return to the meeting.

[0033] In summary, being late to the meeting, disconnecting, or losing focus indicates that participants are out of touch with the current meeting; in other words, they are in a disconnected state. This will trigger a disconnection event, meaning the participants need to catch up on the missed meeting content. Therefore, it's necessary to determine what content the participants missed.

[0034] For step S14, first, determine the exact time when a participant arrives late, disconnects, or becomes distracted; second, determine the time difference between this time and the ongoing meeting; finally, use this time difference to determine how much content the participant needs to review. The specific meeting content can be adjusted based on the participant's late arrival time, determining the size and priority of subsequent supplementary content.

[0035] At this point, because the meeting content is divided into multiple small segments in real time during the live conference, each segment represents a specific important part of the meeting. Based on the participants' lag time, the system extracts the Top-K important segments from the index. These segments contain the key content that participants need to quickly access. The Top-K important segments are selected from multiple segments based on importance scores and participants' needs, ensuring that the supplementary content is the most critical information.

[0036] As one implementation, after determining the atomic blocks that the participants need to catch up on using the disconnection time difference and the index, the method further includes: Multimodal summaries of the atomic blocks are generated based on a large language model. The multimodal summaries required by the participants are determined by the importance score of the meeting. The required multimodal summaries are dynamically pushed to the participants to achieve customized real-time follow-up and retrospection of the meeting.

[0037] In this implementation, to further help attendees quickly understand missed content, the system invokes LLM (Large Language Model) to generate a rapid summary script for key segments. This step extracts and compresses the content of each selected segment into a concise, key summary. The generated summary script includes not only text but may also include images or other multimodal content to ensure the completeness and accuracy of the information. In this way, attendees can quickly understand the missed content by simply glancing at it.

[0038] Furthermore, this method also performs Catch-up audio and video stream rendering: it employs a 1.5x-1.8x resampling technique and provides subtitles to ensure that participants do not miss key information while quickly browsing.

[0039] This method utilizes multimodal data acquisition, including audio, video, and interactive behaviors, combined with real-time video stream playback and text summarization, to enable participants to catch up on missed content in real time during the meeting. Compared to traditional audio transcription or speed-up playback, this solution ensures that users can catch up on the most crucial meeting segments through segmented real-time indexing and importance scoring, and can quickly resume the current meeting progress by playing back at 1.5x-1.8x speed.

[0040] By combining LLM (Learning by Multiplication) to generate rapid summaries, this method not only provides accurate meeting recaps but also dynamically pushes relevant content based on attendees' preferences and needs. Unlike traditional linear playback, this method uses smart summary cards for non-linear playback, allowing users to flexibly choose the duration and importance of the content they catch up on, thus avoiding information overload.

[0041] As the meeting progresses and more important content is generated, a smart recap card pops up after each key segment. This card, presented to attendees, contains replays of the meeting information in both video and text formats. Attendees can use the card to review the content they are catching up on. The card design aims to ensure that attendees can efficiently capture key information from the meeting.

[0042] As one implementation method, questions from participants are received in real time, the time point of the question in the meeting is located, and multimodal meeting content at that time point is provided.

[0043] In this implementation, considering that participants may have questions, the method also provides an intelligent QA function. When participants view the replay content or review the summary, they can ask questions about unclear parts. The system will display a Q&A window, allowing participants to ask questions directly and jump to the corresponding time point of the multimodal meeting content for immediate feedback.

[0044] During the meeting, participants can ask questions via a real-time Q&A function. The system automatically detects the relevance of the questions and provides accurate answers, even jumping to the corresponding timecode based on the question content. This feature not only improves the efficiency of interaction but also greatly enhances the accessibility and engagement of the meeting content, achieving 3. Real-time Q&A and dynamic content feedback.

[0045] For step S15, participants catch up on the atomic blocks that need to be caught up and backtracked in the meeting. When the participants have finished catching up on the content, the catching up mode ends and the current live stream is switched back.

[0046] As one implementation method, during the process of participants catching up on the atomic blocks, multiple indexed atomic blocks, which are divided into multimodal meeting content collected in the current synchronous meeting, are identified as the atomic blocks that participants need to catch up on and backtrack on. After the participants have completed catching up on the atomic blocks that require meeting backtracking, the system seamlessly switches back to the current meeting live stream.

[0047] In this implementation, the method does not provide participants with a backtracking of the content between the disconnected point in time and the current point in time. Instead, it considers that participants may still be unable to effectively absorb the ongoing meeting content during the backtracking process. Therefore, the backtracking portion must also take into account new meeting content generated synchronously in the ongoing meeting while participants are backtracking. Thus, during the backtracking process, new atomic blocks that the meeting needs to catch up with will be identified. This ensures that participants can synchronously return to the currently ongoing meeting after backtracking.

[0048] This method introduces dual-stream parallel playback: one stream for real-time meeting content and the other for rewinding and supplementing missing information. Users can quickly catch up on missed content without interrupting the current live stream, and through intelligent switching, automatically return to the live stream after catching up, ensuring meeting continuity and engagement, and achieving a dual-stream parallel and seamless rewind mechanism. Furthermore, considering that participants may not be able to watch two content sessions simultaneously, the ongoing live stream is still rewound during the rewind process, ensuring that the rewind functionality is effective for participants.

[0049] Specific implementation examples of the above steps are described below: A conference was held with multiple participants to discuss intelligent voice application challenges. Several speakers and numerous attendees participated. Each attendee used a smart device equipped with this method. Each attendee logged into their own account, and information from all attendees was recorded.

[0050] Several speakers delivered presentations at the conference. The equipment captured the speakers' voices, played the PowerPoint slides in real time, recorded the speakers' annotations on the content, and monitored the status of each participant. For example, the data collected from 00:05:00 to 00:12:30 included content on the background of the intelligent voice industry, technology penetration rates in smart homes, in-vehicle systems, and healthcare, as well as commercialization strategies in vertical fields. From 00:15:00 to 00:32:00, the data collected on key technology discussions, cutting-edge technology directions, and performance optimization. From 00:38:00 to 01:00:00, the data collected on interactive content regarding key areas of industrial application and risk control challenges. Finally, the data collected from 01:10:00 to 01:20:00 included conference discussions. As the conference progressed, this application also captured real-time audio, image, and user interaction data through the audio and video capture module. Audio was processed into text using ASR (Automatic Speech Recognition), and video was processed using OCR (Optical Character Recognition) to identify key slide content.

[0051] During the speaker's presentation, this method also detects the behavior of other participants (late login, re-login, eye movements, mouse clicks, etc.) to determine the participants' attention state.

[0052] After obtaining the conference audio and image data, a segmented and vectorized index is established. The conference content is divided into atomic blocks of 2-5 seconds and vectorized. Each atomic block contains key audio, image, and text information, and is scored based on factors such as the emotional intensity of the audio and keyword density. For example, the content collected from 00:15:00 to 00:32:00 regarding key technology discussions, cutting-edge technology directions, and performance optimization is segmented into multiple atomic blocks. Atomic blocks 1-20 contain content related to multimodal voice interaction, atomic blocks 21-50 contain content related to voiceprint and lip movement recognition, atomic blocks 51-74 contain content related to low-resource language models, and atomic blocks 75-255 contain content related to minority language support solutions. The key content of each atomic block segment is statistically analyzed, such as the repetition frequency of a certain keyword, to determine its importance percentage in the conference, thereby determining an importance score. Using the same method, as conference audio is input, the above-mentioned indexed atomic blocks are obtained in real time, and the importance score is continuously determined with each input.

[0053] At 00:15:00, a participant disconnected, indicating a disconnection. When they reconnected, it was already 00:18:00, and they needed to catch up on the missed meeting content. Calculating the time difference between the disconnection and the current meeting progress (assuming the meeting is currently at 00:20:00), we determine that the participant needs to review the content from 00:15:00 to 00:20:00. We then retrieve the key information from this period. Furthermore, using an importance score, we extract the Top-K most important key information from the index, ensuring that the most critical information is quickly provided to the participant.

[0054] As the meeting progresses, suppose a participant is detected distracted at 00:25:00. Determining whether a participant has indeed been distracted takes time, say 4 minutes. If the distraction time is determined to be 00:21:00, then the participant needs to review the content from 00:21:00 to 00:25:00. The atom blocks from this period are retrieved. Then, based on importance scoring, the Top-K most important atom block fragments are extracted from the index, ensuring that the content quickly provided to the participant is the most critical information.

[0055] In addition, the system invokes an LLM script to generate rapid summaries of key segments. This step extracts and compresses the content of each selected segment into a concise, key summary. For example, the summary of a specific segment from 00:15:00 to 00:20:00 could be textual content about a technological breakthrough, graphic markers from a conference speaker, or other modalities. Subtitles are then provided via Catch-up audio and video stream rendering to ensure attendees don't miss crucial information while quickly browsing.

[0056] By monitoring participant behavior (such as lateness, disconnection, or inattentiveness), the system automatically triggers catch-up replays and content pushes, avoiding the hassle of participants actively searching for missed content. The system automatically pushes important content when participants need to catch up, improving the participant experience and interaction efficiency. Once participants have caught up, the system ends the catch-up mode and switches back to the current live stream.

[0057] The above embodiments will be described in a simplified manner, as follows: Figure 3 As shown, the steps include: Step 1: The system uses an audio and video acquisition module to collect real-time audio, image, and user interaction data during the meeting. Audio is processed into text using ASR (Automatic Speech Recognition), and video is identified using OCR (Optical Character Recognition) to recognize key slide content. The system also incorporates user behavior (such as eye movements and mouse clicks) to identify the participants' attention levels.

[0058] Step 2: Chunking and Vectorized Indexing: Divide the meeting content into 2-5 second atomic blocks and vectorize them. Each atomic block will contain key information such as speech, images, and text, and will be scored based on factors such as the emotional intensity of the speech and keyword density.

[0059] Step 3: User Late Arrival / Disconnection / Attention Detection: The system detects whether a user is late to the meeting, disconnects, or becomes distracted to determine if they are missing any part of the meeting content. If the system detects a late arrival or disconnection, it will monitor the time difference between the user and the meeting to determine whether to trigger the catch-up replay function. If a user has not performed any action for an extended period (e.g., is distracted), the system will trigger attention monitoring and provide the user with a catch-up function to help them quickly return to the meeting.

[0060] Step 4: If the system detects that the user's status is out of touch (such as being late, disconnected, or distracted) in the previous step, it will trigger a disconnection event, indicating that the user needs to catch up on the missed meeting content.

[0061] Step 5: Calculate the user's lag time relative to the current meeting. This is the time difference between when the user joined / disconnected / lost focus and the current meeting progress. This time difference will determine how much content the user needs to review, and the system will determine the size and priority of subsequent supplementary content based on the user's lag time.

[0062] Step 6: The system uses a block-level index, dividing the meeting content into multiple smaller segments. Each segment represents a key element of the meeting. Based on the user's time lag, the system extracts the Top-K most important segments from the index. These segments contain the crucial content the user needs to quickly access. The Top-K most important segments are selected from multiple segments based on importance scores and the user's needs, ensuring that the supplementary content is the most critical information.

[0063] Step 7: The system invokes an LLM script to generate a rapid summary of key segments. This step extracts and compresses the content of each selected segment into a short, concise summary. The generated summary script includes not only text but may also include images or other multimodal content, ensuring the completeness and accuracy of the information.

[0064] Step 8: Catch-up audio and video stream rendering: Employs 1.5x-1.8x resampling technology and provides subtitles to ensure that attendees do not miss key information while quickly browsing.

[0065] Step 9: A smart replay card pops up. This card is the interface presented to the user by the system, containing replay information from the meeting, displayed in video and text formats. Users can view the replay content through the card, which is designed to ensure that users can efficiently capture key information from the meeting.

[0066] Step 10: Intelligent QA: When users are watching the replay content or viewing the summary, they can ask questions about parts they don't understand. The system will display a Q&A window, allowing users to ask questions directly and get instant feedback.

[0067] Step 11: Once the user has finished catching up with the content, the system ends the catching-up mode and switches back to the current live stream.

[0068] This method also provides intelligent backtracking cards, including meeting content backtracking cards and gap backtracking cards. Meeting content backtracking cards, for example, capture the overall content of a key technology discussion from 00:15:00 to 00:32:00, including meeting information playback presented in video and text formats. Similarly, gap backtracking cards include backtracking cards for portions of the content from 00:15:00 to 00:20:00, also containing meeting information playback presented in video and text formats. Users can view the content they are catching up on through these cards, designed to ensure users can efficiently capture key information from the meeting. Furthermore, this method allows users to ask questions about unclear parts while viewing backtracking content or summaries. The system displays a Q&A window, allowing participants to directly ask questions and receive immediate feedback.

[0069] As one implementation method, this method also constructs a visual interface for an online conferencing system for real-time meeting tracking and rewinding, such as... Figure 4 As shown, the design leans towards a modern, minimalist, and high-contrast style, allowing users to quickly access key information during meetings. The overall interface comprises four main layout areas: (1) Main video screen / shared content area: This is the central area of ​​the interface, which displays the main video stream of the meeting or the shared screen content. Participants can see the speaker's video or other shared content here.

[0070] (2) Sidebar Tabs: Place the entry tabs (tabs) for important AI-assisted functions such as "intelligent retrospection", "agenda management" and "real-time transcription".

[0071] (3) Bottom toolbar: Place common meeting controls (audio / video switch, screen sharing, recording, raising hands, etc.) and add an "On / Off AI function" button for users to choose whether to display the AI ​​function panel.

[0072] (4) Top status bar: Displays the current meeting title, countdown / progress bar, and global status prompts such as the number of minutes / to-do items currently recorded.

[0073] The key functional tags for "Intelligent Backtracking" include: (5) Video stream display area: Play back the catch-up content (i.e. the part of the meeting that the user missed) at a speed of 1.5x to 1.8x so that the user can quickly catch up on the previous meeting content without affecting the normal meeting progress. Variable speed playback allows users to accelerate information acquisition while ensuring understanding of the content, thereby minimizing the time delay caused by missing some content.

[0074] (6) Current Summary: This area is used to display a summary of the current catch-up content, providing a brief text description to help users quickly understand the key points discussed in the meeting.

[0075] (7) Intelligent Q&A: Real-time Q&A function component. Users can enter questions and the system will provide automatic answers or suggestions.

[0076] As demonstrated by this implementation, by acquiring multimodal data (audio, video, and user interaction data), this method can capture information related to the meeting process in real time and dynamically push it to participating users. This mechanism allows participants to quickly catch up on meeting content even if they are late, disconnected, or distracted. It does not rely solely on single audio or video data, but rather ensures that users do not miss any key information through real-time audio and video synchronization and user behavior monitoring, thus achieving a multimodal data synchronization and real-time catch-up mechanism.

[0077] By leveraging multimodal LLM to generate rapid summaries, key meeting content is compressed and reorganized, enabling attendees who missed the main content to quickly grasp important information. This not only improves the speed of information catching up but also dynamically pushes summary content based on attendee needs, ensuring each attendee receives customized review material, achieving intelligent summary generation and dynamic delivery.

[0078] This method allows participants to simultaneously receive the live conference stream and the follow-up content stream, and seamlessly switch back to the live stream. Once participants have finished catching up, the system automatically returns to the current conference progress, ensuring the continuity of the conference content without affecting other participants.

[0079] This method further enhances the interactivity of the meeting and the accuracy of information acquisition. Participants can ask questions in real time, and the system will intelligently locate the time of the question and automatically jump to the relevant content, effectively reducing the information gaps for participants during the meeting.

[0080] like Figure 5 The diagram shown is a schematic diagram of a real-time meeting catch-up and backtracking system provided in an embodiment of the present invention. The system can execute the real-time meeting catch-up and backtracking method described in any of the above embodiments and is configured in the terminal.

[0081] This embodiment provides a real-time meeting catch-up and backtracking system 10, which includes: a content acquisition module 11, an atomic block division module 12, a disconnection determination module 13, a catch-up and backtracking module 14, and a switching module 15.

[0082] The content acquisition module 11 is used to acquire multimodal meeting content, including voice, images, and user interaction data, in real time, determine the text content of the multimodal meeting content, and simultaneously monitor the user behavior of the participants in real time to determine the attention status of the participants; the atomic block division module 12 is used to divide the real-time acquired multimodal meeting content into multiple indexed atomic blocks according to time, wherein the atomic blocks include some of the meeting's voice, images, user interaction data, and text content, and score the meeting importance of the multiple atomic blocks based on the voice emotion intensity and keyword density; the disconnection determination module 13 is used to determine whether the participants are disconnected based on the attention status, and if the participants are determined to be disconnected, the disconnection time point is determined; the catch-up and backtracking module 14 is used to determine the disconnection time difference based on the disconnection time point and the current meeting time, and determine the atomic blocks that the participants need to catch up and backtrack using the disconnection time difference and the index; the switching module 15 is used to switch back to the current meeting live stream after the participants have caught up with the atomic blocks that need to be caught up and backtracked.

[0083] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the real-time meeting catch-up and rewind method in any of the above method embodiments. In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows: The system collects multimodal meeting content, including voice, images, and user interaction data, in real time, determines the text content of the multimodal meeting content, and simultaneously monitors the user behavior of the participants in real time to determine the attention status of the participants. The real-time collected multimodal conference content is divided into multiple indexed atomic blocks according to time. Each atomic block includes audio, images, user interaction data, and text content from a portion of the conference. The conference importance of each atomic block is scored based on the emotional intensity of the audio and keyword density. Based on the attention state, it is determined whether the participants are out of touch. If it is determined that the participants are out of touch, the time point of the participants' out-of-touching is determined. Based on the disconnection time point and the current meeting time, determine the disconnection time difference, and use the disconnection time difference and the index to determine the atomic blocks that the participants need to catch up on in the meeting; After the participants have completed catching up on the atomic blocks that require meeting backtracking, switch back to the current meeting live stream.

[0084] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of this invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, the real-time meeting catch-up and backtracking method in any of the above method embodiments is executed.

[0085] Figure 6 This is a schematic diagram of the hardware structure of an electronic device for a real-time meeting catch-up and backtracking method provided in another embodiment of this application, as shown below. Figure 6 As shown, the device includes: One or more processors 610 and memory 620, Figure 6 Taking a processor 610 as an example, the device for the real-time conference catch-up and backtracking method may also include an input device 630 and an output device 640.

[0086] The processor 610, memory 620, input device 630, and output device 640 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0087] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the real-time meeting catch-up and rewind method in the embodiments of this application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 620, thereby implementing the real-time meeting catch-up and rewind method in the above-described method embodiments.

[0088] The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 620 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include memory remotely located relative to the processor 610, and these remote memories can be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0089] Input device 630 can receive input numerical or character information. Output device 640 may include display devices such as a display screen.

[0090] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, the real-time meeting catch-up and backtracking method in any of the above method embodiments is executed.

[0091] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0092] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0093] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the real-time meeting catch-up and backtracking method of any embodiment of this invention.

[0094] The electronic devices described in this application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0095] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0096] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0097] (4) Other electronic devices with data processing functions.

[0098] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for real-time meeting backtracking, comprising: The system collects multimodal meeting content, including voice, images, and user interaction data, in real time, determines the text content of the multimodal meeting content, and simultaneously monitors the user behavior of the participants in real time to determine the attention status of the participants. The real-time collected multimodal conference content is divided into multiple indexed atomic blocks according to time. Each atomic block includes audio, images, user interaction data, and text content from a portion of the conference. The conference importance of each atomic block is scored based on the emotional intensity of the audio and keyword density. Based on the attention state, it is determined whether the participants are out of touch. If it is determined that the participants are out of touch, the time point of the participants' out-of-touching is determined. Based on the disconnection time point and the current meeting time, determine the disconnection time difference, and use the disconnection time difference and the index to determine the atomic blocks that the participants need to catch up on in the meeting; After the participants have completed catching up on the atomic blocks that require meeting backtracking, switch back to the current meeting live stream.

2. The real-time meeting catch-up and backtracking method according to claim 1, wherein, The step of determining whether a participant is out of touch based on their attention state, and if it is determined that a participant is out of touch, then the time points at which the participant is out of touch include: When a user is detected to be late to join or disconnect, it is determined that the participant's attention state is missing, and the participant is identified as disconnected. The time point when the participant is late to join or disconnects and reconnects is determined as the time point when the participant is disconnected.

3. The real-time meeting catch-up and backtracking method according to claim 1, wherein, The step of determining whether a participant is out of touch based on their attention state, and if it is determined that a participant is out of touch, further includes identifying the time point of the participant's disconnection: When a user's inattentive behavior is detected, it is determined that the participant's attention state is missing, and thus the participant is out of touch. Based on the time point at which the participant's inattentiveness is detected, the time point of the participant's out of touch is determined.

4. The real-time meeting catch-up and backtracking method according to claim 1, wherein, After determining the atomic blocks that the participants need to catch up on using the disconnection time difference and the index, the method further includes: Multimodal summaries of the atomic blocks are generated based on a large language model. The multimodal summaries required by the participants are determined by the importance score of the meeting. The required multimodal summaries are dynamically pushed to the participants to achieve customized real-time follow-up and retrospection of the meeting.

5. The real-time meeting catch-up and backtracking method according to claim 1, wherein, The step of switching back to the current meeting live stream after the participants have completed catching up on the atomic blocks that require meeting backtracking includes: During the process of participants catching up on the atomic blocks, multiple indexed atomic blocks, which are divided into multimodal meeting content collected in the current synchronous meeting, are identified as the atomic blocks that participants need to catch up on and backtrack on. After the participants have completed catching up on the atomic blocks that require meeting backtracking, the system seamlessly switches back to the current meeting live stream.

6. The real-time meeting catch-up and backtracking method according to claim 1, wherein, The method further includes: The system receives questions from participants in real time, locates the time point of the question in the meeting, and provides multimodal meeting content at that time point.

7. The real-time meeting catch-up and backtracking method according to claim 1, wherein, The method further includes: providing participants with real-time meeting replay functionality using the visual interface of an online conferencing system, wherein the visual interface of the online conferencing system includes: The content area is used to display meeting videos or shared screen content to participants; The tag bar is used to provide participants with tags for meeting replay, meeting management, and real-time transcription. When a participant clicks on the meeting replay tag, a video stream display area, a summary area, and an intelligent Q&A area are generated in the tag bar for meeting playback. The toolbar provides meeting control buttons for participants; The status bar is used to provide meeting participants with the meeting's status information.

8. A real-time meeting tracking and replay system, comprising: The content acquisition module is used to acquire multimodal meeting content, including voice, images and user interaction data, in real time, determine the text content of the multimodal meeting content, and simultaneously monitor the user behavior of the participants in real time to determine the attention status of the participants. The atomic block segmentation module is used to divide the real-time acquired multimodal conference content into multiple indexed atomic blocks according to time. The atomic blocks include some conference audio, images, user interaction data and text content. The conference importance of the multiple atomic blocks is scored based on the emotional intensity of the audio and keyword density. The disconnection determination module is used to determine whether the participants are disconnected based on their attention state. If the participants are found to be disconnected, the module determines the time point of their disconnection. The catch-up and backtracking module is used to determine the time difference of the disconnection based on the disconnection time point and the current meeting time, and to determine the atomic blocks that the participants need to catch up and backtrack on the meeting by using the time difference of the disconnection and the index. The switching module is used to switch back to the current live conference stream after the participants have finished catching up on the atomic blocks that require meeting backtracking.

9. A storage medium storing a computer program product thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-7.

10. A computer program product having instructions embedded on a storage medium, the instructions implementing the steps of the method according to any one of claims 1-7.

11. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-7.

Images