A video conference key information highlighting method and system based on semantic understanding

Abstract

The application relates to the technical field of video processing, in particular to a video conference key information highlighting method and system based on semantic understanding. The video conference key information highlighting system based on semantic understanding comprises a subtitle text acquisition module, a text labeling module and a subtitle highlighting rendering module. The application constructs conference text display features with highlighting density and streaming subtitle text display time interval as input, and determines network processing of the conference text display features by using a starting probability threshold, dynamically generates the starting probability threshold, and enables the highlighting trigger intensity to be adaptively adjusted according to the conference speaking rhythm and key information appearance density, thereby avoiding the highlighting jitter or highlighting sparseness problem caused by the fixed threshold under different conference rhythms, and enabling the system to gradually converge to a highlighting strategy more in line with the current conference content without manual intervention.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, specifically to a method and system for highlighting key information in video conferencing based on semantic understanding. Background Technology

[0002] With the widespread adoption of remote work and online collaboration, video conferencing has become a crucial method for corporate communication, project discussions, and decision-making. To improve meeting participation efficiency, existing video conferencing systems typically integrate real-time captioning functionality. This uses speech recognition technology to convert the speaker's audio into text and display it on the meeting interface, facilitating participants' understanding of the content and enabling them to access information in noisy or cross-language environments. However, current real-time captioning technology primarily focuses on accurate speech-to-text transcription. The captions are usually presented as continuous text, lacking effective differentiation of key information. Participants still need to manually identify content relevant to the meeting topic, decisions, or core conclusions from a large amount of text, resulting in a significant cognitive burden. While some systems organize the meeting content afterward through keyword retrieval, summary generation, or manual annotation, this processing mostly occurs after the meeting ends and cannot provide users with immediate information prompts during the meeting. Summary of the Invention

[0003] This invention captures spoken audio in real-time during video conferences and generates character-by-character streaming subtitles. Upon arrival of each streaming subtitle, it combines it with its preceding subtitle to form a subtitle text sequence, which is then input into a text annotation model using BIO annotation. This yields probability vectors for the start of key information, its internal structure, and non-key information, enabling instant semantic recognition and highlighting of the starting position of key information, ensuring the timeliness and continuity of real-time subtitle display. By introducing a retraining mechanism based on conference topic tag-related corpora into the text annotation model, the model provides higher and more concentrated probability outputs for the start and internal positions of key information within the current conference context. This significantly reduces the probability of general pre-trained models misidentifying irrelevant text fragments as key information in the early stages of the conference, improving the semantic relevance of highlighting triggers. Highlighting activation is controlled by comparing the starting probability value of key information with a starting probability threshold. Furthermore, at different stages of the conference, the model combines a topic tag-related key information database with a dynamically constructed adaptive key information database. The system employs a filtering and matching approach, ensuring that highlighting triggers are constrained by both the model's semantic judgment and the distribution of key information already present in the meeting context. This guarantees continuous highlighting of genuine key information while effectively suppressing reading interference caused by excessive or insufficient highlighting. By constructing meeting text display features with highlight density and streaming subtitle display time intervals as input, and using an initial probability threshold determination network to process these features and dynamically generate an initial probability threshold, the highlighting trigger intensity can adaptively adjust with the meeting's speaking rhythm and the density of key information. This avoids the highlight jitter or sparse highlighting issues caused by fixed thresholds under different meeting rhythms. The system can gradually converge to a highlighting strategy that better suits the current meeting content without manual intervention. This decouples semantic understanding capabilities from display control capabilities, allowing them to work collaboratively. Overall, this improves the accuracy, stability, and readability of key information highlighting in video conferencing, enhancing user efficiency and experience during long video conferences.

[0004] This invention provides a method for highlighting key information in video conferencing based on semantic understanding, comprising: Continuously acquire streaming subtitle text; For each streaming subtitle text, perform the following operations: combine the current streaming subtitle text with the previous N-1 streaming subtitle texts to form a subtitle text sequence, and then feed the subtitle text sequence into a text annotation model that has been retrained through the corpus training set associated with the topic tags of the current video conference for processing, to obtain the subtitle text label probability vector corresponding to the current streaming subtitle text. The subtitle text label probability vector includes the initial probability value of key information, the internal probability value of key information, and the probability value of non-key information. If the initial probability value of key information in the probability vector of the current streaming subtitle text is the highest, the current streaming subtitle text is recorded as the starting text of key information. A highlighting operation is performed based on the initial probability threshold determined by the distribution of the highlighted key information. The highlighting operation adds a highlighting style to the current streaming subtitle text. If the internal probability value of key information in the probability vector of the current streaming subtitle text is the highest, the current streaming subtitle text is recorded as the text inside key information. If the previous streaming subtitle text underwent a highlighting operation, the current streaming subtitle text also undergoes a highlighting operation. If the non-key information probability value in the probability vector of the current streaming subtitle text is the highest, the current streaming subtitle text is recorded as non-key information text, and no highlighting operation is performed.

[0005] Preferably, the highlighting operation is performed based on the initial probability threshold determined by the distribution of the highlighted key information, specifically including the following steps: The initial probability value of the key information is compared with the initial probability threshold. If the initial probability value is greater than the threshold, and the number of adaptive key information in the adaptive key information database does not reach the threshold, the initial text of the key information is matched with the key information database associated with the topic tag of the current video conference. Key information associated with the topic tag of the current video conference is considered as associated key information. All associated key information constitutes the key information database. The matching method is to match the initial text of the key information with the first character of the associated key information in the key information database. If the initial text of the key information matches the first character of the associated key information in the key information database, it is considered a successful match; otherwise, it is considered not a successful match. If the match fails, and if the match succeeds, add a highlight style to the starting text of the key information; otherwise, no action is taken. When the number of adaptive key information in the adaptive key information database reaches the threshold, the starting text of the key information is matched with the database. The matching method is to match the starting text of the key information with the first character of the adaptive key information in the database. If the starting text of the key information matches the first character of the adaptive key information in the database, the match is considered successful; otherwise, the match fails. If the match succeeds, add a highlight style to the starting text of the key information; otherwise, no action is taken. If the starting probability value of the key information is not greater than the starting probability threshold, no action is taken.

[0006] Preferably, the adaptive key information database is constructed in the following manner: The adaptive key information base is initially zero. Highlighted key information is added to the adaptive key information base. The subtitle text sequence is fed into a text annotation model that has been retrained on a corpus training set that is not associated with the topic tags of the current video conference for processing and key information is determined. Key information is the text inside the last consecutive key information starting from the key information start text. The frequency of these key information is then counted. Key information with a frequency higher than the frequency threshold is also added to the adaptive key information base.

[0007] Preferably, determining the initial probability threshold based on the distribution of highlighted key information specifically includes the following steps: Construct a meeting time window with a preset time length preceding the timestamp of the current key information starting text. The preset time length is set by the operator. Obtain the time percentage of the highlighted key information within the meeting time window, which is recorded as the highlight density. Obtain the display time interval of adjacent streaming subtitle text, which is recorded as the meeting text display interval. Combine the highlight density and the meeting text display interval to form the meeting text display feature. Then, send the meeting text display feature to the starting probability threshold determination network for processing to obtain the starting probability threshold.

[0008] Preferably, the text annotation model is retrained using a training set of corpora associated with the topic tags corresponding to the current video conference, specifically including the following steps: Subtitle text sequences, including key information related to topic tags, are sampled from sentences in the corpus training set. These subtitle text sequences are then used as input to the text annotation model. The probability vectors of subtitle text tags corresponding to all characters in the subtitle text sequence are used as the target output. The mean squared error between the probability vectors of subtitle text tags corresponding to all characters predicted by the text annotation model and the probability vectors of subtitle text tags corresponding to all characters in the target output is used to construct the retraining loss value. The parameters of the text annotation model are iteratively optimized using gradient descent to minimize the direction of the retraining loss value, thus completing the retraining process.

[0009] Preferably, pre-training the text annotation model includes the following steps: Several video conference transcripts were acquired and key information was annotated. Subtitle text sequences containing the key information were sampled from all annotated transcripts. These sampled subtitle text sequences were then used to form a pre-training set. The text annotation model was pre-trained using this set. During pre-training, the subtitle text sequences were used as input to the model, and the probability vectors of subtitle text labels corresponding to all characters in the subtitle text sequences were used as the target output. The mean squared error between the predicted subtitle text label probability vectors of all characters in the model's output and the target output subtitle text label probability vectors was used to construct the pre-training loss value. The parameters of the text annotation model were iteratively optimized using gradient descent to complete the pre-training process.

[0010] Preferably, training the network based on the initial probability threshold specifically includes the following steps: Several meeting text display features are obtained and labeled using an initial probability threshold. All labeled meeting text display features are combined into an initial probability threshold training set. The initial probability threshold determination network is trained using the initial probability threshold training set. During training, the initial probability threshold determination loss value is constructed using the difference between the predicted output of the initial probability threshold determination network and the labeled initial probability threshold. The parameters of the initial probability threshold determination network are iteratively optimized using gradient descent to minimize the direction of the initial probability threshold determination loss value.

[0011] This invention also provides a video conferencing key information highlighting system based on semantic understanding, comprising: The subtitle text acquisition module is used to continuously acquire streaming subtitle text; The text annotation module is used to perform the following operations for each streaming subtitle text: combine the current streaming subtitle text with the previous N-1 streaming subtitle texts to form a subtitle text sequence, and then feed the subtitle text sequence into the text annotation model retrained through the corpus training set associated with the topic tags corresponding to the current video conference for processing, to obtain the subtitle text label probability vector corresponding to the current streaming subtitle text. The subtitle text label probability vector includes the initial probability value of key information, the internal probability value of key information, and the probability value of non-key information. The subtitle highlighting module is used to highlight the text in the streaming subtitles. If the initial probability value of key information in the subtitle text label probability vector corresponding to the current streaming subtitle text is the largest, the current streaming subtitle text is marked as the starting text of key information. The highlighting operation is performed based on the initial probability threshold determined by the distribution of the highlighted key information, which adds a highlighting style to the current streaming subtitle text. If the internal probability value of key information in the subtitle text label probability vector corresponding to the current streaming subtitle text is the largest, the current streaming subtitle text is marked as the text inside key information. If the previous streaming subtitle text has undergone a highlighting operation, the current streaming subtitle text will also undergo a highlighting operation. If the non-key information probability value in the subtitle text label probability vector corresponding to the current streaming subtitle text is the largest, the current streaming subtitle text is marked as non-key information text and no highlighting operation is performed.

[0012] The present invention has the following advantages: This invention captures spoken audio in real-time during video conferences and generates character-by-character streaming subtitles. Upon arrival of each streaming subtitle, it combines it with its preceding subtitle to form a subtitle text sequence, which is then input into a text annotation model using BIO annotation. This yields probability vectors for the start of key information, its internal structure, and non-key information, enabling instant semantic recognition and highlighting of the starting position of key information, ensuring the timeliness and continuity of real-time subtitle display. By introducing a retraining mechanism based on conference topic tag-related corpora into the text annotation model, the model provides higher and more concentrated probability outputs for the start and internal positions of key information within the current conference context. This significantly reduces the probability of general pre-trained models misidentifying irrelevant text fragments as key information in the early stages of the conference, improving the semantic relevance of highlighting triggers. Highlighting activation is controlled by comparing the starting probability value of key information with a starting probability threshold. Furthermore, at different stages of the conference, the model combines a topic tag-related key information database with a dynamically constructed adaptive key information database. The system employs a filtering and matching approach, ensuring that highlighting triggers are constrained by both the model's semantic judgment and the distribution of key information already present in the meeting context. This guarantees continuous highlighting of genuine key information while effectively suppressing reading interference caused by excessive or insufficient highlighting. By constructing meeting text display features with highlight density and streaming subtitle display time intervals as input, and using an initial probability threshold determination network to process these features and dynamically generate an initial probability threshold, the highlighting trigger intensity can adaptively adjust with the meeting's speaking rhythm and the density of key information. This avoids the highlight jitter or sparse highlighting issues caused by fixed thresholds under different meeting rhythms. The system can gradually converge to a highlighting strategy that better suits the current meeting content without manual intervention. This decouples semantic understanding capabilities from display control capabilities, allowing them to work collaboratively. Overall, this improves the accuracy, stability, and readability of key information highlighting in video conferencing, enhancing user efficiency and experience during long video conferences. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the structure of a video conferencing key information highlighting system based on semantic understanding, used in an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this invention.

[0015] Example 1: A method for highlighting key information in video conferencing based on semantic understanding, comprising: During a video conference, the speaker's voice signal data is collected and converted into corresponding subtitle text using speech recognition technology. This is a common real-time subtitle technology in existing video conferences. Streaming subtitles refer to the subtitles displayed word by word, meaning that each word spoken by the speaker is displayed at the bottom of the video conference interface through speech recognition. For streaming subtitles, streaming subtitle text is continuously acquired. The streaming subtitle text here is obtained by converting the words spoken by the speaker through speech recognition technology. For each streaming caption, the following operations are performed: the current streaming caption and the previous N-1 streaming captions are combined to form a caption text sequence, which represents a portion of the speaker's speech. The caption text sequence is then fed into a text annotation model retrained using a corpus training set associated with the topic tags of the current video conference for processing. This yields a caption text label probability vector corresponding to the current streaming caption. The caption text label probability vector includes probability values ​​for each of the following: key information start identifier, key information internal identifier, and non-key information identifier, denoted as key information start probability value, key information internal probability value, and non-key information probability value, respectively. The key information start identifier represents the beginning of a key piece of information. Key information refers to text fragments related to the topic of the current video conference or decision-related text fragments, which are set by the operator during model training. The key information internal identifier refers to text fragments within the key information, and the non-key information identifier refers to text fragments not considered key information. The text annotation model here uses the BIO annotation method, which can be implemented using a Transformer model. If the initial probability value of key information in the probability vector of the current streaming subtitle text is the highest, then the current streaming subtitle text is recorded as the starting text of key information. A highlighting operation is then performed based on the initial probability threshold determined by the distribution of the highlighted key information. This highlighting operation adds a highlighting style to the current streaming subtitle text. If the internal probability value of key information in the probability vector of the current streaming subtitle text is the highest, then the current streaming subtitle text is recorded as the text inside key information. If the previous streaming subtitle text underwent a highlighting operation, the current streaming subtitle text will also undergo a highlighting operation. If the non-key information probability value in the probability vector of the current streaming subtitle text is the highest, then the current streaming subtitle text is recorded as non-key information text, and no highlighting operation is performed. It should be noted that the text annotation model was tested using various video conference texts before its implementation. The pre-trained text annotation model, after being retrained using a corpus associated with the topic tags of the current video conference, will have a higher probability value for annotating key information related to the current video conference. If the distribution of highlighted key information is relatively dense, a higher initial probability threshold can be set. When more sparsely highlighted key information is needed, key information related to the current video conference can still be highlighted, while key information with less relevance to the current video conference will not be highlighted. This ensures that the highlighted key information is more representative while maintaining the highlighting effect. Conversely, if the distribution of highlighted key information is relatively sparse, some less relevant keywords can also be highlighted, improving the user experience of highlighting key information in the video conference. Regardless of whether the subtitle text has been highlighted, it will be displayed sequentially at the bottom of the video conference interface. It should be added that when starting a video conference, the operator needs to select relevant topic tags in advance. The system will determine the associated key information based on these topic tags. This can be achieved through a knowledge graph. Developers can build a knowledge graph that associates topic tags with key information. For example, they can obtain the text records of completed video conferences, manually annotate key information in the text records, and then associate these annotated key information with the topic tags corresponding to the video conference. Then, they can query the associated key information in the knowledge graph based on the topic tags provided by the operator. Finally, they can select all statements containing associated key information from all maintained video record texts to form a corpus training set. The highlighting operation is performed based on the initial probability threshold determined by the distribution of the key highlighted information, specifically including the following steps: The initial probability value of key information is compared with the initial probability threshold. If the initial probability value of key information is greater than the initial probability threshold, and the number of adaptive key information in the adaptive key information database has not reached the adaptive key information quantity threshold (set by the operator), the initial text of the key information is matched with the key information database associated with the topic tag corresponding to the current video conference. Key information associated with the topic tag corresponding to the current video conference is considered associated key information. All associated key information constitutes the key information database. The matching method is to match the initial text of the key information with the first character of the associated key information in the key information database. If the initial text of the key information matches the first character of the associated key information in the key information database, the match is considered successful; otherwise, the match is considered unsuccessful. If the match is successful, a highlight style is added to the initial text of the key information; otherwise, no action is taken. When the number of adaptive key information in the adaptive key information database reaches the adaptive key information quantity threshold, the initial text of the key information is matched with the adaptive key information database. The matching method is to match the initial text of the key information with the first character of the associated key information in the database. The first character of the key information in the key information database is matched. If the first character of the key information matches the first character of the key information in the database, the match is considered successful; otherwise, the match is considered unsuccessful. If the match is successful, a highlighting style is added to the first character of the key information; otherwise, no action is taken. If the initial probability value of the key information is not greater than the initial probability threshold, no action is taken. It should be noted that at the beginning of the video conference, when more sparsely highlighted key information is needed, some text fragments that are irrelevant to the current video conference may also meet the condition that the initial probability value of the key information is greater than the initial probability threshold. This is because the text annotation model was pre-trained on various video conference text records before use, and highlighting these text fragments is of little significance. Therefore, the key information database associated with the topic tags of the current video conference is used for filtering to reduce meaningless highlighting in the early stages of the conference. The method is first character matching, which is a fuzzy matching and is only used as a means to reduce meaningless highlighting in the early stages of the conference. In the middle of the conference, the key information already recorded in the video conference can be used for filtering. The adaptive key information base is constructed as follows: Initially, the adaptive key information base is zero. Highlighted key information is added to the adaptive key information base. The subtitle text sequence is fed into a text annotation model retrained on a corpus not associated with the current video conference's topic tags for processing, and key information is determined. Here, key information is defined as the text within the last consecutive key information segment, starting with the key information's initial text. The frequency of these key information segments is then counted, and key information segments with frequencies exceeding a frequency threshold are also added to the adaptive key information base. This frequency threshold is set by the operator. The subtitle text from the current video conference's history is used to determine key information using a text annotation model retrained on a corpus not associated with the current video conference's topic tags, and frequently occurring key information segments are also highlighted. The initial probability threshold is determined based on the distribution of highlighted key information, specifically including the following steps: A meeting time window with a preset time length preceding the timestamp of the current key information starting text is constructed. The preset time length is set by the operator. The time ratio corresponding to the highlighted key information within the meeting time window is obtained and recorded as the highlight density. The display time interval of adjacent streaming subtitle text is obtained and recorded as the meeting text display interval. It should be noted that the display time interval of streaming subtitle text here refers to the time interval between the display of streaming subtitle text on the video conferencing interface. The highlight density and the meeting text display interval are combined to form the meeting text display feature. The meeting text display feature is then fed into the starting probability threshold determination network for processing to obtain the starting probability threshold. The starting probability threshold determination network adopts MLP and analyzes the distribution of highlighted key information through highlight density and meeting text display interval. The text annotation model is retrained using the corpus training set associated with the topic tags of the current video conference. The specific steps include the following: Subtitle text sequences, including key information related to topic tags, are sampled from sentences in the training corpus. These subtitle text sequences are then used as input to the text annotation model, with the subtitle text label probability vectors corresponding to all characters in the subtitle text sequence serving as the target output. Specifically, the probability vectors are as follows: the probability of the initial key information text corresponding to the starting key information text is 1, and the probability of the text text text corresponding to the key information text is 0; the probability of the text text text corresponding to the key information text is 1, and the probability of the text text text corresponding to the key information text is 0; and the probability of the text text text corresponding to the non-key information text is 1, and the probability of the text text text corresponding to the non-key information text is 0. The mean squared error between the subtitle text label probability vectors corresponding to all characters predicted by the text annotation model and the subtitle text label probability vectors corresponding to all characters in the target output is used to construct the retraining loss value. The parameters of the text annotation model are iteratively optimized using gradient descent to minimize the direction of the retraining loss value, thus completing the retraining process.

[0016] Pre-training the text annotation model involves the following steps: Acquire several video conference transcripts and annotate key information within them. Annotation can be done manually or using annotation software. Sample caption text sequences containing key information from all annotated transcripts and compile these sequences into a pre-training set. Use this pre-training set to pre-train the text annotation model. During pre-training, use the caption text sequences as input and the probability vectors of caption text labels corresponding to all characters in the sequence as the target output. Construct the pre-training loss value using the mean squared error between the predicted and target outputs of the text annotation model's probability vectors of caption text labels. Iteratively optimize the parameters of the text annotation model using gradient descent to complete the pre-training. If the accuracy of the text annotation model meets expectations, output the pre-trained model; otherwise, continue pre-training using the pre-training set. The text annotation model then enables semantic understanding of the caption text.

[0017] Training the network to determine the initial probability threshold involves the following steps: Several meeting text display features are obtained. These features are acquired by developers based on pre-experiment video conferences. These features are labeled using an initial probability threshold. During the pre-experiment video conferences, developers continuously adjust the initial probability threshold and score the highlighting effect. The initial probability threshold corresponding to the highest score is used to label the meeting text display feature. All labeled meeting text display features are combined into an initial probability threshold training set. The initial probability threshold determination network is trained using this training set. During training, the difference between the predicted output of the initial probability threshold determination network and the labeled initial probability threshold is used to construct the initial probability threshold determination loss value. The direction for minimizing the initial probability threshold determination loss value is used to iteratively optimize the parameters of the initial probability threshold determination network using gradient descent. If the accuracy of the initial probability threshold determination network meets expectations, the trained initial probability threshold determination network is output; otherwise, the network is trained again using the initial probability threshold training set.

[0018] This application captures spoken audio in real-time during video conferences and generates word-by-word streaming subtitles. Upon arrival of each streaming subtitle, it combines it with its preceding subtitle to form a subtitle text sequence, which is then input into a text annotation model using BIO annotation. This yields probability vectors for the start of key information, its internal structure, and non-key information, enabling immediate semantic recognition and highlighting of the starting position of key information, ensuring the timeliness and continuity of real-time subtitle display. By introducing a retraining mechanism based on conference topic tag-related corpora into the text annotation model, the model provides higher and more concentrated probability outputs for the start and internal positions of key information within the current conference context. This significantly reduces the probability of general pre-trained models misidentifying irrelevant text fragments as key information in the early stages of the conference, improving the semantic relevance of highlighting triggers. Highlighting activation is controlled by comparing the starting probability value of key information with a starting probability threshold. Furthermore, at different stages of the conference, the model combines a topic tag-related key information database with a dynamically constructed adaptive key information database. The system employs a filtering and matching approach, ensuring that highlighting triggers are constrained by both the model's semantic judgment and the distribution of key information already present in the meeting context. This guarantees continuous highlighting of genuine key information while effectively suppressing reading interference caused by excessive or insufficient highlighting. By constructing meeting text display features with highlight density and streaming subtitle display time intervals as input, and using an initial probability threshold determination network to process these features and dynamically generate an initial probability threshold, the highlighting trigger intensity can adaptively adjust with the meeting's speaking rhythm and the density of key information. This avoids the highlight jitter or sparse highlighting issues caused by fixed thresholds under different meeting rhythms. The system can gradually converge to a highlighting strategy that better suits the current meeting content without manual intervention. This decouples semantic understanding capabilities from display control capabilities, allowing them to work collaboratively. Overall, this improves the accuracy, stability, and readability of key information highlighting in video conferencing, enhancing user efficiency and experience during long video conferences.

[0019] Example 2: A video conferencing key information highlighting system based on semantic understanding, such as... Figure 1 As shown, it includes: The subtitle text acquisition module is used to continuously acquire the text of streaming subtitles. The text of streaming subtitles is obtained by converting the words spoken by the speaker through speech recognition technology. The text annotation module is used to combine the current streaming subtitle text with the previous N-1 streaming subtitle texts to form a subtitle text sequence. This subtitle text sequence represents a portion of the speaker's speech text. The subtitle text sequence is then fed into a text annotation model retrained using a corpus training set associated with the topic tags of the current video conference for processing. This results in a subtitle text label probability vector corresponding to the current streaming subtitle text. The subtitle text label probability vector includes probability values ​​for each of the following: key information start identifier, key information internal identifier, and non-key information identifier. These are denoted as key information start probability value, key information internal probability value, and non-key information probability value, respectively. The key information start identifier represents the beginning of a key piece of information. Key information refers to text fragments related to the topic of the current video conference or decision-related text fragments, which are set by the operator during model training. The key information internal identifier refers to text fragments within the key information, and the non-key information identifier refers to text fragments not considered key information. The text annotation model here uses the BIO annotation method and can use a Transformer model. The subtitle highlighting module is used to highlight the text in the streaming subtitles. If the initial probability value of key information in the subtitle text label probability vector corresponding to the current streaming subtitle text is the largest, the current streaming subtitle text is marked as the starting text of key information, and a highlighting operation is performed based on the initial probability threshold determined by the distribution of the highlighted key information. The highlighting operation adds a highlighting style to the current streaming subtitle text. If the internal probability value of key information in the subtitle text label probability vector corresponding to the current streaming subtitle text is the largest, the current streaming subtitle text is marked as the text inside key information. If the previous streaming subtitle text underwent a highlighting operation, the current streaming subtitle text will also undergo a highlighting operation. If the non-key information probability value in the subtitle text label probability vector corresponding to the current streaming subtitle text is the largest, the current streaming subtitle text is marked as non-key information text, and no highlighting operation is performed. It should be noted that the text annotation model is used in... Previously, the text annotation model was pre-trained using various video conference text transcripts. After retraining on a training set associated with the topic tags of the current video conference, the text annotation model has a higher probability of annotating key information related to the current video conference. If the distribution of highlighted key information is relatively dense, a higher initial probability threshold can be set. When more sparsely highlighted key information is needed, key information related to the current video conference can still be highlighted, while key information with less relevance to the current video conference will not be highlighted. This ensures that the highlighted key information is more representative while maintaining the highlighting effect. Conversely, if the distribution of highlighted key information is relatively sparse, some less relevant keywords can also be highlighted, improving the user experience of highlighting key information in video conferences. Regardless of whether the subtitles have been highlighted, they will be displayed sequentially at the bottom of the video conference interface.

[0020] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims. Parts not described in detail in this specification are prior art known to those skilled in the art.

Claims

1. A method for highlighting key information in video conferencing based on semantic understanding, characterized in that, include: Continuously acquire streaming subtitle text; For each streaming subtitle text, perform the following operations: combine the current streaming subtitle text with the previous N-1 streaming subtitle texts to form a subtitle text sequence, and then feed the subtitle text sequence into a text annotation model that has been retrained through the corpus training set associated with the topic tags of the current video conference for processing, to obtain the subtitle text label probability vector corresponding to the current streaming subtitle text. The subtitle text label probability vector includes the initial probability value of key information, the internal probability value of key information, and the probability value of non-key information. If the initial probability value of key information in the probability vector of the current streaming subtitle text is the highest, the current streaming subtitle text is recorded as the starting text of key information. A highlighting operation is performed based on the initial probability threshold determined by the distribution of the highlighted key information. The highlighting operation adds a highlighting style to the current streaming subtitle text. If the internal probability value of key information in the probability vector of the current streaming subtitle text is the highest, the current streaming subtitle text is recorded as the text inside key information. If the previous streaming subtitle text underwent a highlighting operation, the current streaming subtitle text also undergoes a highlighting operation. If the non-key information probability value in the probability vector of the current streaming subtitle text is the highest, the current streaming subtitle text is recorded as non-key information text, and no highlighting operation is performed.

2. The method for highlighting key information in video conferencing based on semantic understanding according to claim 1, characterized in that, The highlighting operation is performed based on the initial probability threshold determined by the distribution of the key highlighted information, specifically including the following steps: The initial probability value of the key information is compared with the initial probability threshold. If the initial probability value is greater than the threshold, and the number of adaptive key information in the adaptive key information database does not reach the threshold, the initial text of the key information is matched with the key information database associated with the topic tag of the current video conference. Key information associated with the topic tag of the current video conference is considered as associated key information. All associated key information constitutes the key information database. The matching method is to match the initial text of the key information with the first character of the associated key information in the key information database. If the initial text of the key information matches the first character of the associated key information in the key information database, it is considered a successful match; otherwise, it is considered not a successful match. If the match fails, and if the match succeeds, add a highlight style to the starting text of the key information; otherwise, no action is taken. When the number of adaptive key information in the adaptive key information database reaches the threshold, the starting text of the key information is matched with the database. The matching method is to match the starting text of the key information with the first character of the adaptive key information in the database. If the starting text of the key information matches the first character of the adaptive key information in the database, the match is considered successful; otherwise, the match fails. If the match succeeds, add a highlight style to the starting text of the key information; otherwise, no action is taken. If the starting probability value of the key information is not greater than the starting probability threshold, no action is taken.

3. The method for highlighting key information in a video conference based on semantic understanding according to claim 2, characterized in that, The adaptive key information database is constructed in the following way: The adaptive key information base is initially zero. Highlighted key information is added to the adaptive key information base. The subtitle text sequence is fed into a text annotation model that has been retrained on a corpus training set that is not associated with the topic tags of the current video conference for processing and key information is determined. Key information is defined as the text inside the last consecutive key information starting from the key information start text. The frequency of all key information is then counted. Key information with a frequency higher than the frequency threshold is also added to the adaptive key information base.

4. The method for highlighting key information in a video conference based on semantic understanding according to claim 3, characterized in that, The initial probability threshold is determined based on the distribution of highlighted key information, specifically including the following steps: Construct a meeting time window with a preset time length preceding the timestamp of the current key information starting text. Obtain the time percentage of the highlighted key information within the meeting time window, denoted as the highlight density. Obtain the display time interval of adjacent streaming subtitle text, denoted as the meeting text display interval. Combine the highlight density and the meeting text display interval to form the meeting text display feature. Then, send the meeting text display feature into the starting probability threshold determination network for processing to obtain the starting probability threshold.

5. The method for highlighting key information in a video conference based on semantic understanding according to claim 4, characterized in that, The text annotation model is retrained using the corpus training set associated with the topic tags of the current video conference. The specific steps include the following: Subtitle text sequences, including key information related to topic tags, are sampled from sentences in the corpus training set. These subtitle text sequences are then used as input to the text annotation model. The probability vectors of subtitle text tags corresponding to all characters in the subtitle text sequence are used as the target output. The mean squared error between the probability vectors of subtitle text tags corresponding to all characters predicted by the text annotation model and the probability vectors of subtitle text tags corresponding to all characters in the target output is used to construct the retraining loss value. The parameters of the text annotation model are iteratively optimized using gradient descent to minimize the direction of the retraining loss value, thus completing the retraining process.

6. The method for highlighting key information in a video conference based on semantic understanding according to claim 5, characterized in that, Pre-training the text annotation model involves the following steps: Several video conference transcripts were acquired and key information was annotated. Subtitle text sequences containing the key information were sampled from all annotated transcripts. These sampled subtitle text sequences were then used to form a pre-training set. The text annotation model was pre-trained using this set. During pre-training, the subtitle text sequences were used as input to the model, and the probability vectors of subtitle text labels corresponding to all characters in the subtitle text sequences were used as the target output. The mean squared error between the predicted subtitle text label probability vectors of all characters in the model's output and the target output subtitle text label probability vectors was used to construct the pre-training loss value. The parameters of the text annotation model were iteratively optimized using gradient descent to complete the pre-training process.

7. The method for highlighting key information in a video conference based on semantic understanding according to claim 6, characterized in that, Training the network to determine the initial probability threshold involves the following steps: Several meeting text display features are obtained and labeled using an initial probability threshold. All labeled meeting text display features are combined into an initial probability threshold training set. The initial probability threshold determination network is trained using the initial probability threshold training set. During training, the initial probability threshold determination loss value is constructed using the difference between the predicted output of the initial probability threshold determination network and the labeled initial probability threshold. The parameters of the initial probability threshold determination network are iteratively optimized using gradient descent to minimize the direction of the initial probability threshold determination loss value.

8. A video conferencing key information highlighting system based on semantic understanding, characterized in that, The system employs a semantic understanding-based video conferencing key information highlighting method according to any one of claims 1-7, including: The subtitle text acquisition module is used to continuously acquire streaming subtitle text; The text annotation module is used to perform the following operations for each streaming subtitle text: combine the current streaming subtitle text with the previous N-1 streaming subtitle texts to form a subtitle text sequence, and then feed the subtitle text sequence into the text annotation model retrained through the corpus training set associated with the topic tags corresponding to the current video conference for processing, to obtain the subtitle text label probability vector corresponding to the current streaming subtitle text. The subtitle text label probability vector includes the initial probability value of key information, the internal probability value of key information, and the probability value of non-key information. The subtitle highlighting module is used to highlight the text in the streaming subtitles. If the initial probability value of key information in the subtitle text label probability vector corresponding to the current streaming subtitle text is the largest, the current streaming subtitle text is marked as the starting text of key information. The highlighting operation is performed based on the initial probability threshold determined by the distribution of the highlighted key information, which adds a highlighting style to the current streaming subtitle text. If the internal probability value of key information in the subtitle text label probability vector corresponding to the current streaming subtitle text is the largest, the current streaming subtitle text is marked as the text inside key information. If the previous streaming subtitle text has undergone a highlighting operation, the current streaming subtitle text will also undergo a highlighting operation. If the non-key information probability value in the subtitle text label probability vector corresponding to the current streaming subtitle text is the largest, the current streaming subtitle text is marked as non-key information text and no highlighting operation is performed.

Images