Methods, devices, programs, and electronic equipment for generating meeting minutes

Abstract

This disclosure relates to a method and apparatus for generating meeting minutes, a computer program product, and an electronic device, belonging to the field of computer technology. The method includes: acquiring multimodal data corresponding to a meeting scenario, and performing spatiotemporal alignment on the multimodal data to determine spatiotemporally aligned multimodal data; the multimodal data includes meeting video, operation trajectories, meeting audio, and interactive text; parsing the spatiotemporally aligned multimodal data based on a multimodal understanding network, and combining it with the role tags of the speakers in the meeting scenario to output multimodal semantic combination information; constructing a decision tree corresponding to the meeting scenario based on the multimodal semantic combination information, and generating meeting minutes corresponding to the meeting scenario based on the decision tree and a knowledge graph associated with the meeting scenario. This disclosure can improve the efficiency and accuracy of meeting minute generation.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a method for generating meeting minutes, a device for generating meeting minutes, a computer program product, and an electronic device. Background Technology

[0002] With the development of digital technology, online meetings play a crucial role in daily production. They are not limited by physical space, reduce personnel movement, and improve work efficiency.

[0003] In some embodiments, meeting minutes can be generated in the following manner. The specific process includes: A data acquisition device saves raw data such as meeting audio and video. Speech recognition capabilities are used to transcribe the incoming audio into a data warehouse. The data warehouse identifies the transcribed text and issues warnings or rejects non-standard information. Support personnel apply natural language understanding technology to break down, complete, and combine the information for correction. The corrected text data is re-diagnosed and scanned again for information standardization. Standardized meeting minutes are generated through the rational arrangement and combination of standardized field information.

[0004] In the above methods, meeting minutes are mainly generated by converting audio recordings into text using speech recognition technology, and then using natural language processing technology to extract keywords for classification and integration. Processing meeting records one by one results in low efficiency and consumes a lot of server resources; single speech transcription cannot capture the operational context. Summary of the Invention

[0005] The purpose of this disclosure is to provide a method, apparatus, computer program product, and electronic device for generating meeting minutes, thereby overcoming, to at least some extent, the problem of low efficiency in generating meeting minutes due to limitations and defects in related technologies.

[0006] According to one aspect of this disclosure, a method for generating meeting minutes is provided, comprising:

[0007] Acquire multimodal data corresponding to the meeting scenario, and perform spatiotemporal alignment on the multimodal data to determine the spatiotemporally aligned multimodal data; the multimodal data includes meeting video, operation trajectory, meeting audio, and interactive text;

[0008] The multimodal understanding network parses spatiotemporally aligned multimodal data and, combined with the speaker's role labels in the meeting scenario, outputs multimodal semantic combination information.

[0009] Based on the multimodal semantic combination information, a decision tree corresponding to the meeting scenario is constructed, and based on the decision tree and the knowledge graph associated with the meeting scenario, meeting minutes corresponding to the meeting scenario are generated.

[0010] In one exemplary embodiment of this disclosure, the multimodal understanding network parses spatiotemporally aligned multimodal data and, in conjunction with the speaker's role labels in the meeting scenario, outputs multimodal semantic combination information, including:

[0011] The spatiotemporally aligned multimodal data is parsed to extract key content features associated with the meeting scenario;

[0012] The target multimodal data corresponding to the key content features are labeled according to the role tags of the speakers in the meeting scenario, and the multimodal semantic combination information is generated; the multimodal semantic combination information is used to represent the target multimodal data corresponding to each role tag.

[0013] In one exemplary embodiment of this disclosure, the feature vectors of the spatiotemporally aligned multimodal data are parsed to extract key content features associated with the meeting scenario, including:

[0014] The intermediate image is obtained by fusing the meeting video and operation trajectory at the target time, and the intermediate text is obtained by fusing the meeting audio and interactive text at the target time.

[0015] Determine the image features of the intermediate image and the text features of the intermediate text;

[0016] An adaptive collaborative attention network is used to determine target image features related to the text features, and target text features are selected based on the target image features.

[0017] The key content features are determined by fusing the target image features and the target text features according to the gating mechanism.

[0018] In one exemplary embodiment of this disclosure, determining the text features of the intermediate text includes:

[0019] Extract the character-level word vectors of the intermediate text, and extract the ordinary-level word vectors of the intermediate text;

[0020] The character-level word vectors and the ordinary-level word vectors are connected at the same position to determine the concatenated text features;

[0021] The spliced ​​text features are encoded using a bidirectional long short-term memory network to determine the text features.

[0022] In one exemplary embodiment of this disclosure, the step of annotating the target multimodal data corresponding to the key content features based on the speaker's role label in the meeting scenario, and generating the multimodal semantic combination information, includes:

[0023] Determine the candidate role tags corresponding to the key content features;

[0024] Based on the candidate role tags, the target multimodal data corresponding to the key content features is decomposed to obtain decomposed data;

[0025] Identify the role tags that are the same as the candidate role tags, and map the decomposition data corresponding to the candidate role tags to the same role tags.

[0026] In one exemplary embodiment of this disclosure, the step of constructing a decision tree corresponding to the meeting scenario based on the multimodal semantic combination information, and generating meeting minutes corresponding to the meeting scenario based on the decision tree and the knowledge graph associated with the meeting scenario, includes:

[0027] The nodes of the decision tree are determined, and based on the knowledge graph and context information, the decomposed data is supplemented to form structured text blocks. The structured text blocks are then mapped to the nodes of the decision tree to construct the decision tree.

[0028] Based on the nodes of the decision tree and the structured knowledge in the knowledge graph, the framework of the meeting minutes is established.

[0029] The structured text blocks corresponding to the nodes of the decision tree are filled into the frame to generate structured information;

[0030] The structured information is converted into text to generate the meeting minutes.

[0031] In one exemplary embodiment of this disclosure, the method further includes:

[0032] A confidence level is calculated on the meeting minutes, and the confidence level is used to determine whether the meeting minutes are a standard text.

[0033] If the meeting minutes are in a standard text format, output the meeting minutes directly.

[0034] If the meeting minutes are non-standard text, the network parameters of the multimodal understanding network are updated until a standard text is obtained.

[0035] According to one aspect of this disclosure, a meeting minutes generation apparatus is provided, comprising:

[0036] A multimodal data acquisition module is used to acquire multimodal data corresponding to the meeting scenario, and to perform spatiotemporal alignment on the multimodal data to determine the spatiotemporally aligned multimodal data; the multimodal data includes meeting video, operation trajectory, meeting audio, and interactive text;

[0037] The parsing module is used to parse spatiotemporally aligned multimodal data based on a multimodal understanding network, and output multimodal semantic combination information by combining the role labels of the speakers in the meeting scenario.

[0038] The generation module is used to construct a decision tree corresponding to the meeting scenario based on the multimodal semantic combination information, and to generate meeting minutes corresponding to the meeting scenario based on the decision tree and the knowledge graph associated with the meeting scenario.

[0039] According to one aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the meeting minutes generation method described in any of the preceding claims.

[0040] According to one aspect of this disclosure, an electronic device is provided, comprising:

[0041] Processor; and

[0042] Memory for storing the executable instructions of the processor;

[0043] The processor is configured to execute the meeting minutes generation method described above by executing the executable instructions.

[0044] The technical solution provided in this disclosure has two advantages. First, it can directly obtain the multimodal semantic combination information of each speaker through multimodal understanding network parsing. Further, it can perform structured processing on the multimodal semantic combination information of each speaker based on decision trees and knowledge graphs to obtain meeting minutes. This eliminates the need to process meeting records one by one, reducing operational steps, improving the efficiency of meeting minutes generation, and reducing resource consumption. Second, since meeting minutes can be generated from multimodal data composed of meeting video, operation trajectories, meeting audio, and interactive text, it avoids the limitations of single speech transcription in capturing context, increasing the comprehensiveness and accuracy of the materials used to generate meeting minutes, and improving the accuracy and reliability of the meeting minutes.

[0045] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0047] Figure 1The diagram illustrates a flowchart of a meeting minutes generation method in related technologies.

[0048] Figure 2 The schematic diagram illustrates a flowchart of a meeting minutes generation method according to an embodiment of the present disclosure.

[0049] Figure 3 The schematic diagram illustrates the overall framework of the multimodal parsing network in an embodiment of this disclosure.

[0050] Figure 4 The schematic diagram illustrates the overall flowchart for generating meeting minutes in an embodiment of this disclosure.

[0051] Figure 5 The schematic diagram illustrates a specific flowchart for generating meeting minutes in an embodiment of this disclosure.

[0052] Figure 6 The schematic diagram illustrates a block diagram of a meeting minutes generation device in an embodiment of this disclosure.

[0053] Figure 7 A schematic block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0054] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0055] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0056] With the development of digital technology, online meetings play a crucial role in daily production. They are not limited by physical space, reduce personnel movement, and improve work efficiency. In some embodiments, they can be achieved through... Figure 1 The method for generating meeting minutes is described in [reference]. Figure 1 As shown, the acquisition device saves raw data such as meeting audio and video. Using Automatic Speech Recognition (ASR) capabilities, the incoming audio is transcribed and stored in the data warehouse. The data warehouse identifies the transcribed text, issuing warnings or rejecting non-standard information. Support personnel apply Natural Language Understanding (NLP) technology to break down, complete, and combine the information for correction. The corrected text data is re-diagnosed and scanned again for information standardization. Through the rational arrangement and combination of standardized field information, standardized meeting minutes are generated.

[0057] In the above methods, meeting minutes are mainly generated by converting audio recordings into text using speech recognition technology, and then using natural language processing technology to extract keywords for classification and integration. Processing meeting records one by one leads to low efficiency and consumes a large amount of server resources, including license channels and storage resources. Single speech transcription cannot capture the operational context (editing trajectory of shared meeting screens and shared documents). Unstructured data leads to the omission of key decision points and relies heavily on support personnel.

[0058] To address the aforementioned technical problems, this disclosure provides a method for generating meeting minutes. (See reference...) Figure 2 As shown, the method for generating meeting minutes mainly includes the following steps:

[0059] In step S210, multimodal data corresponding to the meeting scenario is acquired, and the multimodal data is spatiotemporally aligned to determine the spatiotemporally aligned multimodal data.

[0060] In step S220, the spatiotemporally aligned multimodal data is parsed based on the multimodal understanding network, and multimodal semantic combination information is output by combining the role labels of the speakers in the meeting scenario.

[0061] In step S230, a decision tree corresponding to the meeting scenario is constructed based on the multimodal semantic combination information, and meeting minutes corresponding to the meeting scenario are generated based on the decision tree and the knowledge graph associated with the meeting scenario.

[0062] The technical solution in this embodiment has several advantages. First, it allows for the direct parsing of multimodal semantic combination information for each speaker through a multimodal understanding network. Further, it enables the structured processing of this information using decision trees and knowledge graphs to generate meeting minutes. This eliminates the need to process meeting records individually, reducing operational steps, improving the efficiency of meeting minutes generation, and lowering resource consumption. Second, since meeting minutes can be generated from multimodal data composed of meeting video, operation trajectories, meeting audio, and interactive text, it avoids the limitations of single speech transcription in capturing context, increasing the comprehensiveness and accuracy of the materials used to generate meeting minutes, and improving the accuracy and reliability of the meeting minutes. Third, the use of decision trees and knowledge graphs helps avoid the omission of key decision points.

[0063] Next, refer to Figure 2 The following diagram illustrates the method for generating meeting minutes in an embodiment of this disclosure.

[0064] In step S210, multimodal data corresponding to the meeting scenario is acquired, and the multimodal data is spatiotemporally aligned to determine the spatiotemporally aligned multimodal data.

[0065] In this embodiment, the meeting scenario can be either an offline meeting scenario or an online meeting scenario, without specific limitations. Multimodal data refers to the interaction data stream of participants corresponding to the meeting scenario obtained from the meeting system. This data can include interaction data from multiple data sources, such as meeting recordings, meeting videos, interactive text, and operation trajectories. Operation trajectories refer to the user's screen operation behavior trajectory in the user scenario, such as mouse movement trajectory, keyboard input trajectory, or keyboard deletion trajectory. Based on this, the interaction data from multiple data sources can include various types of meeting videos, operation trajectories, meeting audio, and interactive text. Meeting videos can be screen recordings or other forms of video. Since meetings typically last for a period of time, meeting videos, operation trajectories, meeting audio, and interactive text are all presented as data streams.

[0066] To improve the accuracy of data processing, data preprocessing can be performed on interactive data from multiple data sources to obtain multimodal data. Data preprocessing can include data cleaning and data noise reduction. For example, real-time and integrity checks can be performed on the interactive data collected from multiple data sources to remove irrelevant information and duplicate data, and noise reduction can be performed to determine the multimodal data stream.

[0067] Next, the multimodal data can be time-aligned. Here, time alignment refers to synchronizing the timestamps of the multimodal data and associating spatial operation layers to ensure that semantically coherent contextual multimodal data is formed under a unified spatiotemporal coordinate system.

[0068] In some embodiments, a convolutional neural network can be constructed to perform temporal behavior analysis on multimodal data, thereby achieving timestamp synchronization and spatial operation layer association.

[0069] For example, a convolutional neural network model can be used to apply DTW (Dynamic Time Warping) to multimodal data streams, such as conference video streams, operation trajectory streams, conference audio streams, and interactive text streams, based on time synchronization rules, to achieve temporal alignment between multimodal data. Specifically, the conference video stream, operation trajectory stream, conference audio stream, and interactive text stream can be treated as time series, and the time series can be input into the convolutional neural network model to determine the distance matrix between the time series, thereby outputting the optimal alignment path for multiple time series. Dynamic time warping between multiple time series can then be implemented based on the optimal alignment path to resolve deviations caused by network latency.

[0070] In addition, convolutional neural network models can be used to extract features from multimodal data, obtaining visual, speech, operational, and text features. The features extracted from each modality are concatenated or summed to form joint features. Dynamic programming is used to calculate the cumulative distance matrix between features from each pair of modalities, finding the optimal path that minimizes the total distance. For example, a cumulative distance matrix can be constructed to calculate the distance between features of each modality. Dynamic programming is applied to find the path that minimizes the total distance as the optimal alignment path. The time axis of each modality's data is adjusted according to the optimal alignment path to achieve temporal alignment.

[0071] After achieving temporal alignment of multimodal data, a transformation matrix from screen pixel coordinates to logical coordinates can be established for the operation trajectory stream and conference video stream based on spatial alignment rules. The screen coordinate system mapping is completed through the transformation matrix to achieve spatial alignment, thereby transforming them to the same space.

[0072] Step S120: Based on the multimodal understanding network, the spatiotemporally aligned multimodal data is parsed, and combined with the role labels of the speakers in the meeting scenario, multimodal semantic combination information is output.

[0073] In this embodiment of the disclosure, a multimodal understanding network can be used to perform multimodal understanding on the multimodal data output by the temporal alignment module, thereby achieving feature extraction and cross-modal fusion of the multimodal data. The multimodal data includes conference video, operation trajectories, conference audio, and interactive text. For example, the multimodal understanding network can parse the spatiotemporally aligned multimodal data and, combined with the speaker's role tags in the conference scene, output multimodal semantic combination information.

[0074] In some embodiments, spatiotemporally aligned multimodal data can first be parsed using a multimodal understanding network to extract key content features associated with the meeting scenario. The multimodal understanding network may include a text extraction module, an image processing module, an adaptive collaborative attention network, and a content annotation module. The text extraction module extracts text features, and the image processing module extracts image features. The adaptive collaborative attention network extracts useful target image features and target text features, thereby determining the key content features of the meeting scenario based on these features. The content annotation module decomposes the target multimodal data corresponding to the key content features and determines the candidate role labels to which the target multimodal data belongs, thus annotating the target multimodal data according to role labels. Key content features represent the key features of the meeting scenario. For example, the meeting video and operation trajectory at the target time can be fused to obtain an intermediate image, and the meeting audio and interactive text at the target time can be fused to obtain intermediate text. The target time can be any time within the meeting scenario. However, the meeting video, operation trajectory, meeting audio, and interactive text belong to the same time. Furthermore, feature extraction can be performed on the intermediate text to determine text features, and feature extraction can be performed on the intermediate image to determine image features. An adaptive collaborative attention network is used to determine target image features related to the text features, and target text features are selected based on the target image features. The target image features and target text features are then fused according to a gating mechanism to determine key content features.

[0075] refer to Figure 3As shown, the conference video and operation trajectory can be fused together and input as an intermediate image to the image processing module. The image processing module can use a 16-layer VGGNet to extract image information and transform it into a vector of the same dimension as the text information through a fully connected layer, thereby obtaining image features. For the conference audio and interactive text at the target time, the conference audio can be converted into conference text, and the interactive text and conference text can be fused to obtain intermediate text. For the intermediate text, a convolutional neural network can be used to extract character-level word vectors of the intermediate text, and ordinary-level word vectors of the intermediate text can be extracted according to a word lookup table; the character-level word vectors and ordinary-level word vectors are connected at the same position to determine the feature-rich concatenated text features. Further, the concatenated text features are encoded through a bidirectional long short-term memory network to determine the text features. The text features can be a summary representation of words and their context at the same time. In order to capture past and future contextual information simultaneously, a bidirectional LSTM (Long Short-Term Memory) network is used. The bidirectional LSTM contains two independent LSTMs, which are used to capture past and future information respectively. One LSTM encodes from the beginning to the end of the sentence, and the other LSTM encodes from the end to the beginning. Therefore, at each location, two hidden layer representations can be obtained. These two hidden layer representations are then connected to form the final output, which is the text feature.

[0076] Continue to refer to Figure 3 As shown, after obtaining text features and image features, an adaptive collaborative attention network can be used to determine target image features related to the text features using the text features, and then select target text features using the target image features. For example, visual attention can be used to select useful image features as target image features using the text features output by the LSTM. Specifically, attention operations can be performed on the text features and image features at the current time step to obtain the target image features. Further, text attention can be used to select useful text features as target text features using the representation of image information, i.e., the selected target image features. Specifically, attention operations can be performed on the text features and target image features to obtain the target text features. Here, the target image features can be used to represent image regions strongly related to the word at the current time step, and the target text features can be used to represent words in the text strongly related to the word at the current time step.

[0077] Based on this, target image features and target text features can be fused using a gating mechanism to determine key content features. For example, a filter gate can be used in the gating mechanism to selectively fuse target text features and target image features. The filter gate is a scalar in the range [0,1]. The filter gate is used to determine whether to include target image features in the final fused features; that is, the value of the filter gate determines whether to fuse target image features with target text features. When target image features help improve word performance, the filter gate value is 1, and target text features can be fused with target image features; otherwise, when target image features do not help improve word performance, the filter gate value is 0, and target image features do not need to be fused with target text features. When fusing target image features and target text features, the target text features and target image features can be concatenated to obtain key content features. Based on this, key content features include features from multiple dimensions such as text features and image features, thus making the feature information richer and more comprehensive.

[0078] After obtaining the key content features of the multimodal data, the target multimodal data corresponding to the key content features can be labeled based on conditional random fields and the role labels of the speakers in the meeting scenario, generating multimodal semantic combination information. This multimodal semantic combination information represents the semantic combination information corresponding to each role label. The target multimodal data corresponding to the key content features can be at least a portion of the multimodal data, and may include any one or more combinations of meeting video, operation trajectories, meeting audio, and interactive text. There can be multiple speakers in the meeting scenario; role labels refer to the labels for each speaker, and user labels can be used to represent user identity, thereby distinguishing different speakers. The semantic combination information can be a combination of text and images.

[0079] When labeling target multimodal data corresponding to key content features based on the speaker's role labels in a meeting scenario, a pre-trained CRF (Conditional Random Fields) model is used. The CRF model predicts candidate role labels for the target multimodal data at each time step based on the multimodal feature vectors. For example, the host's speech is labeled "Host," and Guest 1's speech is labeled "Guest 1." The target multimodal data is then decomposed into sub-data corresponding to each candidate role label. Furthermore, sub-data with the same candidate role label can be merged, and the sub-data can be mapped to the same role label, thus labeling the target multimodal data according to each role label. In addition, timestamps can be added to the sub-data of each role label to record the start and end times of the speech, facilitating subsequent review and retrieval.

[0080] In this embodiment of the disclosure, the output multimodal semantic combination information can be the image and text combination information corresponding to each role tag, which may include one or more of the following: meeting video, operation trajectory, meeting audio, and interactive text for each role tag.

[0081] In step S230, a decision tree corresponding to the meeting scenario is constructed based on the multimodal semantic combination information, and meeting minutes corresponding to the meeting scenario are generated based on the decision tree and the knowledge graph associated with the meeting scenario.

[0082] In this embodiment of the disclosure, a decision tree can be constructed for the meeting scenario. First, the nodes of the decision tree can be determined, including root nodes, branch nodes, and leaf nodes. Specifically, the meeting topic can be used as the root node, role tags as branch nodes, and the breakdown data of each role tag as leaf nodes. In other embodiments, keywords can also be used as branch nodes, and each role tag and its breakdown data as leaf nodes.

[0083] After determining the nodes of the decision tree, the decision tree structure is used to construct the root node meeting topic, branch node expansion, and leaf node automatic assignment of responsible persons for the knowledge graph of this field. Based on semantic unit parsing, the text information is decomposed, completed, and standardized and compiled.

[0084] For example, based on the knowledge graph of the domain to which the meeting scenario belongs and the contextual information of the decomposed data, information completion can be performed on the decomposed data to form structured text blocks. These structured text blocks are then mapped to nodes in a decision tree to construct the decision tree, ensuring that each structured text block corresponds to the meeting topic and role label. For example, the decomposed data for each role label may include meeting video, operation trajectory, meeting audio, and interactive text. The meeting video, operation trajectory, and interactive text can be associated with the meeting audio for that role label. Furthermore, the meeting video, operation trajectory, and interactive text can be described in text form to generate descriptive text. This descriptive text is then combined with the meeting audio, using the meeting video, operation trajectory, and interactive text as supplementary information to the meeting audio. For example, if role label A corresponds to a meeting video and interactive text has been input, and an operation trajectory exists, the meeting audio, the target and type of the operation trajectory, the input interactive text, and the content displayed in the meeting video can be connected to obtain a structured text block.

[0085] Based on the nodes of the decision tree and the structured knowledge related to the meeting topic in the knowledge graph, a framework for the meeting minutes is established. For example, a meeting minutes template can be generated based on the nodes of the decision tree and the structured knowledge related to the meeting topic in the knowledge graph, thus serving as the framework for the meeting minutes. Furthermore, the structured text blocks corresponding to each node of the decision tree can be filled into the framework of the meeting minutes to generate structured information. Specifically, for a certain keyword, if there are identical structured text blocks corresponding to different role labels, they can be directly merged and filled into the corresponding positions; if there are different structured text blocks corresponding to different role labels, multiple different structured texts can be displayed, and a voting algorithm can be used to determine the final structured text for filling. Based on this, the structured information can be converted into text to generate meeting minutes in text form.

[0086] After generating meeting minutes, a confidence score can be calculated to determine whether the minutes are canonical text. Further, based on whether the minutes are canonical text, the meeting minutes can be directly output or the network parameters of the multimodal understanding network can be updated to obtain the meeting minutes corresponding to the meeting scenario. First, a rule base can be established from multiple dimensions, such as index information, field similarity, and text canonicity. The meeting minutes can then be matched against the validation rule base; a successful match indicates a reference confidence score of 1 for that dimension. Based on this, the weighted sum of multiple reference confidence scores from different dimensions yields the overall confidence score of the meeting minutes. Index information can be pre-defined keywords or pre-defined rules that the minutes must meet; field similarity is the similarity between fields in the meeting minutes and fields stored in the rule base; and text canonicity is used to assess text coherence and accuracy. The weights of different dimensions are determined based on their importance to the meeting minutes; higher importance results in higher weights.

[0087] If the calculated confidence score is greater than or equal to the confidence score threshold, the meeting minutes are considered to meet the normalization criteria and are considered normalized text, thus they can be directly output. If the calculated confidence score is less than the confidence score threshold, the meeting minutes are considered to not meet the normalization criteria and are considered non-normalized text. To improve accuracy, the process can return to the multimodal understanding network, where the updated multimodal fusion network continues to extract key content features from the multimodal data. Based on the speaker's role labels in the meeting scenario, the target multimodal data corresponding to the key content features is labeled to output multimodal semantic combination information. Then, the meeting minutes are regenerated based on the decision tree and knowledge graph until a complete and normalized meeting minutes are output.

[0088] By capturing user action trajectories (mouse movement, document editing), meeting screen recordings, meeting audio, and interactive text in real time, and utilizing a spatiotemporal alignment engine to achieve multi-source data synchronization, combined with cross-modal attention mechanisms and knowledge graph-enhanced semantic parsing technology, structured meeting minutes are automatically generated. Through causal correlation analysis between user actions and audio content, meeting topics are accurately identified, and a dynamic confidence verification mechanism is introduced to ensure content accuracy. This reduces manual recording costs and significantly improves the authenticity and readability of meeting information.

[0089] The technical solutions in this disclosure effectively reduce the resource consumption of manual modification, improve the execution efficiency of automatic minutes generation, and reduce the cost of generating meeting minutes. Simultaneously, the integration of a spatiotemporal alignment engine, a multimodal understanding network, and a knowledge graph improves the accuracy of the meeting minutes. Furthermore, by fine-tuning the parameters of the multimodal understanding network, standardized text can be generated, enabling real-time model updates and increasing reliability.

[0090] Figure 4 The diagram illustrates the overall flowchart for generating meeting minutes. (See reference) Figure 4 As shown, the system for generating meeting minutes mainly includes: a multi-source data acquisition module, a data preprocessing module, a spatiotemporal alignment engine module, a multimodal understanding network, a structured generation module, and a dynamic verification module. Among them:

[0091] Multi-source data acquisition module: Acquires interactive data streams from participants in the conference system, including conference recordings, screen recordings, interactive text, and operation trajectories. Participants can be speakers in the conference scenario.

[0092] Data preprocessing module: Performs real-time and integrity checks on the collected data, removes irrelevant information and duplicate data, and performs noise reduction to obtain multimodal data.

[0093] Spatiotemporal alignment engine module: Constructs a convolutional neural network to perform temporal behavior analysis on the processed multimodal data to achieve timestamp synchronization and spatial operation layer association, ensuring the formation of semantically coherent contextual multi-source data packets under a unified spatiotemporal coordinate system.

[0094] Multimodal Understanding Network: Based on the Transformer architecture, a hierarchical intramodal and cross-modal attention mechanism is used to achieve cross-modal semantic fusion by weighting the correlation matrix units of keywords, description images and operation trajectories in speech.

[0095] The structured generation module stores professional knowledge from various fields to form a knowledge graph, performs semantic parsing of meeting elements through a decision tree approach, and outputs a structured context summary based on the optimized decision nodes.

[0096] Dynamic verification module: Based on multi-round consistency verification rules, it determines the credibility of structured meeting minutes and whether they conform to the standardized output standards.

[0097] Based on this, the entire process of generating meeting minutes mainly includes the following steps:

[0098] Step S401: The multi-source data acquisition module acquires multimodal data.

[0099] Step S402: The multimodal data is temporally and spatially aligned using the spatiotemporal alignment engine module to obtain spatiotemporally aligned multimodal data.

[0100] Step S403: Input the spatiotemporally aligned multimodal data into the multimodal understanding network and output multimodal semantic combination information. Multimodal semantic combination information refers to the image-text combination information corresponding to each speaker's role label.

[0101] Step S404: In the structured generation module, the multimodal semantic combination information is structured based on the decision tree and the knowledge graph associated with the meeting scenario to generate meeting minutes.

[0102] Step S405: Input the meeting minutes into the dynamic verification module. If the meeting minutes are standard text, output the meeting minutes directly. If the meeting minutes are non-standard text, update the parameters of the multimodal understanding network until a complete and standardized meeting minutes is output.

[0103] Figure 5 The diagram illustrates the specific flowchart for generating meeting minutes. (See reference) Figure 5 As shown, the system can collect meeting recordings, interactive texts, screen recordings, and operation trajectories from participants, and preprocess these data to obtain multimodal data. This multimodal data is then input into a spatiotemporal alignment engine module to obtain time-aligned multimodal data. The time-aligned multimodal data is then input into a multimodal understanding network, which outputs multimodal semantic combination information. This multimodal semantic combination information is then structured using knowledge graphs and decision trees, and the output results are dynamically validated. During the dynamic validation process, the confidence level is fed back to the multimodal understanding network to update its parameters, ultimately outputting meeting minutes.

[0104] In this embodiment, multimodal data from multiple dimensions, including voice, screen operation trajectories (mouse / keyboard), meeting recordings, and interactive text, are integrated to cover all aspects of the meeting, thereby improving the comprehensiveness and reliability of the generated meeting minutes. Integrating user operation trajectories replaces manual screenshot annotation, minimizing human intervention and exponentially reducing time costs. Knowledge graph construction improves intelligent summary retrieval efficiency and enhances the quality of the meeting minutes.

[0105] This disclosure also provides a meeting minutes generation apparatus. (See reference) Figure 6 As shown, the meeting minutes generation device 600 mainly includes the following modules:

[0106] The multimodal data acquisition module 601 is used to acquire multimodal data corresponding to the meeting scenario, and to perform spatiotemporal alignment on the multimodal data to determine the spatiotemporally aligned multimodal data; the multimodal data includes meeting video, operation trajectory, meeting audio, and interactive text;

[0107] The parsing module 602 is used to parse spatiotemporally aligned multimodal data based on a multimodal understanding network, and output multimodal semantic combination information by combining the role labels of the speakers in the meeting scenario.

[0108] The generation module 603 is used to construct a decision tree corresponding to the meeting scenario based on the multimodal semantic combination information, and generate meeting minutes corresponding to the meeting scenario based on the decision tree and the knowledge graph associated with the meeting scenario.

[0109] In one exemplary embodiment of this disclosure, the multimodal understanding network parses spatiotemporally aligned multimodal data and, in conjunction with the speaker's role labels in the meeting scenario, outputs multimodal semantic combination information, including:

[0110] The spatiotemporally aligned multimodal data is parsed to extract key content features associated with the meeting scenario;

[0111] The target multimodal data corresponding to the key content features are labeled according to the role tags of the speakers in the meeting scenario, and the multimodal semantic combination information is generated; the multimodal semantic combination information is used to represent the target multimodal data corresponding to each role tag.

[0112] In one exemplary embodiment of this disclosure, the feature vectors of the spatiotemporally aligned multimodal data are parsed to extract key content features associated with the meeting scenario, including:

[0113] The intermediate image is obtained by fusing the meeting video and operation trajectory at the target time, and the intermediate text is obtained by fusing the meeting audio and interactive text at the target time.

[0114] Determine the image features of the intermediate image and the text features of the intermediate text;

[0115] An adaptive collaborative attention network is used to determine target image features related to the text features, and target text features are selected based on the target image features.

[0116] The key content features are determined by fusing the target image features and the target text features according to the gating mechanism.

[0117] In one exemplary embodiment of this disclosure, determining the text features of the intermediate text includes:

[0118] Extract the character-level word vectors of the intermediate text, and extract the ordinary-level word vectors of the intermediate text;

[0119] The character-level word vectors and the ordinary-level word vectors are connected at the same position to determine the concatenated text features;

[0120] The spliced ​​text features are encoded using a bidirectional long short-term memory network to determine the text features.

[0121] In one exemplary embodiment of this disclosure, the step of annotating the target multimodal data corresponding to the key content features based on the speaker's role label in the meeting scenario, and generating the multimodal semantic combination information, includes:

[0122] Determine the candidate role tags corresponding to the key content features;

[0123] Based on the candidate role tags, the target multimodal data corresponding to the key content features is decomposed to obtain decomposed data;

[0124] Identify the role tags that are the same as the candidate role tags, and map the decomposition data corresponding to the candidate role tags to the same role tags.

[0125] In one exemplary embodiment of this disclosure, the step of constructing a decision tree corresponding to the meeting scenario based on the multimodal semantic combination information, and generating meeting minutes corresponding to the meeting scenario based on the decision tree and the knowledge graph associated with the meeting scenario, includes:

[0126] The nodes of the decision tree are determined, and based on the knowledge graph and context information, the decomposed data is supplemented to form structured text blocks. The structured text blocks are then mapped to the nodes of the decision tree to construct the decision tree.

[0127] Based on the nodes of the decision tree and the structured knowledge in the knowledge graph, the framework of the meeting minutes is established.

[0128] The structured text blocks corresponding to the nodes of the decision tree are filled into the frame to generate structured information;

[0129] The structured information is converted into text to generate the meeting minutes.

[0130] In one exemplary embodiment of this disclosure, the method further includes:

[0131] A confidence level is calculated on the meeting minutes, and the confidence level is used to determine whether the meeting minutes are a standard text.

[0132] If the meeting minutes are in a standard text format, output the meeting minutes directly.

[0133] If the meeting minutes are non-standard text, the network parameters of the multimodal understanding network are updated until a standard text is obtained.

[0134] It should be noted that the specific details of each module in the above-mentioned meeting minutes generation device have been described in detail in the corresponding meeting minutes generation method, so they will not be repeated here.

[0135] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0136] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0137] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0138] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0139] The following reference Figure 7 To describe an electronic device 700 according to such an embodiment of the present disclosure. Figure 7 The electronic device 700 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0140] like Figure 7 As shown, the electronic device 700 is manifested in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, a bus 730 connecting different system components (including storage unit 720 and processing unit 710), and a display unit 740.

[0141] The storage unit stores program code that can be executed by the processing unit 710, causing the processing unit 710 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 710 can perform actions such as... Figure 2 The steps are shown in the figure.

[0142] Storage unit 720 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 7201 and / or cache memory 7202, and may further include a read-only memory (ROM) 7203.

[0143] The storage unit 720 may also include a program / utility 7204 having a set (at least one) program module 7205, such program module 7205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0144] Bus 730 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0145] Electronic device 700 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 760. As shown, network adapter 760 communicates with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0146] It should be noted that some embodiments of this disclosure also provide a computer program product, which includes a computer program that implements the above-described method when executed by a processor.

[0147] In one embodiment, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. For example, the computer program product can be implemented as a non-volatile storage medium storing the computer program, such as read-only memory, NAND flash memory, etc. In one embodiment, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, installation package, or other digital file storing the computer program.

[0148] Computer program code can be written in one or more programming languages. Examples of programming languages ​​include C, Java, and C++. Program code can execute entirely on the user's computing device, partially on the user's computing device, or as a standalone software package. It can also execute partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).

[0149] Computer programs can be carried or transmitted via signals such as electrical, magnetic, optical, electromagnetic, and infrared rays. Electronic devices can convert signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code is used to cause the electronic device to execute (more specifically, to be executed by the processor of the electronic device) the method steps of various exemplary embodiments of this disclosure.

[0150] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0151] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0152] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0153] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0154] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method of generating a meeting summary, characterized by, The method comprises the following steps: acquiring multi-modal data corresponding to a conference scene, and performing spatio-temporal alignment on the multi-modal data to determine spatio-temporally aligned multi-modal data; the multi-modal data comprises conference video, operation track, conference speech, and interactive text; analyzing the spatio-temporally aligned multi-modal data to extract key content features associated with the conference scene; annotating target multi-modal data corresponding to the key content features according to a role label of a speaker of the conference scene to generate multi-modal semantic combination information; the multi-modal semantic combination information is used to represent target multi-modal data corresponding to each role label; constructing a decision tree corresponding to the conference scene based on the multi-modal semantic combination information, and generating a conference minutes corresponding to the conference scene based on the decision tree and a knowledge graph associated with the conference scene; wherein analyzing the spatio-temporally aligned multi-modal data to extract key content features associated with the conference scene comprises: fusing conference video and operation track at a target time to obtain intermediate images, and fusing conference speech and interactive text at the target time to obtain intermediate text; determining image features of the intermediate images, and determining text features of the intermediate text; the text features are used to select useful image features as target image features; using the text features to determine target image features related to the text features by means of an adaptive collaborative attention network, and selecting target text features by means of the target image features; fusing the target image features and the target text features according to a gating mechanism to determine the key content features.

2. The meeting summary generation method of claim 1, wherein, The method further comprises the following steps: extracting character-level word vectors of the intermediate text, and extracting normal-level word vectors of the intermediate text; connecting the character-level word vectors and the normal-level word vectors at the same positions to determine spliced text features; encoding the spliced text features by means of a bidirectional long short-term memory network to determine the text features.

3. The method of claim 1, wherein, The method further comprises the following steps: determining candidate role labels corresponding to the key content features; disassembling target multi-modal data corresponding to the key content features according to the candidate role labels to obtain disassembled data; determining a role label identical to the candidate role labels, and mapping disassembled data corresponding to the candidate role labels to the identical role label.

4. The method of claim 1, wherein, The method further comprises the following steps: determining nodes of the decision tree, performing information completion on the disassembled data based on the knowledge graph and context information to form structured text blocks, and mapping the structured text blocks to the nodes of the decision tree to construct the decision tree; establishing a framework of the conference minutes according to the nodes of the decision tree and structured knowledge in the knowledge graph. filling a structured text block corresponding to a node of the decision tree to the framework to generate structured information; performing text conversion on the structured information to generate the meeting minutes.

5. The method of claim 1, wherein, The method further comprises: performing confidence calculation on the meeting minutes, and determining whether the meeting minutes are standard texts according to the calculated confidence; if the meeting minutes are standard texts, directly outputting the meeting minutes; if the meeting minutes are non-standard texts, updating network parameters of the multi-modal understanding network until standard texts are obtained.

6. A meeting summary generation apparatus characterized by comprising: comprises: a multi-modal data acquisition module, configured to acquire multi-modal data corresponding to a conference scene, perform spatio-temporal alignment on the multi-modal data, and determine spatio-temporally aligned multi-modal data; the multi-modal data comprises conference video, operation trajectory, conference speech, and interactive text; an analysis module, configured to analyze the spatio-temporally aligned multi-modal data, and extract key content features associated with the conference scene; annotating target multi-modal data corresponding to the key content features according to role labels of speakers of the conference scene to generate multi-modal semantic combination information; the multi-modal semantic combination information is used to represent target multi-modal data corresponding to each role label; a generation module, configured to construct a decision tree corresponding to the conference scene based on the multi-modal semantic combination information, and generate meeting minutes corresponding to the conference scene based on the decision tree and a knowledge graph associated with the conference scene; wherein analyzing the spatio-temporally aligned multi-modal data and extracting key content features associated with the conference scene comprises: fusing conference video and operation trajectory of a target time to obtain intermediate images, and fusing conference speech and interactive text of the target time to obtain intermediate text; determining image features of the intermediate images, and determining text features of the intermediate text; the text features are used to select useful image features as target image features; using the text features to determine target image features related to the text features through an adaptive collaborative attention network, and selecting target text features through the target image features; fusing the target image features and the target text features according to a gating mechanism to determine the key content features.

7. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the meeting minutes generation method of any one of claims 1-5.

8. An electronic device, comprising: comprises: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to implement the meeting minutes generation method of any one of claims 1-5 by executing the executable instructions.

Images