Conference summary generation method and system based on multi-modal data fusion

Abstract

The invention provides a conference summary generation method and system based on multi-modal data fusion. The method comprises the following steps: receiving audio and video data of a conference; extracting a content image frame from the video data, and identifying characters in the content image frame to obtain an image text; face recognition is carried out on the video data, and identities and appearing time periods of all participants are obtained; performing voice recognition and speaker separation on the audio data, extracting a speaking time period and speaking content of each speaker, and calculating a speaking weight of each speaker; matching the speaking time periods of the speakers with the appearing time periods of the participants, and determining participants identities corresponding to the speakers; according to the speaking weight and the speaking content of each speaker, identifying the conference role of each participant; and generating a conference summary according to the image text, the speaking content and the participant labels. According to the method, conference key information is extracted from multiple dimensions, and more comprehensive, structured and semantic-rich conference summary generation is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video communication, in particular to a conference summary generation method and system based on multi-modal data fusion. BACKGROUND

[0002] With the development of speech recognition and natural language processing technologies, automatic generation of conference summaries has gradually become an important function in intelligent office systems. Currently, conference summary systems on the market mainly process audio data, usually converting conference speech into text through speech recognition technology, and then combining keyword extraction, syntax analysis or structural templates to summarize the converted content.

[0003] In some more advanced systems, large language model (LLM) technology is also introduced to generate summaries of conference texts by writing prompts. This method can improve the quality of semantic generation, but its effectiveness still depends heavily on the accuracy of audio transcription and the design capabilities of prompts, and it still mainly relies on "pure text processing", failing to effectively utilize other data content in conference audio and video besides audio data, resulting in summaries lacking the ability to link different data sources and enhance semantics.

[0004] At the same time, existing technologies do not further bind speakers with specific identity information, and it is also difficult to automatically identify their roles in the conference. Due to the lack of systematic recognition and fusion mechanisms for information such as the identities of conference participants, the structure of conference content, and the key points of speeches, existing methods still have problems such as one-sided summary content, lack of key semantics, and difficulty in reflecting conference responsibility attribution in practical applications.

[0005] Therefore, it is necessary to provide a technical solution for generating conference summaries to solve the problem of obvious deficiencies in the semantic integrity, information relevance and structured expression of existing conference summary generation methods. SUMMARY

[0006] This invention provides a meeting summary generation method and system based on multimodal data fusion (multimodal data fusion refers to the integration, processing, and analysis of different data sources). It aims to address problems in existing meeting summary generation processes such as reliance on single text, lack of multi-source information integration, inability to accurately identify speakers and their roles, and insufficient semantic extraction. By comprehensively receiving audio and video data from the meeting and integrating multiple technologies such as speech recognition, speaker separation, face recognition, and image text recognition, it extracts key meeting information from multiple dimensions. This determines the speaker's participant identity (e.g., name), meeting role (e.g., moderator, main speaker, general participant), and organizational role (e.g., department affiliation, position), improving the structure and logic of the meeting summary. Furthermore, it fully utilizes image and text content extracted from the video (e.g., meeting PPT, titles, keywords in screen sharing) to enhance semantic context and help confirm the meeting's focus.

[0007] This invention breaks through the technical limitations of traditional meeting summary generation methods that rely on "pure text processing." It not only improves the semantic integrity, structure, and criticality of the summary content, but also enables the traceability of meeting responsibility attribution. It is suitable for intelligent minutes generation needs in various complex meeting scenarios.

[0008] In a first aspect, the present invention provides a method for generating conference summaries based on multimodal data fusion, characterized in that the method includes:

[0009] Receive audio and video data from the meeting;

[0010] Content image frames are extracted from the video data, and the text in the content image frames is recognized to obtain image text;

[0011] Facial recognition is performed on the video data to obtain the identity of each participant and the time period of their appearance in the meeting;

[0012] The audio data is subjected to speech recognition and speaker separation to extract the speaking time and content of each speaker and to calculate their speaking weight.

[0013] The speaker's speaking time period is matched with the time period of the participants' appearance, and the participant identity corresponding to each speaker is determined based on the identity of each participant.

[0014] Based on the attendee identity corresponding to each speaker, and according to the speaking weight and speaking content of each speaker, the meeting role of each attendee is identified, and the attendee identity and meeting role constitute the attendee label;

[0015] A meeting summary is generated based on the image text, the speech content, and the participants' annotations.

[0016] In a second aspect, the present application further provides a conference summary generation system based on multi-modal data fusion, characterized in that the system comprises:

[0017] an apparatus for receiving audio data and video data of a conference;

[0018] an image extraction apparatus for extracting content image frames from the video data and recognizing text in the content image frames to obtain image text;

[0019] an identity recognition apparatus for performing face recognition on the video data to obtain identities of each participant and appearance time periods of the participants in the conference;

[0020] a content extraction apparatus for performing speech recognition and speaker separation on the audio data, extracting speech time periods and speech content of each speaker, and calculating speech weights of the speakers;

[0021] an apparatus for matching the speech time periods of the speakers with the appearance time periods of the participants and determining identities of the participants corresponding to each speaker based on identities of the participants;

[0022] an apparatus for identifying conference roles of each participant based on the speech weights and the speech content of each speaker according to the identities of the participants corresponding to each speaker, wherein the identities and the conference roles of the participants constitute participant annotations;

[0023] a generation apparatus for generating a conference summary according to the image text, the speech content, and the participant annotations.

[0024] The conference summary generation method and system based on multi-modal data fusion provided by the application have the advantages that: first, the face recognition technology based on deep learning is combined with automatic extraction of preset face databases or identity information in videos to improve the accuracy of identity recognition of conference participants, ensure efficient matching of speakers and identities, and solve the problem that it is difficult to accurately distinguish speakers by relying on traditional voice recognition; second, by fusing audio speech content, video image text, and identity and role information of conference participants, semantic enhancement and multi-dimensional information mutual verification of conference content are realized, the deficiencies of single voice recognition text are made up, the semantic background and accuracy of the summary are enhanced, the credibility and practicality of the summary are improved, and the semantic integrity and highlight of the summary are significantly improved; third, conference roles such as the host, the main speaker, and ordinary participants are recognized and labeled, department affiliation and duty information in the organizational structure are combined, the structured expression of conference content is enhanced, the responsibility affiliation and speech importance are helped to be determined, the speech content is reasonably classified and highlighted, the structure of the generated conference summary is more clear, the content is more rich, and the semantic integrity and expression effect are improved; fourth, the application supports processing of real-time conference data and offline recorded data, has wide application range and high flexibility, and can meet the conference summary generation requirements in different scenarios; fifth, the application supports displaying the conference summary on a graphical interface, users can search, modify, export, and share, and the modification of the speech content can trigger automatic updating, and dynamic maintenance and personalized customization of the summary are realized. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only some embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor under the premise of these drawings.

[0026] FIG. 1 It is a conference summary generation method flowchart based on multi-modal data fusion provided by the embodiment of the application.

[0027] FIG. 2 It is a conference summary generation system block diagram based on multi-modal data fusion provided by the embodiment of the application. DETAILED DESCRIPTION

[0028] The technical solutions of the application will be further described in detail below by means of the drawings and embodiments. SUMMARY

[0030] As described above, the application provides a conference summary generation method and system based on multi-modal data fusion, which extracts conference key information from multiple dimensions to realize more comprehensive, structured, and semantically rich conference summary generation.

[0031] Exemplary method

[0032] FIG. 1 FIG. 1 is a flowchart of a conference summary generation method based on multi-modal data fusion according to an embodiment of the present application. The embodiment includes the following steps:

[0033] S101: receiving audio data and video data of a conference.

[0034] The audio data and the video data include real-time data and / or offline data.

[0035] The real-time data refers to video and audio data collected and processed synchronously during the conference, which is used for real-time transcription and summary generation while listening and watching, and is convenient for participants to follow up the conference content in real time.

[0036] The offline data refers to data processed by importing recorded audio and video files after the conference, which does not require real-time performance and allows more complex algorithms to be used for in-depth analysis and processing of recorded conference materials to meet the needs of later archiving and review. For example, for conference audio and video files recorded before the popularization of intelligent summary technology, the present application can import these offline files into the system for processing to realize intelligent analysis and summary generation of historical conference content.

[0037] S102: extracting content image frames from the video data and recognizing the text in the content image frames to obtain image text.

[0038] The purpose of extracting the content image frames is to identify the text information in the PPT and other presentation content.

[0039] Based on the text density, clarity and / or change rate of the image content in the video data, the content image frames are extracted from the video data at a fixed frame rate. Specifically, content image frames with high text density, high clarity and / or high change rate are extracted. For example, image frames with high text density often contain more key information, such as topic titles, important data, chart descriptions, etc. in the conference, which are crucial for generating accurate and valuable conference summaries. Then, the text in the content image frames is recognized by a text recognition technology to obtain image text. The text recognition technology can be an optical character recognition technology.

[0040] For example, during the conference, when there are more written content on the PPT slides or whiteboard, the image frames containing text can be automatically extracted from the video, and the text in the image frames is recognized to obtain image text.

[0041] By extracting content image frames from the conference video data, the text information displayed during the conference, such as slide content, whiteboard writing, and screen sharing content, can be effectively captured. These text information usually carries important topics, data support, and discussion highlights of the conference. Extracting and identifying these text content can provide rich semantic background and context support for the conference summary, enhancing the accuracy and integrity of the summary. Meanwhile, combining with the semantic comparison of the speech content can help to confirm the conference highlights and improve the coverage of key topics and conclusions in the summary, ultimately generating a high-quality conference summary with more structured and information-related characteristics.

[0042] S103: Perform face recognition on the video data to obtain the identities of the conference participants and their appearance time periods in the conference.

[0043] The invention confirms the identities of the conference participants in two ways depending on whether there is a preset face database. The preset face database refers to a set of conference participant face image feature data with identity information established and stored before the conference starts, which is used for subsequent face recognition and identity matching during the conference.

[0044] As an optional embodiment, when there is a preset face database, face detection is performed on the complete image frames at a fixed frame rate. The frame rate can be adaptively set according to the image stability and personnel frequency of the conference scene to balance processing efficiency and recognition accuracy. Based on a deep learning model, the feature vector of the detected face image is extracted, which can uniquely represent the facial features of an individual. The extracted face feature vector is compared with the feature vectors in the preset face database to calculate the similarity score. If the matching score exceeds the preset threshold, it is determined that it is a certain conference participant known in the face database, achieving identity recognition of the conference participant. For faces that do not match successfully, they can be considered as unregistered identities or subsequent processing. By combining the time axis of the complete image frames of the video, the repeated appearance of the same face feature vector in subsequent frames is tracked and recognized to determine the appearance time period of the conference participant in the entire conference. The behavior trajectory of the conference participant on the time axis is constructed through continuous matching of face features.

[0045] If there is no preset face database, the initial identity mapping needs to be established by extracting the text information (nicknames, IDs, etc.) from the identity image frames appearing in the conference video. The preset face database is mainly used for scenarios where the list of known personnel before the conference, such as internal company meetings, formal reviews, etc.

[0046] As another optional embodiment, when there is no preset face database, the method before step S103 further comprises extracting an identity image frame (referring to an image containing user identity information text in the picture, such as a user name appearing in the corner of a window in a video conference) from the video data, and identifying the text content in the identity image frame to extract user identity information, which includes a user name, a nickname and / or an ID. Step S103 specifically comprises performing face detection on the identity image frame at a fixed frame rate; extracting a feature vector of the detected face image based on a deep learning model; associating the face feature vector with the user identity information extracted from the image in the frame to establish a mapping relationship between the participant identity and the face feature, and determine the participant identity; and tracking and identifying repeated appearance of the same face feature vector in subsequent frames in combination with the complete image frame timeline of the video to determine the appearance period of the participant in the conference.

[0047] In summary, the present application supports an accurate recognition mode based on a preset face database, and also supports a flexible recognition mode of automatically establishing identity binding through identity image frame and text recognition under the condition of no preset face database. The former is suitable for a closed scene with known participants and pre-collected information, and has high recognition accuracy and processing efficiency; the latter is suitable for an open, temporary or remote conference scene, and has stronger automation and real-time adaptability. The two modes can be dynamically switched according to the conference scene to ensure efficient and accurate participant recognition under different conditions, and provide a solid foundation for subsequent conference role analysis and summary generation.

[0048] S104: performing speech recognition and speaker separation on the audio data, extracting the speaking period and content of each speaker, and calculating the speaking weight thereof.

[0049] Specifically, first, the original audio signal in the audio data is processed for noise reduction, silence detection, signal enhancement, etc. to improve the accuracy of speech recognition and speaker separation; the audio data is divided into multiple sound source segments according to the speaker by using a voiceprint recognition model; the time stamp corresponding to each sound source segment is generated to obtain the speaking period; each sound source segment is assigned a temporary identifier, such as "Speaker A", "Speaker B", etc.; speech recognition processing is performed on each sound source segment to extract the corresponding speech content; the correspondence between the speaking period and the speech content is established; the speech behavior characteristics of each speaker are counted, including the total speaking time, the speaking frequency, the average speaking time and / or the speech speed; the semantic features are analyzed based on the speech content using natural language processing methods, including the keyword density, the professional term usage frequency and / or the theme relevance; the speech behavior characteristics and the semantic features of each speaker are integrated to calculate the corresponding speech weight, which is used to represent the speaking influence or importance of the speaker in the meeting. The higher the weight, the higher the speaking right or information contribution in the meeting, for example, the main speaker.

[0050] S105: The speaking period of the speaker is matched with the appearance period of the meeting participant, and the identity of each meeting participant is determined based on the identity of each meeting participant.

[0051] To improve the matching accuracy, the consistency between the speaker and the face can be further verified by combining visual features such as mouth shape change, pronunciation moment facial action, image region sound source positioning, etc. in the matching process, thereby enhancing the robustness of identification.

[0052] Once the matching is successful, the system binds the speech segment with the identity (such as name, nickname, ID, etc.) of the corresponding meeting participant, realizes the mapping of speaker A to the real personnel identity, and eliminates anonymity.

[0053] S106: Based on the identity of each speaker corresponding to the meeting participant, the meeting role of each meeting participant is identified according to the speech weight and the speech content of each speaker, and the identity and the meeting role of the meeting participant constitute the meeting participant annotation.

[0054] The meeting role includes but is not limited to the host, the main speaker and / or the ordinary participant.

[0055] For example, in a technical solution review meeting, speaker A corresponds to the participant "Zhang San", whose speaking frequency is high, the total time length accounts for more than 30%, the speaking speed is stable, and the speaking content involves the overall conference structure arrangement, such as "please start introducing by someone" and "we will summarize the opinions again at the end", and Zhang San is determined to be the host of the meeting. Speaker B corresponds to the participant "Li Si", and the total time length accounts for more than 50%. His speech is concentrated in the core issue paragraph of the meeting, and the semantic content contains high-density technical keywords (such as "algorithm process", "index improvement", and "data set division"), and the use of technical terms is high and the duration is long, so it is determined that he is the main speaker. Speaker C corresponds to "Wang Wu", who only asks a short question "Can this index be tested by another algorithm?" in the middle segment, and the speaking time is short and there is no obvious theme extension, so it is identified as an ordinary participant.

[0056] Through conference role recognition, important technical elaboration of the main speaker Li Si can be preferentially retained in the generation of the summary, the conference structural language of the host Zhang San can be retained, and the short speeches of ordinary participants can be condensed or omitted as appropriate, so as to realize the generation of a more structured, clear-level, and highlighted conference summary content.

[0057] The participant labeling also includes the organizational role of the participant; the method further comprises:

[0058] Matching the participant identity with the pre-stored organizational structure information to label the organizational role of the participant.

[0059] The organizational structure information includes but is not limited to department division, position level (such as director, manager, engineer), and project group to which the personnel belong of an enterprise or institution.

[0060] The identity, conference role, and organizational role of the participant have core significance in the generation of the conference summary. The identity information identifies "who" the speaker is, providing basic attribution for the speaking content; the conference role reflects its responsibility positioning in the meeting, such as the host usually leading the topic advancement, the main speaker elaborating on the key content, and the ordinary participant mostly supplementing or responding, which helps to judge the importance and context relationship of the speech; the organizational role shows the hierarchical and functional background of the speaker in the enterprise or institution, such as department affiliation and position information, which can further infer the business relevance and representativeness of the viewpoint, and users can also choose to view only the speaking points of a certain department or a specific position, improving the usability and retrieval efficiency of the conference record. The combination of the three can achieve more accurate speech classification, speech weight evaluation, and key content extraction, making the generated conference summary more structured, clear, and semantically rich, and fitting the actual use scenario, thereby improving the usability and credibility of the automatic summary.

[0061] Compared with the traditional conference summary without marking the identities of the participants, the conference roles and the organizational roles, the conference summary generated by the present application has significant advantages. First, the clear marking of the identities of the participants enables the summary content to accurately correspond to specific persons, avoids ambiguous information, and improves the traceability and responsibility attribution clarity of the summary. Second, the identification of the conference roles helps to distinguish the functions and speaking focuses of different speakers, such as the guidance of the moderator, the report of the main speaker and the supplement of the ordinary participant, and enhances the structure and logic of the summary. Finally, in combination with the organizational roles, the speaking content can reflect the hierarchy and duty background within the organization, and enhance the semantic depth and business relevance of the summary. The fusion of these information makes the conference summary not only more comprehensive and accurate, but also more convenient for the management to quickly understand the conference key points, decision basis and subsequent follow-up, greatly improving the practical application value of the conference record.

[0062] S107: generating a conference summary according to the image text, the speaking content and the participant marking.

[0063] The speaking content is subjected to natural language processing to extract semantic information, including keywords, theme sentences and important expressions, so as to accurately capture the essential content of the conference discussion; based on semantic clustering and context analysis, the speaking content is classified to identify the key issues and discussion focuses appearing in the conference, ensuring that the summary covers the conference key points rather than scattered information; the participant marking, speaking frequency and / or keyword density are integrated to evaluate the summary weight of each segment of the speaking content, ensuring that important speaking is given priority; the image text and the speaking content are subjected to semantic comparison to identify consistent information, which assists in confirming the summary key points and improves the accuracy and integrity of the summary, for example, the PPT image extracted from the conference video identifies the content with the title of "Channel Comparison Analysis", which is semantically consistent with the speaking content of the speaker Wang San, further strengthening the summary priority of this part of the content; based on the pre-defined or automatically generated prompt words, the summary key points are integrated to extract summary candidate texts from the speaking content with higher weights, and the summary candidate texts are subjected to language reconstruction in combination with the semantic information, the key issues and the discussion focuses, to generate a structured conference summary text with clear logic, which includes but is not limited to the conference theme, the key personnel speaking points containing the participant marking, the discussion issues and the conference conclusion.

[0064] Compared with the prior art of generating a conference summary based only on speech transcription or based on text keyword extraction, the present application has significant advantages in multi-dimensional information fusion and deep semantic understanding, which are embodied in the following aspects:

[0065] Firstly, not only the speech information in the audio data is utilized, but also the image information in the video data (such as slides, whiteboards, screen sharing, etc.) is integrated. Through image text recognition and semantic comparison, the key content consistent with the image is captured, thereby making up for the defect of missing image core content in the traditional summary based on speech only.

[0066] Secondly, through face recognition, identity information extraction and organizational structure matching, the identity of the speaker, the conference role and the organizational role are comprehensively labeled. When generating the summary, it can clearly indicate "who said what", greatly enhancing the traceability of the meeting record and the clarity of task allocation.

[0067] Thirdly, by analyzing the speech behavior characteristics (such as duration, frequency, speech rate) and semantic characteristics (such as keyword density, term usage, theme correlation) of each speaker, the speech weight is calculated comprehensively, thereby prioritizing the extraction of core content of key speakers, significantly improving the representativeness and quality of the summary.

[0068] Finally, with the help of semantic clustering and context analysis, the themes of different stages in the meeting can be automatically summarized, and a structured summary with clear structure and logical reasoning can be output, including the meeting theme, main issues, key speech content and conclusions, which is much better than the traditional "running account" summary.

[0069] The conference summary can not only be output in text form, but also be displayed in an interactive way in the graphical user interface (GUI). Users can search, modify, export (export to common formats such as Word, PDF, TXT) or share the content of the conference summary according to their needs, improving the readability and operability of the summary content. Users can directly share the summary content through enterprise communication systems, email or cloud platforms to realize the quick distribution of meeting minutes and improve the efficiency of meeting closed-loop processing.

[0070] By searching, modifying, exporting and sharing the conference summary through the graphical user interface, users are liberated from traditional static documents, realizing the visualization, structuring and efficient use of meeting records, and significantly improving the intelligent level of conference management and information transmission.

[0071] The method further includes automatically re-executing the step of generating the conference summary when the user modifies the speech content, obtaining an updated conference summary, effectively improving the flexibility and accuracy of the conference summary, enabling users to make personalized adjustments according to actual needs, while automatically ensuring that content updates take effect in a timely manner, greatly optimizing the conference summary and information management process.

[0072] Exemplary system

[0073] Correspondingly, the embodiment of the present application also provides a conference summary generation system based on multi-modal data fusion. FIG. 2 is a conference summary generation system based on multi-modal data fusion provided by the embodiment of the present application, as shown in the figure, the system 100 provided by the embodiment of the present application comprises: FIG. 2 FIG. 2

[0074] means for receiving audio data and video data of a conference;

[0075] image extraction device 101 for extracting content image frames from the video data and recognizing the text in the content image frames to obtain image text;

[0076] identity recognition device 102 for performing face recognition on the video data to obtain the identity of each participant and the appearance time period of each participant in the conference;

[0077] content extraction device 103 for performing speech recognition and speaker separation on the audio data, extracting the speaking time period and speaking content of each speaker, and calculating the speaking weight of each speaker;

[0078] means for matching the speaking time period of the speaker with the appearance time period of the participant, and determining the identity of the corresponding participant of each speaker based on the identity of each participant;

[0079] means for identifying the conference role of each participant based on the speaking weight and speaking content of each speaker according to the identity of the corresponding participant of each speaker, wherein the identity and conference role of the participant constitute the participant annotation;

[0080] generation device 104 for generating a conference summary according to the image text, the speaking content and the participant annotation.

[0081] The participant annotation further comprises an organizational role of the participant; the system 100 further comprises:

[0082] means for matching the identity of the participant with pre-stored organizational structure information to annotate the organizational role of the participant.

[0083] The audio data and the video data comprise real-time data and / or offline data.

[0084] The image extraction device 101 further comprises:

[0085] a unit for extracting content image frames from the video data at a fixed frame rate based on the text density, definition and / or change rate of image content in the video data;

[0086] ​A unit for recognizing text in the content image frame using text recognition technology to obtain image text.

[0087] When a preset face database exists, the identity recognition device 102 further includes:

[0088] A unit for performing face detection on complete image frames at a fixed frame rate;

[0089] Units used to extract feature vectors from detected face images based on deep learning models;

[0090] A unit used to compare the extracted facial feature vectors with feature vectors in a preset facial database to determine the identity of the attendees;

[0091] This unit is used to combine the complete image frame timeline of the video to track and identify the repeated occurrence of the same facial feature vector in subsequent frames, and to determine the time period in which the participants appeared in the meeting.

[0092] When a preset face database does not exist, the identity recognition device 102 further includes:

[0093] Used to extract identity image frames from the video data and to identify the text content in the identity image frames in order to extract user identity information, wherein the user identity information includes units of username, nickname and / or ID;

[0094] A unit for performing face detection on the identity image frames at a fixed frame rate;

[0095] Units used to extract feature vectors from detected face images based on deep learning models;

[0096] This unit is used to associate facial feature vectors with user identity information extracted from the same frame image, establish a mapping relationship between the identity of the attendee and facial features, and determine the identity of the attendee.

[0097] This unit is used to combine the complete image frame timeline of the video to track and identify the repeated occurrence of the same facial feature vector in subsequent frames, and to determine the time period in which the participants appeared in the meeting.

[0098] The content extraction device 103 further includes:

[0099] A unit used to divide audio data into multiple sound source segments according to the speaker using a voiceprint recognition model;

[0100] Units used to generate timestamps corresponding to each sound source segment and obtain the speaking time period;

[0101] A unit used to assign temporary identifiers to each sound source segment;

[0102] a unit for performing speech recognition on each sound source segment to extract corresponding speech content;

[0103] a unit for establishing a correspondence between the speech time period and the speech content;

[0104] a unit for statistically analyzing speech behavior features of each speaker;

[0105] a unit for analyzing semantic features based on the speech content;

[0106] a unit for synthesizing speech behavior features and semantic features of each speaker to calculate corresponding speech weights.

[0107] The speech behavior features include total speech duration, speech frequency, average speech duration, and / or speech speed;

[0108] The semantic features include keyword density, professional term usage frequency, and / or topic relevance.

[0109] The conference roles include moderator, presenter, and / or ordinary participant.

[0110] The generation device 104 further includes:

[0111] a unit for performing natural language processing on the speech content to extract semantic information;

[0112] a unit for classifying speech content based on semantic clustering and context analysis to identify key issues and discussion focuses in the conference;

[0113] a unit for evaluating the summary weight of each speech content by synthesizing participant annotations, speech frequency, and / or keyword density;

[0114] a unit for performing semantic comparison between image text and the speech content to identify consistent information, and assisting in confirming summary highlights;

[0115] a unit for extracting summary candidate text from speech content with higher weights based on pre-defined or automatically generated prompt words, synthesizing the summary highlights assisted in confirming, combining the semantic information, the key issues, and the discussion focuses to perform language reconstruction on the summary candidate text, and generating structured conference summary text, the summary text including conference theme, key personnel speech points containing participant annotations, discussion issues, and conference conclusions.

[0116] The conference summary supports display in a graphical user interface, and users can search, modify, export, or share the content of the conference summary as needed.

[0117] The system 100 further comprises means for automatically re-executing the step of generating the conference summary to obtain an updated conference summary after the user modifies the speech content.

[0118] It should be noted that, although the operations of the conference summary generation method based on multi-modal data fusion of the present application are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all of the illustrated operations must be performed to achieve desirable results. Additionally or alternatively, certain steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be split into a plurality of steps.

[0119] Furthermore, although several means, units, or modules of the conference summary generation system based on multi-modal data fusion are mentioned in the above detailed description, such division is merely exemplary and not mandatory. Indeed, according to an embodiment of the present application, the features and functionalities of two or more modules described above can be embodied in one module. Conversely, the features and functionalities of one module described above can be further divided into a plurality of modules.

[0120] While the principles and spirit of the present application have been described with reference to several specific embodiments, it should be understood that the present application is not limited to the disclosed specific embodiments, and that the division of aspects does not imply that features from these aspects cannot be combined to benefit. Such division is merely for ease of expression. The present application is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

[0121] The present application provides: 1. A conference summary generation method based on multi-modal data fusion, characterized in that the method comprises: receiving audio data and video data of a conference; extracting content image frames from the video data, and recognizing text in the content image frames to obtain image text; performing face recognition on the video data to obtain identities of participants and their appearance time periods in the conference; performing speech recognition and speaker separation on the audio data to extract speech time periods and speech content of each speaker, and calculate speech weights of each speaker; matching the speech time periods of the speakers with the appearance time periods of the participants, and determining identities of the participants corresponding to each speaker based on the identities of the participants; identifying conference roles of the participants based on the speech weights and speech content of each speaker according to the identities of the participants corresponding to each speaker, the identities and conference roles of the participants constituting participant annotations; According to the image text, the speech content, and the participant annotation, a conference summary is generated. 2. The conference summary generation method based on multi-modal data fusion according to item 1, wherein the participant annotation further comprises an organizational role of the participant; and the method further comprises: matching the participant identity with pre-stored organizational structure information to annotate the organizational role of the participant. 3. The conference summary generation method based on multi-modal data fusion according to item 1 or item 2, wherein the audio data and the video data comprise real-time data and / or offline data. 4. The conference summary generation method based on multi-modal data fusion according to item 1 or item 2, wherein the step of extracting content image frames from the video data and recognizing the text in the content image frames to obtain image text comprises: extracting content image frames from the video data at a fixed frame rate based on the text density, clarity, and / or change rate of the image content in the video data; recognizing the text in the content image frames by a text recognition technology to obtain image text. 5. The conference summary generation method based on multi-modal data fusion according to item 1 or item 2, wherein when there is a pre-set face database, the step of performing face recognition on the video data to obtain the identity of each participant and the appearance time period of the participant in the conference comprises: performing face detection on complete image frames at a fixed frame rate; extracting a feature vector of the detected face image based on a deep learning model; comparing the extracted face feature vector with a feature vector in the pre-set face database to determine the participant identity; tracking and recognizing repeated appearance of the same face feature vector in subsequent frames in combination with the time axis of the complete image frames of the video to determine the appearance time period of the participant in the conference. 6. The conference summary generation method based on multi-modal data fusion according to item 1 or item 2, wherein when there is no pre-set face database, before the step of performing face recognition on the video data to obtain the identity of each participant and the appearance time period of the participant in the conference, the method further comprises: extracting identity image frames from the video data and recognizing the text content in the identity image frames to extract user identity information, wherein the user identity information comprises a user name, a nickname, and / or an ID; The face recognition of the video data obtains the identity of each participant and the appearance time period of the participant in the conference, and the specific steps are as follows: Face detection is performed on the identity image frame at a fixed frame rate; A feature vector of the detected face image is extracted based on a deep learning model; The face feature vector is associated with the user identity information extracted from the same image frame, a mapping relationship between the participant identity and the face feature is established, and the participant identity is determined; The repeated appearance of the same face feature vector in subsequent frames is tracked and recognized in combination with the complete image frame time axis of the video, and the appearance time period of the participant in the conference is determined. 7. The conference summary generation method based on multi-modal data fusion according to item 1 or item 2, wherein the steps of performing speech recognition and speaker separation on the audio data, extracting the speaking period and content of each speaker, and calculating the speaking weight of each speaker specifically include: The audio data is divided into multiple sound source segments according to the speaker using a voiceprint recognition model; Time stamps corresponding to each sound source segment are generated to obtain the speaking period; Each sound source segment is assigned a temporary identifier; Speech recognition processing is performed on each sound source segment to extract the corresponding speaking content; The speaking period and the speaking content are associated; The speech behavior characteristics of each speaker are counted; The semantic features are analyzed based on the speaking content; The speech behavior characteristics and semantic features of each speaker are integrated to calculate the corresponding speaking weight. 8. The conference summary generation method based on multi-modal data fusion according to item 7, wherein the speech behavior characteristics include total speaking time, speaking frequency, average speaking time, and / or speech speed; The semantic features include keyword density, professional term usage frequency, and / or theme relevance. 9. The conference summary generation method based on multi-modal data fusion according to item 1 or item 2, wherein the conference roles include the host, the main speaker, and / or the ordinary participant. 10. The conference summary generation method based on multi-modal data fusion according to item 1 or item 2, wherein the steps of generating a conference summary based on the image text, the speaking content, and the participant label are as follows: Natural language processing is performed on the speaking content to extract semantic information; Based on semantic clustering and context analysis, the speech content is classified, and the key issues and discussion focuses in the meeting are identified; By integrating the participant annotations, speech frequency, and / or keyword density, the summary weight of each segment of speech content is evaluated; The image text and the speech content are semantically compared to identify consistent information, which assists in confirming the summary focus; Based on predefined or automatically generated cue words, the summary candidate text is extracted from the speech content with higher weight based on the auxiliary-confirmed summary focus. The summary candidate text is language-reconstructed in combination with the semantic information, key issues, and discussion focuses to generate structured meeting summary text, which includes the meeting theme, key personnel speech points containing participant annotations, discussion topics, and meeting conclusions. 11. The meeting summary generation method based on multi-modal data fusion according to item 1 or item 2, characterized in that the meeting summary supports display in a graphical user interface, and users can search, modify, export, or share the content of the meeting summary as needed. 12. The meeting summary generation method based on multi-modal data fusion according to item 1 or item 2, characterized in that the method further comprises: when the user modifies the speech content, automatically re-executing the step of generating the meeting summary to obtain an updated meeting summary. 13. A meeting summary generation system based on multi-modal data fusion, characterized in that the system comprises: a device for receiving audio data and video data of a meeting; an image extraction device for extracting content image frames from the video data and recognizing the text in the content image frames to obtain image text; an identity recognition device for performing face recognition on the video data to obtain the identity of each participant and their appearance time period in the meeting; a content extraction device for performing speech recognition and speaker separation on the audio data to extract the speech time period and speech content of each speaker and calculate their speech weight; a device for matching the speech time period of the speaker with the appearance time period of the participant and determining the corresponding participant identity of each speaker based on the identity of each participant; a device for identifying the meeting role of each participant based on the speech weight and speech content of each speaker according to the corresponding participant identity of each speaker, wherein the identity and meeting role of the participant constitute the participant annotations; a generation device for generating a meeting summary based on the image text, speech content, and participant annotations. 14. The conference summary generation system based on multi-modal data fusion according to item 13, wherein the attendee label further comprises an organizational role of the attendee; and the system further comprises: a device for matching the attendee identity with pre-stored organizational structure information to label the organizational role of the attendee. 15. The conference summary generation system based on multi-modal data fusion according to item 13 or 14, wherein the audio data and the video data comprise real-time data and / or offline data. 16. The conference summary generation system based on multi-modal data fusion according to item 13 or 14, wherein the image extraction device further comprises: a unit for extracting content image frames from the video data at a fixed frame rate based on the text density, definition and / or variation rate of image content in the video data; a unit for recognizing the text in the content image frames by text recognition technology to obtain image text. 17. The conference summary generation system based on multi-modal data fusion according to item 13 or 14, wherein when there is a pre-set face database, the identity recognition device further comprises: a unit for performing face detection in the complete image frames at a fixed frame rate; a unit for extracting a feature vector of the detected face image based on a deep learning model; a unit for comparing the extracted face feature vector with the feature vectors in the pre-set face database to determine the identity of the attendee; a unit for tracking and recognizing the repeated appearance of the same face feature vector in subsequent frames in combination with the time axis of the complete image frames of the video to determine the appearance period of the attendee in the conference. 18. The conference summary generation system based on multi-modal data fusion according to item 13 or 14, wherein when there is no pre-set face database, the identity recognition device further comprises: a unit for extracting identity image frames from the video data and recognizing the text content in the identity image frames to extract user identity information, the user identity information comprising a user name, a nickname and / or an ID; a unit for performing face detection in the identity image frames at a fixed frame rate; a unit for extracting a feature vector of the detected face image based on a deep learning model; The unit is configured to associate the face feature vector with the user identity information extracted from the image in the same frame, establish a mapping relationship between the identity of the participant and the face feature, and determine the identity of the participant. The unit is configured to track and identify the repeated appearance of the same face feature vector in subsequent frames in combination with the complete image frame timeline of the video, and determine the appearance period of the participant in the meeting. 19. The conference summary generation system based on multi-modal data fusion according to item 13 or 14, wherein the content extraction device further comprises: A unit configured to divide the audio data into multiple sound source segments according to the speaker using a voiceprint recognition model; A unit configured to generate a timestamp corresponding to each sound source segment to obtain a speaking period; A unit configured to assign a temporary identifier to each sound source segment; A unit configured to perform speech recognition processing on each sound source segment to extract the corresponding speaking content; A unit configured to establish a correspondence between the speaking period and the speaking content; A unit configured to statistically analyze the speech behavior characteristics of each speaker; A unit configured to analyze the semantic characteristics based on the speaking content; A unit configured to integrate the speech behavior characteristics and semantic characteristics of each speaker to calculate the corresponding speaking weight. 20. The conference summary generation system based on multi-modal data fusion according to item 19, wherein the speech behavior characteristics include total speaking time, speaking frequency, average speaking time, and / or speech speed; The semantic characteristics include keyword density, professional term usage frequency, and / or theme relevance. 21. The conference summary generation system based on multi-modal data fusion according to item 13 or 14, wherein the conference roles include moderator, presenter, and / or ordinary participant. 22. The conference summary generation system based on multi-modal data fusion according to item 13 or 14, wherein the generation device further comprises: A unit configured to perform natural language processing on the speaking content to extract semantic information; A unit configured to classify the speaking content based on semantic clustering and context analysis, and identify the key issues and discussion focuses in the meeting; A unit configured to evaluate the summary weight of each segment of speaking content by integrating participant annotation, speaking frequency, and / or keyword density; A unit configured to perform semantic comparison between the image text and the speaking content, identify the consistent information, and assist in confirming the summary focus. The unit for generating structured meeting summary text based on pre-defined or automatically generated prompt words, synthesizing auxiliary confirmation, extracting summary candidate text from higher weight speech content, language reconstruction of the summary candidate text combined with the semantic information, the key issues and the discussion focus, generating structured meeting summary text, the summary text includes meeting theme, key personnel speech points containing participant annotation, discussion issues and meeting conclusion. 23. The conference summary generation system based on multi-modal data fusion according to item 13 or 14, characterized in that the conference summary supports display in a graphical user interface, and users can search, modify, export or share the content of the conference summary as needed. 24. The conference summary generation system based on multi-modal data fusion according to item 13 or 14, characterized in that the system further comprises a device for automatically re-executing the step of generating a conference summary after the user modifies the speech content, and obtaining an updated conference summary.

Claims

1. A method for generating conference summaries based on multimodal data fusion, characterized in that, The method includes: Receive audio and video data from the meeting; Content image frames are extracted from the video data, and the text in the content image frames is recognized to obtain image text; Facial recognition is performed on the video data to obtain the identity of each participant and the time period of their appearance in the meeting; The audio data is subjected to speech recognition and speaker separation to extract the speaking time and content of each speaker and to calculate their speaking weight. The speaker's speaking time period is matched with the time period of the participants' appearance, and the participant identity corresponding to each speaker is determined based on the identity of each participant. Based on the attendee identity corresponding to each speaker, and according to the speaking weight and speaking content of each speaker, the meeting role of each attendee is identified, and the attendee identity and meeting role constitute the attendee label; A meeting summary is generated based on the image text, the speech content, and the participants' annotations.

2. The meeting summary generation method based on multimodal data fusion according to claim 1, characterized in that, The participant labeling also includes the participant's organizational role; the method also includes: The identities of the attendees are matched with pre-stored organizational structure information to identify the organizational roles of the attendees.

3. The meeting summary generation method based on multimodal data fusion according to claim 1 or 2, characterized in that, The audio data and the video data include real-time data and / or offline data.

4. The meeting summary generation method based on multimodal data fusion according to claim 1 or 2, characterized in that, The steps of extracting content image frames from the video data and recognizing the text in the content image frames to obtain image text include: Based on the text density, sharpness, and / or rate of change of the image content in the video data, content image frames are extracted from the video data at a fixed frame rate. The text in the content image frame is identified using text recognition technology to obtain the image text.

5. The meeting summary generation method based on multimodal data fusion according to claim 1 or 2, characterized in that, When a pre-defined facial database exists The specific steps for performing facial recognition on the video data to obtain the identity of each participant and the time period of their appearance in the meeting are as follows: Face detection is performed on complete image frames at a fixed frame rate; Feature vectors of detected face images are extracted based on deep learning models; The extracted facial feature vectors are compared with feature vectors in a pre-set facial database to determine the identity of the attendees. By combining the complete image frame timeline of the video, the repeated appearance of the same facial feature vector in subsequent frames is tracked and identified to determine the time period in which the participants appeared during the meeting.

6. The meeting summary generation method based on multimodal data fusion according to claim 1 or 2, characterized in that, When there is no pre-set face database Before the step of performing facial recognition on the video data to obtain the identity of each participant and the time period of their appearance in the meeting, the method further includes: The identity image frames are extracted from the video data, and the text content in the identity image frames is identified to extract user identity information, which includes username, nickname and / or ID. The specific steps for performing facial recognition on the video data to obtain the identity of each participant and the time period of their appearance in the meeting are as follows: Face detection is performed on the identity image frames at a fixed frame rate; Feature vectors of detected face images are extracted based on deep learning models; By associating facial feature vectors with user identity information extracted from the same frame image, a mapping relationship between the identity of the attendees and facial features is established to determine the identity of the attendees. By combining the complete image frame timeline of the video, the repeated appearance of the same facial feature vector in subsequent frames is tracked and identified to determine the time period in which the participants appeared during the meeting.

7. The meeting summary generation method based on multimodal data fusion according to claim 1 or 2, characterized in that, The steps of performing speech recognition and speaker separation on the audio data, extracting the speaking time and content of each speaker, and calculating their speaking weight specifically include: Audio data is divided into multiple sound source segments according to the speaker using a voiceprint recognition model; Generate timestamps corresponding to each sound source segment to obtain the speaking time period; Each sound source segment will be assigned a temporary identifier; Speech recognition processing is performed on each sound source segment to extract the corresponding speech content; Establish a correspondence between speaking time periods and speaking content; Statistical analysis of the speech behavior characteristics of each speaker; Analyze semantic features based on the content of the speech; By combining the speech behavior features and semantic features of each speaker, the corresponding speech weight is calculated.

8. The meeting summary generation method based on multimodal data fusion according to claim 7, characterized in that, The speech behavior characteristics include total speaking time, speaking frequency, average speaking time and / or speaking speed; Semantic features include keyword density, frequency of use of technical terms, and / or topic relevance.

9. The meeting summary generation method based on multimodal data fusion according to claim 1 or 2, characterized in that, The roles in the meeting include moderator, speaker, and / or general participant.

10. A conference summary generation system based on multimodal data fusion, characterized in that, The system includes: Device for receiving audio and video data from a conference; An image extraction device is used to extract content image frames from the video data and to recognize the text in the content image frames to obtain image text; An identity recognition device is used to perform facial recognition on the video data to obtain the identity of each participant and the time period of their appearance in the meeting; The content extraction device is used to perform speech recognition and speaker separation on the audio data, extract the speaking time and content of each speaker, and calculate their speaking weight. A device for matching the speaking time of the speaker with the appearance time of the participants, and determining the participant identity corresponding to each speaker based on the identity of each participant; A device for identifying the meeting role of each participant based on the participant's identity, the speaker's speaking weight, and the content of their speech, wherein the participant's identity and meeting role constitute the participant label; A generation device for generating a meeting summary based on the image text, the speech content, and the participants' annotations.

Images