Conference summarization method, apparatus, electronic device, and storage medium

Abstract

The present application provides a conference summarization method, an apparatus, an electronic device, and a storage medium. The method comprises: displaying a conference interface of a target conference, and initiating recording of the conference interface of the target conference; recognizing content displayed on the conference interface corresponding to the target conference to obtain a screen content recognition result; on the basis of a speech recognition result and the screen content recognition result, performing timeline alignment to obtain aligned multi-source input data; and inputting the multi-source input data into a conference summarization large model to output a summarization result of the target conference. The method can improve the accuracy and comprehensiveness of conference summarization.

Description

Conference summary method, device, electronic device and storage medium TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and more particularly, to a conference summary method, device, electronic device and storage medium. BACKGROUND

[0002] At present, an effective conference summary solution is basically to recognize the speech of the attendees, and then input the conference text content recognized by speech into a large model for conference summary. The development of large model technology can greatly improve the accuracy of conference summary. However, due to the diversity of conference types and the complexity of conference scenarios, the accuracy of speech recognition is generally difficult to reach more than 90%, resulting in errors in the text input into the large model, which makes it difficult to ensure the accuracy of the conference summary. Moreover, the conference summary considering only the speech information is difficult to provide comprehensive conference summary information for users. Therefore, how to improve the accuracy and comprehensiveness of conference summary has become a technical problem to be solved.

[0003] SUMMARY

[0004] The present application provides a conference summary method, device, electronic device and storage medium, which can improve the accuracy and comprehensiveness of conference summary.

[0005] In a first aspect, a conference summary method is provided, comprising: displaying a conference interface of a target conference, starting conference interface recording of the target conference; recognizing conference speech of the target conference to obtain a speech recognition result; recognizing display content of the conference interface corresponding to the target conference to obtain a screen content recognition result; performing time axis alignment according to the speech recognition result and the screen content recognition result to obtain aligned multi-source input data; inputting the multi-source input data into a conference summary large model to output a summary result of the target conference.

[0006] In the above technical solution, after starting conference interface recording of the target conference, the conference speech of the target conference is recognized, and the display content of the conference interface corresponding to the target conference is recognized, that is, when summarizing the target conference, not only the conference speech is considered, but also the display content of the conference interface is considered, which provides multi-dimensional reference for the summary of the target conference, and is conducive to improving the comprehensiveness of the conference summary. According to the speech recognition result and the screen content recognition result, the time axis is aligned to obtain the aligned multi-source input data, which is conducive to providing stable and reliable input data to the conference summary large model to overcome the problem of difficult speech recognition in complex conferences. Therefore, inputting the multi-source input data into the conference summary large model is conducive to obtaining an accurate summary result, and improving the accuracy and comprehensiveness of the summary result of the target conference.

[0007] With reference to the first aspect, in some possible implementation manners, the display content of the conference interface includes board content, and the screen content recognition result includes a board recognition result; and the identifying the display content of the conference interface corresponding to the target conference to obtain the screen content recognition result includes: identifying the board content written by the user in the target conference to obtain the board recognition result.

[0008] In the technical solution described above, the screen content recognition result includes the board recognition result, which provides stable and reliable multi-source input data for the conference summary model, and is conducive to improving the accuracy and reliability of the obtained conference summary result.

[0009] With reference to the first aspect and the implementation manners described above, in some possible implementation manners, the identifying the board content written by the user in the target conference to obtain the board recognition result includes: if it is detected that the user has an intention to erase the board content in the target conference, identifying the board content to obtain the board recognition result.

[0010] In the technical solution described above, when it is detected that the user has an intention to erase the board content in the target conference, the board content is identified, which can identify the board content at a reasonable time, without the need to identify the board content in real time, and is conducive to avoiding the loss of the board content.

[0011] With reference to the first aspect and the implementation manners described above, in some possible implementation manners, the identifying the display content of the conference interface corresponding to the target conference to obtain the screen content recognition result includes: detecting whether the display content of the conference interface changes; and in the case where the display content changes, identifying the changed display content to obtain the screen content recognition result.

[0012] In the technical solution described above, considering that the display content of the conference interface changes in the conference process, and does not have so many scene changes like videos captured by real-world cameras. Therefore, when identifying the display content on the conference interface, it can be identified without real-time identification, but in the case where the display content of the conference interface changes, the changed display content is identified to obtain the screen content recognition result, which is conducive to avoiding repeated identification, thereby saving computing power, and can avoid missing identification of the display content, and ensure the effectiveness of the identification.

[0013] In some possible implementation manners, in combination with the first aspect and the foregoing implementation manners, the identifying the changed display content to obtain the screen content recognition result includes: obtaining a target image frame displayed by the conference interface; the target image frame is an image frame other than a first image frame among the image frames displayed by the conference interface; comparing the target image frame with a previous image frame of the target image frame to obtain a first type of text region in the target image frame that has changed compared with the previous image frame; performing text recognition on the first type of text region to obtain a text recognition result of the first type of text region; and obtaining the screen content recognition result according to the text recognition result.

[0014] In some possible implementation manners, in combination with the first aspect and the foregoing implementation manners, the method further includes: if there is a second type of text region in the target image frame that has not changed compared with the previous image frame, using a text recognition result of the second type of text region in the previous image frame as a text recognition result of the second type of text region in the target image frame.

[0015] In the foregoing technical solution, if there is a second type of text region in the target image frame that has not changed compared with the previous image frame, the text recognition result of the text region in the previous image frame can be reused in the target image frame, without the need to perform text detection and recognition again, which is beneficial to saving the time required for text recognition of the second type of text region and speeding up the processing time of the system.

[0016] In some possible implementation manners, in combination with the first aspect and the foregoing implementation manners, the obtaining the screen content recognition result according to the text recognition result includes: performing layout analysis on the target image frame to obtain a plurality of block regions; determining an entity relationship between text entities in each block region according to the text recognition result; and for each block region, splicing the text entities having the entity relationship according to the entity relationship between the text entities in the block region to obtain text structured content of the target image frame, and using the text structured content as the screen content recognition result.

[0017] In the foregoing technical solution, the layout analysis algorithm is introduced to perform layout analysis on the target image frame to obtain a plurality of block regions. The text entity relationship in each block region is predicted, and then the text entities having the entity relationship are spliced, which is beneficial to effectively aggregating the text content in the same block region. The entity relationship between the text entities is used to form effective and accurate text structured content, which provides more reliable input for the conference summary large model, and is beneficial to improving the usability of the conference summary result.

[0018] In a possible implementation manner of the first aspect, the conference summary large model is trained based on the following manner: obtaining unlabeled original training data; obtaining labeled target training data; wherein the target training data is labeled with instructions generated based on the target training data; and pre-training the conference summary large model based on the original training data and the target training data.

[0019] In the technical solution, the labeled target training data is a high-quality instruction data pair, which is equivalent to adding a high-quality instruction data pair to the pre-training of the conference summary large model, so as to enable the conference summary large model to learn more knowledge in the training process, thereby improving the accuracy of the conference summary result based on the conference summary large model.

[0020] In a possible implementation manner of the first aspect, the conference summary large model is trained based on the following manner: obtaining unlabeled original training data; obtaining labeled target training data; wherein the target training data is labeled with instructions generated based on the target training data; and pre-training the conference summary large model based on the original training data and the target training data.

[0021] In a possible implementation manner of the first aspect, the conference summary large model is trained based on the following manner: obtaining unlabeled original training data; obtaining labeled target training data; wherein the target training data is labeled with instructions generated based on the target training data; and pre-training the conference summary large model based on the original training data and the target training data.

[0022] In the technical solution, the labeled target training data is a high-quality instruction data pair, which is equivalent to adding a high-quality instruction data pair to the pre-training of the conference summary large model, so as to enable the conference summary large model to learn more knowledge in the training process, thereby improving the accuracy of the conference summary result based on the conference summary large model.

[0023] In a possible implementation manner of the first aspect, the display content of the conference interface includes annotation content, and the screen content recognition result includes an annotation content recognition result; and the identifying the display content of the conference interface corresponding to the target conference to obtain a screen content recognition result includes: identifying annotation content of the conference interface corresponding to the target conference to obtain an annotation content recognition result.

[0024] In the technical solution, the user annotation is added to capture the personalized attention points of the participants, and the depth and personalized elements of the summary are increased. In the process of generating the conference summary, multi-source information input is constructed by combining conference speech recognition and user annotation content recognition. The multi-source information can complement each other, and the influence of recognition errors of a single information source on the accuracy of the summary result is reduced. Even if the accuracy of the speech recognition result is less than 90%, the annotation content recognition result in the screen content recognition result can be used as a check and supplement to jointly improve the credibility of the overall information. The more rich and structured information input is provided for the conference summary large model, which helps the conference summary large model to better understand and learn the conference content, and thus to output a more accurate, comprehensive and clear summary result.

[0025] In a possible implementation manner of the first aspect, the annotation content includes annotation content of a text type and / or annotation content of a non-text type; and the identifying the annotation content to obtain an annotation content recognition result includes: identifying the annotation content of the text type to obtain a text annotation recognition result; and / or identifying the annotation content of the non-text type to obtain a non-text annotation recognition result.

[0026] In the technical solution, the annotation content of the text type is recognized, so that the explicit text feedback, opinion summary or keyword annotation of the participant can be captured, and the text information of the conference content is directly enriched. Meanwhile, the annotation content of the non-text type is recognized, so that the non-text feedback of the participant in the conference process can be understood. The non-text feedback can reflect the emotional attitude, emphasis or logical relationship of the participant to a certain extent. The recognition of the annotation content of the text type and / or the annotation content of the non-text type makes the conference summary closer to the actual atmosphere and intention of the conference, and improves the depth and accuracy of the summary. In combination with the recognition of the text and non-text annotation, the conference summary can be presented in a more diversified form, which not only includes detailed text analysis, but also integrates the interpretation of visual elements such as charts and symbols, so that the conference summary is more vivid and easy to understand. Meanwhile, the comprehensive recognition of the annotation content encourages the participant to participate in the conference in a diversified manner. Whether the annotation content of the text type is entered by text or the annotation content of the non-text type is marked by graphics or symbols, it can be effectively recognized and integrated into the summary result of the conference, so as to enhance the interactivity and participation, improve the flexibility of the conference tool and user satisfaction, and be beneficial to enhancing the user experience and further improving the quality and practicality of the conference summary.

[0027] In combination with the first aspect and the above implementation manner, in some possible implementation manners, the above-mentioned recognition of the annotation content of the non-text type to obtain the non-text annotation recognition result comprises: obtaining an annotation trajectory feature of the annotation content of the non-text type; determining a target trajectory feature matched with the annotation trajectory feature from trajectory features in a trajectory feature library; each trajectory feature in the trajectory feature library corresponds to a trajectory recognition result represented by the trajectory feature; and taking the trajectory recognition result corresponding to the target trajectory feature as the non-text annotation recognition result.

[0028] In the technical solution, the annotation trajectory feature of the annotation content of the non-text type is obtained, and trajectory matching is performed among the trajectory features in the trajectory feature library. This method can deeply understand the morphological details of the annotation, reduce misrecognition, and improve the accuracy of recognition. Moreover, the trajectory feature library contains a plurality of predefined trajectory features and corresponding recognition results, which makes it possible to adapt to the recognition of diversified non-text annotation, improves the compatibility of different user annotation habits, and makes the conference summary more comprehensively reflect various annotation information.

[0029] In some possible implementation manners, in combination with the first aspect and the foregoing implementation manners, the time axis alignment according to the speech recognition result and the screen content recognition result to obtain the aligned multi-source input data comprises: determining actual meanings represented by the annotation recognition result in the target meeting; and performing time axis alignment according to the actual meanings represented by the speech recognition result and the annotation recognition result in the target meeting to obtain the aligned multi-source input data.

[0030] In the technical solution, by analyzing the actual meanings represented by the annotation recognition result in the target meeting, the recognition of the annotation content is not limited to the surface, but the actual meanings represented behind, such as logical relationships of confirmation, emphasis, negation, and doubt, are analyzed in depth. This deep understanding helps to more accurately reflect the focus of the meeting discussion and the attitude of the participants, so that the summary result is more insightful. Through the time axis alignment, the spatiotemporal consistency of information from different sources is ensured, which helps to construct a coherent meeting narrative, so that the summary result of the target meeting can be reproduced according to the natural process of the meeting, and the logicality and traceability are enhanced. Moreover, this multi-dimensional information integration manner avoids misunderstanding or omission that may be caused by a single information source, and ensures that the meeting summary is comprehensive and accurate.

[0031] In a second aspect, a meeting summary apparatus is provided, which comprises: a meeting interface recording module configured to display a meeting interface of a target meeting and start recording the meeting interface of the target meeting; a speech recognition module configured to recognize meeting speech of the target meeting to obtain a speech recognition result; a screen content recognition module configured to recognize display content of a meeting interface corresponding to the target meeting to obtain a screen content recognition result; a time axis alignment module configured to perform time axis alignment according to the speech recognition result and the screen content recognition result to obtain aligned multi-source input data; and a meeting summary module configured to input the multi-source input data into a meeting summary large model to output a summary result of the target meeting.

[0032] In some possible implementation manners, in combination with the second aspect, the display content of the meeting interface comprises board content, and the screen content recognition result comprises a board recognition result; and the screen content recognition module is specifically configured to recognize board content written by a user in the target meeting to obtain the board recognition result.

[0033] In some possible implementation manners, in combination with the second aspect and the foregoing implementation manners, the board recognition module is specifically configured to, if it is detected that the user has an intention to erase the board content in the target meeting, recognize the board content to obtain the board recognition result.

[0034] With reference to the second aspect and the implementation manner thereof, in a possible implementation manner, the screen content recognition module is specifically configured to: detect whether the display content of the conference interface changes; and in a case where the display content changes, recognize the changed display content to obtain the screen content recognition result.

[0035] With reference to the second aspect and the implementation manner thereof, in a possible implementation manner, the screen content recognition module is specifically configured to: obtain a target image frame displayed by the conference interface; the target image frame is an image frame other than a first image frame among the image frames displayed by the conference interface; compare the target image frame with a previous image frame of the target image frame to obtain a first type of text region in the target image frame that changes compared with the previous image frame; perform text recognition on the first type of text region to obtain a text recognition result of the first type of text region; and obtain the screen content recognition result according to the text recognition result.

[0036] With reference to the second aspect and the implementation manner thereof, in a possible implementation manner, the apparatus further includes a multiplexing module configured to, if there is a second type of text region in the target image frame that does not change compared with the previous image frame, use a text recognition result of the second type of text region in the previous image frame as a text recognition result of the second type of text region in the target image frame.

[0037] With reference to the second aspect and the implementation manner thereof, in a possible implementation manner, the screen content recognition module is specifically configured to: perform layout analysis on the target image frame to obtain a plurality of block regions; determine an entity relationship between text entities in each of the block regions according to the text recognition result; and for each of the block regions, splice the text entities having the entity relationship according to the entity relationship between the text entities in the block region to obtain text structured content of the target image frame, and use the text structured content as the screen content recognition result.

[0038] With reference to the second aspect and the implementation manner thereof, in a possible implementation manner, the apparatus further includes a training module configured to: obtain original training data that is not labeled; obtain target training data that is labeled; the target training data is labeled with an instruction generated based on the target training data; and pre-train the conference summary large model according to the original training data and the target training data.

[0039] In some possible implementation manners, in combination with the second aspect and the foregoing implementation manners, the training module is specifically configured to perform instruction generation on the foregoing original training data based on the instruction generation model to obtain a set of valid instruction data pairs, wherein the set of valid instruction data pairs includes a plurality of valid instruction data pairs, each of the valid instruction data pairs includes candidate training data and a valid instruction generated based on the candidate training data, and the candidate training data is training data in the original training data that can generate a valid instruction; and the target training data that is labeled is obtained by screening the plurality of valid instruction data pairs.

[0040] In some possible implementation manners, in combination with the second aspect and the foregoing implementation manners, the training module is specifically configured to perform quality scoring on the plurality of valid instruction data pairs to obtain a quality score of each valid instruction data pair; and the target instruction data pair that meets a preset quality requirement is obtained by screening the plurality of valid instruction data pairs according to the quality score, and the target instruction data pair is taken as the target training data that is labeled.

[0041] In some possible implementation manners, in combination with the second aspect and the foregoing implementation manners, the display content of the conference interface includes annotation content, and the screen content recognition result includes an annotation content recognition result; and the screen content recognition module is specifically configured to recognize the annotation content of the conference interface corresponding to the target conference to obtain the annotation content recognition result.

[0042] In some possible implementation manners, in combination with the second aspect and the foregoing implementation manners, the screen content recognition module is specifically configured to recognize the annotation content of the text type to obtain a text annotation recognition result, and / or recognize the annotation content of the non-text type to obtain a non-text annotation recognition result.

[0043] In some possible implementation manners, in combination with the second aspect and the foregoing implementation manners, the screen content recognition module is specifically configured to obtain annotation trajectory features of the annotation content of the non-text type, determine a target trajectory feature that matches the annotation trajectory features from trajectory features in a trajectory feature library, wherein each trajectory feature in the trajectory feature library corresponds to a trajectory recognition result that is represented by the trajectory feature, and take the trajectory recognition result corresponding to the target trajectory feature as the non-text annotation recognition result.

[0044] In some possible implementation manners, in combination with the second aspect and the foregoing implementation manners, the time axis alignment module is specifically configured to determine actual meanings represented by the annotation recognition result in the target conference, and perform time axis alignment according to the voice recognition result and the actual meanings represented by the annotation recognition result in the target conference to obtain the aligned multi-source input data.

[0045] In a third aspect, an electronic device is provided, including a memory and a processor. The memory is configured to store executable program code, and the processor is configured to invoke and run the executable program code from the memory, so that the electronic device executes the method in the first aspect.

[0046] In a fourth aspect, a computer program product is provided, which includes computer program code. When the computer program code is run on a computer, the computer is caused to execute the method in the implementation manner of the first aspect.

[0047] In a fifth aspect, a computer-readable storage medium is provided, which stores computer program code. When the computer program code is run on a computer, the computer is caused to execute the method in the implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0048] FIG. 1 is a schematic diagram of a scenario at the beginning of an online meeting according to an embodiment of the present application;

[0049] FIG. 2 is a schematic diagram of a scenario in the process of an online meeting according to an embodiment of the present application;

[0050] FIG. 3 is a schematic diagram of a scenario at the end of an online meeting according to an embodiment of the present application;

[0051] FIG. 4 is a schematic diagram of a scenario of a meeting summary according to an embodiment of the present application;

[0052] FIG. 5 is a schematic diagram of another scenario of a meeting summary according to an embodiment of the present application;

[0053] FIG. 6 is a schematic flowchart of a meeting summary method according to an embodiment of the present application;

[0054] FIG. 7 is a schematic diagram of a time axis marked with a speech recognition result according to an embodiment of the present application;

[0055] FIG. 8 is a schematic diagram of a process of globally sorting and text splicing text content in a text region to obtain a screen content recognition result according to an embodiment of the present application;

[0056] FIG. 9 is a schematic diagram of a process of obtaining text structured content of a target image frame according to an embodiment of the present application;

[0057] FIG. 10 is a schematic diagram of another implementation process of obtaining text structured content of a target image frame according to an embodiment of the present application;

[0058] FIG. 11 is a schematic diagram of a structure of a meeting summary large model according to an embodiment of the present application;

[0059] FIG. 12 is a schematic diagram of an interface after a conference according to an embodiment of the present application;

[0060] FIG. 13 is a schematic diagram of another conference summary method according to an embodiment of the present application;

[0061] FIG. 14 is a schematic diagram of a comment content according to an embodiment of the present application;

[0062] FIG. 15 is a schematic diagram of a conference summary device according to an embodiment of the present application;

[0063] FIG. 16 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] The technical solutions in the present application will be described in detail below with reference to the drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B: "and / or" in the text is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist together, and B exists alone, and in addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0065] Hereinafter, the terms "first" and "second" are only for descriptive purposes, and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more features.

[0066] With the development of intelligent technology, the types of electronic products that people encounter in daily life are becoming increasingly diverse. Among them, interactive electronic products based on touch technology have a good human-computer interaction experience and are increasingly integrated with comprehensive functions. Interactive tablets are one of the representative integrated devices, which are suitable for group interaction occasions such as conferences, teaching, and business displays. Such devices integrate projectors, video conferencing, and other functions.

[0067] There is a demand for automatically generating a conference minutes after a conference is conducted by using an interactive tablet. At present, a conference minutes summary scheme with good effect is basically to recognize the speech of the conference participants and convert it into text, and then input the converted text content into a large model to summarize the conference minutes. The development of large model technology can greatly improve the accuracy of conference summary, but because of the diversity of conference types and the complexity of conference scenes, the accuracy of speech recognition is generally difficult to reach more than 90%, which leads to errors in the text input into the large model, and it is difficult to ensure that the conference summary always has high accuracy. Moreover, the conference summary considering only the speech information is also difficult to provide comprehensive conference minutes information for the user, for example, the user only displays on the screen without using language to speak out the scene. Therefore, how to improve the accuracy and comprehensiveness of the conference summary has become a technical problem to be solved.

[0068] To at least solve the above technical problems, the embodiments of the present application provide a conference summary method applied to an electronic device. The electronic device can be an interactive device such as the above-mentioned interactive tablet and conference terminal, which is not limited in the embodiments. For ease of description, the electronic device is taken as a conference terminal in the following description.

[0069] In the embodiments, not only the conference speech but also the display content of the conference interface is considered, which provides multi-dimensional reference for the summary of the target conference and is conducive to improving the comprehensiveness of the conference summary. By time axis alignment of the speech recognition result and the screen content recognition result, the aligned multi-source input data is obtained, which is conducive to providing stable and reliable input data for the conference summary large model and improving the accuracy of the obtained conference summary result.

[0070] The application scenario of the embodiments can be that after the conference is ended, the conference summary results such as conference minutes, conference to-do list and conference speech record related to the conference are automatically generated.

[0071] FIG. 1 is a schematic diagram of a scene when an online conference starts according to an embodiment of the present application.

[0072] For example, the scene in FIG. 1 includes a conference terminal 110 and a conference display area 120. The conference display area 120 includes a camera picture 121, a conference display picture 122, a time length information 123 of the conference, and a virtual button 124 for starting recording.

[0073] The camera picture 121 is a video picture taken by the camera of the conference terminal, which is usually a video picture of the conference participant. The conference display picture 122 is the current demonstration picture of the conference, which is usually the conference demonstration picture. The time length information 123 of the conference is used to indicate the time length of the conference. The virtual button 124 for starting recording is used to trigger the recording instruction of the conference terminal.

[0074] Exemplarily, at the beginning of the online conference, the interface is shown as in FIG. 1; when the virtual button 124 is clicked, the interface is shown as in FIG. 2.

[0075] FIG. 2 is a schematic diagram of a scene in the online conference process according to an embodiment of the present application.

[0076] Exemplarily, the conference display area 120 in FIG. 2 includes a camera picture 121, a conference display picture 122, a duration information 123 of the conference, a virtual button 127 for pausing recording, a virtual button 128 for ending recording, and a subtitle display area 129.

[0077] Among them, the conference terminal 110, the conference display area 120, the camera picture 121, the conference display picture 122, and the duration information 123 of the conference are the same as in the embodiment of FIG. 1; the virtual button 127 is used to trigger a pause recording instruction; the virtual button 128 is used to trigger an end recording instruction; and the subtitle display area 129 is used to display real-time subtitles.

[0078] Exemplarily, during the online conference process, when the virtual button 128 is clicked, the interface is shown as in FIG. 3.

[0079] FIG. 3 is a schematic diagram of a scene at the end of the online conference according to an embodiment of the present application.

[0080] Exemplarily, the conference display area 120 in FIG. 3 includes a window 130, a virtual button 131, and a virtual button 132. The window 130 is a display window after responding to the end recording instruction; the virtual button 131 is used to trigger a save file instruction; and the virtual button 132 is used to trigger a conference summary generation instruction.

[0081] Exemplarily, when the virtual button 124 is clicked, the interface is shown as in FIG. 2; when the virtual button 127 is clicked, the conference recording is paused; when the virtual button 128 is clicked, the interface is shown as in FIG. 3. When the virtual button 131 is clicked, the recording file is saved; and when the virtual button 132 is clicked, the conference summary is generated, and a display interface shown in FIG. 4 is displayed.

[0082] FIG. 4 is a schematic diagram of a conference summary according to an embodiment of the present application.

[0083] Exemplarily, different display items in the displayed conference summary include intelligent chapters, speaker records, opinion statistics, conference minutes, conference to-do lists, and conference recording pictures. The different display items in the conference summary are described below.

[0084] For example, the smart chapter is a chapter extracted according to the key content in different stages during the conference. When the corresponding chapter name is clicked, the corresponding content of the chapter is displayed on the conference recording picture above the smart chapter. For example, when the first chapter is clicked, the detailed content of the smart writing experience research and practice of the first chapter is displayed in the upper area of the smart chapter, and the detailed content is the recording content between the time period (0:36-5:12) when the conference starts to record.

[0085] For example, the speaker record is a display of all speech content of each speaker. The speech content of different speakers is distinguished by the voiceprint features of the speakers, and the text display of the speech content is performed, and the speech content of different speakers corresponds to the speech time. For example, speaker 1 speaks at 1 minute after the start of the conference, speaker 2 speaks at 11 minutes after the start of the conference, and speaker 3 speaks at 16 minutes after the start of the conference.

[0086] For example, the opinion statistics is a summary of the speech content of each speaker, and a summary of the speech content of each speaker and an attitude summary of the speech content of the speaker, so that the user can know the attitude and main point of view of each speaker. In addition, the attitudes of all speakers can be analyzed to obtain the proportion of the attitudes of all speakers, as shown in FIG. 4, the proportion of speakers with positive attitude is 55%, the proportion of speakers with neutral attitude is 30%, and the proportion of speakers with negative attitude is 15%.

[0087] It should be noted that the speech record is a display of all speech content of each speaker, and the opinion statistics is a summary of the speech content of each speaker and an overall analysis of the attitudes of all speakers.

[0088] For example, the initial display content in the lower right area of FIG. 4 is a conference minutes, and the content in the conference minutes includes a conference summary and conference content. The conference summary is a highly summarized text obtained according to the speech information of different speakers, so that the user can quickly understand the key theme content of the conference according to the conference summary. The conference content is a summary of important content in the conference by using a point-by-point text, so that the user can know the main content of the conference discussion. When the user clicks the conference to do, a display interface as shown in FIG. 5 is displayed.

[0089] FIG. 5 is another scene diagram of a conference summary provided by an embodiment of the present application.

[0090] FIG. 5 is substantially the same as the display items in FIG. 4, except that the display content of the lower right area in FIG. 4 is a meeting minutes, and the display content of the lower right area in FIG. 5 is a meeting to-do list, such as to-do items 1-5 shown in FIG. 5. Specifically, by obtaining time information in the meeting, the text corresponding to the time information can be extracted to obtain the meeting to-do list, so that the user can know the to-do items to be completed after the meeting.

[0091] For example, as shown in FIG. 4 and FIG. 5, the meeting summary scenario further includes an exit button, a save button, and a re-summarize button. The exit button is used to trigger an instruction to exit the meeting summary, the save button is used to trigger an instruction to save the summary result, and the re-summarize button is used to trigger an instruction to re-perform the meeting summary. When the user clicks the exit button, the conference device exits the meeting summary; when the user clicks the save button, the conference device saves the current summary result; and when the user clicks the re-summarize button, the conference device re-executes the process of the meeting summary and re-generates the summary result of the meeting.

[0092] FIG. 6 is a schematic flowchart of a meeting summary method provided by an embodiment of the present application.

[0093] For example, as shown in FIG. 6, the method includes:

[0094] Step 200: display a meeting interface of a target meeting, and start recording the meeting interface of the target meeting.

[0095] Step 201: recognize the meeting voice of the target meeting to obtain a voice recognition result.

[0096] Step 202: recognize the display content of the meeting interface corresponding to the target meeting to obtain a screen content recognition result.

[0097] Step 203: perform time axis alignment based on the voice recognition result and the screen content recognition result to obtain aligned multi-source input data.

[0098] Step 204: input the multi-source input data into a meeting summary large model to output a summary result of the target meeting.

[0099] In the embodiment shown in FIG. 6, after starting the conference interface recording of the target conference, the conference voice of the target conference is recognized, and the display content of the conference interface corresponding to the target conference is recognized, that is, when summarizing the target conference, not only the conference voice but also the display content of the conference interface will be considered, which provides multi-dimensional reference for the summary of the target conference, and is beneficial to improve the comprehensiveness of the conference summary. According to the voice recognition result and the screen content recognition result, the time axis is aligned to obtain the aligned multi-source input data, which is beneficial to provide stable and reliable input data for the conference summary large model to overcome the problem of difficult voice recognition in complex conferences. Therefore, the multi-source input data is input into the conference summary large model, which is beneficial to obtain an accurate summary result and improve the accuracy and comprehensiveness of the summary result of the target conference.

[0100] The implementation of each step in the embodiment shown in FIG. 6 will be described below.

[0101] In step 200, the conference interface of the target conference displayed by the conference terminal can refer to the above-mentioned FIG. 1. When the conference terminal detects that the user clicks the virtual button 124, the conference interface recording of the target conference is started, so that the conference terminal can record during the development of the target conference to obtain conference recording information. The conference recording information includes conference voice, screen recording information and other information.

[0102] In step 201, after the target conference ends, the conference voice in the conference recording information is recognized by the voice-to-text technology, the conference voice is converted into conference text, and the conference text is taken as the voice recognition result. Specifically, the conference voice can be obtained by the conference terminal, and speaker recognition and voice content recognition are performed according to the conference voice to obtain the voice recognition result corresponding to each speaker in the target conference.

[0103] Optionally, during the recording of the target conference, the recorded conference voice can be recognized in real time by the voice-to-text technology to obtain the voice recognition result.

[0104] In a possible implementation manner, in order to facilitate the time axis alignment in the subsequent step 203, after obtaining the voice recognition result, the voice recognition result can be marked with a time axis to obtain the correspondence between the voice recognition result and the time point. For example, the time axis marked on the voice recognition result can refer to FIG. 7, from which it can be seen that the correspondence between the voice recognition result corresponding to different speakers (the host, Xiao A and Xiao B) and the time point. And it can also be seen from FIG. 4 and FIG. 5 that the speech content of different speakers is marked with the corresponding speech time, and the speech content is the voice recognition result and the speech time is the time point.

[0105] In step 202, the conference terminal has a display screen, and a display interface of the display screen can be a conference interface. The conference terminal can record content displayed on the conference interface during development of the target conference to obtain screen recording information. That is, the conference recording information can include the screen recording information in addition to the conference voice. The screen recording information includes a plurality of image frames displayed on the display screen during the conference. The plurality of image frames can be automatically obtained through the conference software / screen projection software on the conference terminal. After the target conference ends, the display content of the conference interface can be recognized according to the screen recording information to obtain a screen content recognition result.

[0106] The display content on the conference interface can include screen projection content during the conference. The screen projection content can be presentation content prepared for the target conference. The presentation content can be a conference PPT (PowerPoint), a conference document, or the like. Optionally, the display content on the conference interface can include at least the presentation content of the target conference, and the corresponding screen content recognition result can include at least a recognition result of the presentation content.

[0107] It can be understood that the screen projection content during the conference is basically structured content that is highly summarized and thought. The screen projection content can include main content of the conference and information related to a conference goal, and the like. Therefore, the recognition of the screen projection content in this embodiment can stably provide reliable references for conference summaries. The screen content recognition result can be a text recognition result of the display content of the conference interface.

[0108] In a possible implementation manner, recognizing the display content on the conference interface includes a plurality of image frames displayed on the display screen during the conference, inputting a text detection model to output a text region in the image frames, and then performing text recognition on text in the text region to obtain the screen content recognition result. The text detection model can use PixelLink. PixelLink is a deep learning-based text detection model. PixelLink can generate a pixel-level text instance segmentation result, which is helpful to accurately locate and segment the text region in the image without being affected by the text layout. This is of great significance to improve the accuracy and robustness of text detection in complex scenarios.

[0109] In an exemplary embodiment, the implementation manner of step 202 includes detecting whether the display content of the conference interface changes, and recognizing the changed display content to obtain the screen content recognition result when the display content changes.

[0110] Specifically, since the screen recording information includes a plurality of image frames displayed on the display screen during the conference process, whether the display content on the conference interface changes can be determined by the difference between the pixel values of two adjacent image frames. For example, the pixel values of two adjacent image frames can be calculated respectively, and then the number of changed pixel points can be counted according to the pixel values of the two adjacent image frames. If the number of changed pixel points exceeds a preset proportion of the total number of pixel points on the conference interface, it is determined that the display content on the conference interface has changed. The preset proportion can be preset, for example, 5%, but the present embodiment is not limited in this regard.

[0111] If the display content on the conference interface changes, the text detection algorithm is called again to perform text detection on the display content on the conference interface to obtain a text region, and then the text recognition algorithm is called to recognize the text content in the text region to obtain a screen content recognition result.

[0112] In a possible implementation manner, whether the display content on the conference interface changes can be activated according to a specific time or a specific external operation (such as a page turning action on the PPT). That is, whether the display content on the conference interface changes can be detected in real time, or the step of detecting whether the display content on the conference interface changes can be performed when the specific time or the specific external operation is detected.

[0113] In a possible implementation manner, if the display content on the conference interface does not change, the text recognition result obtained last time when the display content is recognized can be reused. For example, according to the pixel difference between the third image frame and the second image frame of the display screen, it is determined that the display content on the conference interface does not change, and then the text recognition result of the second image frame can be used as the text recognition result of the third image frame, without the need to perform text recognition on the third image frame again.

[0114] In the present embodiment, it is considered that the change of the display content on the conference interface during the conference process does not have so many scene changes as the video captured by the real-world camera. Therefore, when the display content on the conference interface is recognized, the recognition can be performed in real time, or the changed display content is recognized to obtain a screen content recognition result when the display content on the conference interface changes, which is beneficial to avoid repeated recognition, thereby saving computing power, and can avoid missed recognition of the display content, and ensure the effectiveness of the recognition.

[0115] In an exemplary embodiment, the above-mentioned recognition of the changed display content to obtain a screen content recognition result includes the following S11 to S14:

[0116] S11: Obtain a target image frame displayed on the conference interface.

[0117] The target image frame is an image frame displayed on the conference interface, except for the first image frame. That is, the target image frame is an image frame displayed on the conference interface, except for the first image frame, in the entire conference process. For example, the target image frame can be a slide in a conference PPT, except for the first slide.

[0118] It can be understood that, for the first image frame, because there is no previous image frame, full-page recognition is performed on the first image frame to obtain text recognition results of each text region in the first image frame.

[0119] S12: Comparing the target image frame with the previous image frame of the target image frame to obtain a first type of text region in the target image frame that has changed compared with the previous image frame.

[0120] S13: Performing text recognition on the first type of text region that has changed to obtain text recognition results of the first type of text region.

[0121] Specifically, a classic CTC (Connectionist Temporal Classification, sequence labeling algorithm) can be used to build a text recognition model. Considering that the displayed text in the conference is basically stable printed characters, a full CNN (Fully Convolutional Network, full convolutional neural network) module can be directly used for feature extraction and front-back modeling. The above text recognition model can be used to perform text recognition on the first type of text region that has changed to obtain text recognition results of the text region.

[0122] In a possible implementation manner, the manner of obtaining the first type of text region can include the following steps: first, determining a change region that has changed compared with the previous image frame. Then, performing text region detection on the target image frame to obtain each text region in the target image frame. Then, a target text region in the change region is taken as the first type of text region that has changed compared with the previous image frame.

[0123] In a possible implementation manner, the manner of obtaining the first type of text region can include the following steps S21 to S24:

[0124] S21: Calculating the difference between the target image frame and the previous image frame to determine a change region that has changed compared with the previous image frame, and performing morphological dilation on the change region to obtain the change region after morphological dilation.

[0125] Since the target image frame can contain other forms of content besides text content, such as picture form content, the change region here is not limited to text changes, and any form of difference between the target image frame and the previous image frame will cause the formation of the change region.

[0126] S22: Calculate the minimum circumscribed rectangle of the change region after morphological dilation, and generate a first coordinate set representing the change region based on the minimum circumscribed rectangle. The first coordinate set can be denoted as Mask_vary.

[0127] S23: Perform text region detection on the target image frame to obtain a second coordinate set representing the text region. The second coordinate set can be denoted as Mask_pixellink.

[0128] The text region detection can use the PixelLink described above. PixelLink can be used to segment the text region in the target image frame, and the second coordinate set includes the position coordinates of each text region in the target image frame.

[0129] S24: Intersect the first coordinate set and the second coordinate set, and the result of the intersection is the target text region in each text region that is located in the change region. The target text region is the first type of text region in the target image frame that has changed compared to the previous image frame.

[0130] In an exemplary embodiment, if there is a second type of text region in the target image frame that has not changed compared to the previous image frame, the text recognition result of the second type of text region in the previous image frame is used as the text recognition result of the second type of text region in the target image frame.

[0131] That is, in addition to the first type of text region that has changed compared to the previous image frame, there can also be a second type of text region in the target image frame that has not changed compared to the previous image frame. For the second type of text region that has not changed, the recognition result of the previous image frame can be directly reused without the need for further recognition.

[0132] In one possible implementation, after S24, S25 and S26 can be further included as follows:

[0133] S25: Determine the non-change region in the target image frame other than the change region.

[0134] The non-change region is a region in which the target image frame does not change compared with the previous image frame. The first coordinate set of the change region is denoted as Mask_vary, and the third coordinate set of the non-change region is denoted as Mask_inv_vary.

[0135] S26: determining a second type of text region located in the non-change region in each text region of the previous image frame, and reusing the recognition content of the second type of text region in the previous image frame.

[0136] The second type of text region is a text region in the target image frame that does not change compared with the previous image frame. Specifically, the intersection of the coordinate set of each text region of the previous image frame and the third coordinate set Mask_inv_vary is obtained, and the result of the intersection is the coordinate of the second type of text region. That is, if the coordinate of a text region in the previous image frame falls within the third coordinate set Mask_inv_vary, the text region is regarded as the second type of text region, and the recognition result of the second type of text region in the previous image frame is reused in the target image frame.

[0137] In the embodiment, if the second type of text region that does not change compared with the previous image frame exists in the target image frame, the text recognition result of the text region in the previous image frame can be reused in the target image frame, without the need for text detection and recognition again, which is beneficial to save the time required for text recognition of the second type of text region and accelerate the processing time of the system.

[0138] S14: obtaining a screen content recognition result according to the text recognition result.

[0139] The text recognition result includes the text recognition result of each text region in the target image frame. Each text region in the target image frame includes the first type of text region and / or the second type of text region. Correspondingly, the text recognition result includes the text recognition result of the first type of text region and / or the text recognition result of the second type of text region. Thus, in this step, if only the first type of text region exists in the target image frame, the screen content recognition result is obtained according to the text recognition result of the first type of text region. If only the second type of text region exists in the target image frame, the screen content recognition result is obtained according to the text recognition result of the second type of text region. If the first type of text region and the second type of text region exist in the target image frame at the same time, the screen content recognition result is obtained according to the text recognition result of the first type of text region and the text recognition result of the second type of text region.

[0140] In a possible implementation manner, the text content of the image frame and the position coordinates of the text region where the text content is located are obtained through the text detection and recognition described above. According to the position coordinates of the text region, the text content in the text region is globally sorted and spliced to obtain a screen content recognition result.

[0141] Specifically, the screen content recognition result of the target image frame can be obtained by splicing the text recognition result described above in a global sorting manner from left to right and from top to bottom. For example, referring to FIG. 8, which is a process schematic diagram of globally sorting and splicing the text content in the text region to obtain a screen content recognition result. The text region in FIG. 8 includes table 1 and table 2. According to the text recognition result of table 1 and table 2, the screen content recognition result 301 is obtained by splicing in a global sorting manner from left to right and from top to bottom.

[0142] Alternatively, the screen content recognition result of the target image frame can also be obtained by splicing the text recognition result described above in a global sorting manner from top to bottom and from left to right. For example, referring to FIG. 3, the text recognition result of table 1 and table 2 in FIG. 3 is spliced in a global sorting manner from top to bottom and from left to right to obtain the screen content recognition result 302.

[0143] As can be seen from the above description and FIG. 8, the global sorting manner from top to bottom and from left to right will result in the text content of different tables being connected in an interleaved manner. And the global sorting manner from left to right and from top to bottom will result in different text entities not being spliced in a correct mapping relationship or order, and thus an incorrect text output is generated. That is, the above two manners are both prone to cause text information misplacement, which will further cause the conference summary large model to deviate from the understanding of the content. Therefore, it is particularly important to analyze the layout of the image frame and structure the content. Based on this, the embodiment provides a solution that can avoid text information misplacement. The following will be specifically introduced:

[0144] In another possible implementation manner, the obtaining of the screen content recognition result according to the text recognition result includes the following S31 to S33:

[0145] S31: performing layout analysis on the target image frame to obtain a plurality of block regions.

[0146] The layout analysis is a process of intelligently distinguishing and positioning different block regions in the target image frame, and the purpose is to identify and segment each independent functional region, and each independent functional region can be understood as a block region. Each independent functional region can include a text region, a directory region, a title region, a table region, a list region, and the like.

[0147] Specifically, the document image understanding framework can be PP-Structure, and the PP-Structure is mainly applied to layout analysis and table recognition of a picture form document. The process of the layout analysis uses deep learning and computer vision technology to analyze complex graphic-text mixed layout, so as to obtain a plurality of block regions of text, directory, title, table, list, and the like in the image frame.

[0148] S32: According to the text recognition result, the entity relationship between the text entities in each block region is determined.

[0149] Specifically, after the layout analysis is completed, the text entities (such as a person, a place, an event, an organization, and the like) and the relationships between the text entities are further analyzed according to the text content in different block regions, that is, the text recognition result, so as to obtain the entity relationship between the text entities in each block region. For example, for each block region, text entity recognition can be performed first, that is, according to the text recognition result, meaningful text entities are identified in each block region, such as identifying a person's name, an event, a time, and the like. Then, relationship extraction is performed, that is, the relationships between different text entities are predicted and extracted, such as the relationship between a person and a position, the relationship between an event and a time, the relationship between a time and an event, and the like, so as to understand the internal structure of each block region, and then the entity relationship between the text entities in each block region is obtained.

[0150] S33: For each block region, according to the entity relationship between the text entities in the block region, the text entities with the entity relationship are spliced, the text structured content of the target image frame is obtained, and the text structured content is taken as a screen content recognition result.

[0151] For example, refer to FIG. 9, which is a schematic diagram of a process of obtaining text structured content. The panel area in FIG. 9 includes a table area, and the text entities in the table area include: a person entity, an event entity, and a time entity. The person entity includes: “A student” and “B student”. The event entity includes: “1. Completed item 1”, “2. Completed item 2”, “1. Promoted item 1”, and “2. Promoted item 2”. The time entity includes: “Time”, “Afternoon”, “Date”, and “2024.2.1”. Among them, there is an entity relationship between the text entities “A student”, “1. Completed item 1”, and “2. Completed item 2”. There is an entity relationship between the text entities “B student”, “1. Promoted item 1”, and “2. Promoted item 2”. There is an entity relationship between the text entities “Time”, “Afternoon”, “Date”, and “2024”. Based on this, the text entities with entity relationships are spliced to obtain text structured content 401.

[0152] For further understanding, refer to FIG. 10, which shows an implementation process of obtaining text structured content of a target image frame. As can be seen from a in FIG. 10, the text entities with entity relationships are connected by arrows, such as the text entities “A student”, “1. Completed item 1”, and “2. Completed item 2”, which are connected by arrows, indicating that they have an entity relationship. The text entities “B student”, “1. Promoted item 1”, and “2. Promoted item 2” are connected by arrows, indicating that they have an entity relationship. The text entities “C student”, “1. XXXXXXX”, and “2. XXXXXXX” are connected by arrows, indicating that they have an entity relationship. The text entities “D student”, “1. XXXXXXX”, and “2. XXXXXXX” are connected by arrows, indicating that they have an entity relationship. The text recognition result obtained by text recognition on a in FIG. 10 can be as shown in b in FIG. 10. The text entities with entity relationships in a in FIG. 10 are spliced to obtain text structured content, which can be referred to in c in FIG. 10. As can be seen from c in FIG. 10, the text entities with entity relationships in each panel area are spliced to obtain text structured content without information misplacement.

[0153] In this embodiment, the layout analysis algorithm is introduced to perform layout analysis on the target image frame to obtain a plurality of panel areas. The text entity relationship in each panel area is predicted, and then the text entities with entity relationships are spliced, which is conducive to effective aggregation of the text content in the same panel area. Through the entity relationship between the text entities, effective and accurate text structured content is formed, which provides a more reliable input for the conference summary large model, and is conducive to improving the usability of the conference summary result.

[0154] In step 203, time axis alignment is performed according to the speech recognition result and the screen content recognition result, to obtain aligned multi-source input data.

[0155] Specifically, the speech recognition result and the screen content recognition result can be aligned according to the time axis shown in FIG. 7, to obtain the aligned multi-source input data. The aligned multi-source input data includes the speech recognition result and the screen content recognition result after time alignment. The time axis alignment aims to establish a corresponding relationship between the speech recognition result, the screen content recognition result and the time point in the conference process, so that the speech recognition result and the screen content recognition result can be arranged in the same time sequence.

[0156] In a possible implementation manner, the screen content recognition result includes a presentation content recognition result. It is considered that the split method of the screen content (such as a conference PPT) is relatively clear. Therefore, the duration of each page can be taken as a reference for time axis alignment processing, and the speech recognition result and the presentation content recognition result within the duration of each page can be uniformly classified into a page structure information, to form the following alignment form of Page ID-Speech Content-Page Content:

[0157] In an exemplary embodiment, the display content of the conference interface includes board content, and the screen content recognition result includes a board recognition result. Correspondingly, the implementation manner of step 202 includes recognizing the board content written by the user in the target conference to obtain the board recognition result. Further, the implementation manner of step 203 includes performing time axis alignment according to the board recognition result in the speech recognition result and the screen content recognition result, to obtain the aligned multi-source input data. In this embodiment, the aligned multi-source input data at least includes the speech recognition result and the board recognition result.

[0158] For example, if the screen content recognition result includes the presentation content recognition result and the board recognition result, the aligned multi-source input data includes the speech recognition result, the presentation content recognition result and the board recognition result.

[0159] It can be understood that during the process of the target conference, the user can switch the conference interface currently displayed on the display screen to a whiteboard application interface, and then input the board content in the whiteboard application interface by handwriting. The board content also has a certain reference effect on the conference summary, and therefore the board content written by the user on the whiteboard application interface in the target conference can be recognized to obtain the board recognition result.

[0160] Specifically, the text region in which the board content in the whiteboard application interface is located can be detected based on the PixelLink scheme. Then, the board content in the text region is recognized based on a text recognition algorithm to obtain a board recognition result. Considering that the handwritten board content has a larger scale change in the width direction, a CNN (Convolutional Neural Network) + BI-LSTM (Bidirectional Long Short-Term Memory) text recognition scheme can be used to recognize the board content in this embodiment.

[0161] After obtaining the board recognition result, the speech recognition result, the presentation content recognition result, and the board recognition result can be time axis aligned to obtain aligned multi-source input data. For example, the page ID of the conference PPT can be used as a reference for time axis alignment processing, and the speech recognition result, the presentation content recognition result, and the board recognition result within the duration of each page can be uniformly classified as one layout structure information to obtain the aligned multi-source input data.

[0162] In this embodiment, the board recognition result is further added on the basis of the speech recognition result and the screen content recognition result, which provides stable and reliable multi-source input data for the conference summary large model, and is beneficial to improve the accuracy and reliability of the obtained conference summary result.

[0163] In a possible implementation, considering that the board content appears relatively less frequently in most conference scenarios, the board recognition result can also not be time axis aligned, and only the speech recognition result and the presentation content recognition result can be time axis aligned. In the subsequent step 204, the board recognition result, the aligned speech recognition result, and the presentation content recognition result are input into the conference summary large model.

[0164] In an exemplary embodiment, the above-mentioned identifying the board content written by the user in the target conference to obtain the board recognition result includes: if it is detected that the user has an intention to erase the board content in the target conference, identifying the board content to obtain the board recognition result.

[0165] It can be understood that, in the process of the user writing the board content in the whiteboard application interface, the user may erase the handwritten content. In this embodiment, the identification logic of the board content can be triggered every time it is detected that the user has an intention to erase the board content. Specifically, when it is detected that the user triggers the action of erasing the board content, it can be determined that the user has an intention to erase the board content, and thus the timing when the board content is identified is determined as the starting timing.

[0166] In this embodiment, when it is detected that the user has the intention to erase the board content in the target meeting, the board content is recognized, which can realize the recognition of the board content at a reasonable time, without the need for real-time recognition of the board content, and also helps to avoid the loss of board content.

[0167] In step 204, the conference summary large model can be based on a large language model, and the large language model is fine-tuned in combination with conference data in a conference scenario to obtain a conference summary large model for performing conference summary. The conference data in the conference scenario mentioned above can be the multi-source input data mentioned above. The prompt instruction for the target meeting is input into the conference summary large model to output a summary result for responding to the prompt instruction. The prompt instruction for the target meeting can be pre-designed for user selection, or can be designed by the user according to actual needs.

[0168] The large language model can be an open source large language model such as LLaMA2, Baichuan, or other large language models similar in structure and containing multiple transformer structures.

[0169] In one possible implementation, when a target condition for generating a summary result of a target meeting is met, the multi-source input data can be input into the conference summary large model to output the summary result of the target meeting. The summary result of the target meeting can include any one or more of the intelligent chapters, speaker records, opinion statistics, meeting minutes, and meeting to-do lists mentioned above.

[0170] In another possible implementation, when a target condition for generating a summary result of a target meeting is met, the multi-source input data and a prompt instruction for the target meeting can be input into the conference summary large model to output a summary result for responding to the prompt instruction. In this example, the prompt instruction can be a question for the target meeting, and some prompt instructions can be pre-designed for user selection, or the user can design the prompt instruction according to actual needs, thereby facilitating the generation of a summary result expected by the user.

[0171] For example, the target condition can be that the end of the target meeting is detected, that is, after the end of the target meeting is detected, the summary result of the target meeting can be automatically generated. The target condition can also be that the user clicks the virtual button 132, which is used to trigger an instruction to generate a conference summary, so that the conference terminal responds to the instruction and generates a summary result of the target meeting.

[0172] When the user clicks the virtual button 132, which is used to trigger an instruction to generate a conference summary, the conference terminal responds to the instruction and generates a summary result of the target meeting.

[0173] For example, the interface diagram of the summary result of the target meeting can refer to FIG. 4 and FIG. 5. When the user is not satisfied with the summary result, the meeting terminal can be instructed to generate a new summary result of the target meeting by clicking the re-summarize button. When the user expects to save the summary result, the meeting terminal can be instructed to save the current summary result by clicking the save button. When the user expects to exit the meeting summary, the meeting terminal can be instructed to exit the interface of the summary result by clicking the exit button.

[0174] It should be noted that FIG. 4 and FIG. 5 are only one possible interface diagram of the summary result. In specific implementation, according to the needs of the user for the summary result, any one or more of the intelligent chapter, the speaker record, the opinion statistics, the meeting minutes, and the meeting to-do list can be displayed in the interface of the summary result, and other summary results for responding to the prompt instructions input by the user can also be displayed.

[0175] In the example embodiment, the conference summary large model is trained based on the following manner: obtaining unlabeled original training data; obtaining labeled target training data; wherein the target training data is labeled with instructions generated based on the target training data; and pre-training the conference summary large model based on the original training data and the target training data.

[0176] The labeling refers to instruction labeling, and the original training data and the target training data are both text training data. The unlabeled original training data can be understood as training data that is not labeled with instructions, i.e., training data that does not form an instruction data pair. The labeled target training data can be understood as training data that is labeled with instructions, and the target training data and the instructions generated based thereon can form an instruction data pair.

[0177] In one possible implementation, the labeled target training data can be derived from artificial labeling and open-source instruction data sets, and the labeled target training data belongs to high-quality instruction data pairs. Therefore, in this embodiment, high-quality instruction data pairs are added to the pre-training of the conference summary large model, which helps the conference summary large model to learn more knowledge during the training process, thereby improving the accuracy of the conference summary result based on the conference summary large model.

[0178] In an example embodiment, the obtaining the labeled target training data includes: generating instructions based on the instruction generation model, to obtain a set of effective instruction data pairs, wherein the set of effective instruction data pairs includes a plurality of effective instruction data pairs, each of the plurality of effective instruction data pairs includes candidate training data and an effective instruction generated based on the candidate training data, the candidate training data is training data in the original training data that can generate an effective instruction; and filtering the plurality of effective instruction data pairs to obtain the labeled target training data.

[0179] The instruction generation model can use an open source large model for generating corresponding instructions based on text data. For example, a prompt for instructing the large model to generate corresponding instructions based on text data is constructed, and the prompt is used to generate instructions for the unlabeled original training data. Since not all original training data can generate appropriate instructions, in this embodiment, the instruction generation model can be allowed to generate a None instruction for original training data that cannot generate appropriate instructions when constructing the prompt. When generating instructions based on a certain original training data, a None instruction is obtained, which indicates that the original training data cannot generate appropriate instructions, i.e., the None instruction generated by the original training data is an invalid instruction. After generating instructions for all original training data, the None instructions can be filtered to obtain effective instructions, each effective instruction and the candidate training data that generates the effective instruction form an effective instruction data pair, thereby obtaining a set of effective instruction data pairs including a plurality of effective instruction data pairs.

[0180] In a possible implementation, the plurality of effective instruction data pairs can be filtered based on a preset filtering rule, and the filtered effective instruction data pairs are used as the labeled target training data.

[0181] In an example embodiment, the filtering the plurality of effective instruction data pairs to obtain the labeled target training data includes: performing quality scoring on the plurality of effective instruction data pairs to obtain a quality score of each effective instruction data pair; and filtering a target instruction data pair that meets a preset quality requirement from the plurality of effective instruction data pairs according to the quality score, and using the target instruction data pair as the labeled target training data.

[0182] The quality score of the effective instruction data pair can represent the correlation between the candidate training data and the effective instruction included in the effective instruction data pair, for example, the higher the correlation between the candidate training data and the effective instruction, the higher the quality score of the effective instruction data pair.

[0183] In a possible implementation, the open-source quality scoring model can be used to score the quality of a plurality of effective instruction data pairs, to obtain quality scores of the effective instruction data pairs, and the effective instruction data pair with a quality score higher than a preset score threshold can be taken as the target instruction data pair meeting the preset quality requirement.

[0184] In another possible implementation, a plurality of different quality scoring models can be used to score the quality of each effective instruction data pair respectively, to obtain quality scores obtained based on different quality scoring models. According to the quality scores obtained based on different quality scoring models, the target instruction data pair meeting the preset quality requirement can be screened from the plurality of effective instruction data pairs. For example, the effective instruction data pair with a quality score higher than a preset score threshold obtained by different quality scoring models can be taken as the target instruction data pair meeting the preset quality requirement. Or, the arithmetic mean or weighted mean of the quality scores obtained by different quality scoring models is calculated, and the effective instruction data pair with an arithmetic mean or weighted mean greater than a preset average value threshold can be taken as the target instruction data pair meeting the preset quality requirement.

[0185] In this embodiment, according to the quality scores of the effective instruction data pairs, the target instruction data pair with high quality can be further screened from the effective instruction data pairs, and then when the conference summary large model is trained, the conference summary large model can learn high-quality knowledge, thereby improving the accuracy of the conference summary result obtained based on the conference summary large model.

[0186] In a possible implementation, the target instruction data pair meeting the preset quality requirement screened above, the manually annotated training data, the open-source instruction data set, and the unlabeled original training data can be fused at a preset ratio to pre-train the conference summary large model, so that the conference summary large model can learn more knowledge during the pre-training process.

[0187] In a possible implementation, the structure diagram of the conference summary large model can be referred to FIG. 11. According to an input token sequence prefix t1, t2, t3,...t k , the probability of the next token t k+1 appearing is predicted, that is, p(t k+1 |t1, t2, t3,...t k ). Through the way of word chain, the cycle is repeated until the generated token sequence reaches a certain length or meets other cycle ending conditions, and a complete string t1, t2, t3,...t k ,t k+1 ,...t n is obtained.

[0188] Specifically, the step of generating the conference summary result based on the conference summary large model includes the following S41-S48:

[0189] S41: input the prompt instruction prompt for the target conference. The prompt serves as the initial text for the generation process and can be a paragraph, a sentence, a phrase, or just a word.

[0190] S42: text segmentation: the input initial text is segmented using a preset segmentation algorithm to obtain a token sequence. The preset segmentation algorithm can be tokenizer, BPE (Byte Pair Encoding), WordPiece, etc.

[0191] S43: obtain Token embedding according to the token sequence.

[0192] S44: obtain Position embedding according to the token sequence

[0193] S45: token encoding: combine Token embedding and Position embedding to generate the final embedding vector, input the embedding vector into the encoding layer composed of multiple Transformer decoder layers to obtain the corresponding token encoding.

[0194] S46: pass the output of the last Transformer decoder layer to the softmax layer to obtain the probability distribution of each token in the token sequence.

[0195] S47: determine the next token according to the probability distribution of the softmax layer, the selection strategy can be greedy sampling, beam search, etc., and add the selected next token to the input token sequence.

[0196] S48: repeat generation: re-input the updated input sequence into the model and repeat steps S43-S47 until the predetermined generation length is reached or other loop termination conditions are met, and output the conference summary result.

[0197] In the above manner, the conference summary large model generates tokens one by one to ensure that the generated conference summary result is natural and coherent under the given prompt.

[0198] In a possible implementation manner, the conference summary large model can also be expanded with a Chinese vocabulary. It can be understood that the original vocabulary of LLaMA only contains a small number of Chinese characters. When tokenizing Chinese characters based on the original vocabulary, one Chinese character is often divided into multiple tokens (2-3 tokens can be combined into one Chinese character), which significantly reduces the efficiency of encoding and decoding. Moreover, if a language that has not appeared or has appeared very little in pre-training is not learned sufficiently, it will also affect the accuracy of the model in generation. Therefore, in this embodiment, a word segmentation model is trained based on the SentencePiece tool to obtain a Chinese vocabulary, and then the Chinese vocabulary is merged with the original vocabulary of LLaMA to obtain a merged and expanded Chinese vocabulary by combining the vocabularies of them.

[0199] Sentencepiece is an open source natural language processing tool used to implement the word segmentation task in text preprocessing. It uses a BPE-based algorithm to perform word segmentation on texts in various languages. BPE is a data-based compression algorithm that compresses text by encoding the most frequently occurring substrings. In the word segmentation task, the BPE algorithm can split longer words into multiple subwords, reducing the size of the vocabulary and increasing the generalization ability of the model, which helps to improve the performance of the model.

[0200] The specific implementation manner of expanding the Chinese vocabulary is described below. The manner of expanding the Chinese vocabulary can include S51 to S53 as follows:

[0201] S51: Prepare training corpus. In this embodiment, high-quality open source data such as Wudao, Chinese Wikipedia, Chinese Web Text, etc. can be used for training. The data format is one sentence per line.

[0202] S52: Train a word segmentation model based on Sentencepiece to obtain a Chinese vocabulary.

[0203] In this embodiment, the BPE word segmentation manner is used, the vocabulary size is set to 20000, the language list supported by the model is set to Chinese and English, the value of the parameter split_digits is set to True to keep consistent with the word segmentation of LLaMA, and the value of the parameter byte_fallback is set to True.

[0204] split_digits is used to specify whether to split all digit characters into separate units, such as splitting '2023' into '2', '0', '2', and '3' independent subword units. The advantage is to reduce the number of digits in the vocabulary, and all digits can be represented. The disadvantage is that the number of tokens will increase, and one '2023' is 4 tokens.

[0205] The above byte_fallback is used to specify whether to decompose unknown or rare characters into UTF-8 (8-bit Unicode Transformation Format) bytes for representation. For example, assume the phrase "chi mei wang liang" appears too few times in the training corpus and is not in the final vocabulary. Without setting the byte_fallback parameter to True, an OOV (Out Of Vocabulary) will occur. If byte_fallback is set to True, this phrase will be tokenized using UTF-8 encoding as: "0xE9 0xAD 0x91 0xE9 0xAD 0x85 0xE9 0xAD 0x8D 0xE9 0xAD 0x89", which is split into 12 tokens.

[0206] When training the tokenization model with the above parameters, SentencePiece will calculate the most frequently occurring substrings in the text and encode them. This encoding process is an iterative one until the training conditions of the model are met, and finally a vocabulary of size 20,000 is obtained.

[0207] S53: Merge the Chinese vocabulary obtained in S52 with the original vocabulary of LLaMa.

[0208] LLaMa has a vocabulary of 32,000, which already contains rich semantic information. Therefore, by merging the Chinese vocabulary obtained in S52 with the original vocabulary of LLaMa, retaining all the original vocabulary of LLaMa and adding words that do not exist in the original vocabulary, an extended Chinese vocabulary is obtained.

[0209] In one possible implementation, it is also possible to add biases to the QKV (Query, Key, Value) layers of transformers to improve the extrapolation ability of the model.

[0210] In one possible implementation, it is also possible to add noise to the Embedding layer to improve the effect of instruction fine-tuning. The core is that after the input passes through the Embedding layer, a uniformly distributed noise is added, and the sampling range of the noise is between, where is an adjustable parameter, L is the input length, and d is the dimension of the embedding layer.

[0211] In a possible implementation manner, a linear interpolation method can also be added for length expansion. The key idea is to directly reduce the position index to match the maximum position index with the context window limit in the pre-training stage. In other words, in order to adapt to more input tokens, the position encoding is interpolated at adjacent integer positions, taking advantage of the fact that the position encoding can be applied to non-integer positions, instead of extrapolating outside the training positions. Linear interpolation can easily achieve a very long context window, and only a very short fine-tuning time is needed for the model to fully adapt to the greatly expanded context window.

[0212] To further facilitate the understanding of the embodiments of the present application, the following will be illustrated in conjunction with FIG. 12 and FIG. 13:

[0213] FIG. 12 is a schematic diagram of an interface after a meeting according to an embodiment of the present application. FIG. 12 includes: a screen recording information display area 701, a text transcription browsing area 702, a meeting attachment area 703, and a meeting summary function area 704. Among them, the screen recording information display area 701 is used to display the screen recording information of the target meeting interface during the development process, and the screen recording information can be in the form of a playable video. The text transcription browsing area 702 is used to display the speech recognition result converted from the conference voice, that is, the text content converted from the conference voice. The meeting attachment area 703 is used to display the attachment content such as screenshots, blackboard content and uploaded files during the meeting process. The meeting summary function area 704 includes an AI assistant dialogue area 7041, which is used for the user to input a prompt instruction for the target meeting. The meeting summary function area 704 is used to display the meeting summary result that can reply to the prompt instruction. Among them, the prompt instruction input by the user can be a prompt instruction edited and input by the user, or a selected prompt instruction from a plurality of selectable prompt instructions displayed in the meeting summary function area 704.

[0214] In a specific implementation, the interface schematic diagram in FIG. 12 can also display the meeting theme, host, meeting time and other content of the target meeting. And the interface schematic diagram can also include a toolbar 705, which can be composed of a series of icons or buttons, each icon or button representing a specific function or command. The user can quickly access these functions by clicking the icons on the toolbar 705, without having to find them through menus one by one, thereby improving work efficiency and user interactivity.

[0215] In this embodiment, more complete conference recording information is provided to the user, and a more comprehensive summary is made based on this, improving the efficiency of review and the accuracy of summary. In the conference recording information, the display content of the conference interface, the conference voice, the board content of the whiteboard application interface, and the identification of key time points, information and events, such as starting / ending screen projection, screenshot, annotation, multi-person conversation, and file transmission, are included.

[0216] FIG. 13 is a schematic flowchart of another conference summary method provided by the embodiments of the present application, which includes the following steps:

[0217] Step 801: Import pre-conference information (conference theme input).

[0218] Step 802: Conference voice pickup.

[0219] Step 803: Display content pickup of the conference interface of the display screen.

[0220] Step 804: Board content pickup of the whiteboard application interface.

[0221] Step 805: Voice recognition is performed on the picked-up conference voice to obtain a voice recognition result.

[0222] Step 806: Conference interface display change detection.

[0223] Step 807: Conference interface display content structured recognition to obtain a screen content recognition result.

[0224] In the case where the display content change of the conference interface is detected, the display content of the conference interface is structured and recognized to obtain a screen content recognition result.

[0225] Step 808: Structured recognition is performed on the picked-up board content to obtain a board recognition result.

[0226] Step 809: Time axis alignment is performed according to the voice recognition result and the screen content recognition result.

[0227] Step 810: The board recognition result, the aligned voice recognition result and screen content recognition result, and the prompt instruction prompt for the conference are input into a conference summary large model.

[0228] Step 811: Output the conference summary result for reply prompt.

[0229] In this embodiment, the recognition of the screen projection content (i.e., the recognition of the presentation content of the conference interface) and the recognition of the board content of the whiteboard application interface are introduced as supplements to the conference summary. Through text detection, text recognition, and text structuring, stable and reliable input data are provided for the conference summary large model to overcome the difficulty of speech recognition in complex conferences and improve the usability of the conference summary results. In addition, an efficient and low-latency content recognition scheme is proposed according to the display characteristics of the conference screen projection content. When a screen change is detected, the changed display content is recognized to obtain a screen content recognition result. When no screen change is detected, no recognition is performed, which can effectively reduce the problem of repeated recognition, reduce the overall resource consumption of the algorithm on the hardware, and improve the overall response speed.

[0230] In some embodiments, the display content of the conference interface includes annotation content, and the screen content recognition result includes annotation content recognition result. Correspondingly, the implementation manner of step 202 includes recognizing the annotation content of the conference interface corresponding to the target conference to obtain the annotation content recognition result. Further, the implementation manner of step 203 includes performing time axis alignment according to the speech recognition result and the annotation content recognition result in the screen content recognition result to obtain the aligned multi-source input data. In this embodiment, the aligned multi-source input data at least includes the aligned speech recognition result and the annotation content recognition result.

[0231] During the process of the target conference, there may be a case where a user inputs annotation content, for example, in order to emphasize the key points in the originally displayed presentation content on the conference interface, the user may mark the key points by circling, underlining, checking, etc. during the conference. At this time, the conference terminal can receive the annotation content input by the user for the target conference, and recognize the annotation content to obtain the annotation content recognition result.

[0232] In a possible implementation manner, during the process of the target conference, the user can switch the conference interface currently displayed on the display screen to an annotation interface, and then input the annotation content on the annotation interface through handwriting or stylus input. For example, the conference software is provided with a floating or fixed annotation toolbar. After the user selects a tool in the annotation toolbar, the display screen can display the annotation interface to allow the user to add annotation content on the annotation interface.

[0233] In another possible implementation manner, during the process of the target conference, the user can select a brush tool or directly add annotation content on the originally displayed presentation content on the conference interface through handwriting without switching to the annotation interface.

[0234] Optionally, in the process of adding the annotation content by the user, a prompt information of adding the annotation content can be generated, such as a prompt information appearing at the edge or corner of the screen when the annotation starts and ends, informing the participant that the annotation activity is currently being performed.

[0235] Optionally, the annotation content added by the user can be displayed or not displayed at any time, that is, after the user finishes adding the annotation, the user can select to display or not display the annotation content according to actual needs.

[0236] Exemplarily, the annotation content can include: a text type annotation content and / or a non-text type annotation content. The text type annotation content and the non-text type annotation content are classified according to different annotation expression forms, and the two types of annotation content are introduced as follows:

[0237] The text type annotation content is mainly expressed by writing text to express thoughts, analysis, evaluation or feelings about the target meeting, such as thoughts, analysis, evaluation or feelings about the presentation content in the target meeting.

[0238] The non-text type annotation content mainly relies on symbols, marks, graphics or visual elements to convey information, and does not directly use text to express. The non-text type annotation content can include: a symbol type annotation and a highlight mark type annotation. Among them, the symbol type annotation refers to using symbols such as underlines, wavy lines, check marks, cross marks, arrow pointing, circle selection, etc. to mark key words or parts that need special attention. The highlight mark type annotation refers to using different colors of highlights to distinguish different types of points or information levels.

[0239] Exemplarily, a schematic diagram of the annotation content can be referred to FIG. 14. In FIG. 14, the presentation content of the target meeting is in a directory page. The text type annotation content input by the user in the directory page can be annotation content 1101. The annotation content 1101 is annotation content input by the user in a handwriting manner, so its font is different from the font of the text in the presentation content. The annotation content 1101 indicates that “Part 3” is the key content of the meeting. The non-text type annotation content input by the user in the directory page can be annotation content 1102 and annotation content 1103. The annotation content 1102 and the annotation content 1103 belong to the symbol type annotation. The annotation content 1102 is essentially a cross mark, which can represent that “Part 1” is not the key content of the meeting. The annotation content 1103 is essentially a circle selection symbol, which can represent that “Part 4” is a direction that still needs to be continuously paid attention to after the meeting.

[0240] In a specific implementation, there can be only text type of annotation content, only non-text type of annotation content, or both text type and non-text type of annotation content in the same page of presentation content, which is not limited in the embodiment.

[0241] In an example embodiment, the annotation content input by the user for the target conference received by the conference terminal includes text type of annotation content and / or non-text type of annotation content; the identifying of the annotation content to obtain annotation content identification result includes: identifying the text type of annotation content to obtain text annotation identification result; and / or, identifying the non-text type of annotation content to obtain non-text annotation identification result.

[0242] Specifically, when the conference terminal receives the text type of annotation content, the text type of annotation content can be identified based on text recognition technology to obtain the text annotation identification result. For example, in FIG. 14, the text annotation identification result of the annotation content 1101 is “highlight”. When the conference terminal receives the non-text type of annotation content, the non-text type of annotation content can be identified based on the annotation trajectory features of the non-text type of annotation content to obtain the non-text annotation identification result. For example, in FIG. 14, the non-text annotation identification result of the non-annotation content 1102 is “cross”, and the non-text annotation identification result of the non-annotation content 1103 is “circle”.

[0243] In the embodiment, by identifying the text type of annotation content, the explicit text feedback, summary of opinions or keyword annotation of the participants can be captured, directly enriching the text information of the conference content. At the same time, by identifying the non-text type of annotation content, the non-text form of feedback of the participants in the conference can be understood, which can reflect the emotional attitude, emphasis or logical relationship of the participants to a certain extent. By identifying the text type of annotation content and / or the non-text type of annotation content, the conference summary can be closer to the actual atmosphere and intention of the conference, and the depth and accuracy of the summary can be improved. In addition, by combining the identification of text and non-text annotations, the conference summary can be presented in more diversified forms, not only containing detailed text analysis, but also integrating the interpretation of visual elements such as charts and symbols, making the conference summary more vivid and easy to understand. At the same time, the comprehensive identification of the annotation content encourages participants to use diversified ways to participate in the conference, whether it is text type of annotation content by typing text, or non-text type of annotation content by using graphical markers and symbols, which can be effectively identified and integrated into the summary results of the conference, enhancing the interactivity and participation, improving the flexibility of the conference tool and user satisfaction, which is conducive to enhancing the user experience and further improving the quality and practicality of the conference summary.

[0244] In the exemplary embodiments, the implementation manner of identifying the non-text type of comment content to obtain the non-text comment recognition result includes S61-S63 as follows:

[0245] S61: Obtain a comment track feature of the non-text type of comment content.

[0246] Specifically, the conference terminal can capture the behavior action of the user adding comments on the screen of the conference terminal through a stylus, mouse movement, or an optical tracking device. These behavior actions include, but are not limited to, drawing lines, circling points, smearing, highlighting, and the like. The conference terminal can record the starting coordinates, ending coordinates, moving speed, pressure intensity, and operation time sequence and the like parameters of each step of operation by using sensor technology and motion tracking algorithms. These parameters jointly constitute the feature vector of the comment track feature, laying a foundation for subsequent track feature matching. In order to ensure the accuracy and robustness of the comment track feature, smoothing processing, noise filtering and the like preprocessing algorithms can also be implemented.

[0247] S62: Among the track features in the track feature library, determine a target track feature matched with the comment track feature.

[0248] Among the track features in the track feature library, each track feature corresponds to a track recognition result represented by the track feature. Each track feature in the track feature library is associated with a specific non-text comment style (i.e., the track recognition result). For example, for straight lines, curves, circles, squares, arrows, cross marks, check marks and the like basic graphics, unique and distinguishable feature vectors are formed. After obtaining the comment track feature of the non-text type of comment content input by the user, a matching algorithm such as nearest neighbor classification, support vector machine (SVM), or deep learning based matching algorithm (such as cosine similarity, Euclidean distance calculation) can be used to search for the most matched target track feature in the track feature library.

[0249] S63: Take the track recognition result corresponding to the target track feature as the non-text comment recognition result.

[0250] It can be understood that once the target track feature most matched with the comment track feature is found in the track feature library, the conference terminal can take the track recognition result associated with the target track feature as the non-text comment recognition result.

[0251] In this embodiment, by obtaining the annotation track features of the non-text type annotation content, and performing track matching in each track feature of the track feature library, this method can deeply understand the morphological details of the annotation, reduce misrecognition, and improve the accuracy of recognition. And the track feature library established contains a variety of pre-defined track features and their corresponding recognition results, which makes it possible to adapt to the recognition of diversified non-text annotations, improves the compatibility of different user annotation habits, and makes the meeting summary more comprehensive reflect various annotation information.

[0252] In this embodiment, since the screen content recognition result includes the annotation content recognition result, when performing time axis alignment, the time axis alignment is performed according to the speech recognition result and the annotation content recognition result.

[0253] Specifically, the speech recognition result and the screen content recognition result including the annotation content recognition result can be aligned according to the time axis shown in FIG. 7 to obtain aligned multi-source input data. The aligned multi-source input data includes: time-aligned speech recognition result and screen content recognition result. The time axis alignment aims to establish a corresponding relationship between the speech recognition result and the screen content recognition result in the conference process and the time point, so that the speech recognition result and the screen content recognition result can be arranged in the same time sequence.

[0254] In a possible implementation manner, the screen content recognition result includes: presentation content recognition result and annotation content recognition result. Considering that the screen projection content (such as conference PPT) is relatively clear in the segmentation method. Therefore, the duration of each page of the speech recognition result, the presentation content recognition result and the annotation content recognition result can be uniformly classified as one layout structure information based on the page ID of the conference PPT as the reference of the time axis alignment processing, to form the following alignment form: Page ID (Page ID) - Speech Content (Speech Content) - Presentation Content (Page Content) - Annotation Content (Annotate Content):

[0255] In an exemplary embodiment, in the case where the screen content recognition result includes the annotation recognition result, the implementation manner of performing time axis alignment according to the speech recognition result and the screen content recognition result to obtain aligned multi-source input data includes the following S71 to S72:

[0256] S71: determining the actual meaning represented by the annotation recognition result in the target conference.

[0257] The actual meaning can be understood as the actual intention of the user when inputting the annotation content. For example, natural language processing and semantic understanding techniques can be used to deeply analyze the annotation recognition result, which can involve lexical context analysis, syntax analysis, entity recognition, sentiment analysis, and the like, to understand the contextual meaning, direction, and emotional color of the annotation content. For example, by identifying the shape and position of specific symbols and graphics in the annotation, the intention expressed by the annotation, such as instructions, questions, emphasis, or criticism, can be inferred. In addition, combined with conference background information such as the topic, speech content, and identity of the participants, the actual meaning of the annotation content representation can be further refined to ensure that it is correctly understood in the conference context.

[0258] In a possible implementation, for the text annotation recognition result, the manner of determining the actual meaning represented by the text annotation recognition result in the target conference can include S81-S83 as follows:

[0259] S81: Perform semantic analysis on the text annotation recognition result to obtain a semantic analysis result.

[0260] Specifically, semantic analysis includes lexical semantic understanding, syntax structure analysis, sentiment analysis, entity recognition, and the like, to the deep meaning of the text annotation recognition result. For example, the text annotation recognition result is "highlight" in FIG. 14, and the semantic analysis result is that some content needs to be highlighted.

[0261] S82: Determine the first target presentation content to which the annotation content of the text type points.

[0262] Specifically, the first target presentation content can be determined according to the annotation position of the annotation content of the text type. The first target presentation content is the presentation content closest to the annotation position. For example, referring to FIG. 14, among the presentation contents Part 1 to Part 4, Part 3 is closest to the annotation position of the annotation content 1101, and therefore, the presentation content Part 3 can be determined as the first target presentation content.

[0263] In a specific implementation, the first target presentation content can also be determined in combination with the conference context information corresponding to the annotation content of the text type. The conference context information includes the speech recognition result of the conference speech in a predetermined time period before and after the user input of the annotation content of the text type. It can be understood that when inputting the annotation content of the text type, the user usually issues speech information to explain and describe the input annotation content of the text type, for example, the speech information can be "the focus of this conference is on Part 3". Therefore, in combination with the conference context information, it can be analyzed that the first target presentation content to which the annotation content 1101 points is Part 3.

[0264] S83: Determine the actual meaning represented by the text annotation recognition result in the target meeting according to the semantic analysis result and the target presentation content.

[0265] Specifically, the semantic analysis result and the first target presentation content are associated to obtain the actual meaning represented by the text annotation recognition result in the target meeting. For example, referring to FIG. 14, the semantic analysis result of the annotation content 1101 is: some content needs to be paid attention to. The target presentation content pointed to by the annotation content 1101 is Part 3, and the actual meaning represented by the annotation content 1101 in the target meeting is: the content in Part 3 needs to be paid attention to.

[0266] In this embodiment, the accurate interpretation of the meaning of the text annotation recognition result in the meeting context is ensured. The text annotation recognition result not only reflects the literal meaning of the annotation, but also reflects its specific role in the meeting discussion context, such as promoting decision support, feedback, or suggestions, etc., providing deep insights for the generation of the summary result of the target meeting.

[0267] In a possible implementation, for a non-text annotation recognition result, the manner of determining the actual meaning represented by the non-text annotation recognition result in the target meeting can include S91 to S93 as follows:

[0268] S91: Perform preliminary meaning analysis on the non-text annotation recognition result to obtain a preliminary meaning analysis result.

[0269] Specifically, for non-text annotation recognition results such as symbolic annotations, highlight mark annotations, etc., the preliminary meaning conveyed by the non-text annotation recognition result can be determined through image sentiment analysis technology to obtain a preliminary meaning analysis result. Different symbols can convey specific meanings in different contexts. The following are some meanings that common symbols may express:

[0270] Underline, representing emphasis, in text, underlining is usually used to emphasize the importance of a word or phrase, indicating parts that need special attention.

[0271] Wavy line, representing uncertainty or approximation: in mathematics or science, a wavy line can represent an estimate or approximation. In text, it may express a relaxed or casual attitude, or in social media, it represents a smile or relaxed expression.

[0272] Check mark, representing affirmation or correctness, usually indicating confirmation, correctness or completion, giving a positive and affirmative emotion. In daily communication, it can express the meaning of agreement or satisfaction.

[0273] Cross mark, representing negation or error, usually indicating negation, error, incompleteness, no special attention, etc.

[0274] Arrow pointing, representing direction indication, arrow explicitly indicates direction, can guide attention or indicate sequence. Emotionally, the upward arrow may represent the meaning of progress or growth, and the downward arrow may represent the meaning of decline or decrease.

[0275] Circle selection, representing selection or highlighting, in a list or option, circle selection represents the selected or highlighted item, which can express the meaning of importance or preference.

[0276] In practical application, the meaning of the above common symbols may vary depending on the context, and the specific environment in which the symbol appears should also be considered when understanding and interpreting the symbol.

[0277] Referring to FIG. 14, the annotation content 1102 is a cross, and in combination with the specific environment in which it appears, it can be determined that the corresponding preliminary meaning analysis result is: there is content that does not need special attention. The annotation content 1103 is a highlight symbol, and in combination with the specific environment in which it appears, it can be determined that the corresponding preliminary meaning analysis result is: there is content that needs continuous attention.

[0278] S92: Determine the second target presentation content pointed to by the non-text type annotation content.

[0279] Wherein, the way to determine the second target presentation content is similar to the way to determine the first target presentation content, which can be determined according to the annotation position of the non-text type annotation content, or can be determined in combination with the meeting context information corresponding to the non-text type annotation content.

[0280] For example, referring to FIG. 14, the annotation position of the annotation content 1102 is closest to the presentation content Part 1, so the presentation content Part 1 can be determined as the second target presentation content. For another example, the voice information issued by the user when inputting the annotation content 1103 is "the direction that still needs continuous attention after this meeting is Part 4", therefore, in combination with the meeting context information, it can be analyzed that the second target presentation content pointed to by the annotation content 1103 is Part 4.

[0281] S93: Determine the actual meaning represented by the non-text annotation recognition result in the target meeting according to the preliminary meaning analysis result and the second target presentation content.

[0282] Specifically, the preliminary meaning analysis result and the second target presentation content are associated to obtain the actual meaning represented by the non-text annotation recognition result in the target meeting. For example, referring to FIG. 14, the preliminary meaning analysis result of the annotation content 1102 is that there is content that does not need special attention. The target presentation content pointed to by the annotation content 1102 is Part 1, and the actual meaning represented by the annotation content 1102 in the target meeting is that the non-key content of the meeting is Part 1. The preliminary meaning analysis result of the annotation content 1103 is that there is content that needs to be continuously paid attention to. The target presentation content pointed to by the annotation content 1103 is Part 4, and the actual meaning represented by the annotation content 1103 in the target meeting is that the content that needs to be continuously paid attention to after the meeting is Part 4.

[0283] S72: Time axis alignment is performed according to the actual meaning represented by the speech recognition result and the annotation recognition result in the target meeting to obtain aligned multi-source input data.

[0284] Specifically, time stamp alignment can be achieved by time stamp synchronization. The actual meaning of the speech recognition result and the annotation recognition result both have corresponding time stamps, ensuring that each data point accurately corresponds to the event corresponding to the actual time in the meeting process. In combination with the above alignment form of Page ID-Speech Content-Page Content-annotate Content, in S72, the actual meaning represented by the annotation recognition result in the target meeting can be taken as annotate Content.

[0285] In this embodiment, the accurate interpretation of the meaning of the non-text annotation recognition result in the meeting context is ensured. The non-text annotation recognition result not only reflects the preliminary meaning of the annotation, but also reflects the specific role of the annotation in the meeting discussion context, i.e., the actual meaning, providing deep insights for the generation of the summary result of the target meeting.

[0286] It can be understood that the presentation content recognition result and the annotation content recognition result both belong to the screen content recognition result, but the presentation content recognition result focuses on the recognition result of the presentation content such as the conference PPT and the conference document on the display screen, and the annotation content recognition result focuses on the recognition result of the annotation content input by the user on the display screen. In specific implementation, the presentation content and the annotation content can be located in different layers on the display screen, which means that the annotation can be added, deleted or modified without interfering with the main visual content, i.e., the presentation content. For example, in a conference, the user can make annotations on a separate layer while the presentation slides are being displayed, and the participants can clearly see the original presentation content and focus on the annotation content. Placing the presentation content and the annotation content in different layers is conducive to improving the interactivity of the conference and the flexibility of the conference content display.

[0287] For example, the display content of the conference interface can include any one or a combination of the following: presentation content of the target conference, annotation content of the target conference, and board writing content of the target conference. The corresponding screen content recognition result can include any one or a combination of the following: presentation content recognition result, board writing recognition result, and annotation content recognition result. Therefore, the aligned multi-source input data includes, in addition to the speech recognition result, any one or a combination of the following: presentation content recognition result, board writing recognition result, and annotation content recognition result.

[0288] In view of the fact that traditional conference summary methods relying only on speech-to-text often overlook the importance of conference presentation content, in this embodiment, by recognizing the display content of the conference interface, it is ensured that visual information such as presentation content, annotation content and board content displayed in the conference is also included in the conference summary, making the summary result more complete. The addition of annotation content and board content can capture the individualized attention of the participants, increasing the depth and personalized elements of the summary. In the process of generating the conference summary, by combining conference speech recognition, presentation content recognition, board content recognition, annotation content recognition, etc., a multi-source information input is constructed. Such multi-source information can complement each other, reducing the impact of single information source recognition errors on the accuracy of the summary result. Even if the accuracy of the speech recognition result is less than 90%, the screen content recognition results such as presentation content recognition results, board content recognition results, and annotation content recognition results can be used as verification and supplement to improve the credibility of the overall information. Furthermore, by aligning the time axis according to the above speech recognition results and screen content recognition results, the aligned multi-source input data is obtained, ensuring that the information from different sources is accurately corresponding in the time dimension, which means that each key point in the conference process can be accurately associated with the corresponding speech content and visual content such as presentation content, board content, and annotation content, making the logical structure of the multi-source input data clear and the time sequence coherent. The aligned multi-source input data provides more rich and structured information input for the conference summary large model, which helps the conference summary large model better understand and learn the conference content, thereby outputting a more accurate, comprehensive and clear summary result.

[0289] FIG. 15 is a structural schematic diagram of a conference summary device provided by an embodiment of the present application.

[0290] For example, as shown in FIG. 15, the device 900 includes:

[0291] The conference interface recording module 901 is configured to display the conference interface of the target conference and start recording the conference interface of the target conference.

[0292] The speech recognition module 902 is configured to recognize the conference speech of the target conference to obtain a speech recognition result.

[0293] The screen content recognition module 903 is configured to recognize the display content of the conference interface corresponding to the target conference to obtain a screen content recognition result.

[0294] The time axis alignment module 904 is configured to align the time axis according to the speech recognition result and the screen content recognition result to obtain aligned multi-source input data.

[0295] The conference summary module 905 is configured to input the multi-source input data into a conference summary large model to output a summary result of the target conference.

[0296] In a possible implementation manner, the display content of the conference interface includes board content, and the screen content recognition result includes a board recognition result; the device 900 further includes a board recognition module configured to recognize the board content written by the user in the target conference to obtain the board recognition result.

[0297] In a possible implementation manner, the board recognition module is specifically configured to, if it is detected that the user has an intention to erase the board content in the target conference, recognize the board content to obtain the board recognition result.

[0298] In a possible implementation manner, the screen content recognition module 903 is specifically configured to detect whether the display content of the conference interface changes; and in the case where the display content changes, recognize the changed display content to obtain a screen content recognition result.

[0299] In a possible implementation manner, the screen content recognition module 903 is specifically configured to obtain a target image frame displayed by the conference interface; the target image frame is an image frame other than a first image frame among all image frames displayed by the conference interface; compare the target image frame with a previous image frame of the target image frame to obtain a first type of text region in the target image frame that changes compared with the previous image frame; perform text recognition on the first type of text region to obtain a text recognition result of the first type of text region; and obtain a screen content recognition result according to the text recognition result.

[0300] In a possible implementation manner, the device 900 further includes a multiplexing module configured to, if there is a second type of text region in the target image frame that does not change compared with the previous image frame, use the text recognition result of the second type of text region in the previous image frame as the text recognition result of the second type of text region in the target image frame.

[0301] In a possible implementation manner, the screen content recognition module 903 is specifically configured to perform layout analysis on the target image frame to obtain a plurality of board regions; determine an entity relationship between text entities in each of the board regions according to the text recognition result; and for each of the board regions, splice the text entities having the entity relationship according to the entity relationship between the text entities in the board region to obtain text structured content of the target image frame, and use the text structured content as the screen content recognition result.

[0302] In a possible implementation, the apparatus 900 further includes a training module, configured to: obtain unlabeled original training data; and obtain labeled target training data, wherein the target training data is labeled with instructions generated based on the target training data; and perform pre-training to obtain the conference summary large model based on the original training data and the target training data.

[0303] In a possible implementation, the training module is specifically configured to: generate instructions based on the original training data by using an instruction generation model to obtain a set of valid instruction data pairs, wherein the set of valid instruction data pairs includes a plurality of valid instruction data pairs, each valid instruction data pair includes candidate training data and a valid instruction generated based on the candidate training data, and the candidate training data is training data in the original training data that can generate a valid instruction; and select the labeled target training data from the plurality of valid instruction data pairs.

[0304] In a possible implementation, the training module is specifically configured to: perform quality scoring on the plurality of valid instruction data pairs to obtain a quality score of each valid instruction data pair; select a target instruction data pair that meets a preset quality requirement from the plurality of valid instruction data pairs based on the quality score, and use the target instruction data pair as the labeled target training data.

[0305] In a possible implementation, the display content of the conference interface includes annotation content, and the screen content recognition result includes an annotation content recognition result. The screen content recognition module 903 is specifically configured to recognize annotation content of a conference interface corresponding to the target conference to obtain an annotation content recognition result.

[0306] In a possible implementation, the screen content recognition module 903 is specifically configured to: recognize the annotation content of the text type to obtain a text annotation recognition result; and / or recognize the annotation content of the non-text type to obtain a non-text annotation recognition result.

[0307] In a possible implementation, the screen content recognition module 903 is specifically configured to: obtain annotation trajectory features of the annotation content of the non-text type; determine a target trajectory feature that matches the annotation trajectory features from trajectory features in a trajectory feature library, wherein each trajectory feature in the trajectory feature library corresponds to a trajectory recognition result that characterizes the trajectory feature; and use the trajectory recognition result corresponding to the target trajectory feature as the non-text annotation recognition result.

[0308] In a possible implementation, the timeline alignment module 904 is specifically configured to determine actual meanings represented by the annotation recognition result in the target conference, and perform timeline alignment according to the voice recognition result and the actual meanings represented by the annotation recognition result in the target conference, to obtain the aligned multi-source input data.

[0309] It should be noted that all related content of each step involved in the method embodiments can be referred to the function description of the corresponding function module, and will not be repeated here.

[0310] FIG. 16 is a structural schematic diagram of an electronic device provided in an embodiment of the present application.

[0311] For example, as shown in FIG. 16, the electronic device 1600 includes a memory 1601 and a processor 1602, where the memory 1601 stores executable program code 16011, and the processor 1602 is configured to invoke and execute the executable program code 16011 to perform a conference summary method.

[0312] The embodiment can divide the function modules of the electronic device according to the method examples, for example, each function module can be divided, or two or more functions can be integrated into one processing module, and the integrated module can be implemented in the form of hardware. It should be noted that the division of the modules in the embodiment is illustrative, and is only a logical function division, and another division mode can be used in actual implementation.

[0313] In the case of dividing each function module according to each function, the electronic device can include a conference interface recording module, a voice recognition module, a screen content recognition module, a timeline alignment module, a conference summary module, and the like. It should be noted that all related content of each step involved in the method embodiments can be referred to the function description of the corresponding function module, and will not be repeated here.

[0314] The electronic device provided in the embodiment is used to perform the conference summary method, and thus the same effect as the implementation method can be achieved.

[0315] In the case of using the integrated unit, the electronic device can include a processing module and a storage module. The processing module can be used to control and manage the actions of the electronic device. The storage module can be used to support the electronic device to execute program codes and data.

[0316] The processing module can be a processor or a controller, which can realize or execute various exemplary logical blocks, modules and circuits represented in combination with the disclosure. The processor can also be a combination of computing functions, such as one or more microprocessor combinations, digital signal processing (DSP) and microprocessor combinations, etc. The storage module can be a memory.

[0317] The embodiment further provides a computer readable storage medium, which stores computer program codes, and when the computer program codes are run on a computer, the computer executes the related method steps to realize the conference summary method in the above embodiment.

[0318] The embodiment further provides a computer program product, which, when run on a computer, causes the computer to execute the related steps to realize the conference summary method in the above embodiment.

[0319] In addition, the electronic device provided by the embodiment of the present application can be a chip, a component or a module, and the electronic device can include a connected processor and a memory; wherein the memory is used to store instructions, and when the electronic device is running, the processor can call and execute the instructions to make the chip execute the conference summary method in the above embodiment.

[0320] The electronic device, the computer readable storage medium, the computer program product or the chip provided by the embodiment can be used to execute the corresponding method provided above, and thus the beneficial effects achieved thereby can refer to the beneficial effects of the corresponding method provided above, which will not be described here again.

[0321] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the above division of functional modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0322] In the embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented by other ways. For example, the device embodiments described above are only schematic, and the division of the modules or units is only a logical function division, and there can be another division way in actual implementation, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection between each other can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0323] The above merely provides the application's specific implementation, but the application's protection scope is not limited thereto, any person skilled in the art can easily think of changes or replacements within the application's disclosed technology range, which should be covered within the application's protection scope. Therefore, the application's protection scope should be subject to the protection scope of the claims.

Claims

1. A meeting summarization method, characterized by, The method is applied to an electronic device, and the method comprises: displaying a conference interface of a target conference, starting conference interface recording of the target conference; recognizing conference speech of the target conference to obtain speech recognition results; recognizing display content of the conference interface corresponding to the target conference to obtain screen content recognition results; aligning a time axis according to the speech recognition results and the screen content recognition results to obtain aligned multi-source input data; inputting the multi-source input data into a conference summary large model to output summary results of the target conference.

2. The method of claim 1, wherein, The display content of the conference interface comprises board content, and the screen content recognition results comprise board recognition results. The recognizing of the display content of the conference interface corresponding to the target conference to obtain screen content recognition results comprises: recognizing board content written by a user in the target conference to obtain board recognition results.

3. The method of claim 2, wherein, The recognizing of the board content written by the user in the target conference to obtain board recognition results comprises: if an intention of the user to erase the board content in the target conference is detected, recognizing the board content to obtain board recognition results.

4. The method of claim 1, wherein, The recognizing of the display content of the conference interface corresponding to the target conference to obtain screen content recognition results comprises: detecting whether the display content of the conference interface changes; in the case where the display content changes, recognizing the changed display content to obtain screen content recognition results.

5. The method of claim 4, wherein, The recognizing of the changed display content to obtain screen content recognition results comprises: obtaining a target image frame displayed by the conference interface; wherein the target image frame is an image frame other than a first image frame among each image frame displayed by the conference interface; comparing the target image frame with a previous image frame of the target image frame to obtain a first type of text region in the target image frame that changes compared with the previous image frame; performing text recognition on the first type of text region to obtain a text recognition result of the first type of text region; obtaining screen content recognition results according to the text recognition result.

6. The method of claim 5, wherein, The method further comprises: if a second type of text region that does not change compared with the previous image frame exists in the target image frame, taking a text recognition result of the second type of text region in the previous image frame as a text recognition result of the second type of text region in the target image frame.

7. The method of claim 5, wherein, The obtaining of screen content recognition results according to the text recognition result comprises: performing layout analysis on the target image frame to obtain a plurality of block regions; determining entity relationships between text entities in each block region according to the text recognition result; for each block region, splicing text entities with entity relationships according to the entity relationships between the text entities in the block region to obtain text structured content of the target image frame, and taking the text structured content as the screen content recognition result.

8. The method according to any one of claims 1 to 7, characterized in that, The conference summary large model is trained based on the following manner: obtaining original training data without annotation; obtain annotated target training data, wherein the target training data is annotated with instructions generated based on the target training data; perform pre-training based on the original training data and the target training data to obtain the conference summary large model.

9. The method of claim 8, wherein, The obtaining annotated target training data comprises: generating instructions based on the instruction generation model to obtain an effective instruction data pair set from the original training data, wherein the effective instruction data pair set comprises a plurality of effective instruction data pairs, each of which comprises candidate training data and an effective instruction generated based on the candidate training data, and the candidate training data is training data in the original training data that can generate an effective instruction; selecting from the plurality of effective instruction data pairs to obtain the annotated target training data.

10. The method of claim 9, wherein, The selecting from the plurality of effective instruction data pairs to obtain the annotated target training data comprises: scoring the quality of the plurality of effective instruction data pairs to obtain a quality score of each effective instruction data pair; selecting target instruction data pairs that meet a preset quality requirement from the plurality of effective instruction data pairs based on the quality score, and taking the target instruction data pairs as the annotated target training data.

11. The method of claim 1, wherein, The display content of the conference interface includes annotation content, and the screen content recognition result includes annotation content recognition result; The identifying the display content of the conference interface corresponding to the target conference to obtain a screen content recognition result comprises: identifying the annotation content of the conference interface corresponding to the target conference to obtain an annotation content recognition result.

12. The method of claim 11, wherein, The annotation content includes text type annotation content and / or non-text type annotation content; The identifying the annotation content to obtain an annotation content recognition result comprises: identifying the text type annotation content to obtain a text annotation recognition result; and / or, identifying the non-text type annotation content to obtain a non-text annotation recognition result.

13. The method of claim 12, wherein, The identifying the non-text type annotation content to obtain a non-text annotation recognition result comprises: obtaining annotation trajectory features of the non-text type annotation content; determining a target trajectory feature matching the annotation trajectory features from trajectory features in a trajectory feature library, wherein each trajectory feature in the trajectory feature library corresponds to a trajectory recognition result represented by the trajectory feature; taking a trajectory recognition result corresponding to the target trajectory feature as the non-text annotation recognition result.

14. The method of claim 11, wherein, The performing time axis alignment based on the speech recognition result and the screen content recognition result to obtain aligned multi-source input data comprises: determining the actual meaning represented by the annotation recognition result in the target conference; performing time axis alignment based on the speech recognition result and the actual meaning represented by the annotation recognition result in the target conference to obtain aligned multi-source input data.

15. A meeting summary apparatus characterized by comprising: The apparatus comprises: a conference interface recording module configured to display a conference interface of a target conference and start recording the conference interface of the target conference; The voice recognition module is configured to recognize conference voice of the target conference to obtain a voice recognition result. The screen content recognition module is configured to recognize display content of a conference interface corresponding to the target conference to obtain a screen content recognition result. The timeline alignment module is configured to perform timeline alignment according to the voice recognition result and the screen content recognition result to obtain aligned multi-source input data. The conference summary module is configured to input the multi-source input data into a conference summary large model to output a summary result of the target conference.

16. An electronic device, comprising: The electronic device comprises: a memory configured to store executable program code; a processor configured to call and run the executable program code from the memory, so that the electronic device executes the method according to any one of claims 1 to 14. The computer readable storage medium stores a computer program, when the computer program is executed, the method according to any one of claims 1 to 14 is realized.

17. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, when the computer program is executed, the method according to any one of claims 1 to 14 is realized.

Images