Meeting summary generation method and device based on large language model, medium and product

Abstract

The invention discloses a conference summary generation method and device based on a large language model, a medium and a product. The conference summary generation method based on the large language model comprises the steps that audio data of conference participants are acquired, the audio data are converted into a speaking text according to a voice conversion model, and the speaking text comprises speaking content of the conference participants; face images of the conference participants are recognized, face matching is conducted on the face images through a face database, and identity information of the conference participants is obtained; obtaining a timestamp according to the speaking time record of the conference participant, associating the speaking content with the identity information according to the timestamp, and synthesizing a conference speaking record; and inputting the conference speaking record into the large language model to generate a conference summary.

Description

Technical Field

[0001] The present application relates to the technical field of large language models, and in particular to a method, device, medium and product for generating meeting minutes based on a large language model. Background Art

[0002] Most companies hold meetings of varying sizes in their daily work. In order to facilitate reporting to superiors or conveying and distributing meeting content to relevant personnel, a meeting minutes is usually required after the meeting. A qualified meeting minutes needs to record the process, basic content, and important matters of the meeting, and also summarize the spirit and core content of the meeting. At the same time, it is required to be accurate and well-structured.

[0003] Traditional meeting minutes require a dedicated person to take minutes. However, the standards for these minutes have high requirements for the person taking minutes, especially when the meeting content involves professional content or when the participants use multiple languages. This will place higher requirements on the person taking minutes. Cultivating such meeting minutes skills requires resources, and setting up a dedicated person to take minutes is a bit of a luxury for human resources.

[0004] In order to solve the above problems, the existing technology obtains the information data corresponding to the list of participants when creating a meeting, and pre-processes the list of participants and the corresponding information data; identifies the speakers entering the meeting based on voice recognition, face recognition and pre-processed data, determines the identity information of the speakers, and dynamically synchronizes the speakers' speech content to the screen and the conference system in real time. Record the text corresponding to the speaker's speech content, extract keywords, and output the corresponding meeting minutes.

[0005] The above method can improve the efficiency and accuracy of meeting minutes by using AI voice recognition and outputting the corresponding meeting minutes in combination with the speaker's identity. However, the above algorithm only transcribes the speaker's speech from voice into text and extracts keywords to generate meeting minutes. This results in a need to pre-define a template for each type of meeting and fill in keywords. The generated meeting minutes are not accurate, and each type of meeting has a template, which leads to a lack of generalization ability and difficulty in adapting to diverse meeting scenarios.

[0006] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0007] The main purpose of this application is to provide a method, device, medium and product for generating meeting minutes based on a large language model, aiming to solve the technical problems that the existing solutions generate meeting minutes based on keywords of speech transcription text, the generated minutes content is not accurate enough, and it is difficult to adapt to various meeting scenarios.

[0008] To achieve the above objectives, this application proposes a method for generating meeting minutes based on a large language model, including:

[0009] Acquire audio data of speakers among the participants, and convert the audio data into speech text according to the speech conversion model, wherein the speech text includes the speech content of the speaker;

[0010] Identify the speaker's face image, use the face database to perform face matching on the face image, and obtain the speaker's identity information;

[0011] Obtain a timestamp based on the speaker's speech time record, associate the speech content and identity information based on the timestamp, and synthesize the meeting speech record;

[0012] Input the meeting speech records into the large language model to generate meeting minutes.

[0013] Preferably, the method for generating meeting minutes, inputting the meeting speech record into the large language model, and generating the meeting minutes comprises:

[0014] Get text files related to the meeting;

[0015] Perform format conversion and semantic extraction on text files to obtain semantic text files;

[0016] The semantic text file and the meeting speech record are input into the large language model to generate the meeting minutes.

[0017] Preferably, the method for generating meeting minutes, the step of obtaining audio data of speakers among all meeting participants, and converting the audio data into speech text according to the speech conversion model, comprises:

[0018] Obtain video images of all participants in the meeting and obtain the location information of the speaker based on the video image positioning;

[0019] According to the speaker's location information, select the audio receiving device closest to the speaker in the meeting as the audio receiving source;

[0020] Perform signal processing on the audio data collected by the sound source;

[0021] The speech conversion model is used to perform feature extraction and text matching on the audio data after signal processing to obtain the speech text.

[0022] Preferably, the method for generating meeting minutes, the step of identifying the face image of the speaker, performing face matching on the face image according to the face database, and obtaining the speaker identity of the speaker, comprises:

[0023] Use video recording equipment to capture video images of all participants in the meeting;

[0024] For each participant, multiple frames of face images are extracted from the video images;

[0025] Use deep learning models to detect facial feature points on face images and obtain language-related facial feature points;

[0026] Calculate the distance vectors of facial feature points in multiple frames of face images, and use the distance vectors to identify the speaking actions of the participants;

[0027] Combining the speaking actions and audio data of the participants, locate the speaker among all the participants;

[0028] The speaker's face image is matched with the face images in the face database to obtain the speaker's identity information.

[0029] Preferably, in the above-mentioned method for generating meeting minutes, the steps of obtaining a timestamp according to the speaker's speech time record, associating the speech content and identity information according to the timestamp, and synthesizing the meeting speech record include:

[0030] Starting from the recognition of the speaker's face image, the speaker's speech start time is recorded to generate a timestamp;

[0031] Associate the timestamp, speech content and identity information to obtain the speaker's speech record;

[0032] The continuous speech records of the same speaker are concatenated and merged to obtain a merged conference speech record, wherein the timestamp of the merged conference speech record is the timestamp corresponding to the earliest speech record.

[0033] Preferably, in the above-mentioned method for generating meeting minutes, the step of inputting the meeting speech record into the large language model to generate the meeting minutes includes:

[0034] Use conference-related text concepts as semantic feature vectors to build a semantic representation space;

[0035] Use the semantic representation space to perform machine learning on the speech content in the conference speech record, match the semantic feature vector, and obtain the semantic features corresponding to the speech content;

[0036] The semantic features are merged into the meeting speech records and input into the large language model to generate meeting minutes.

[0037] Preferably, in the above-mentioned method for generating meeting minutes, the step of inputting the meeting speech record into the large language model and generating the meeting minutes includes:

[0038] Get the meeting type and historical speech records of historical meetings;

[0039] For the same type of meeting, historical meeting samples are obtained from historical speech records, wherein the historical meeting samples include historical topic records and historical program records of the same topic;

[0040] Input historical topic records into the large language model, use historical solution records as a validation set to supervise the large language model and generate a knowledge graph model;

[0041] Input the meeting speech records into the knowledge graph model to obtain the meeting minutes.

[0042] In addition, to achieve the above-mentioned purpose of the present application, the present application also provides a meeting minutes generation device based on a large language model, the device including: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the meeting minutes generation method based on a large language model as provided in any of the above-mentioned technical solutions.

[0043] In addition, to achieve the above-mentioned purpose of the present application, the present application also provides a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the meeting minutes generation method based on a large language model as provided in any of the above technical solutions are implemented.

[0044] In addition, to achieve the above-mentioned purpose of the present application, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the meeting minutes generation method based on a large language model provided by any of the above-mentioned technical solutions.

[0045] One or more technical solutions proposed in this application have at least the following technical effects:

[0046] By acquiring the audio data of the participants, converting the audio data into speech text, and using the face database to perform face matching on the face images, the identity information of the participants can be obtained; in this way, the timestamp can be used to associate the above speech content and identity information to synthesize the meeting speech record; finally, the meeting minutes are generated through the large language model. The above method only needs to input the meeting speech record into the large language model to automatically and efficiently generate high-quality meeting minutes. On the one hand, there is no need to set up a special meeting recorder, reducing the waste of human resources; on the other hand, this solution generates meeting minutes based on the large language model, the generated content is accurate, and the generalization ability is strong, which can adapt to a variety of meeting scenarios, and solves the problem that the existing technology generates meeting minutes that are not accurate enough, lacks generalization ability, and is difficult to adapt to the needs of a variety of meeting scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0049] Figure 1 A schematic diagram of a conference room layout architecture for generating meeting minutes provided in an embodiment of the present application;

[0050] Figure 2 A flowchart of a first method for generating meeting minutes based on a large speech model provided in an embodiment of the present application;

[0051] Figure 3 A flowchart of a second method for generating meeting minutes based on a large language model provided in an embodiment of the present application;

[0052] Figure 4 for Figure 3 A flowchart of a method for converting audio data into speech text provided by the illustrated embodiment;

[0053] Figure 5 for Figure 3 A flowchart of a method for matching faces and obtaining speaker identities provided by the illustrated embodiment;

[0054] Figure 6 yes Figure 3 A schematic flow chart of a method for synthesizing conference speech records provided by the illustrated embodiment;

[0055] Figure 7 It is a flowchart of the first method for generating meeting minutes provided in the embodiment of the present application;

[0056] Figure 8 It is a flowchart of a second method for generating meeting minutes provided in an embodiment of the present application;

[0057] Fig. 9 It is a flowchart of a third method for generating meeting minutes provided in an embodiment of the present application;

[0058] Fig.10 It is a flowchart of a third method for generating meeting minutes based on a large language model provided in an embodiment of the present application;

[0059] Fig.11 It is a structural diagram of a meeting minutes provided in an embodiment of the present application;

[0060] Fig.12 It is a flowchart of a method for generating meeting minutes based on a large language model provided in an embodiment of the present application.

[0061] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0062] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0063] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0064] The existing technology has the following technical problems: the existing technical solution for creating meeting minutes usually outputs the corresponding meeting minutes through AI voice recognition and combining the speaker's identity. This method only transcribes the speaker's speech content from voice into text, extracts keywords to generate meeting minutes, which results in the need to pre-define a template for each type of meeting and fill in keywords; which in turn results in the generated meeting minutes being inaccurate, and each type of meeting has a template, resulting in a lack of generalization ability and difficulty in adapting to diverse meeting scenarios.

[0065] To solve the above technical problems, this application provides a solution based on a large language model, combined with technologies such as speaker face detection, face recognition and voice conversion, which can automatically generate meeting minutes through multimodal inputs such as video, voice, and text. On the one hand, there is no need to specially set up the role of a secretary; on the other hand, the generated minutes are accurate enough and have strong generalization capabilities, and can be applied to a variety of meeting scenario requirements.

[0066] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions, a computer-readable storage medium, etc.

[0067] In order to better understand the above technical solution, the exemplary embodiments of the present application will be described in more detail with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments shown here. On the contrary, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0068] To achieve the above technical objectives, see Figure 1 , Figure 1A schematic diagram of a conference room layout architecture for generating meeting minutes is provided in an embodiment of the present application. Figure 1 As shown, the layout architecture of the conference room includes: two cameras 1 and multiple microphones 2. Among them, the two cameras 1 are respectively facing a row of participants for real-time recording, and the video data obtained is analyzed in real time by the background computer. Multiple microphones 2, one for each seat, each microphone 2 respectively picks up the speech of the participant at the corresponding position, and the audio data is also converted into voice by the background computer. Then, the video data obtained by the camera is analyzed by face detection and face recognition technology to obtain the identity information of the speaker. Combined with the recorded timestamp and the text obtained by voice conversion of the speech content, a complete meeting speech record can be synthesized. Finally, the meeting speech record is summarized using a large language model to obtain the meeting minutes. If there is a presentation, it can also be input into the large language model as auxiliary information. Through the above scheme, high-quality meeting minutes can be automatically and efficiently generated after each meeting.

[0069] To achieve the above purpose, see Figure 2 , Figure 2 A flowchart of a method for generating meeting minutes based on a large speech model provided in an embodiment of the present application. The method for generating meeting minutes corresponds to Figure 1 The conference room layout architecture is shown in Figure 1. Figure 2 As shown, combined Figure 1 The conference room layout shown in the figure, the method for generating meeting minutes includes the following steps:

[0070] A101: Face detection and face recognition; the camera obtains video data of a face, then detects facial features, compares the facial features with face images in a face database, and performs face recognition to obtain the identity of the speaker.

[0071] A102: Voice conversion; each microphone picks up the speech of the attendee at the corresponding position, and the audio data is also converted into voice by the background computer.

[0072] A103: Convert PPT to PDF: Convert presentations from PPT format to PDF, thereby obtaining readable documents in a format that can be processed by the large language model.

[0073] A104: After combining, the speaker identity and speech content are combined and sorted by time to obtain the meeting speech record.

[0074] A105: A large speech model inputs the above-mentioned meeting speech records and readable documents into the large language model to finally generate meeting minutes.

[0075] It should be noted that the multimodal data input into the large language model mainly consists of the following parts:

[0076] Meeting speech record: Record the speech process of the participants in the meeting in text form. The content of the meeting speech record includes the timestamp, speaker identity and corresponding speech content. The schematic diagram of the meeting speech record is as follows Fig.11 As shown, it is used to provide analysis and summary to the large language model to generate meeting minutes.

[0077] The meeting speech record includes the timestamp, speaker identity and corresponding speech content; it should be noted that:

[0078] Timestamp: The time when the participant started speaking. It is recorded simultaneously during face recognition and is used to sort the speech content in chronological order.

[0079] Speaker identity: used to distinguish the speeches of different participants. Use the images taken by the camera for face recognition and match them with the records in the face database to obtain the speaker's identity information.

[0080] Speech content: Record the speech content of different participants during the meeting. Use a microphone to record in real time during the meeting, and then convert the audio data into text through the speech conversion model.

[0081] In addition, the above-mentioned presentation: the presentation document used by the host during the meeting is converted into a format suitable for reading by the large language model and then input as auxiliary information for the meeting speech record.

[0082] Specifically, to achieve the above purpose, see Figure 3 , Figure 3 A flowchart of a method for generating meeting minutes based on a large language model is provided. The method for generating meeting minutes includes:

[0083] S110: Acquire audio data of speakers among the participants, and convert the audio data into speech text according to the speech conversion model, wherein the speech text includes the speech content of the speaker. Figure 1 As shown, cameras are arranged in the conference room, and each seat corresponds to a microphone, so that each microphone can pick up the audio of the attendee at that seat and obtain the audio data of the attendee. The speech conversion model in the embodiment of the present application can use the NLP algorithm to convert the speech semantics. Of course, the speech conversion software can also be directly used to convert the speech text. In the above manner, a text format that can be processed by a large model can be obtained.

[0084] Specifically, as a preferred embodiment, Figure 4As shown, in the above-mentioned method for generating meeting minutes, step S110: obtaining audio data of the participants, and converting the audio data into speech text according to the speech conversion model, includes:

[0085] S111: Obtain video images of all participants in the conference, and obtain positioning information of speakers among all participants based on video image positioning; Figure 1 In the conference room layout shown in the figure, the two cameras in the conference room are facing a row of participants respectively, and can record in real time to obtain video images of all participants. In this way, by detecting the video image, such as the changes in mouth shape and the swing amplitude of body movements in multiple consecutive frames of images, the positioning information of the speaker among all participants can be obtained.

[0086] S112: According to the location information of the speaker, select the sound receiving device closest to the speaker in the conference as the sound receiving source; Figure 1 As can be seen from the conference room layout shown, each seat in the conference room corresponds to a microphone, so that the microphone acts as a sound receiving device. After the position of the speaker is determined, the microphone acts as a sound receiving source to extract the speaker's audio data.

[0087] S113: performing signal processing on the audio data picked up by the sound receiving source; the signal processing here includes signal noise removal, signal enhancement, and conversion from analog signal to digital signal.

[0088] S114: Use the speech conversion model to extract features and match text on the audio data after signal processing to obtain speech text. For example, the NLP algorithm extracts text that corresponds to the speech, and then uses the semantic conversion model and other methods to perform semantic processing and adjust the logical order of the text in combination with the context, thereby obtaining the speech text in text form.

[0089] The technical solution provided in the embodiment of the present application combines the positioning information of the speaker among all the participants with the video image to determine the speaker's position, then selects the closest sound receiving device to the speaker as the sound receiving source to capture the audio data, then performs signal processing on the audio data, and uses the speech conversion model to perform feature extraction and text matching on the audio data after signal processing, so as to obtain the speech text. In this way, the speaker can be accurately located, the speaker's audio data can be efficiently and accurately converted into speech, and a speech text that highly reflects the speaker's speech content can be obtained.

[0090] Figure 3 The technical solution provided by the illustrated embodiment, after converting the audio data into speech text according to the speech conversion model, further includes the following steps:

[0091] S120: Identify the speaker's face image, use the face database to perform face matching on the face image, and obtain the speaker's identity information. Figure 2 As can be seen from the process shown, the face database stores the face images and identity information of all participants. By using the face database to match the face images of all participants, the identity information of the speaker can be obtained.

[0092] Specifically, as a preferred embodiment, Figure 5 As shown, the above-mentioned method for generating meeting minutes, step S120: identifying the face image of the speaker, performing face matching on the face image according to the face database, and obtaining the speaker identity of the speaker, includes:

[0093] S121: Using a video recording device to capture video images of all participants in the meeting;

[0094] S122: For each participant, extract multiple frames of face images from the video image; the continuous multiple frames of face images can reflect the details such as facial features and facial movements of the participant over time.

[0095] S123: Use a deep learning model to detect facial feature points on the face image to obtain language-related facial feature points; here, a graph model can be used to detect facial feature points on the face image, identify facial feature points, and specifically mark multiple facial feature points related to facial movements and language. In multiple consecutive frames of face images, if a participant speaks, the movement amplitude of the multiple facial feature points will change dramatically. Here, a deep learning model can be used to learn and verify the facial feature points, thereby obtaining the distance vector of the same facial feature points in multiple consecutive frames of face images.

[0096] S124: Calculate the distance vector of facial feature points in multiple frames of facial images, and use the distance vector to identify the speaking actions of the participants; if the participant speaks, the movement amplitude of the facial feature points related to the language in the continuous multiple frames of facial images will become larger, and its distance vector will change greatly, so that the speaking actions of the participants, such as the facial expressions when speaking, can be identified through the change amplitude of the distance vector.

[0097] S125: Combining the speaking actions and audio data of the participants, locating the speaker among all the participants; combining the speaking actions and audio data of the above participants, for example, comparing the audio data with the sound characteristics of each participant in the audio database, or after determining the position of the speaker through the speaking action, using the microphone at that position to collect the sound amplitude of the audio data, the speaker among all the participants can be accurately located.

[0098] S126: Perform face matching on the speaker's face image with the face images in the face database to obtain the speaker's identity information. By matching the speaker's face image with the face images in the face database, the speaker's identity can be accurately determined and then sent to the large language model to generate meeting minutes.

[0099] The technical solution provided by the embodiment of the present application uses a video recording device, such as a camera, to capture video images of all participants in the meeting, and then extracts multiple frames of facial images from the video images for each participant. It is then possible to use a deep learning model to detect and obtain voice-related facial feature points, and then calculate the distance vector between the facial feature points in the above-mentioned multiple facial images, identify the speaking actions of the participants, and further combine the above-mentioned speaking actions with audio data to accurately locate the position of the speaker. Then, the speaker's facial image is matched with the facial image in the facial database to obtain accurate speaker identity information.

[0100] Figure 3 The technical solution provided by the illustrated embodiment, after performing face matching on the face image using the face database to obtain the identity information of the attendees, further includes:

[0101] S130: Obtain a timestamp based on the speaker's speech time record, associate the speech content and identity information according to the timestamp, and synthesize the meeting speech record. The timestamp is the time when the participant starts speaking. It can be recorded synchronously when performing face recognition, and the timestamp is used to sort the speech content in chronological order. The above timestamp can be used to associate the speech content and identity information, so as to uniformly generate the corresponding speech file and input it into the large language model.

[0102] Specifically, as a preferred embodiment, Figure 6 As shown, in the above-mentioned method for generating meeting minutes, step S130: obtaining a timestamp according to the speaker's speech time record, associating the speech content and identity information according to the timestamp, and synthesizing the meeting speech record, specifically includes:

[0103] S131: Starting from the recognition of the speaker's facial image, the speaker's speech start time is recorded to generate a timestamp.

[0104] S132: Associating the timestamp, speech content and identity information to obtain the speaker's speech record.

[0105] S133: The continuous speech records of the same speaker are spliced ​​and merged to obtain the merged meeting speech record, wherein the timestamp of the merged meeting speech record is the timestamp corresponding to the earliest speech record. The continuous speech records of the same speaker may be interrupted in the middle, and there may be problems such as interval time. In this way, the content of multiple continuous speech records of the same speaker are spliced ​​and merged to obtain the meeting speech record of the same speaker. Assist in determining the content and purpose of the speaker's speech at the meeting.

[0106] The technical solution provided by the embodiment of the present application records the start time of the speaker's speech by identifying the speaker. From the start time of the speech, it is possible to associate the speech content with the speaker's identity information, determine the speaker's speech content and speech duration, and generate the speaker's speech record. Further content splicing and record merging of continuous speech records can obtain the speaker's conference speech record. In this way, the speaker can be classified according to his / her identity to obtain speech records of different participants.

[0107] Figure 3 The technical solution provided by the embodiment shown in the figure further includes: after associating the speech content and identity information according to the timestamp to synthesize the conference speech record

[0108] S140: Input the meeting speech record into the large language model to generate meeting minutes.

[0109] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data that can generate natural language text or understand the meaning of language text. Large language models have superb generalization capabilities and can extract, summarize, and expand information content. For meeting minutes, inputting the meeting speech records into the large language model can generate meeting minutes with accurate content that adapts to the meeting scene and meeting theme.

[0110] Specifically, as a preferred embodiment, Figure 7 As shown, the above step S140: inputting the meeting speech record into the large language model to generate the meeting minutes specifically includes:

[0111] S141: Obtaining text files related to the meeting; the text files may include text about the subject of the meeting, as well as presentations used by the host, etc. By obtaining the text files and inputting them into the large language model, the meeting minutes can be generated.

[0112] S142: Perform format conversion and semantic extraction on the text file to obtain a semantic text file; here, a semantic model can be used to perform format conversion and semantic extraction on the text file, on the one hand to obtain a text format that can be processed by a large language model, and on the other hand to use the semantic model to perform semantic extraction on the text file, summarize the content, and generate the main content reflected by the text file, thereby obtaining a voice text file.

[0113] S143: Input the semantic text file and the meeting speech record into the large language model to generate the meeting minutes.

[0114] The technical solution provided in the embodiment of the present application obtains a semantic text file by performing format conversion and semantic extraction on a text file, and inputs the semantic text file together with the above-mentioned conference speech record into a large language model, which can further enrich the processing content of the large language model and obtain more accurate meeting minutes.

[0115] In addition, as a preferred embodiment, Figure 8 As shown, in the above-mentioned method for generating meeting minutes, step S140: inputting the meeting speech record into the large language model to generate the meeting minutes includes:

[0116] S145: Using the text concepts related to the meeting as semantic feature vectors, a semantic representation space is established. By using the text concepts as semantic feature vectors, a semantic expression composed of the text concepts related to the meeting can be obtained, i.e., the above-mentioned semantic feature vectors. As the content of the semantic representation space is gradually enriched, the content of various types of meetings can be reflected, and the texts required for various meetings can be represented.

[0117] S146: Using the semantic representation space to perform machine learning on the speech content in the conference speech record, matching the semantic feature vector, and obtaining the semantic features corresponding to the speech content. By performing machine learning on the speech content in the conference speech record, it is possible to match the above speech content with the semantic feature vector, accurately represent the semantics expressed by each sentence and each word in the speech content, and obtain the semantic features corresponding to the speech content.

[0118] S147: Merge the semantic features into the meeting speech record and input them into the big language model to generate meeting minutes. By merging the semantic features into the meeting speech record and inputting them into the big language model, the big language model can generate more accurate meeting minutes that are in line with the meeting theme.

[0119] The technical solution provided in the embodiment of the present application uses conference-related text concepts as semantic feature vectors to establish a semantic representation space, and then uses the semantic representation space to perform subsequent learning on the speech content, so as to match the semantic feature vectors in the semantic representation space to obtain semantic features, and then merge the semantic features into the conference speech records and input them into the large language model, so as to obtain more accurate meeting minutes with semantically consistent content.

[0120] In addition, as a preferred embodiment, Fig. 9 As shown, in the above-mentioned method for generating meeting minutes, the meeting speech record is input into the large language model, and the steps of generating meeting minutes include:

[0121] S148: Obtain the meeting type and historical speech records of historical meetings;

[0122] S149: For the same meeting type, obtaining historical meeting samples from historical speech records, wherein the historical meeting samples include historical topic records and historical program records of the same topic;

[0123] S1410: Input the historical topic records into the large language model, use the historical solution records as a verification set to perform supervised learning on the large language model, and generate a knowledge graph model;

[0124] S1411: Input the meeting speech records into the knowledge graph model to obtain the meeting minutes.

[0125] The knowledge graph model here is generated through the meeting types and historical speech records of historical meetings. For the same type of meetings, historical meeting samples are extracted from the historical speech records to obtain historical subject records and historical plan records, which are input into the large language model to obtain the knowledge graph model of the same type of meeting. This knowledge graph can reflect the knowledge content and topic summaries involved in the same type of meetings. In this way, by inputting the meeting speech records into the knowledge graph model, more professional meeting minutes can be obtained.

[0126] In summary, the technical solution provided by the above-mentioned embodiments of the present application obtains the audio data of the participants, converts the audio data into speech text, and obtains the identity information of the participants by performing face matching on the face images using the face database; in this way, the above-mentioned speech content and identity information can be associated by using the timestamp to synthesize the meeting speech record; finally, the meeting minutes are generated through the large language model. The above-mentioned method only needs to input the meeting speech record into the large language model to automatically and efficiently generate high-quality meeting minutes. On the one hand, there is no need to set up a special meeting recorder, reducing the waste of human resources; on the other hand, this solution generates meeting minutes based on the large language model, and the generated content is accurate and has strong generalization ability. It can adapt to a variety of meeting scenarios, which solves the problem that the content of meeting minutes generated by the existing technology is not accurate enough, lacks generalization ability, and is difficult to adapt to the needs of a variety of meeting scenarios.

[0127] Also, see Fig.10 , Fig.10 A flowchart of a method for generating meeting minutes based on a large language model provided in an embodiment of the present application. Figure 1 As can be seen from the conference room architecture shown, the meeting minutes generation method includes:

[0128] B101: Real-time video: Cameras are distributed in the corners of the conference room and can record real-time video of two rows of participants.

[0129] B102: Extract frames from the video recorded by the camera to obtain multiple consecutive frames.

[0130] B103: Record the timestamp for each frame.

[0131] B104: Detect facial feature points for each frame. Here, distance vectors can be detected for facial feature points in multiple consecutive frames to obtain feature points related to language. B105: Identify the action of the speaker. The speaker's facial expressions and body movements will change when speaking. After detecting facial feature points, the action of the speaker can be identified.

[0132] B106: Positioning the speaker. Here, the speaker can be positioned in combination with the voice data extracted by the camera and microphone.

[0133] B107: Face recognition confirms the speaker's identity. After obtaining the face image, the speaker's identity can be confirmed by combining the face-identity information in the face database.

[0134] B108: Real-time sound recording. This allows you to use the microphone closest to the speaker to record the sound in real time.

[0135] B109: Noise reduction processing: In order to improve the audio quality, multiple sound sources can be used to input audio data, and then the audio data can be subjected to noise reduction processing to obtain denoised audio information, thereby further accurately determining the position of the speaker.

[0136] B110: Select the nearest sound source input. After determining the speaker's position, you can select the nearest sound source.

[0137] B111: Perform voice conversion on the audio data input by a single sound receiving source to obtain speech text.

[0138] B112: Synthesize conference speech records; by combining the above speaker identities and speech texts, the conference speech records can be synthesized.

[0139] B113: Input the big language model; input the meeting speech records into the big language model to generate meeting minutes.

[0140] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the method for generating meeting minutes based on a large language model of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0141] In addition, to achieve the above-mentioned purpose of this application, see Fig.12 The present application also provides a device for generating meeting minutes based on a large language model, the device for generating meeting minutes based on a large language model comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating meeting minutes based on a large language model of any of the above-mentioned embodiments.

[0142] Reference below Fig.12 , which shows a schematic diagram of the structure of a large language model-based conference minutes generation device suitable for implementing the embodiment of the present application. The large language model-based conference minutes generation device in the embodiment of the present application can include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.12 The meeting minutes generation device based on the large language model shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0143] like Fig.12 As shown, the conference minutes generation device based on the large language model can include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the conference minutes generation device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can enable the meeting minutes generation device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a meeting minutes generation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0144] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0145] Compared with the prior art, the beneficial effects of the meeting minutes generation device based on a large language model provided by the present application are the same as the beneficial effects of the meeting minutes generation method based on a large language model provided by the above-mentioned embodiment, and the other technical features in the meeting minutes generation device based on a large language model are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0146] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0147] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

[0148] In addition, to achieve the above-mentioned purpose of the present application, the present application also provides a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the meeting minutes generation method based on a large language model as provided in any of the above technical solutions are implemented.

[0149] The computer-readable storage medium provided in the present application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0150] Computer program code for performing the operations of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0151] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes also be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs a specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0152] The modules involved in the embodiments of the present application can be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0153] The readable storage medium provided in the present application is a computer-readable storage medium. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the meeting minutes generation method provided in the above embodiment, which will not be repeated here.

[0154] In addition, to achieve the above-mentioned purpose of the present application, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the method for generating meeting minutes based on a large language model provided in any of the above-mentioned embodiments are implemented. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the method for generating meeting minutes based on a large language model provided in the above-mentioned embodiments, and will not be repeated here.

[0155] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for generating meeting minutes based on a large language model, characterized in that: include: Acquire audio data of a speaker among all the participants of the meeting, and convert the audio data into speech text according to a speech conversion model, wherein the speech text includes the speech content of the speaker; Identify the face image of the speaker, perform face matching on the face image using a face database, and obtain the identity information of the speaker; Obtain a timestamp based on the speaker's speech time record, and use the timestamp to associate the speech content and identity information to synthesize the conference speech record; The meeting speech records are input into the large language model to generate meeting minutes.

2. The method according to claim 1, characterized in that The step of inputting the conference speech record into the large language model to generate the meeting minutes includes: Get text files related to the meeting; Performing format conversion and semantic extraction on the text file to obtain a semantic text file; The semantic text file and the conference speech record are input into the large language model together to generate the meeting minutes.

3. The method according to claim 1, characterized in that The step of obtaining audio data of speakers among all conference participants and converting the audio data into speech text according to the speech conversion model includes: Acquire video images of all participants in the meeting, and obtain location information of the speaker based on the video image positioning; According to the positioning information of the speaker, selecting a sound receiving device closest to the speaker in the conference as a sound receiving source; Performing signal processing on the audio data picked up by the sound receiving source; The speech conversion model is used to perform feature extraction and text matching on the audio data after signal processing to obtain the speech text.

4. The method according to claim 1, characterized in that The step of identifying the face image of the speaker, performing face matching on the face image using a face database, and obtaining the identity information of the speaker includes: Use video recording equipment to capture video images of all participants in the meeting; For each participant, extracting multiple frames of face images from the video image; Using a deep learning model to detect facial feature points on the face image to obtain language-related facial feature points; Calculating the distance vectors of the facial feature points in the multiple frames of facial images, and using the distance vectors to identify the speaking actions of the participants; Combining the speaking action of the participant and the audio data, locating the speaker among all the participants; The face image of the speaker is matched with the face images in the face database to obtain the identity information of the speaker.

5. The method according to claim 1, characterized in that The step of obtaining a timestamp according to the speaker's speech time record, and using the timestamp to associate the speech content and identity information to synthesize the conference speech record includes: Starting from recognizing the speaker's facial image, the speaker's speech start time is recorded to generate the timestamp; Associating the timestamp, speech content and identity information to obtain the speech record of the speaker; The continuous speech records of the same speaker are concatenated and merged to obtain a merged conference speech record, wherein the timestamp of the merged conference speech record is the timestamp corresponding to the earliest speech record.

6. The method according to claim 1, characterized in that The step of inputting the conference speech record into the large language model to generate the meeting minutes includes: Use conference-related text concepts as semantic feature vectors to build a semantic representation space; Using the semantic representation space to perform machine learning on the speech content in the conference speech record, matching the semantic feature vector, and obtaining the semantic features corresponding to the speech content; The semantic features are merged into the conference speech record and input into the large language model to generate the conference minutes.

7. The method according to claim 1, characterized in that The step of inputting the conference speech record into the large language model to generate the meeting minutes includes: Get the meeting type and historical speech records of historical meetings; For the same meeting type, obtaining historical meeting samples from the historical speech records, wherein the historical meeting samples include historical topic records and historical program records of the same topic; Inputting the historical topic records into the large language model, using the historical solution records as a validation set to perform supervised learning on the large language model, and generating a knowledge graph model; The meeting speech records are input into the knowledge graph model to obtain the meeting minutes.

8. A device for generating meeting minutes based on a large language model, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for generating meeting minutes based on a large language model as described in any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the method for generating meeting minutes based on a large language model as described in any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method for generating meeting minutes based on a large language model as claimed in any one of claims 1 to 7 are implemented.

Images