Meeting minutes processing method, equipment and storage medium

Abstract

The present application discloses a method, device and storage medium for processing meeting minutes, relates to the field of voice conference technology, and discloses a method for processing meeting minutes, including: sending a meeting request to a server to create a meeting, collecting audio frames through a conference recording component and sending them to the server; receiving meeting minutes information obtained by the server based on the audio frames, ASR technology and a pre-trained large language model; obtaining a user's meeting query request; displaying matching content of the meeting minutes information according to the meeting query request; realizing real-time display of meeting content, being able to summarize the meeting content from a semantic level, and timely pushing meeting completion information, so as to facilitate users to understand the meeting content in real time and efficiently.

Description

Technical Field

[0001] The present application relates to the field of voice conferencing technology, and in particular to a conference minutes processing method, device and storage medium. Background Art

[0002] Meeting minutes are a document that records the process and results of a meeting. They play an important role and provide important information for meeting participants. Notes record meeting resolutions and action plans. In modern society, meetings are an important means for companies and organizations to make decisions and cooperate. With the increasing frequency of corporate meetings and remote collaboration, the generation and management of meeting minutes has become an important need. Traditional methods of generating meeting minutes have many shortcomings, such as the recording terminal equipment is not universal, real-time recording cannot display the meeting content in real time, the meeting generated content is simple, and the completion information cannot be pushed in time after the meeting is completed. These problems seriously affect the efficiency and accuracy of meeting minutes, and it is difficult to meet the needs of modern enterprises for efficient and accurate meeting minutes.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a meeting minutes processing method, device and storage medium, aiming to solve the technical problems that traditional real-time recording of meeting minutes cannot display the meeting content in real time and cannot promptly push user-concerned information after the meeting is processed.

[0005] To achieve the above purpose, the present application proposes a method for processing meeting minutes, which is applied to a client, and the method includes:

[0006] Send a meeting request to the server to create a meeting, and use the meeting recording component to collect audio frames and send them to the server;

[0007] Receiving the meeting minutes information obtained by the server based on the audio frame, ASR technology and a pre-trained large language model;

[0008] Receive conference query requests from users;

[0009] The matching content of the meeting minutes information is displayed according to the meeting query request.

[0010] In one embodiment, the step of sending a meeting request to a server to create a meeting and collecting audio frames through a meeting recording component and sending them to the server includes:

[0011] Send a user message to the server to create a conference ID;

[0012] Receive the conference creation success flag and conference ID from the server, and establish a connection with the server through the STOMP protocol;

[0013] Request user authorization to access general recording devices;

[0014] Initialize the recording function, record the user ID of the conference speaker and collect audio data;

[0015] The collected audio data is converted into audio frames through the conference recording component and sent to the server.

[0016] In one embodiment, the receiving server obtains the meeting minutes information based on the audio frame, ASR technology and a pre-trained large language model, including:

[0017] When the audio frame is a real-time audio frame, receiving real-time meeting minutes information obtained by the server based on the real-time audio frame and the ASR real-time service;

[0018] When the audio frame is an offline audio frame after the end of the meeting, the receiving server obtains complete meeting minutes information based on the offline audio frame, the ASR offline service and the pre-trained large language model.

[0019] In one embodiment, displaying matching content of the meeting minutes information according to the meeting query request includes:

[0020] Match the user subscription message, topic or user ID with the received meeting minutes information and display it on the user interface;

[0021] When the complete meeting minutes information is received, the user is notified of the meeting completion.

[0022] In addition, to achieve the above purpose, the present application also proposes a meeting minutes processing method, which is applied to the server, and the method includes:

[0023] Receive audio frames sent by the client;

[0024] Meeting minutes information obtained through the audio frames, ASR technology and a pre-trained large language model;

[0025] The matching content of the meeting minutes information is sent to the client according to the user's query request.

[0026] In one embodiment, after the step of receiving the audio frame sent by the client, the step further includes:

[0027] Buffer the received audio frames into the server's memory or temporary storage;

[0028] Pre-process the cached audio frames for noise reduction and compression.

[0029] In one embodiment, the step of obtaining the meeting minutes information by using the audio frame, ASR technology and a pre-trained large language model includes:

[0030] When the audio frame is a real-time audio frame;

[0031] Convert the preprocessed real-time audio frames into feature vectors;

[0032] The feature vector is subjected to speech recognition through ASR real-time service technology to generate real-time meeting minutes information.

[0033] In one embodiment, the step of obtaining the meeting minutes information by using the audio frame, ASR technology and a pre-trained large language model further includes:

[0034] When the audio frame is an offline audio frame after the conference ends;

[0035] The pre-processed offline audio frames are converted into offline texts through ASR offline service technology;

[0036] Load a pre-trained large language model;

[0037] Combining the offline text and the text or instruction input by the user into a complete prompt;

[0038] The prompt is sent to a pre-trained large language model, and complete meeting minutes information generated by semantic extraction of the large language model is received.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a meeting minutes processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the meeting minutes processing method described above.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the meeting minutes processing method described above are implemented.

[0041] One or more technical solutions proposed in this application have at least the following technical effects:

[0042] Send a meeting request to the server to create a meeting, collect audio frames through the meeting recording component and send them to the server; receive the meeting minutes information obtained by the server based on the audio frames, ASR technology and a pre-trained large language model; receive the user's meeting query request; display the matching content of the meeting minutes information according to the meeting query request; solve the problem that real-time recording cannot display the meeting content in real time, the meeting generated content is simple, and the completion information cannot be pushed in time after the meeting processing is completed. Compared with the existing technology, it can realize real-time display of meeting content, summarize the meeting content from a semantic level, and push meeting completion information in time, so that users can understand the meeting content in real time and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 A flowchart diagram of the first embodiment of the method for processing meeting minutes of this application;

[0046] Figure 2 A flowchart diagram of the second embodiment of the method for processing meeting minutes of this application;

[0047] Figure 3 A flowchart diagram of the third embodiment of the method for processing meeting minutes of this application;

[0048] Figure 4 A flowchart diagram of the fourth embodiment of the method for processing meeting minutes of this application;

[0049] Figure 5 A flowchart diagram of the fifth embodiment of the method for processing meeting minutes provided in the present application embodiment;

[0050] Figure 6 A flowchart diagram of the sixth embodiment of the method for processing meeting minutes of the present application;

[0051] Figure 7 A flow chart of Embodiment 7 of the method for processing meeting minutes provided in the present application;

[0052] Figure 8 A brief schematic diagram of Embodiment 8 of the method for processing meeting minutes of the present application;

[0053] Fig. 9 A brief schematic diagram of Embodiment 9 of the method for processing meeting minutes of the present application;

[0054] Fig.10 This is a schematic diagram of the module structure of the meeting minutes processing device of the embodiment of the present application;

[0055] Fig.11 This is a schematic diagram of the device structure of the hardware operating environment involved in the meeting minutes processing method in this application.

[0056] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0057] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0058] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0059] In this embodiment, for ease of description, the following description is made with the client as the execution subject.

[0060] In the existing technology, the traditional method of generating meeting minutes has many shortcomings, such as the recording terminal equipment is not universal, the real-time recording cannot display the meeting content in real time, the generated meeting content is simple, and the completion information cannot be pushed in time after the meeting is completed. These problems seriously affect the efficiency and accuracy of meeting minutes, and it is difficult to meet the needs of modern enterprises for efficient and accurate meeting minutes.

[0061] The main solution of the embodiment of the present application is: applied to the client, sending a meeting request to the server to create a meeting, collecting audio frames through the conference recording component and sending them to the server; receiving the meeting minutes information obtained by the server based on the audio frames, ASR technology and a pre-trained large language model; receiving the user's meeting query request; displaying the matching content of the meeting minutes information according to the meeting query request; solving the problem that real-time recording cannot display the meeting content in real time, the meeting generated content is simple, and the completion information cannot be pushed in time after the meeting processing is completed. Compared with the existing technology, it realizes real-time display of meeting content, can summarize the meeting content from a semantic level, and push meeting completion information in time, so that users can understand the meeting content in real time and efficiently.

[0062] Based on this, the present application embodiment provides a method for processing meeting minutes, which is applied to the client. Figure 1 , Figure 1 This is a flowchart of the first embodiment of the method for processing meeting minutes of this application.

[0063] In this embodiment, the meeting minutes processing method includes steps S10 to S40:

[0064] Step S10: Send a meeting request to the server to create a meeting, and collect audio frames through the conference recording component and send them to the server.

[0065] It should be noted that the recording of the meeting minutes after the meeting is created can be based on a general recording device, such as an HTML5 recording component; the configuration of the general recording device should meet the requirements of the meeting minutes system, such as configuring the sampling rate, bit depth and number of channels of the recording device, and the audio quality can be ensured by setting the gain and noise suppression parameters of the recording device.

[0066] Specifically, you can use the WebRTC API or MediaDevices API to obtain the audio stream to ensure that the audio stream can be transmitted to the meeting minutes system in real time. In addition, extract real-time audio frames from the audio stream. Usually, audio data is extracted at fixed time intervals. You can use the MediaRecorder API or a custom audio processing module to split the audio stream into audio frames. Pre-process the extracted audio frames, such as noise reduction, gain adjustment, echo cancellation, etc., to improve audio quality.

[0067] Step S20: receiving the meeting minutes information obtained by the server based on the audio frame, ASR technology and the pre-trained large language model.

[0068] It should be noted that the client and the server can realize real-time transmission and reception of audio frames through protocols such as STMOP protocol, WebSocket or HTTP / 2; the real-time audio frames are sent to the server for caching and preprocessing, and the received audio frames can be stored in the buffer of the server to ensure the continuity and integrity of the audio data; the ASR technology is the abbreviation of "automatic speech recognition", which can analyze audio signals such as voice conversations, speeches, telephone recordings, etc., identify the voice content therein, and convert it into a readable text format; the pre-trained large language model mainly refers to a large language model (LLM), which is usually based on deep learning technology, such as the Transformer architecture, and can process and generate natural language text.

[0069] It can be provided by loading a pre-trained ASR model to convert audio frames into text; the complete meeting minutes information refers to the meeting minutes information that has been processed by semantic extraction.

[0070] Specifically, the ASR technology can be used for speech-to-text conversion of conference recordings in this application, as well as real-time recording of voice content to generate meeting minutes; the ASR technology is applied to the ASR real-time service technology and ASR offline service technology of this application; the pre-trained large language model can be used in the generation of meeting minutes to extract meeting keywords, generate meeting summaries, chapter overviews, key points reviews, summaries of each speaker, to-do items and other information, and store them in a database, which can improve the efficiency and accuracy of meeting minutes generation.

[0071] Step S30: receiving a conference query request from a user.

[0072] Specifically, the client receives a user's conference query request, and the request content may include the meeting date, meeting topic, speaker name, keywords, etc.

[0073] Step S40: displaying matching content of the meeting minutes information according to the meeting query request.

[0074] It should be noted that the client receives and parses the conference query request, generates matching content by querying the database of the server, receives the matching content returned by the server, and displays the generated matching content to the user.

[0075] Specifically, when a conference participant or user sends a conference query request on the client, the system can identify and parse the request content, record the query request in the system, and query the database for relevant meeting records based on the query request, such as using SQL query statements or other database query tools. The query request can meet multiple condition queries, generate a list of meeting records that meet the conditions, and return the generated query results to the conference participant or user.

[0076] This embodiment provides a method for processing meeting minutes, which is applied to a client, sends a meeting request to a server to create a meeting, collects audio frames through a meeting recording component and sends them to the server; receives meeting minutes information obtained by the server based on the audio frames, ASR technology and a pre-trained large language model; receives a user's meeting query request; and displays matching content of the meeting minutes information according to the meeting query request. This method solves the problem that real-time recording cannot display meeting content in real time, the meeting generated content is simple, and completion information cannot be pushed in time after the meeting processing is completed. It can summarize the meeting content from a semantic level and push meeting completion information in time, so that users can understand the meeting content in real time and efficiently.

[0077] Further, refer to Figure 2 The second embodiment of the method for processing meeting minutes of this application provides a flow chart based on the above Figure 2The illustrated embodiment further refines the step of "sending a meeting request to the server to create a meeting, and collecting audio frames through the meeting recording component and sending them to the server" in step S10, including steps A201 to A205:

[0078] Step A201: Send a user message to the server to create a conference ID;

[0079] It should be noted that the client can send a request to create a conference interface to the server through HTTP or WebSocket protocol, and the server can generate a unique conference ID and update the conference record based on the user message. The user message can include necessary information such as user ID, conference topic, and list of participants.

[0080] Step A202: After receiving the conference creation success identifier and conference ID returned by the server, a connection is established with the server through the STOMP protocol.

[0081] It should be noted that the STOMP protocol is a simple text-oriented message protocol used for message transmission between the client and the server. It has the characteristics of simplicity, cross-platform, flexibility and compatibility, and can be applied to application scenarios such as real-time messaging, event-driven architecture, microservice communication and the Internet of Things. The client receives the HTTP or WebScoket response returned by the server, which should include the identification of successful meeting creation and the meeting ID, and parses the response content to extract the identification of successful meeting creation and the meeting ID. The client can connect to the STOMP server through TCP, send the STOMPCONNECT command, and request to establish a connection with the server. The server responds with the CONNECTED command to confirm that the connection is successful, so that the client can send the SECD command to send messages to the server. At the same time, the client uses the SUBSCRIBE command to subscribe to a specific target address and receive real-time messages from the server.

[0082] Step A203: Request the user to authorize access to the universal recording device.

[0083] It should be noted that requesting user authorization to access the general recording device is a key step to ensure the normal operation of the meeting minutes system. The client can display the authorization request on the user interface to prompt the user to authorize access to the general recording device. At this time, the authorization request can explain the reason and purpose of the client's need to access the recording device to improve the user's trust. The user can confirm the authorization request on the interface to allow the client to access the general recording device.

[0084] Step A204: Initialize the recording function, record the conference speaking user ID and collect audio data.

[0085] It should be noted that when initializing the recording function, it is necessary to detect available general recording devices, such as microphones, voice recorders, etc. when accessing general recording devices. Multiple recording devices should be supported to ensure the versatility and compatibility of the devices. A list of available recording devices should be displayed on the user interface, and users can select the default device or manually select a specific device. Users can set recording parameters as needed, such as sampling rate, bit rate, audio format, etc., to ensure the quality of recording. The identification of the speaking user ID can be obtained by logging into the system, scanning a QR code, voice recognition, or microphone positioning, and the speaking user ID is associated with the corresponding audio data and recorded and stored in the database or local file on the server. The client collects the audio data of the conference speech in real time through the connected recording device.

[0086] Step A205: The collected audio data is converted into audio frames through the conference recording component and sent to the server.

[0087] It should be noted that the client pre-processes the collected audio data through the conference recording component, such as noise reduction, gain adjustment, etc., and divides the pre-processed audio data into multiple audio frames, where the size and frame rate of the audio can be set according to actual needs; by encoding each audio frame, standard audio frame data is generated, and then the encoded audio frames are sent to the server frame by frame.

[0088] In this embodiment, the client sends a user message to the server to create a meeting ID; receives the meeting creation success mark and meeting ID fed back by the server, and establishes a connection with the server through the STOMP protocol; requests user authorization to access a general recording device; initializes the recording function, records the conference speaking user ID and collects audio data; converts the collected audio data into audio frames through the conference recording component and sends them to the server; through automation and real-time messaging, the time for manual recording and sorting is reduced, and the efficiency of generating meeting minutes is improved; at the same time, by recording the conference speaking user ID and collecting audio data, the possibility of human error is reduced, and the content of the meeting can be accurately recorded.

[0089] Further, refer to Figure 3 The third embodiment of the method for processing meeting minutes of this application provides a flow chart based on the above Figure 3 The illustrated embodiment further refines the step of "receiving the meeting minutes information obtained by the server based on the audio frame, ASR technology and the pre-trained large language model" in step S10, including steps A301-302:

[0090] Step A301: When the audio frame is a real-time audio frame, the receiving server obtains real-time meeting minutes information based on the real-time audio frame and the ASR real-time service.

[0091] It should be noted that the ASR real-time service analyzes real-time audio frames, identifies the speech content therein, and converts it into a readable text format; pre-processes the received real-time audio frames, extracts the feature vector of the real-time audio stream from the pre-processed audio stream using a feature extraction library, converts the real-time audio frames into real-time text based on the feature vector, integrates the real-time text into the meeting minutes, ensures the integrity and consistency of the meeting minutes, generates real-time meeting minutes information based on the integrated text, including the meeting topic, participants, discussion content, conclusion points, etc., and displays the generated real-time meeting minutes information in real time on the user interface of the client.

[0092] Step A302: When the audio frame is an offline audio frame after the end of the meeting, the receiving server obtains complete meeting minutes information based on the offline audio frame, the ASR offline service and the pre-trained large language model.

[0093] It should be noted that the ASR offline service can achieve offline conversion of speech to text without a network connection through steps such as audio collection, preprocessing, feature extraction, speech recognition and post-processing. The offline audio frames after the meeting are sent to the server, which preprocesses the offline audio frames, extracts features to achieve speech recognition and obtains offline text, inputs the offline text into a pre-trained large language model, performs semantic analysis and summary, extracts key information and conclusion points, generates complete meeting minutes information, stores the generated complete meeting minutes information in the server's database, and sends it to the client for display in the user interface.

[0094] In this embodiment, the real-time recording and display of the meeting content makes it easier for participants to understand the progress of the meeting and the discussion content in real time, while the processing of offline audio frames and the generation of complete meeting minutes information can be performed in an unstable network or no network environment, ensuring the continuity and reliability of the meeting records.

[0095] In a feasible embodiment, displaying matching content of the meeting minutes information according to the meeting query request includes:

[0096] Match the user subscription message, topic or user ID with the received meeting minutes information and display it on the user interface;

[0097] When the complete meeting minutes information is received, the user is notified of the meeting completion.

[0098] It should be noted that the client allows users to subscribe to specific messages, topics or user IDs to receive relevant meeting minutes information, wherein the user's subscription messages are stored in the server's database. When the client receives the meeting minutes information, it matches it according to the user's subscription information and displays the matching results in the user interface so that the user can view the meeting minutes information that meets the subscription conditions; when the client receives the complete meeting minutes information, it detects whether the meeting is completed, wherein the criteria for meeting completion include the meeting end time, completeness of the meeting minutes, etc.; the client generates a meeting completion notification, wherein the notification content includes the meeting subject, meeting time, meeting location, participants, meeting minutes link, etc., and pushes it to users who have subscribed to relevant messages, topics or user IDs, so that they can view the notification content in the user interface.

[0099] Specifically, the user subscribes to the "Project Progress Report" message through the client application interface. The client stores the user's subscription message in the server's database. When the meeting minutes information of the "Project Progress Report Meeting" is received, it is matched according to the user's subscription message. If it is found that the user has subscribed to the message, the matching result will be displayed in the user interface, and the user can view the meeting minutes information of the "Project Progress Report Meeting".

[0100] The present application embodiment provides a method for processing meeting minutes, which is applied to the server. Figure 4 , Figure 4 This is a flowchart of the fourth embodiment of the meeting minutes processing method of this application.

[0101] In this embodiment, the meeting minutes processing method includes steps S100 to S300:

[0102] Step S100: receiving audio frames sent by the client.

[0103] It should be noted that the server can use protocols such as WebSocket or HTTP / 2 to achieve real-time transmission and reception of real-time audio frames from the client, and store the received audio frames in the server's buffer, such as using a memory database or a cache system to store audio frame data; at the same time, the audio processing library can be used to pre-process the stored audio frames, such as noise reduction, gain adjustment, echo cancellation, etc.

[0104] Step S200: obtaining meeting minutes information through the audio frame, ASR technology and a pre-trained large language model.

[0105] It should be noted that by loading pre-trained ASR models, such as Google Speech-to-Text, Microsoft Azure Speech Service, etc., ASR offline service technology support is provided; by loading pre-trained large language models, semantic analysis and summarization are performed to extract key information and conclusion points.

[0106] Specifically, use the model loading tool to load the pre-trained ASR model, input the pre-processed audio frames into the ASR model, use the ASR offline service technology in the ASR model to perform speech recognition, generate the corresponding offline text, and post-process the generated offline text, such as language model correction, punctuation addition, semantic analysis, etc., to improve the readability and accuracy of the offline text, and store it in the server database. At the same time, load the pre-trained large language model, input the text generated by ASR into the large language model, analyze and summarize the text, and generate a complete meeting minutes based on the analysis results of the large language model, such as the meeting topic, participants, discussion content, conclusion points, to-do items, etc., and use the message push system, such as email, SMS, in-app notification, etc. to push the meeting minutes information to the client user.

[0107] Step S300: sending the matching content of the meeting minutes information to the client according to the user's query request.

[0108] It should be noted that the complete meeting minutes information is generated by combining the offline text and the text or instruction input by the user into a complete prompt, and sending the prompt to the ASR large language model. By querying the meeting query request stored in the database of the server, using SQL query statements or other database query tools to query the corresponding matching content in the complete meeting minutes information, and sending the matching content to the client, the generated matching content is displayed to the user.

[0109] Specifically, when a conference participant or user sends a conference query request on the client, the system can identify and parse the request content, record the query request in the system, and query the database for relevant meeting records based on the query request, such as using SQL query statements or other database query tools. The query request can meet multiple condition queries, generate a list of meeting records that meet the conditions, and return the generated query results to the conference participant or user.

[0110] In this embodiment, the client device can collect audio frames in real time through a microphone, compress and transmit them, the server loads the ASR model, converts the received real-time audio frames into offline text, and performs post-processing and storage, obtains the offline text from the database, combines user input, generates meeting minutes information through a large language model, and returns it to the client. At the same time, the server loads the ASR model, converts the received real-time audio frames into real-time text, and performs post-processing and real-time transmission. Through the above steps, efficient and accurate speech recognition and meeting minutes generation can be achieved to meet the needs of modern conference and voice interaction systems.

[0111] Further, refer to Figure 5 The fifth embodiment of the method for processing meeting minutes of this application provides a flow chart based on the above Figure 5 The illustrated embodiment further includes steps A501 to A502 after the step of "receiving an audio frame sent by the client" in step S100:

[0112] Step A501: Cache the received audio frames into the memory or temporary storage of the server;

[0113] Step A502: pre-process the cached audio frames by noise reduction and compression.

[0114] It should be noted that the noise reduction can be performed by compressing the audio frames through an audio compression algorithm, such as Opus and AAC; the preprocessing can be performed by preprocessing the audio frames through an audio processing library, such as FFmpeg and WebRTC.

[0115] Further, refer to Figure 6 The sixth embodiment of the method for processing meeting minutes of this application provides a flow chart based on the above Figure 6 The illustrated embodiment further subdivides the step of "obtaining the meeting minutes information by using the audio frame, ASR technology and a pre-trained large model" in step S200, and when the audio frame is a real-time audio frame, includes steps A601 to A602:

[0116] Step A601: converting the preprocessed real-time audio frame into a feature vector;

[0117] It should be noted that feature extraction libraries, such as Librosa and PyAudioAnalysis, can be used to extract feature vectors, such as Mel-frequency cepstral coefficients (MFCC) and linear predictive coding (LPC), from preprocessed real-time audio frames.

[0118] Step A602: Performing speech recognition on the feature vector through ASR real-time service technology to generate real-time meeting minutes information;

[0119] It should be noted that a pre-trained ASR model can be loaded from the model storage path, and the extracted feature vectors can be analyzed using ASR real-time service technology to convert real-time audio frames into real-time text; a text processing library can be used to organize and format the real-time text content to obtain real-time meeting minutes information, and the real-time meeting minutes information can be stored in a database and sent to the client.

[0120] Further, refer to Figure 7 The seventh embodiment of the method for processing meeting minutes of this application provides a flow chart based on the above Figure 7 The illustrated embodiment further refines the step of "obtaining the meeting minutes information by using the audio frame, ASR technology and a pre-trained large model" in step S200. When the audio frame is an offline audio frame after the end of the meeting, it also includes steps A701 to A704:

[0121] Step A701: converting the pre-processed offline audio frame into offline text through ASR offline service technology;

[0122] Specifically, the pre-processed offline audio frame is obtained through the database, the pre-trained ASR model is loaded, the offline audio frame is converted into offline text using the offline service technology of the ASR model, and the format and content of the offline text are verified to be valid;

[0123] Step A702: Load the pre-trained large model;

[0124] It should be noted that you can use model loading tools (such as TensorFlow, PyTorch) to load the ASR model to ensure the availability and performance of the model.

[0125] Step A703: combining the offline text and the text or instruction input by the user into a complete prompt;

[0126] It should be noted that the complete prompt can be obtained through a text processing library, and the combined prompt is formatted to ensure that it meets the input requirements of the large language model.

[0127] Step A704: Send the prompt to a pre-trained large language model, and receive the complete meeting minutes information generated by semantic extraction of the large model.

[0128] It should be noted that by sending the assembled prompt to the large language model through API calls, offline text can obtain meeting keywords, meeting summary, chapter overview, key points review, speaker summary and to-do items through semantic extraction to generate complete meeting minutes information.

[0129] In one embodiment, after the client establishes a connection with the server, the client can subscribe to messages, topics or user IDs, collect audio frames in real time, and send them to the server in real time. The server uses a model loading tool to load a pre-trained ASR model, and inputs the received real-time audio frames into the ASR model through the ASR real-time service technology to perform speech recognition, generate corresponding real-time text, and send the real-time text to the user who subscribed to the message, topic or user ID on the client, and display it in a table. Figure 8 As shown in the figure, the text message contains the structure of time + speaker + speech content; the client receives real-time text messages by subscribing to messages, topics or user IDs, and displays them with front-end rendering. The server pushes messages through the APP to notify users that the meeting is completed, as shown in the figure. Fig. 9 As shown, the client can query the meeting ID, keywords, meeting summary, chapter overview, key points review, speaker summary, to-do information, etc. At the same time, it can understand the summary and start time of a single key point, and view it in a targeted and selective manner; when faced with the speaker sequence ID that you are interested in or concerned about during the meeting, you can then understand their name and the content summary of the meeting, the start time and the topic of the speech, etc.

[0130] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the method for processing meeting minutes of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0131] In addition, this application also provides a meeting minutes processing device, please refer to Fig.10 , the conference minutes processing device comprises:

[0132] Audio acquisition and preprocessing module: responsible for acquiring audio frames from the meeting minutes recording and preprocessing the audio frames;

[0133] Meeting minutes generation module: receives audio frames, calls ASR technology and large language models to generate meeting minutes information,

[0134] Content display module: by receiving the user's meeting query request, searching for matching content in the meeting minutes information and displaying and pushing it to the client.

[0135] The device provided by the present application adopts the conference minutes processing method in the above embodiment to solve the technical problems of conference minutes processing. Compared with the prior art, the beneficial effects of the conference minutes processing device provided by the present application are the same as the beneficial effects of the method provided by the above embodiment, and the other technical features in the conference minutes processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0136] In addition, the present application provides a meeting minutes processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the meeting minutes processing method in the above-mentioned embodiment one.

[0137] Reference below Fig.11 , which shows a schematic diagram of the structure of a conference minutes processing device suitable for implementing the embodiment of the present application. The conference minutes processing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.11 The conference minutes processing device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0138] like Fig.11As shown, the conference minutes processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the conference minutes processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the conference minutes processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a conference minutes processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0139] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0140] The conference minutes processing device provided by the present application adopts the conference minutes processing method in the above embodiment, which can solve the technical problems of conference minutes processing. Compared with the prior art, the beneficial effects of the conference minutes processing device provided by the present application are the same as the beneficial effects of the conference minutes processing method provided by the above embodiment, and the other technical features in the conference minutes processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0141] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0142] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0143] In addition, the present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the meeting minutes processing method in the above-mentioned embodiment.

[0144] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0145] The computer-readable storage medium may be included in the conference minutes processing device; or may exist independently without being assembled into the conference minutes processing device.

[0146] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the meeting minutes processing device, the meeting minutes processing device enables the meeting minutes processing device to: collect real-time audio frames through meeting minutes recording; send the real-time audio frames to the server for caching and preprocessing, and receive complete meeting minutes information generated by the server based on the real-time audio frames, ASR offline service technology, and user-input text or instructions; receive real-time text generated by the server based on the real-time audio frames and ASR real-time service technology, and display and push messages.

[0147] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0148] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0149] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0150] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned method for processing meeting minutes, and can solve the technical problems that the real-time recording in the meeting minutes processing cannot display the meeting content in real time, and the information that users are concerned about cannot be pushed in time after the meeting processing is completed. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the method for processing meeting minutes provided by the above-mentioned embodiment, and will not be repeated here.

[0151] In addition, the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned meeting minutes processing method when executed by a processor.

[0152] The computer program product provided by this application can solve the technical problem that the real-time recording in the meeting minutes processing cannot display the meeting content in real time, and the information that users are concerned about cannot be pushed in time after the meeting processing is completed. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the meeting minutes processing method provided by the above embodiment, which will not be repeated here.

[0153] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for processing meeting minutes, characterized in that: Applied to a client, the method comprises: Send a meeting request to the server to create a meeting, and use the meeting recording component to collect audio frames and send them to the server; Receiving the meeting minutes information obtained by the server based on the audio frame, ASR technology and a pre-trained large language model; Receive conference query requests from users; The matching content of the meeting minutes information is displayed according to the meeting query request.

2. The method according to claim 1, characterized in that The step of sending a meeting request to the server to create a meeting and collecting audio frames through the meeting recording component and sending them to the server includes: Send a user message to the server to create a conference ID; Receive the conference creation success flag and conference ID from the server, and establish a connection with the server through the STOMP protocol; Request user authorization to access general recording devices; Initialize the recording function, record the user ID of the conference speaker and collect audio data; The collected audio data is converted into audio frames through the conference recording component and sent to the server.

3. The method according to claim 2, characterized in that The receiving server obtains the meeting minutes information based on the audio frame, ASR technology and the pre-trained large language model, including: When the audio frame is a real-time audio frame, receiving real-time meeting minutes information obtained by the server based on the real-time audio frame and the ASR real-time service; When the audio frame is an offline audio frame after the end of the meeting, the receiving server obtains complete meeting minutes information based on the offline audio frame, the ASR offline service and the pre-trained large language model.

4. The method according to claim 3, characterized in that The displaying of the matching content of the meeting minutes information according to the meeting query request includes: Match the user subscription message, topic or user ID with the received meeting minutes information and display it on the user interface; When the complete meeting minutes information is received, the user is notified of the meeting completion.

5. A method for processing meeting minutes, characterized in that: Applied to the server, the method further includes: Receive audio frames sent by the client; Meeting minutes information obtained through the audio frames, ASR technology and a pre-trained large language model; The matching content of the meeting minutes information is sent to the client according to the user's query request.

6. The method according to claim 5, characterized in that After the step of receiving the audio frame sent by the client, the method further includes: Buffer the received audio frames into the server's memory or temporary storage; Pre-process the cached audio frames for noise reduction and compression.

7. The method according to claim 6, characterized in that The step of obtaining the meeting minutes information by using the audio frame, ASR technology and a pre-trained large language model includes: When the audio frame is a real-time audio frame; Convert the preprocessed real-time audio frames into feature vectors; The feature vector is subjected to speech recognition through ASR real-time service technology to generate real-time meeting minutes information.

8. The method according to claim 7, characterized in that The step of obtaining the meeting minutes information by using the audio frame, ASR technology and a pre-trained large language model also includes: When the audio frame is an offline audio frame after the conference ends; The pre-processed offline audio frames are converted into offline texts through ASR offline service technology; Load a pre-trained large language model; Combining the offline text and the text or instruction input by the user into a complete prompt; The prompt is sent to a pre-trained large language model, and complete meeting minutes information generated by semantic extraction of the large language model is received.

9. A conference minutes processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for processing meeting minutes according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the meeting minutes processing method according to any one of claims 1 to 8 are implemented.

Images