Smart conference interaction method and device based on large language model, and medium

Abstract

The invention relates to an intelligent conference interaction method and device based on a large language model and a medium, and the method comprises the steps: converting a conference voice into a text through a first large language model, and carrying out the slicing of the text; vectorizing conference text slices based on a natural language processing technology; storing the vectorized conference text slice into a conference domain of the corresponding conference; using the second large language model to receive a user question, and combining with a question and answer historical record to generate a composite question; searching related conference texts in the conference domain through similarity calculation and returning answers; through similarity calculation and networking query, relevant content is retrieved and returned for recommendation; and adjusting a similarity calculation function according to the feedback of the user for the answer and the recommendation. Compared with the prior art, the conference voice is recognized in real time and the content is retrieved in real time by using the large language model, so that a user can crossly understand and learn similar conference content, and the conference efficiency and the intelligent degree are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent conference technology, and in particular to an intelligent conference interaction method, device and medium based on a large language model. Background Art

[0002] Conferences are an important way of communication and cooperation in modern society, involving multiple fields such as enterprises, academia, government, and associations. As the core tool for conference organization and management, conference systems play a vital role in ensuring the quality and efficiency of conferences. Some existing conference systems already contain speech-to-text modules. Through speech recognition technology, the conference speech can be converted into text, which can be easily recorded and searched for later. However, the existing speech recognition technology is mainly divided into statistical model-based and traditional deep learning-based. Among them, traditional deep learning has become the mainstream speech recognition technology. However, the existing speech recognition technology still has problems such as low accuracy, low semantic understanding ability, and limited ability to handle accents and noise. In addition, the existing conference system also lacks intelligent dialogue function and cannot communicate and retrieve the content of the conference intelligently. A very small number of systems have implemented dialogue modules, but they are also rule-based dialogues. This technical solution mainly answers users' questions and solves users' needs through pre-set rules and knowledge bases. However, rule-based dialogue systems usually require expert knowledge and a large amount of manually annotated data to support their dialogue and decision-making capabilities. Therefore, they lack flexibility and adaptability and cannot adapt to complex and changing scenarios.

[0003] Existing technical solutions still have room for improvement and optimization in converting speech to text. Although rule-based dialogue systems can solve some specific problems well, they cannot adapt to complex and changing scenarios. More flexible and adaptive technical solutions are needed to support the development of smart conferences.

[0004] Traditional conference systems mainly use technical solutions such as speech-to-text technology based on statistics and deep learning, and rule-based dialogue systems. These solutions still have some room for improvement. First, the accuracy of speech-to-text conversion of such solutions is not high, which will affect the efficiency and accuracy of smart conferences. Secondly, rule-based dialogue systems usually require expert knowledge and a large amount of manually annotated data to support their dialogue and decision-making capabilities. They lack flexibility and adaptability and cannot adapt to complex and changing scenarios, resulting in poor dialogue results. In addition, traditional conference systems are often only able to perform simple recording and query, lack intelligent recommendation functions, and users cannot cross-understand and learn similar conference content, forming knowledge islands. All of these limit the efficiency and intelligence of meetings. Summary of the invention

[0005] The purpose of the present invention is to provide an intelligent conference interaction method based on a large language model, which utilizes the large language model to recognize conference voice in real time and perform real-time content retrieval, so that users can cross-understand and learn similar conference content, thereby improving the efficiency and intelligence of the meeting.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A smart conference interaction method based on a large language model, the method comprising:

[0008] Using the No.1 language model, the conference speech is converted into text and sliced ​​for processing;

[0009] Slice and vectorize conference texts based on natural language processing technology;

[0010] The vectorized conference text slices are stored in the conference domain of the corresponding conference;

[0011] Using the second largest language model, we receive user questions and generate compound questions based on the question and answer history records;

[0012] Based on the second largest language model and the generated compound question, relevant conference texts are retrieved in the conference domain through similarity calculation and the answers are returned;

[0013] Based on the second largest language model and the user's historical meeting records and behaviors, combined with the information of the current meeting, through similarity calculation and online query, relevant content is retrieved and recommendations are returned;

[0014] Adjust the function used for similarity calculation based on user feedback on answers and recommendations.

[0015] Furthermore, the process of converting the conference speech into text and slicing it includes:

[0016] Input the conference audio to be converted into text;

[0017] Configure the first language model according to one or more languages ​​to be used in the meeting;

[0018] Segment the conference audio by time and create indexes for the segmented conference audio;

[0019] Call the API of the first language model to perform speech recognition on the segmented voice files and convert the voice into text;

[0020] Merge the text files obtained from different slices into one overall text file.

[0021] Furthermore, the conference voice to be converted into text includes recordings of historical meetings and voice input of real-time meetings.

[0022] Furthermore, the vectorization includes converting the conference text slices into high-dimensional vector representations.

[0023] Furthermore, the vector representation includes the semantics of the text slice, context information, and the person to whom the conference speech belongs.

[0024] Furthermore, the conference domain is a vector database storing vectorized conference text slices. Conference domains of different conferences are independent of each other and are provided with domain keywords according to the conference type.

[0025] Furthermore, the similarity calculation process includes:

[0026] Slice and vectorize the generated compound problem to determine the areas involved in the compound problem;

[0027] According to the fields involved in the compound question, the conference text slice vector is extracted from the conference domain with similar field keywords, and the similarity is calculated with the composite question slice vector. The vector dot product and vector modulus between the slice vectors are calculated using the similarity calculation function to measure the angle between the slice vectors.

[0028] Screening and outputting the conference text slice vector that is closest to the composite question slice vector after calculation;

[0029] The similarity calculation function includes cosine similarity or Euclidean distance.

[0030] Furthermore, the answers and recommendations are texts obtained by reorganizing and arranging the output results obtained by the large language model.

[0031] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned intelligent conference interaction method based on a large language model are implemented.

[0032] A computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the above-mentioned smart conference interaction method based on a large language model.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention adopts speech recognition technology based on a large language model to realize the function of converting the speech data of the meeting into text data, thereby improving the accuracy of speech conversion, the ability to understand semantics, and the ability to handle accents and noise. When storing text, the concept of conference domain is used to store texts of different fields and meetings by domain, effectively distinguishing and managing text data in different meetings, and improving the accuracy of text data semantics; similarity calculation is used to match similarity, thereby realizing more efficient similarity query and matching; after the retrieval is completed, user feedback is accepted to adjust the function used for similarity calculation, thereby realizing self-adaptation and liberalization;

[0035] 2. When searching, the present invention generates compound questions based on the question and answer history records, uses the compound questions for searching, and better understands the needs and intentions of users. After searching, the present invention combines the user's historical meeting records, behaviors and information of the current meeting, and retrieves related content and returns recommendations through similarity calculation and online query, so that users can cross-understand and learn similar meeting content, breaking the knowledge island phenomenon in traditional conference systems and improving the efficiency and intelligence of meetings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of the method of the present invention;

[0037] Figure 2 This is an example diagram of speech-to-text conversion of the present invention;

[0038] Figure 3 This is an example diagram of text slicing of the present invention;

[0039] Figure 4 This is an example diagram of text vectorization of the present invention. DETAILED DESCRIPTION

[0040] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0041] This embodiment discloses a smart conference interaction method based on a large language model, the method steps are as follows: Figure 1 shown.

[0042] Step S1, using the first language model to perform speech recognition, converting the conference speech into text and slicing it.

[0043] The specific speech conversion and text slicing process is as follows: Figure 2 As shown, including:

[0044] Input conference voice to be converted into text, where the conference voice includes recordings of historical conferences and voice input of real-time conferences;

[0045] Configure the open source Massively Multilingual Speech Model (MMS) model according to one or more languages ​​that need to be used in the meeting;

[0046] The conference audio is segmented by time, and indexes are created for the segmented conference audio. The segmentation example is as follows: Figure 3 As shown;

[0047] Call the API of the Massively Multilingual Speech Model to perform speech recognition on the segmented speech files and convert the speech into text;

[0048] Merge the text files obtained from different slices into one overall text file.

[0049] Step S2, vectorizing the conference text slices based on natural language processing technology.

[0050] In this embodiment, the text2vec-base-chinese model is used for vectorization processing, and the SentenceEmbeddings (sentence vectorization) technology based on the text2vec-base-chinese model is as follows: Figure 4 As shown in the figure, each text segment in the conference text is converted into a high-dimensional vector of fixed dimension, which can capture the semantic and contextual information of the sentence. Different from WordEmbeddings (word vectorization), SentenceEmbeddings considers the semantic meaning of the entire sentence instead of considering each word in the sentence separately.

[0051] During the vectorization process of the text2vec-base-chinese model, the sentence is encoded using a Transformer-based method, which maps the sentence to a 768-dimensional high-dimensional vector space for subsequent tasks such as text matching or semantic search.

[0052] Step S3, storing the vectorized conference text slices in the conference domain corresponding to the conference.

[0053] The conference domain is a vector database that stores vectorized conference text slices, and the conference domains of different conferences are independent of each other.

[0054] Since different conferences may involve different topics and fields, the same text may have different semantics and importance in different conferences. If the vectorized texts of all conferences are stored in a vector database, it will cause ambiguity and misunderstanding. By creating an independent namespace for each conference, the text data in different conferences can be effectively distinguished and managed, and the accuracy of the semantics of the text data can be improved.

[0055] In this embodiment, the open source Chroma vector database is used to store high-dimensional vector representations of conference texts to facilitate subsequent content retrieval and answer generation. The Chroma vector database is a database based on vectorization technology that can efficiently store a large amount of vector data and support complex similarity calculations and retrieval operations.

[0056] Step S4, using the second largest language model, receiving user questions, and generating compound questions based on the question and answer history records.

[0057] In this embodiment, the second largest language model used is the locally deployed open source large language model ChatGLM-6B. During multiple rounds of conversations, users ask natural language questions, and the locally deployed open source large language model ChatGLM-6B combines the chat history with the content of this round of conversation to generate compound questions. The generated compound questions facilitate the model to better understand the needs and intentions of the participants in the future.

[0058] Step S5, based on the second largest language model and the generated compound question, relevant conference texts are retrieved in the conference domain through similarity calculation and the answer is returned.

[0059] The similarity calculation process includes:

[0060] Slice and vectorize the generated compound problem to determine the areas involved in the compound problem;

[0061] According to the fields involved in the compound question, the conference text slice vector is extracted from the conference domain with similar field keywords, and the similarity is calculated with the composite question slice vector. The cosine value between the slice vectors is calculated using the cosine similarity calculation function to measure the angle between the slice vectors.

[0062] The conference text slice vector whose calculated cosine value is closest to 1 is filtered and output.

[0063] Taking the two-dimensional cosine distance as an example, the formula is:

[0064]

[0065] Among them, A and B are both text slice vectors. The closer the calculated value of cosine_similarity is to 1, the more similar the text is to text B.

[0066] In this embodiment, the cosine similarity calculation is actually performed based on the 768-dimensional high-dimensional vector space mapped based on text2vec-base-chinese.

[0067] Step S6, based on the second largest language model and the user's historical meeting records and behaviors, combined with the information of the current meeting, through similarity calculation, retrieve relevant meeting content in the meeting domain and recommend it to the user.

[0068] The answers and recommendations returned to the user are texts that are reorganized and arranged by the large language model based on the retrieved output results.

[0069] Step S7: adjusting the function used for similarity calculation according to the user's feedback on the answer and the recommendation.

[0070] In this embodiment, the similarity calculation uses the cosine similarity calculation using the cosine formula to calculate the similarity, but in actual applications, the similarity calculation function can be customized, the settings can be customized to return the most relevant N types of conference documents, and the recommendation model can be continuously optimized and adjusted based on the feedback from the participants.

[0071] If the above method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0072] Example 2

[0073] Based on the method steps of the above-mentioned embodiment 1 and applied to actual work, this embodiment discloses a smart conference system based on a large language model.

[0074] To ensure data security, the smart conference system of this embodiment uses the open source large language model ChatGLM-6B and private cloud deployment. The system will not pass private information of the meeting to a third party in the form of an API, and all interactions are within the system. At the same time, all data is stored on a private cloud server to ensure the privacy and security of the data. Through these measures, the system can provide users with safe, intelligent, and personalized conference services, and improve user satisfaction and participation.

[0075] In a meeting, the smart conference system needs to record meeting content, intelligently analyze meeting content, and provide personalized meeting recommendations and intelligent dialogue functions.

[0076] This intelligent conference system uses the Massively Multilingual Speech Model for speech recognition, converts all conference speech data into text data, slices the converted text data, and generates text fragments suitable for vector database storage and retrieval. For the generated text fragments, text2vec-base-chinese is used to vectorize the text, generate 768-dimensional high-dimensional vectors, and store them in the Chroma vector database.

[0077] This intelligent conference system communicates with users by establishing a chat box, using the chatbot MeetBot based on ChatGLM-6B to interact in natural language, answer questions raised by users, and provide relevant conference information and the most relevant conference content. The recommendation system module uses the historical meeting records and behaviors of participants, combined with the information of the current meeting, and based on the search function of the vector database to recommend relevant conference content and services.

[0078] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A smart conference interaction method based on a large language model, characterized in that: The method comprises: Using the No.1 language model, the conference speech is converted into text and sliced ​​for processing; Slice and vectorize conference texts based on natural language processing technology; The vectorized conference text slices are stored in the conference domain of the corresponding conference; Using the second largest language model, we receive user questions and generate compound questions based on the question and answer history records; Based on the second largest language model and the generated compound question, relevant conference texts are retrieved in the conference domain through similarity calculation and the answers are returned; Based on the second largest language model and the user's historical meeting records and behaviors, combined with the information of the current meeting, through similarity calculation and online query, relevant content is retrieved and recommendations are returned; Adjust the function used for similarity calculation based on user feedback on answers and recommendations.

2. According to claim 1, a smart conference interaction method based on a large language model is characterized in that: The process of converting the conference speech into text and slicing it includes: Input the conference audio to be converted into text; Configure the first language model according to one or more languages ​​to be used in the meeting; Segment the conference audio by time and create indexes for the segmented conference audio; Call the API of the first language model to perform speech recognition on the segmented voice files and convert the voice into text; Merge the text files obtained from different slices into one overall text file.

3. According to claim 2, a smart conference interaction method based on a large language model is characterized in that: The conference voice to be converted into text includes recordings of historical conferences and voice input in real-time conferences.

4. According to claim 1, a smart conference interaction method based on a large language model is characterized in that: The vectorization includes converting conference text slices into high-dimensional vector representations.

5. According to claim 4, a smart conference interaction method based on a large language model is characterized in that: The vector representation includes the semantics of the text slice, context information, and the person to whom the conference speech belongs.

6. The method for intelligent conference interaction based on a large language model according to claim 1, characterized in that: The conference domain is a vector database that stores vectorized conference text slices. The conference domains of different conferences are independent of each other and are provided with domain keywords according to the conference type.

7. The method for intelligent conference interaction based on a large language model according to claim 1, characterized in that: The process of similarity calculation includes: Slice and vectorize the generated compound problem to determine the areas involved in the compound problem; According to the fields involved in the compound question, the conference text slice vector is extracted from the conference domain with similar field keywords, and the similarity is calculated with the composite question slice vector. The vector dot product and vector modulus between the slice vectors are calculated using the similarity calculation function to measure the angle between the slice vectors. Screening and outputting the conference text slice vector that is closest to the composite question slice vector after calculation; The similarity calculation function includes cosine similarity or Euclidean distance.

8. The method for intelligent conference interaction based on a large language model according to claim 1, characterized in that: The answers and recommendations are texts that are reorganized and arranged by a large language model based on the retrieved output results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the smart conference interaction method based on a large language model as described in any one of claims 1-8 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, executes the steps of the smart conference interaction method based on a large language model as described in any one of claims 1-8.

Images