Dialogue data processing method and system and computer equipment

Abstract

The invention relates to a dialogue data processing method and system and computer equipment. Comprising the following steps: through a two-stage identification process, firstly, carrying out preliminary identification by using a large model, then, carrying out data screening and vector database query based on real-time related content indexes, and finally, carrying out identification by using the large model again, so that a target result can be obtained more accurately. Meanwhile, the content indexes related to the real-time performance of the demand are considered in the identification process, so that the demand of the user can be quickly responded, and the latest information and result can be provided.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a conversation data processing method, system and computer device. Background Art

[0002] The large models in the field of artificial intelligence technology have shown extraordinary capabilities in language understanding, interaction and reasoning, and are widely used in natural language processing fields such as dialogue, information support and decision assistance, and code generation. Therefore, enhancing the accuracy of the content generated by large models has become one of the most important issues.

[0003] Most of the existing large-model question-answering systems are generally optimized. While improving the generalized reasoning ability, their performance in specific professional fields is difficult to meet the baseline requirements. For example, in question-answering scenarios with relevant background knowledge such as digital economy, technological innovation, and financial analysis, it is often necessary to input a large amount of pre-information during question-answering to improve the accuracy of the answer, which leads to a decline in user experience and conversation effect. In addition, the time cost of large model training is high, and the real-time information level of large model knowledge is limited by the novelty of the pre-trained model. This leads to problems such as difficulty in extraction and inaccurate search or unreliable single search results when encountering users' real-time requirements. Summary of the invention

[0004] Based on this, the purpose of this application is to provide a conversation data processing method, apparatus and computer equipment that ensure more accurate answer quality, so as to solve the technical problems mentioned in the above background technology.

[0005] In a first aspect, the present application provides a method for processing conversation data, including:

[0006] Acquiring user conversation data and a determined analysis mode, wherein the analysis mode is associated with a model control parameter;

[0007] Preliminary recognition of the conversation data is performed through the target large model and the model control parameters to obtain an initial result; the initial result includes content indicators related to the real-time nature of the demand;

[0008] Based on the content index, the associated data is obtained from the vector database, and the associated data is re-identified through the target large model to obtain the target result.

[0009] In one embodiment, the target big model is obtained by pre-training with multiple corpus data sets; there is a hierarchical relationship between different corpus data sets; the hierarchical relationship is reflected by the degree of features represented by the data; the training process of the target big model includes: determining the current corpus data set, and training the initial big model with the current corpus data set to obtain the current big model; determining the upper-level corpus data set of the current corpus data set, and training the current big model with the upper-level corpus data set until the target big model is iteratively trained.

[0010] In one embodiment, the initial large model is trained by the current corpus data set to obtain the current large model, including: preprocessing the current corpus data set; the preprocessing includes at least one of corpus batch processing, corpus classification structuring and cleaning; determining the training parameters of the initial expert large model; the training parameters include at least one of learning rate, sampling parameters and network parameters; updating the training parameters by the preprocessed current corpus data set until the training stop condition is reached, thereby obtaining the trained current large model.

[0011] In one embodiment, the current corpus data set is preprocessed, including: if the data in the current corpus data set is complex text, converting the complex text into initial text, and using a local large model to supplement the initial text to obtain a valid text; and structuring the valid text into corpus classification according to the domain focus and corpus characteristics.

[0012] In one embodiment, the above method also includes: determining the number of rounds corresponding to the current corpus data set; when the number of rounds is greater than a preset value, determining the sub-models corresponding to the initial large model when the number of rounds is trained; and selecting the current large model from the multiple sub-models.

[0013] In one embodiment, the content indicator includes a content phrase and a relevance indicator; the vector database includes multiple sets; and obtaining related data from the vector database based on the content indicator includes: when the relevance indicator is greater than a preset value, querying the related data from the set of the vector database according to the content phrase.

[0014] In one embodiment, the method further includes: when the correlation index is greater than a target value or the associated data queried from the set of vector databases is empty, triggering an update operation on the vector database; the updated vector database is used to obtain new associated data.

[0015] In one embodiment, a vector database is initialized based on a local large model; the highest-level corpus data set in the training target large model is the initial corpus of the vector database; an update operation on the vector database is triggered, including: determining keywords after word segmentation of the content phrase; performing an online search based on the keywords, and determining update data based on a preset weight list; generating interactive instructions associated with the update data so that the update data is stored in a corresponding collection of the vector database.

[0016] In one embodiment, the initial result also includes an initial answer; the above method also includes: when the relevance index is less than a preset value, formatting the initial answer and calling a renderer to display the user interface.

[0017] In a second aspect, the present application also provides a dialogue data processing system, the system comprising a front end, a target large model, a vector database and a back end, wherein:

[0018] A front end, for acquiring user conversation data and determining an analysis mode, wherein the analysis mode is associated with a model control parameter;

[0019] A target large model, used to perform preliminary recognition on the dialogue data based on the model control parameters to obtain an initial result; the initial result includes content indicators related to the real-time nature of the demand;

[0020] A backend, used for making an interface call based on the initial result to trigger a query instruction or an update instruction for the vector database;

[0021] A vector database, for responding to a query instruction and determining associated data according to the content indicator;

[0022] The target large model is also used to re-identify the associated data to obtain the target result.

[0023] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above conversation data processing method when executing the computer program.

[0024] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above conversation data processing method are implemented.

[0025] The above-mentioned conversation data processing method, system, computer device and readable storage medium, through a two-stage recognition process, first use the target large model for preliminary recognition, then perform data screening and vector database query based on real-time related content indicators, and finally send back the target large model for re-recognition, so as to obtain the target result more precisely and accurately. In addition, the recognition process starts from the positioning of the self-trained target large model, and on the basis of the vector database supplemented with expert knowledge plug-ins in professional fields, it can gradually update the data during the user conversation process, making up for the shortcoming of insufficient real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is an application schematic diagram of a conversation data processing system in one embodiment;

[0027] Figure 2 is an application flow chart of a conversation data processing system in one embodiment;

[0028] Figure 3 is a flow chart of a method for processing conversation data in one embodiment;

[0029] Figure 4 is a general architecture diagram of a conversation data processing system in one embodiment;

[0030] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] The present embodiment provides a conversation data processing system, including a front end, a target big model, a vector database and a back end, wherein: the front end is used to obtain the user's conversation data and a determined analysis mode, and the analysis mode is associated with a model control parameter; the target big model is used to perform a preliminary identification of the conversation data based on the model control parameter to obtain an initial result; the back end is used to make an interface call based on the initial result to trigger a query instruction or an update instruction to the vector database; the vector database is used to respond to the query instruction and determine the associated data according to the content indicator; the target big model is also used to identify the associated data again to obtain the target result.

[0033] Specifically, Figure 1The application diagram of the conversation data processing system is described. The front end is a Web front-end cloud service that provides interactive page source files, including an interactive interface for input boxes, an option menu for analysis mode, a control list of historical conversations, and an HTTP AJAX communication part that is transparent to users. It provides instructions for encoding user input text into a format that can be understood by the target large model, as well as a Web interface for obtaining user conversation data and analysis modes during system interaction.

[0034] Among them, HTTP AJAX communication is a technology for exchanging data with the server and updating part of the web page without reloading the entire page. Cloud service is a microservice architecture that contains many loosely coupled and independently deployable small components or services in a single application. These services have their own technology stack, including databases and data management models; communicate with each other through combinations such as event streams and message brokers; and are organized according to business capabilities, with service separators usually called bounded contexts.

[0035] Target large model: By receiving input request instructions, such as analyzing model control parameters associated with the mode, understanding user needs and performing preliminary recognition of conversation data, generating human-friendly language results, recognizing and executing daily conversations, professional conversations, data collection and processing, report analysis functions, and text generation and translation tasks, initial results are obtained.

[0036] Large models refer to machine learning models with a large number of parameters, which are used to process and generate natural language text. Large language models usually have millions or even billions of parameters, which enables them to capture the nuances of language and perform well in various natural language processing tasks. They can perform tasks such as language understanding, language generation, information retrieval, and knowledge question answering.

[0037] Backend: Receives the dialog request initiated by the Web interface proxy user, identifies the task type based on the initial result, calls the corresponding interface to control the input command of the target large model, and encodes and sends back the input result of the target large model. If the input command corresponds to the query command when real-time requirements are met, the vector database is searched and the associated data is determined according to the query request generated by the target large model. When the input command corresponds to the update command associated with data collection and processing, a network search is initiated through the background, and the vector database is updated with the search results.

[0038] Therefore, the process of front-end and back-end interaction requests based on the dialogue data processing system can be as follows: Figure 2As shown. Start the Web front-end cloud service. The target large model receives the request instruction sent by the front end and determines whether it is a dialogue request related to the dialogue. If not, it returns an error message and returns to the initial process of starting the Web front-end cloud service until a new request instruction sent by the front end is obtained. If so, parse the model control parameters attached to the request instruction, such as the analysis mode association, and perform a preliminary identification of the dialogue data in the request instruction to obtain the initial result. The back end identifies the task type based on the initial result, calls the corresponding interface to control the input instruction of the target large model, and returns a json string until the request processing is completed, and the result is rendered and displayed on the Web interface according to different visualization types.

[0039] In one embodiment, the front end of the conversation data processing system is based on application frameworks such as Vue3 and Vue-Router to create a SPA application with smooth page switching, and a naive UI component library to implement a simple and easy-to-use interactive interface. For example, the axios library is used to initiate Ajax conversation requests and communicate with the back end. Modern languages ​​such as Typescript and SCSS are used to improve development efficiency. The compiled files of the front end are hosted by Nginx, a high-performance HTTP and reverse proxy server.

[0040] Among them, SPA, or Single Page Application, is a modern web application design pattern that interacts with users by dynamically rewriting the current page, avoiding page reloads. HTTP (Hypertext Transfer Protocol) is an application layer protocol for distributed, collaborative, hypermedia information systems.

[0041] In one embodiment, the target big model of the dialogue data processing system is trained based on the Lora fine-tuning method. For example, ChatGLM3 with excellent Chinese capabilities is selected as the benchmark big model, and data documents in various formats are cleaned with the help of libraries such as pdfplumber, python-docx, and json. The data is further refined and classified with the help of a big model such as Wenxin Yiyan, and the training set is organized. The corpus data sets are divided into basic, professional, and divergent categories for three rounds of Lora parameter training. The preferred models are determined from the multiple sub-models generated, and finally the merged big model file is exported using the export_ml.py script.

[0042] In one embodiment, the backend of the conversation data processing system uses, for example, FastAPI+uvicorn as an asynchronous server to configure and load the target large model and tokenizer. The pre-trained target large model is loaded through AutoTokenizer and AutoModel, and the designated port is listened to for conversation requests from the front end. The API endpoint uses the Pydantic model to define the data format of requests and responses, improving type safety and document quality. The CORS middleware is configured to allow barrier-free access to the front end.

[0043] Among them, FastAPI and Uvicorn are two popular Python tools for building high-performance modern web applications. FastAPI is a modern, high-performance web framework for building APIs, while Uvicorn is a lightweight ASGI server. AutoTokenizer is a class that automatically loads a tokenizer that matches a specific model; AutoModel is a class that automatically selects and loads a corresponding pre-trained model based on the model name or path. Pydantic is a fast data model validation component that is often used with FastAPI to improve interface security and robustness.

[0044] Therefore, the conversation data processing system implements an interactive WEB interface with front-end and back-end separation based on various application frameworks. Through three rounds of progressive fine-tuning of the chatGLM3 open source big model, the target big model of think tank consultation is realized. Finally, based on the chroma vector database, a searchable plug-in knowledge base is implemented, and the target big model is linked to embed documents. The back-end API gateway receives the response data of the front-end request to call the target big model interface, and links the vector database agent to query, maintain and update the database knowledge.

[0045] In one embodiment, Figure 3 As shown, a conversation data processing method is provided, which is applied to a conversation data processing system and is implemented through linkage between a front end, a back end, and a vector database in the conversation data processing system, and includes the following steps:

[0046] Step 302, obtaining the user's conversation data and the determined analysis mode.

[0047] Among them, the analysis mode is associated with the model control parameters.

[0048] Specifically, the conversation data processing system provides an easy-to-use interactive interface, namely a Web interface. When the user starts the Web service, he can enter the interactive interface and input the conversation data through the dialog box to achieve normal interaction with the target large model. By clicking on the settings of the interactive interface, you can see the author's related information, and the interactive interface has various option menus and control lists, which can realize functions such as user conversation history, optional context association, light and dark mode, creativity mode, and custom prompt words. When the operation is triggered in response to the user's selection of a function, the selected analysis mode is determined.

[0049] For example, when the selected analysis mode is the creativity mode, the representation needs to focus on free expression, imagination and diverse content, which is suitable for tasks that require innovative thinking and unique insights, such as writing, story creation and advertising copywriting. This mode is highly open, and the answers are more emotional and personalized, which is suitable for application scenarios that establish emotional connections with users. When the selected analysis mode is the non-creativity mode, it emphasizes accuracy, precision and structure, which is suitable for tasks that require clear and authoritative answers, such as technical support, academic research, think tank consulting and data query. The answers of the large model in the non-creativity mode are more conservative, objective and neutral, ensuring the reliability and fairness of the information.

[0050] Therefore, by selecting different analysis modes, the model control parameters corresponding to the target large model will also be different. Users can obtain the most suitable content and suggestions from the target large model based on the model control parameters according to specific needs and application scenarios.

[0051] In one embodiment, by clicking a conversation button in the interactive interface, one can adjust whether to allow the large model to maintain context when answering.

[0052] In one embodiment, the target big model also supports a code interpreter function, which can generate corresponding codes according to the needs and display them in the form of code blocks rendered based on Markdown syntax. Markdown is a lightweight markup language that allows people to write documents in a plain text format that is easy to read and write, and then convert them into valid XHTML (or HTML) documents.

[0053] Step 304, preliminarily identify the dialogue data through the target large model and model control parameters to obtain initial results.

[0054] Among them, the initial results include content indicators related to real-time requirements; the target large model is pre-trained through multiple corpus data sets; there is a hierarchical relationship between different corpus data sets; the hierarchical relationship is reflected by the degree of features represented by the data; the multiple corpus data sets at least include the lowest level basic corpus data set, the second level professional corpus data set and the highest level divergent corpus data set.

[0055] Specifically, after receiving the user's determination of whether it is a creativity mode, whether it is a context association mode, etc., the front-end interactive interface determines to adopt different model control parameters, initiates an Ajax dialogue request to the back-end, and waits for the target large model to respond.

[0056] For example, in the creativity mode, increase the parameter Temperature to 0.8, set it to Top-K (50+) and Top-P (0.95), and increase the Repetition_penalty parameter to a certain preset value to reduce content repetition. Among them, the Temperature parameter is used to control the randomness of the model when generating text. When the parameter value is less than 1, the model will tend to select common words with higher probabilities, making the generated text more fluent and reasonable. The Top-K parameter controls the diversity of the generated text by limiting the model to select only from the K words with the highest probability at each step. The Top-P parameter retains all words before the cumulative probability reaches a certain threshold P. The Repetition_penalty parameter increases the probability penalty for repeated words, and the model is encouraged to generate new words to avoid repeated phrases or patterns in the text.

[0057] The backend listens to a fixed port based on the server that provides an interface for the application to receive the user's dialogue request, and then calls the target large model interface to conduct the dialogue. When the dialogue data is submitted to the target large model, the target large model performs preliminary recognition and returns the initial result. The initial result at this time corresponds to the pre-defined content template, including the initial answer and content indicators.

[0058] In one embodiment, the content template returned by the target macro model is predefined as: answer text and real-time information. The real-time information specifically includes real-time content and real-time relevance. The real-time relevance is 0-1.

[0059] Step 306, acquiring related data from the vector database based on the content index, and re-identifying the related data through the target large model to obtain the target result.

[0060] The content index includes content phrases corresponding to the real-time content in the content template and relevance indexes corresponding to the real-time relevance in the content template; the vector database includes multiple sets; each set is pre-created by a distance calculation method of an embedded space.

[0061] In one embodiment, acquiring associated data from a vector database based on a content index includes: when the relevance index is greater than a preset value, querying associated data from a set of vector databases according to content phrases.

[0062] Specifically, based on the comparison difference between the correlation index and the preset value, it is determined whether it is necessary to trigger a query instruction for the vector database. When the correlation index is greater than the preset value, it is determined that the conversation data is characterized by a normal degree of real-time requirements. The preset value is an indicator preset by the user, such as 0.3. Then the query instruction for the vector database can be triggered, and the related data can be queried from the collection of the vector database according to the content phrase. After splicing the queried related data, the corresponding interface is called to control the input instruction of the target large model, and then the target large model is called a second time to re-identify the related data to obtain the target result, and the answer text in the target result is returned to the user as the front-end response result.

[0063] In one embodiment, the method further includes: when the correlation index is greater than a target value or the associated data queried from the set of vector databases is empty, triggering an update operation on the vector database.

[0064] The updated vector database is used to obtain new associated data; the target value is greater than the preset value, and the target value is an indicator preset by the user, such as 0.8.

[0065] Specifically, based on the difference between the correlation index and the target value, or the data queried from the vector database set, it is determined whether it is necessary to trigger the update operation of the vector database. When the correlation index is greater than the target value, it is judged that the user's problem is a strongly related real-time requirement, or the content of the associated data queried from the vector database set is small or empty. At this time, it is necessary to make an interface call based on the initial result, activate the background to initiate a network search, so as to trigger the update instruction of the vector database and implement the update operation of the vector database.

[0066] Therefore, to ensure the response speed, the user will get effective feedback the next time he asks a similar real-time demand, and then obtain new related data, realizing the near real-time characteristics of the target big model. That is, the dialogue data processing system completes the maintenance and update of near real-time data during the communication between the user and the target big model.

[0067] In one embodiment, the method further includes: when the relevance index is less than a preset value, formatting the initial answer and calling a renderer to display the user interface.

[0068] Specifically, the initial answer in the initial result corresponds to the answer text in the content template. If the relevance index is less than the preset value, the target large model directly returns the initial answer obtained after preliminary recognition to the user. The front end receives the initial answer, for example, parses it in Markdown format, and calls, for example, a Markdown-it renderer to generate HTML text from the result to display the interactive interface.

[0069] In one embodiment, the query instruction and update instruction in the input instruction of the target big model are set as prompt instructions through, for example, the Function function. Function refers to function calling, which is a technology used to obtain reliable structured data from big model interaction. By pre-defining functions for the big model to choose and outputting function parameters, the function structure output expected by the developer is obtained.

[0070] In the above-mentioned conversation data processing method, through a two-stage recognition process, the target large model is first used for preliminary recognition, then data screening and vector database query are performed based on real-time related content indicators, and finally the target large model is sent back for re-recognition, so that the target result can be obtained more precisely and accurately. In addition, the recognition process starts from the positioning of the self-trained target large model, and on the basis of the vector database supplemented with expert knowledge plug-ins in professional fields, the data can be gradually updated during the user conversation process, making up for the shortcomings of insufficient real-time performance and improving practicality. Compared with temporary online search, the solution based on database query can reduce the time to respond to user questions.

[0071] In addition, through the determined analysis mode, it can adapt to the rigorous answer research scenario and the idea exploration and creation scenario respectively, and meet the needs of users in a refined manner. It supports storage of historical conversations, optional context association, light and dark mode, conversation export, etc., and the rendering output supports a variety of rich text formats, which is friendly for long-term experience. The ease of use and the positioning of the target large model complement each other, improving the user experience and enhancing user stickiness.

[0072] In one embodiment, the training process of the target large model includes: determining the current corpus data set, and training the initial large model through the current corpus data set to obtain the current large model; determining the upper-level corpus data set of the current corpus data set, and training the current large model through the upper-level corpus data set until the target large model is iteratively trained.

[0073] Specifically, the process of acquiring multiple corpus data sets includes: first determining the target professional field, and then collecting texts with different degrees of characteristics from various sources to obtain three types of corpus data sets. Scientific and accurate subject knowledge is used as the foundation of model thinking, that is, as the lowest level of basic corpus data sets; classic articles in the field, papers on field integration and application, reports and progress reports by cutting-edge scholars, and supporting interpretations are used as field learning materials, that is, as the second-level professional corpus data sets; major progress frontier news, predictive field development prospect articles and opinions are used as creative thinking divergent materials, that is, as the highest level of divergent corpus data sets.

[0074] First, according to the hierarchical relationship of the corpus dataset, determine the current corpus dataset of the lowest level that has not yet participated in the model training, and use the current corpus dataset to perform the current round of Lora fine-tuning training, that is, add a parameter matrix of low-rank structure in the specific layer of the initial large model. These matrices are multiplied with the weight matrix of the initial large model to adjust the representation ability of the model so that it can better adapt to the new learning objectives. Then determine the upper-level corpus dataset, that is, follow the basic, professional, and divergent corpus datasets preprocessed in the previous round, and perform three rounds of Lora fine-tuning training, step by step and iterate until the target large model is trained. Among them, Lora fine-tuning is a technology for optimizing large machine learning models by introducing low-rank matrices to adapt to new tasks or datasets.

[0075] In this embodiment, a high-performance, highly adaptable deep learning model parameter update method based on low-rank approximate decomposition of the weight matrix is ​​used to focus on adjusting a small part of the parameters in the model instead of comprehensively retraining the entire model. This method can improve training efficiency and reduce the consumption of computing resources while maintaining or improving the performance of the model. After three rounds of progressive fine-tuning, the target large model achieves self-awareness in identity recognition, and can focus on fields such as digital economy, technological innovation, and financial research, and can make decisions and analysis more efficiently.

[0076] In one embodiment, based on multiple corpus data sets, the target big model also clearly integrates the various research results of Peking University Changsha Institute of Computing and Digital Economy, highlighting the professional knowledge of the target big model. The target big model can accurately analyze some reports with regional characteristics, and review the development history and important nodes of the institute. Through review, it helps users understand the development of the institute, fully combine the results of the institute, and come up with corresponding answers and suggestions.

[0077] In one embodiment, the initial large model is trained by the current corpus data set to obtain the current large model, including: preprocessing the current corpus data set; determining the training parameters of the initial expert large model; updating the training parameters by the preprocessed current corpus data set until the training stop condition is reached, thereby obtaining the trained current large model.

[0078] Among them, preprocessing includes at least one of corpus batch processing, corpus classification structuring and cleaning; training parameters include at least one of learning rate, sampling parameters and network parameters. For example, the data sources used in the example of consulting big models are: financial entries in Chinese encyclopedias, financial QA data sets, financial data sets in DuReader, classic reports of research institutes (covering the fields of digital economy and technological innovation, specifically including big data, advanced computing, artificial intelligence, digital finance, data elements, data security, digital new infrastructure, audio and video industry, advanced manufacturing industry, etc.), and publicly published articles by research institutes with local regional characteristics of Hunan and Changsha, such as think tank research articles in the fields of digital economy and technological innovation, independent intellectual property reports, etc.

[0079] Specifically, the text that has undergone three rounds of processing can become a high-quality corpus data set for training the target large model. For each of the multiple corpus data sets in each round of training, the training steps can be performed through the following process. Install the environment dependency and CUDA driver, and select ChatGLM3-6B as the training base, for example. The training parameters after configuring the training Yaml file include learning rate, sampling parameters, batch_size, network parameters, etc., fill in the Lora training script to complete the logic of loading the model application configuration training and outputting the merged model, and then update the training parameters through the preprocessed current corpus data set until the training stop condition is reached, and the trained current large model is obtained.

[0080] Among them, the CUDA driver provides a complete software ecosystem, including programming language extensions, memory management, parallel programming models, tool chains, and library support to achieve high-performance computing. YAML files are usually used for configuration files, data exchange, and defining parameters and configuration settings in applications.

[0081] In this embodiment, by focusing the generalized answer output of the large model on the professional content in the niche field, the problem that the large model is unable to meet the needs of researchers and scholars for intelligent assistants with professional qualities due to generalized answers is solved. Fine-grained output alleviates the pain point of the large model's general and empty answers, improves the quality and depth of the answers, and expands the practical value space.

[0082] In one embodiment, the current corpus data set is preprocessed, including: if the data in the current corpus data set is complex text, the complex text is converted into initial text, and the initial text is supplemented by a local large model to obtain a valid text; according to the domain focus and corpus characteristics, the valid text is structurized into corpus classification.

[0083] Specifically, we first perform corpus batch processing to convert various reports in the current corpus dataset into initial text form, such as converting PDF, docx, and json files into txt files, where json and docx are extracted using Python's built-in and third-party json / docx parsers, respectively. PDF is a complex text, and we need to consider both processing efficiency and accuracy. We use Python's pdfplumber to convert complex text into initial text. Since the initial text extracted from complex text is often fragmented and incomplete, it needs to be refined and supplemented with other initial texts using a local large model to obtain valid text. Pdfplumber is a Python library for extracting text, tables, and other elements from PDF documents.

[0084] Next, the corpus classification and structuring process is carried out according to the field focus and corpus characteristics. Taking the economic field report as an example, it is disassembled and structured according to the background, development case, data support, main point, and title. Then data cleaning is required, and through manual observation, the analysis corpus without clear classification or reliable source in the data set is eliminated.

[0085] In this embodiment, complex text is converted into initial text, and further supplemented and structured, which can ensure the consistency of text content and facilitate the management and maintenance of the knowledge base. After data cleaning, the accuracy and relevance of the content are improved, the interference to the model is reduced, and the problem of slow convergence of large data set training is alleviated.

[0086] In one embodiment, the above method also includes: determining the number of rounds corresponding to the current corpus data set; when the number of rounds is greater than a preset value, determining the sub-models corresponding to the number of rounds of training for the initial large model; and selecting the current large model from multiple sub-models.

[0087] The number of rounds is determined in advance by setting the round parameter epoch. If the round parameter epoch is set to be greater than the preset value 1, the sub-models corresponding to the number of rounds of training for the initial large model are determined, and a current large model is loaded for testing and selected from the generated multiple sub-models, so that when the three rounds of corpus data sets are respectively subjected to three progressive Lora trainings, they are all based on the best and preferred sub-models trained by the previous level corpus data set.

[0088] In this embodiment, through multiple rounds of training and screening, it can be ensured that the finally selected model has good performance on various data, thereby improving the generalization ability of the model on unknown data and reducing the risk of overfitting of a single model to the training data.

[0089] In one embodiment, the process of initializing the vector database is as follows: pre-install a vector database, such as the Chroma database, and configure it to be persistent, and use the local large model as the document embeddings algorithm. Then create a database collection (Chroma Collection), and customize the distance calculation method of the embedding space through metadata parameters, such as using cosine similarity or Euclidean distance. If the document of the corpus is large, it is split, and the document embeddings are calculated by calling the local large model API, and the document related information is added to the collection as metadata.

[0090] The Embeddings algorithm is a method for converting discrete data points into continuous vector representations, such as words, sentences, images, etc. This vectorized representation can capture the semantic or contextual relationship between data points. Creating a database collection refers to the process of creating a collection when using the Chroma vector database, such as specifying the name of the collection and an optional embedding function.

[0091] In one embodiment, triggering an update operation on a vector database includes: determining keywords after segmenting content phrases; performing an online search based on the keywords, and determining update data based on a preset weight list; and generating interactive instructions associated with the update data so that the update data is stored in a corresponding collection of the vector database.

[0092] Among them, the vector database is initialized based on the local large model; the highest-level corpus dataset in the training target large model is the initial corpus of the vector database, such as forward-looking reports and newer corpora.

[0093] Specifically, when the backend recognizes the real-time need to update the knowledge base during the user dialogue with the target big model, it will start the background process network search. At this time, the content phrases are processed by, for example, jieba word segmentation, and the length is limited as keywords for network search. For example, the Request library is used to request the search engine and the preset list of high-quality websites in the professional field, and the content of the high-quality sites in the preset weight list is given priority. The determined update data is aggregated and the interactive instructions are generated and submitted to the target big model for refinement. After the structured documents are generated, they are stored in the corresponding collection of the vector database to achieve asynchronous and reliable updates, so that the system can complete the maintenance and update of the near real-time database during the communication between the user and the target big model.

[0094] Among them, jieba is a powerful Python Chinese word segmentation library that supports multiple word segmentation modes, including precise mode, full mode and search engine mode, and can perform part-of-speech tagging and add custom dictionaries. Request library refers to a third-party library in Python for initiating HTTP requests.

[0095] In this embodiment, based on the support of the vector database, the target large model system can retrieve the knowledge database to obtain near-real-time information, and during use, continuously supplement and update the database information, breaking through the limitations of the training set information timeliness and maintaining the near-real-time feature level of question and answer. At the same time, compared with temporary online search, this solution has the advantages of shorter answer delay, less impact from search channels, higher information robustness, and knowledge that can be screened and refined.

[0096] In one embodiment, the functional summary of the dialogue data processing system mainly includes providing an interactive front-end web interface for returning more divergent results in, for example, creativity mode and more conservative results in non-creativity mode. The functional summary also includes providing a target large model based on three rounds of progressive fine-tuning to assist decision-making, which is used to learn some general knowledge by using various corpus data sets such as open source general data in the fields of economics and finance; learning macro decision-making knowledge by using text data such as various research results of the research institute; and retrieving knowledge outside the corpus data set by using a searchable plug-in knowledge base.

[0097] In one embodiment, the dialog data processing system is a near real-time professional dialog system based on the ChatGLM target large model. Figure 4 As shown, Figure 4 This is the overall architecture diagram of the dialogue data processing system. It includes data sources, data input, data cleaning, data processing, data storage and visualization solutions. Data sources include open source general data sets, text data and external vector databases. The source of text data is research institute report data, etc. All data sources are imported into the memory to complete the data input, and the data cleaning process is completed by cleaning and refining the target large model. The cleaned data is used directly as the input command to control the target large model during the data processing process, such as the Prompt command. After obtaining the recognition result of the target large model, it is cached locally on the client and finally visualized on the Web interface. At the same time, the cleaned data is also used as a corpus data set for Lora fine-tuning training during the data processing process, and the trained target large model is exported and stored. Finally, the target result recognized by the target large model is visualized through the Web interface.

[0098] In one embodiment, the exported target large model is a bin model, which is usually used to store weight parameters of deep learning models, especially in the PyTorch framework. The advantage of this file format is its smaller size and faster loading speed, which makes it widely used in production environments.

[0099] To summarize, this system is based on an open source big model with excellent Chinese capabilities. It collects and pre-processes data in a targeted manner, and uses an efficient fine-tuning algorithm for progressive fine-tuning training to obtain the target big model. The model can deeply understand the data characteristics and semantic information of texts in professional fields such as digital economy and technological innovation. By constructing and maintaining an additional knowledge vector database, the target big model is endowed with the ability to retrieve real-time knowledge outside the training set, which fills a major shortcoming of the big model and enables the provision of specific, accurate and cutting-edge personalized answers and suggestions for consulting work in professional fields in basic dialogue scenarios.

[0100] At the same time, products that integrate the dialogue data processing system have an easy-to-use Web interface framework and practical output strategies that can switch between creativity and precision modes. They are suitable for the daily work of policy makers, think tank researchers, consultants, and corporate managers, and can provide all-round information support and decision-making assistance. At the same time, they have information reports for data collection and processing, intelligent question-answering and dialogue systems, report analysis functions, and text generation and translation functions.

[0101] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0102] Based on the same inventive concept, the embodiment of the present application also provides a conversation data processing device for implementing the conversation data processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more conversation data processing device embodiments provided below can refer to the above limitations on the conversation data processing method, and will not be repeated here.

[0103] In one embodiment, a conversation data processing device is provided, comprising: a data acquisition module, a preliminary recognition module and a secondary recognition module, wherein;

[0104] The data acquisition module is used to obtain the user's conversation data and determine the analysis mode, and the analysis mode is associated with the model control parameters.

[0105] The preliminary identification module is used to perform preliminary identification on the dialogue data through the target large model and model control parameters to obtain initial results; the initial results include content indicators related to the real-time nature of the demand.

[0106] The re-identification module is used to obtain related data from the vector database based on content indicators, and re-identify the related data through the target large model to obtain the target result.

[0107] Each module in the above-mentioned dialogue data processing can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0108] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O), a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for processing conversation data is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0109] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0110] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0111] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0112] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0113] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited thereto.

[0114] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for processing conversation data, characterized in that: The method comprises: Acquiring user conversation data and a determined analysis mode, wherein the analysis mode is associated with a model control parameter; Preliminary recognition of the conversation data is performed through the target large model and the model control parameters to obtain an initial result; the initial result includes content indicators related to the real-time nature of the demand; Based on the content index, the associated data is obtained from the vector database, and the associated data is re-identified through the target large model to obtain the target result.

2. The method according to claim 1, characterized in that: The target large model is obtained by pre-training with a variety of corpus data sets; There is a hierarchical relationship between different corpus datasets; The hierarchical relationship is reflected by the degree of characteristics represented by the data; The training process of the target large model includes: Determine a current corpus data set, and train an initial large model using the current corpus data set to obtain a current large model; A corpus data set at a previous level of the current corpus data set is determined, and the current large model is trained using the corpus data set at the previous level until a target large model is iteratively trained.

3. The method according to claim 2, characterized in that The training of the initial large model by the current corpus data set to obtain the current large model includes: Preprocessing the current corpus data set; the preprocessing includes at least one of corpus batch processing, corpus classification and structuring, and cleaning; Determining the training parameters of the initial expert large model; the training parameters include at least one of a learning rate, a sampling parameter and a network parameter; The training parameters are updated through the preprocessed current corpus data set until the training stop condition is reached, thereby obtaining the trained current large model.

4. The method according to claim 3, characterized in that: The preprocessing of the current corpus data set includes: If the data in the current corpus data set is a complex text, convert the complex text into an initial text, and use the local large model to supplement the initial text to obtain a valid text; According to the focus of the field and the characteristics of the corpus, the valid text is classified and structured.

5. The method according to claim 2, characterized in that: The method further comprises: Determine the number of rounds corresponding to the current corpus data set; When the number of rounds is greater than a preset value, determining the sub-models corresponding to the initial large model when the initial large model is trained for the number of rounds; A current large model is selected from the plurality of sub-models.

6. The method according to claim 1, characterized in that The content index includes a content phrase and a relevance index; the vector database includes a plurality of sets; and obtaining associated data from the vector database based on the content index includes: When the relevance index is greater than a preset value, querying the associated data from a set of vector databases according to the content phrase; The method further includes: when the correlation index is greater than a target value or the associated data queried from the set of vector databases is empty, triggering an update operation on the vector database; the updated vector database is used to obtain new associated data.

7. The method according to claim 6, characterized in that The vector database is initialized based on the local large model; the highest-level corpus data set in the target large model is used as the initial corpus of the vector database; The triggering of the update operation on the vector database includes: Determining keywords after segmenting the content phrases; Conduct online searches based on keywords and determine updated data based on a preset weight list; Generate an interactive instruction associated with the update data so that the update data is stored in a corresponding set of the vector database.

8. The method according to claim 6, characterized in that The initial result also includes an initial answer; the method also includes: when the relevance index is less than a preset value, format parsing the initial answer and calling a renderer to display the user interface.

9. A conversation data processing system, characterized in that: The system includes a front end, a target large model, a vector database and a back end, wherein: The front end is used to obtain the user's conversation data and determine the analysis mode, and the analysis mode is associated with the model control parameters; A target large model, used to perform preliminary recognition on the dialogue data based on the model control parameters to obtain an initial result; the initial result includes content indicators related to the real-time nature of the demand; A backend, used for making an interface call based on the initial result to trigger a query instruction or an update instruction for the vector database; A vector database, for responding to a query instruction and determining associated data according to the content indicator; The target large model is also used to re-identify the associated data to obtain the target result.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images