A real-time statistical method and system for high-frequency input problems based on vector database

Abstract

The present invention provides a real-time statistical method and system for high-frequency input problems based on a vector database. The method has good adaptability and computational efficiency in the face of various text inputs, effectively avoiding the problems of long running time and large resource occupation of clustering algorithms and the influence of the preset number of categories on the clustering algorithms. When facing high-dimensional text data vectors, extremely low memory resources can be used, with high clustering efficiency, and the stability of the clustering algorithm is also improved. The method can automatically perform high-accuracy clustering based on semantic high-dimensional information, ensuring high statistical efficiency. The present invention makes full use of the summarizing ability of a large language model to summarize multiple input texts in a field, and the high-frequency problems obtained based on this method have high representativeness and professionalism. The present invention can count high-frequency problems in different time periods in real time, and if re-clustering is required, there is no need to perform repeated clustering operations, thereby improving the statistical freedom and statistical efficiency of high-frequency problem statistics.

Description

Technical Field

[0001] The present invention belongs to the technical field of data statistics, and in particular relates to a real-time statistics method and system for high-frequency input problems based on a vector database. Background Art

[0002] Statistical analysis of high-frequency questions can effectively extract hot issues that users are concerned about, making it easier for staff to conduct in-depth mining based on hot issues, measure the importance of content or topics, and collect and update information on related topics in a timely manner. Especially in the knowledge base question-answering system based on retrieval enhancement generation, the real-time statistical method of high-frequency questions can effectively feedback to managers the problem areas that users are concerned about, increase the frequency of knowledge base updates, and avoid the update of invalid knowledge. While greatly improving management efficiency, users also get a better user experience.

[0003] To achieve statistics on high-frequency questions, traditional solutions usually use manual statistics or keyword frequency statistics. Manual statistics are time-consuming and labor-intensive, and keyword frequency statistics lack important semantic information. With the rapid development of artificial intelligence technology, statistical solutions based on question text encoding clustering have become increasingly popular. Although this type of solution takes semantic information into account, the clustering method has poor clustering effect and high clustering cost when facing a large amount of high-dimensional data, and the clustering needs to be re-clustered when the clustering conditions are modified. Some classification algorithms also require the number of categories to be preset, which is very limited when facing a variety of problems, such as K-Means clustering algorithm, DBSCAN clustering algorithm, etc. Therefore, it is urgent to design and implement a high-accuracy question frequency statistics solution with no fixed categories, high degrees of freedom, and real-time clustering of high-dimensional data in the scenario where users input a large number of questions. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a real-time statistical method and system for high-frequency input problems based on a vector database. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0005] In a first aspect, the present invention provides a real-time statistical method for high-frequency input problems based on a vector database, comprising:

[0006] S100, obtaining a current question text encoding vector database and receiving a target text input by a user at the current time;

[0007] S200, vectorizing and encoding the target text to obtain a first high-dimensional semantic feature vector;

[0008] S300, based on the semantic similarity between the first high-dimensional semantic feature vector and a plurality of second high-dimensional semantic feature vectors retrieved from the question text encoding vector database at the current time, confirming the cluster to which the target text conforms, and updating it to the cluster to obtain an updated question text encoding vector database;

[0009] S400, searching for a target cluster that meets the query condition in the updated question text encoding vector database, selecting a query question text in the target cluster and inputting it into the large language model to obtain an output question text that summarizes all query question texts;

[0010] S500: Display the output question text to the user.

[0011] In a second aspect, the present invention provides a real-time statistical system for high-frequency input problems based on a vector database, comprising:

[0012] An input module is configured to obtain a question text encoding vector database at the current time and receive a target text input by a user at the current time;

[0013] An encoding module, configured to perform vectorization encoding on the target text to obtain a first high-dimensional semantic feature vector;

[0014] An updating module is configured to determine the cluster to which the target text conforms by calculating the semantic similarity between the first high-dimensional semantic feature vector and the second high-dimensional semantic feature vector in the question text encoding vector database at the current time, and update the target text to the cluster to obtain an updated question text encoding vector database;

[0015] A query module is configured to search for a target cluster that meets the query condition in the updated question text encoding vector database, select a query question text from the target cluster and input it into the large language model to obtain an output question text that summarizes all query question texts;

[0016] The display module is configured to display the output question text to the user.

[0017] Beneficial effects:

[0018] 1. The present invention provides a real-time statistical method and system for high-frequency problems based on a vector database. Compared with traditional manual statistics or solutions based on word frequency statistics of high-frequency problems, the present invention can automatically perform high-accuracy clustering based on semantic high-dimensional information, while improving the clustering effect and ensuring higher statistical efficiency.

[0019] 2. The present invention provides a real-time statistical method and system for high-frequency problems based on a vector database. Compared with the solution of combining text encoding with traditional clustering methods, the present invention can use extremely low memory resources when facing high-dimensional text data vectors, has higher clustering efficiency, and also improves the stability of the clustering algorithm.

[0020] 3. The present invention provides a real-time statistical method and system for high-frequency problems based on a vector database. Compared with traditional fixed-category and non-fixed-category clustering algorithms, the present invention has better adaptability and computational efficiency in the face of a variety of text inputs, effectively avoiding the problems of long running time and large resource occupation of clustering algorithms, and also avoiding the influence of the preset number of categories on the clustering algorithm.

[0021] 4. The present invention provides a real-time statistical method and system for high-frequency problems based on a vector database. When extracting high-frequency problems, the present invention does not select a high-frequency text as the high-frequency problem of this category, but makes full use of the summarizing ability of the large language model to summarize multiple input texts in the field. The high-frequency problems obtained based on this method are highly representative and professional.

[0022] 5. The present invention provides a real-time statistical method and system for high-frequency problems based on a vector database. Compared with existing statistical solutions for high-frequency problems, the present invention can count high-frequency problems in different time periods in real time without repeated clustering operations, which improves the statistical freedom and efficiency of high-frequency problem statistics.

[0023] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of a real-time statistical method for high-frequency input problems based on a vector database provided by the present invention;

[0025] Figure 2 It is a structural schematic diagram of a real-time statistical system for high-frequency input problems based on a vector database provided by the present invention. DETAILED DESCRIPTION

[0026] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0027] like Figure 1 As shown, the present invention provides a real-time statistical method for high-frequency input problems based on a vector database, including:

[0028] S100, obtaining a current question text encoding vector database and receiving a target text input by a user at the current time;

[0029] The current question text encoding vector database includes multiple question texts, dimensional semantic feature vectors corresponding to the question texts, dates of the question texts and IDs of the question texts; and the multiple question texts belong to different clusters according to semantic similarity.

[0030] For the convenience of distinction, the present invention uses the question text input by the user at that time as the target text, and the question text encoding vector stored in the database is called the question text.

[0031] The present invention needs to cyclically update the question text coding vector database. At the beginning of the cycle, that is, in the first cycle, it is necessary to build and initialize the question text coding vector database based on the Milvus database. At this time, the question text coding vector database is empty. The question text coding vector database is used for structured storage of information such as question text ID, question text, question text coding, question text input date, etc., wherein the question text ID is managed by the Milvus database automatically, so in the subsequent structured data, the structured data does not need to manage and add the question text ID.

[0032] S200, vectorizing and encoding the target text to obtain a first high-dimensional semantic feature vector;

[0033] In this step, the question text embedding coding model can be used to embed the target text into a vector encoding to obtain a high-dimensional semantic feature vector of the target text, and the high-dimensional semantic feature vector is used as the first high-dimensional semantic feature vector.

[0034] In this step, the question text embedding coding model can use the embedding coding model bge-m3 to embed the question text into a vector, and the output dimension is 1024. At this time, the high-dimensional semantic feature vector obtained has the semantic information of the target text. In this step, the high-dimensional semantic feature vector of the target text is obtained and combined with the target text, the ID of the target text, the input date of the target text and other information to form structured data, and then stored in the question text encoding vector database. During storage, the question text ID will be managed by the database itself, and no operation is required. If the database cannot manage the question text ID, the question text ID needs to be added, and the unique question text ID generated by the time of question text input is added to the structured data.

[0035] S300, based on the semantic similarity between the first high-dimensional semantic feature vector and a plurality of second high-dimensional semantic feature vectors retrieved from the question text encoding vector database at the current time, confirming the cluster to which the target text conforms, and updating it to the cluster to obtain an updated question text encoding vector database;

[0036] The present invention uses the updated question text encoding vector database as the question text encoding vector database for the next time, so that the question text encoding vector database can be cyclically updated by repeating S100-S300 for the next time.

[0037] S400, searching for a target cluster that meets the query condition in the updated question text encoding vector database, selecting a query question text in the target cluster and inputting it into the large language model to obtain an output question text that summarizes all query question texts;

[0038] S500: Display the output question text to the user.

[0039] As an optional implementation manner of the present invention, S300 includes:

[0040] S310, setting a statistical time period for high-frequency issues, with days as the minimum statistical unit;

[0041] For example, the required statistical time period is set, with days as the minimum statistical unit, the statistical start date and end date are set, and the time filter condition is automatically generated. For example, the start date October 17, 2024 is set as "2024-10-17", and the end date October 21, 2024 is set as "2024-10-21". At this time, the filter condition is automatically generated:

[0042] [“2024-10-17”,”2024-10-18”,”2024-10-19”,”2024-10-20”,”2024-10-21”]

[0043] S320, using the statistical time period as a filtering condition, filtering the second high-dimensional semantic feature vectors in the question text encoding vector database of the current time, and obtaining a plurality of second high-dimensional semantic feature vectors that meet the filtering condition;

[0044] S330, calculating the similarity scores between the first high-dimensional semantic feature vector and the second high-dimensional semantic feature vector, and selecting the second high-dimensional semantic feature vectors with top N confidences as N search results according to the principle of high to low confidence;

[0045] The database search parameters use the default parameters L2 metric parameters, IVF_FLAT memory index parameters, no score threshold filtering is set, and the number of retrievals is limited to N vectors. The search results must include the cluster category number, the original question text, and the similarity score;

[0046] S340, inputting the N search results and the first high-dimensional semantic feature vector into a semantic search ranking model, obtaining an output result of the semantic search ranking model in the current cycle, and using the output result as a final semantic similarity score; the output result includes a semantic similarity score between the target text at the current time and the N input question texts corresponding to the N search results;

[0047] In order to further improve the clustering effect, this step introduces the semantic retrieval ranking model bge-reranker-v2-m3 to further sort the N search results and improve the judgment effect of the semantic similarity of the search results. The output result of the retrieval ranking model is used as the final semantic similarity score. The score range is [0,1]. The higher the score, the more semantically similar it is. The search results are sorted based on the semantic similarity score.

[0048] S350, using the final semantic similarity score, confirm the cluster to which the target text at the current time belongs, and update it to the cluster to obtain an updated question text encoding vector database.

[0049] As an optional implementation manner of the present invention, S350 includes:

[0050] S351, for each final semantic similarity score, determining whether the final semantic similarity score exceeds a semantic similarity threshold, and if so, confirming that the target text at the current time hits the cluster to which the corresponding input question text belongs;

[0051] S352, counting the number of times each cluster is hit by the target text at the current time, and using the number and the final semantic similarity score to calculate the mean semantic similarity score of each cluster; the mean semantic similarity score is expressed by the formula:

[0052]

[0053] Among them, score_i represents the semantic similarity score of the i-th input question text in any cluster, n represents the total number of clusters hit by the target text, and e_score represents the mean semantic similarity score;

[0054] S353, selecting the cluster with the highest mean semantic similarity score as the cluster to which the target text at the current time belongs, and updating it to the cluster to obtain an updated question text encoding vector database.

[0055] S354, if each final semantic similarity score does not exceed the semantic similarity threshold, it is confirmed that the target text at the current time does not hit any cluster; a new cluster is created in the encoding vector database of the question text at the current time, and the target text that does not hit any cluster is assigned to the new cluster.

[0056] The present invention sets a default similarity score threshold of 0.8. If the similarity score threshold is exceeded, it is considered that the target text input by the user hits the cluster, and the number of hits of the cluster is increased by 1 and added to the total semantic similarity score of the cluster. The semantic similarity total score of the cluster is divided by the number of hits of the cluster to calculate the mean semantic similarity between the cluster and the target text input by the user. The clusters are sorted in descending order by the semantic similarity mean, and the highest score is considered as the clustering result of the text. If no cluster is hit, it is considered that the target text belongs to a new category, and a new set is created in the database as the storage location of the cluster. The maximum number of clusters is 4095. If it exceeds this value, the clusters with the smallest number and the longest establishment time are deleted based on the number of elements in the cluster and the order of cluster establishment.

[0057] As an optional implementation manner of the present invention, S400 includes:

[0058] S410, setting a statistical time period for high-frequency questions, and taking days as the minimum statistical unit, and using the statistical time period as a filtering condition; using the filtering condition to filter the updated question text encoding vector database to obtain a number of clusters that meet the filtering condition;

[0059] S420, sorting all clusters according to the number of question texts in each cluster and the time when the cluster was established to obtain sorting results of all clusters;

[0060] As an optional implementation manner of the present invention, S420 includes:

[0061] S421, for each cluster, counting the number of question texts in the cluster and the time when the cluster was established;

[0062] S422, sorting all clusters according to the number of question texts. If there are re-sorted clusters with the same number of question texts during the sorting process, sorting all re-sorted clusters according to the creation time of the clusters to obtain the sorting results of all clusters.

[0063] The present invention counts the number of elements in each cluster within the time period, and sorts all clusters in descending order based on the number of elements. When the number of elements in the clusters is consistent, the clusters with the consistent number are sorted based on the order of cluster establishment, with the newly established cluster being ranked first.

[0064] S430, generating a text list for each cluster according to the principle of sorting the question texts in the time when they were stored in the question text encoding vector database first and then in the last;

[0065] S440, reordering the text list of each cluster based on the binary idea to obtain a reordered text list;

[0066] Each cluster is statistically analyzed to obtain a text list. The closer to the end of the list, the shorter the time the text has entered the question bank. At this time, the text list of each cluster is reordered based on the idea of ​​​​half-half, that is, the question text is recursively extracted from the back, middle and front of the text list to form a new text list, for example:

[0067] A list of original questions in a cluster [question text 1, question text 2, question text 3, question text 4, question text 5, question text 6]

[0068] The list of questions sorted by the binary idea [Question text 6, question text 4, question text 1, question text 5, question text 3, question text 2].

[0069] S450, selecting the first cluster in the cluster sorting result as the target cluster, and selecting a plurality of question texts ranked first in the target cluster in the re-sorted text list as query question texts;

[0070] S460, inputting the query question text into the large language model to obtain a question text that summarizes all query question texts.

[0071] The large language model uses the Qwen2-7B-Instruct large language model. In order to take into account both reasoning efficiency and computing resource service capabilities, the maximum number of tokens needs to be set. For example, the maximum token is set to 4096. At this time, the prompt word occupies 790 tokens, and the cluster enters the large language model for question summary. The sum of tokens is 3306. At this time, if the target cluster has more elements in the question list, they cannot all be input into the large language model. At this time, by calculating the tokens of each question text, when the sum of the tokens of an element and the previous n elements exceeds 3306, only the first n elements combined with the prompt word are input into the large model.

[0072] The question summary prompt word template based on the Qwen2-7B-Instruct large language model can refer to the following template:

[0073] "question_summary":(

[0074] "The following is a single or multiple user input texts extracted from the question and answer database. Please condense and summarize the content of the text list into a professional main content, express it in natural language, and the number of words is within 15 characters.\n"

[0075] "You need to follow the following rules.\n"

[0076] "1. Check the output format. If it does not conform to the Json format, re-output it according to Json.\n"

[0077] "2. Check the output word count. If it exceeds 20 words, re-output the condensed result.\n"

[0078] "3. If the summary cannot be performed or there is no valid information, the extraction result will be NULL.\n\n"

[0079] "Example:\n"

[0080] "Text list: [\"What are the specific steps to deploy torch-gpu on a physical machine? \",\"What hardware and operating system environments need to be prepared when deploying torch-gpu in a containerized manner? \",\"Torch-gpu containerized deployment steps\"]\n"

[0081] "Output the main content of multiple paragraphs in Json format:\n"

[0082] "{\"output\":\"torch-gpu deployment steps and environment requirements\"}\n"

[0083] "Text list: {{input_text}}\n"

[0084] "Output the main content of multiple paragraphs in Json format:\n" )

[0086] The present invention S400 obtains the high-frequency problems of each cluster within the statistical time period and the statistical frequency of the corresponding problems. If statistics are required for other time periods, the start time and the end time are reset to modify the statistical time period without repeated clustering operations.

[0087] like Figure 2 As shown, the present invention provides a real-time statistical system for high-frequency input problems based on a vector database, comprising:

[0088] An input module 21 is configured to obtain a current question text encoding vector database and receive a target text input by a user at the current time;

[0089] An encoding module 22, configured to perform vectorization encoding on the target text to obtain a first high-dimensional semantic feature vector;

[0090] The updating module 23 is configured to confirm the cluster to which the target text conforms based on the semantic similarity between the first high-dimensional semantic feature vector and a plurality of second high-dimensional semantic feature vectors retrieved from the question text encoding vector database at the current time, and update the cluster to obtain an updated question text encoding vector database in the cluster;

[0091] The query module 23 is configured to search for a target cluster that meets the query condition in the updated question text encoding vector database, select a query question text in the target cluster and input it into the large language model to obtain a question text that summarizes all query question texts;

[0092] The display module 25 is configured to display the question text to the user.

[0093] The device embodiment of the present invention corresponds to the method embodiment, and the implementation details of each module correspond to the method implementation details, which will not be repeated here.

[0094] The present invention provides a real-time statistical method and system for high-frequency input problems based on a vector database. The method has good adaptability and operation efficiency in the face of various text inputs, effectively avoids the problems of long running time and large resource occupation of clustering algorithms, and also avoids the influence of the preset number of categories on the clustering algorithm. When facing high-dimensional text data vectors, extremely low memory resources can be used, with high clustering efficiency, and the stability of the clustering algorithm is also improved. The method can automatically perform high-accuracy clustering based on semantic high-dimensional information, and ensure high statistical efficiency while improving the clustering effect. When extracting high-frequency problems, the present invention makes full use of the summarizing ability of a large language model to summarize multiple input texts in the field, and the high-frequency problems obtained based on this method have high representativeness and professionalism. For the problem that re-clustering is required when setting new clustering conditions, the present invention can count high-frequency problems in different time periods in real time, and there is no need to perform repeated clustering operations, thereby improving the statistical freedom and statistical efficiency of high-frequency problem statistics.

[0095] It is worth noting that the terms "first" and "second" in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0096] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality of components or steps.

[0097] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A real-time statistical method for high-frequency input problems based on a vector database, characterized in that: include: S100, obtaining a current question text encoding vector database and receiving a target text input by a user at the current time; S200, vectorizing and encoding the target text to obtain a first high-dimensional semantic feature vector; S300, based on the semantic similarity between the first high-dimensional semantic feature vector and a plurality of second high-dimensional semantic feature vectors retrieved from the question text encoding vector database at the current time, confirming the cluster to which the target text conforms, and updating it to the cluster to obtain an updated question text encoding vector database; S400, searching for a target cluster that meets the query condition in the updated question text encoding vector database, selecting a query question text in the target cluster and inputting it into the large language model to obtain an output question text that summarizes all query question texts; S500: Display the output question text to the user.

2. The real-time statistical method for high-frequency input problems based on vector database according to claim 1 is characterized in that: S200 includes: The target text is embedded in a vector and encoded using a question text embedding coding model to obtain a high-dimensional semantic feature vector of the target text, and the high-dimensional semantic feature vector is used as a first high-dimensional semantic feature vector.

3. The real-time statistical method for high-frequency input problems based on vector database according to claim 1 is characterized in that: The current question text encoding vector database includes multiple question texts, dimensional semantic feature vectors corresponding to the question texts, dates of the question texts and IDs of the question texts; the multiple question texts belong to different clusters according to semantic similarity.

4. The real-time statistical method for high-frequency input problems based on vector database according to claim 3 is characterized in that: S300 includes: S310, setting a statistical time period for high-frequency issues, with days as the minimum statistical unit; S320, using the statistical time period as a filtering condition, filtering the second high-dimensional semantic feature vectors in the question text encoding vector database of the current time, and obtaining a plurality of second high-dimensional semantic feature vectors that meet the filtering condition; S330, calculating the similarity scores between the first high-dimensional semantic feature vector and the second high-dimensional semantic feature vector, and selecting the second high-dimensional semantic feature vectors with top N confidences as N search results according to the principle of high to low confidence; S340, inputting the N search results and the first high-dimensional semantic feature vector into a semantic search ranking model, obtaining an output result of the semantic search ranking model in the current cycle, and using the output result as a final semantic similarity score; the output result includes a semantic similarity score between the target text at the current time and the N input question texts corresponding to the N search results; S350, using the final semantic similarity score, confirm the cluster to which the target text at the current time belongs, and update it to the cluster to obtain an updated question text encoding vector database.

5. The real-time statistical method for high-frequency input problems based on vector database according to claim 4 is characterized in that: S350 includes: S351, for each final semantic similarity score, determining whether the final semantic similarity score exceeds a semantic similarity threshold, and if so, confirming that the target text at the current time hits the cluster to which the corresponding input question text belongs; S352, counting the number of times each cluster is hit by the target text at the current time, and using the number and the final semantic similarity score to calculate the mean semantic similarity score of each cluster; the mean semantic similarity score is expressed by the formula: Among them, score_i represents the semantic similarity score of the i-th input question text in any cluster, n represents the total number of clusters hit by the target text, and e_score represents the mean semantic similarity score; S353, selecting the cluster with the highest mean semantic similarity score as the cluster to which the target text at the current time belongs, and updating it to the cluster to obtain an updated question text encoding vector database.

6. The real-time statistical method for high-frequency input problems based on vector database according to claim 5 is characterized in that: After S351, the real-time statistical method for high-frequency input problems based on the vector database further includes: S354, if each final semantic similarity score does not exceed the semantic similarity threshold, it is confirmed that the target text at the current time does not hit any cluster; a new cluster is created in the encoding vector database of the question text at the current time, and the target text that does not hit any cluster is assigned to the new cluster.

7. The real-time statistical method for high-frequency input problems based on vector database according to claim 1 is characterized in that: After S300, the real-time statistical method for high-frequency input problems based on a vector database further includes: The updated question text encoding vector database is used as the question text encoding vector database for the next time.

8. The real-time statistical method for high-frequency input problems based on vector database according to claim 1 is characterized in that: S400 includes: S410, setting a statistical time period for high-frequency questions, and taking days as the minimum statistical unit, and using the statistical time period as a filtering condition; using the filtering condition to filter the updated question text encoding vector database to obtain a number of clusters that meet the filtering condition; S420, sorting all clusters according to the number of question texts in each cluster and the time when the cluster was established to obtain sorting results of all clusters; S430, generating a text list for each cluster according to the principle of sorting the question texts in the time when they were stored in the question text encoding vector database first and then in the last; S440, reordering the text list of each cluster based on the binary idea to obtain a reordered text list; S450, selecting the first cluster in the cluster sorting result as the target cluster, and selecting a plurality of question texts ranked first in the target cluster in the re-sorted text list as query question texts; S460, inputting the query question text into the large language model to obtain an output question text summarizing all query question texts.

9. The real-time statistical method for high-frequency input problems based on vector database according to claim 8 is characterized in that: S420 includes: S421, for each cluster, counting the number of question texts in the cluster and the time when the cluster was established; S422, sorting all clusters according to the number of question texts. If there are re-sorted clusters with the same number of question texts during the sorting process, sorting all re-sorted clusters according to the creation time of the clusters to obtain the sorting results of all clusters.

10. A real-time statistical system for high-frequency input problems based on a vector database, characterized in that: include: An input module is configured to obtain a question text encoding vector database at the current time and receive a target text input by a user at the current time; An encoding module, configured to perform vectorization encoding on the target text to obtain a first high-dimensional semantic feature vector; An updating module is configured to determine the cluster to which the target text conforms by calculating the semantic similarity between the first high-dimensional semantic feature vector and the second high-dimensional semantic feature vector in the question text encoding vector database at the current time, and update the target text to the cluster to obtain an updated question text encoding vector database; A query module is configured to search for a target cluster that meets the query condition in the updated question text encoding vector database, select a query question text from the target cluster and input it into the large language model to obtain an output question text that summarizes all query question texts; The display module is configured to display the output question text to the user.

Images