Data clustering method and device based on large model, medium and computer equipment

Abstract

The invention discloses a data clustering method and device based on a large model, a medium and computer equipment, and relates to the technical field of big data processing and financial science and technology. The method comprises the following steps: acquiring to-be-processed data, and randomly sampling the to-be-processed data for multiple times to obtain multiple groups of sample data; performing clustering processing on each group of sample data through a pre-trained large model to obtain a plurality of initial category names of each group of sample data; performing dimension reduction processing on the plurality of initial category names of the plurality of groups of sample data through the large model to obtain a plurality of target category names of the to-be-processed data; and performing classification processing on the data in the to-be-processed data one by one through the plurality of target category names to obtain a clustering result of the to-be-processed data. The method can prevent the input data from exceeding the maximum context window limit supported by the large model, can reduce the computing resource consumption of the large model, and improves the accuracy of the clustering result.

Description

Technical Field

[0001] The present application relates to the field of big data processing and financial technology, and in particular to a data clustering method, device, medium and computer equipment based on a large model. Background Art

[0002] As an important unsupervised learning method, clustering algorithm has been widely used in big data processing in insurance systems, banking systems, trading systems, order systems and other fields. The core goal of clustering algorithm is to automatically divide a set of data into multiple categories so that the data in the same category have high similarity, while the data in different categories have low similarity. At present, traditional clustering methods such as K-means and hierarchical clustering can provide general solutions for data clustering, but they will face problems such as low computational efficiency and insufficient accuracy when dealing with complex feature spaces and large-scale data sets.

[0003] In recent years, large models (large language models) have been gradually applied to clustering algorithms due to their powerful natural language understanding and generation capabilities. The method of clustering using large model technology mainly relies on prompt word technology, that is, by designing reasonable prompt words, the large model can understand the user's input intention and generate clustering results that meet the requirements. This method can not only realize the automatic grouping of data, but also automatically generate descriptive group names based on the characteristics of the data, thereby improving the interpretability of the clustering results.

[0004] However, in order for the large model to fully understand and process the entire data set, all relevant data often needs to be input at once during clustering. For data sets with tens of thousands or even more entries, even if each piece of data only occupies dozens of characters, the amount of data that needs to be processed may easily exceed the maximum context window limit supported by most existing large models. In addition, even if the input length limit can be overcome, it still requires a huge computational cost, and the accuracy of the clustering results is often low. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a data clustering method, apparatus, medium and computer equipment based on a large model, the main purpose of which is to solve the technical problems that the data input amount of the large model in the clustering task is likely to exceed the context window limit, the computational cost required for clustering is high, and the clustering results are inaccurate.

[0006] According to one aspect of the present application, a data clustering method based on a large model is provided, the method comprising:

[0007] Acquire data to be processed, and perform multiple random sampling on the data to be processed to obtain multiple groups of sample data;

[0008] Through the pre-trained large model, each group of sample data is clustered to obtain multiple initial category names for each group of sample data;

[0009] Through the large model, multiple initial category names of multiple groups of sample data are subjected to dimensionality reduction processing to obtain multiple target category names of the data to be processed;

[0010] By using multiple target category names, each data in the data to be processed is classified one by one to obtain a clustering result of the data to be processed.

[0011] According to another aspect of the present application, a data clustering device based on a large model is provided, the device comprising:

[0012] A data sampling module is used to obtain the data to be processed and perform multiple random sampling on the data to be processed to obtain multiple groups of sample data;

[0013] The data clustering module is used to perform clustering processing on each group of sample data through a pre-trained large model to obtain multiple initial category names for each group of sample data;

[0014] A data dimension reduction module, used to perform dimension reduction processing on multiple initial category names of multiple groups of sample data through the large model to obtain multiple target category names of the data to be processed;

[0015] The data classification module is used to classify each data in the data to be processed one by one according to multiple target category names to obtain a clustering result of the data to be processed.

[0016] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned large model-based data clustering method is implemented.

[0017] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned large model-based data clustering method when executing the program.

[0018] By means of the above technical scheme, a data clustering method and device based on a large model, a storage medium and a computer device provided in an embodiment of the present application first obtain the data to be processed, and perform multiple random sampling on the data to be processed to obtain multiple groups of sample data, and then perform clustering processing on each group of sample data through the pre-trained large model to obtain multiple initial category names of each group of sample data, wherein the initial category can represent the initial clustering result of each group of sample data, and then perform dimensionality reduction processing on the multiple initial category names of the multiple groups of sample data through the large model to obtain multiple target category names of the data to be processed, wherein the target category name can represent the common clustering result of the multiple groups of sample data, and finally, through the multiple target category names, classify each data in the data to be processed one by one to obtain the clustering result of the data to be processed. The above method converts the clustering task of the data to be processed with a large amount of data into the classification task of each data in the data to be processed. On the one hand, it can avoid the input data from exceeding the maximum context window limit supported by the large model, thereby reducing the requirements of the clustering task on the large model, and on the other hand, it can reduce the computing resource consumption of the large model and improve the accuracy of the clustering result by multiple simple calls.

[0019] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 A schematic diagram of an application environment of a data clustering method based on a large model provided in an embodiment of the present application is shown;

[0022] Figure 2 A schematic diagram of a process of a data clustering method based on a large model provided in an embodiment of the present application is shown;

[0023] Figure 3 A schematic diagram of the structure of a data clustering device based on a large model provided in an embodiment of the present application is shown;

[0024] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The data clustering method based on a large model provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, the client can communicate with the server through the network. The server can receive the data to be processed through the client, and perform multiple random sampling on the data to be processed to obtain multiple groups of sample data, and then perform clustering processing on each group of sample data through the pre-trained large model to obtain multiple initial category names of each group of sample data, and then perform dimensionality reduction processing on the multiple initial category names of the multiple groups of sample data through the large model to obtain multiple target category names of the data to be processed, and finally, through the multiple target category names, classify each data in the data to be processed one by one to obtain the clustering result of the data to be processed, and feed back the clustering result of the data to be processed to the client. In the present invention, by converting the clustering task of the data to be processed with a large amount of data into the classification task of each data in the data to be processed, on the one hand, it can avoid the input data exceeding the maximum context window limit supported by the large model, thereby reducing the requirements of the clustering task on the large model, and on the other hand, it can reduce the computing resource consumption of the large model by multiple simple calls, and improve the accuracy of the clustering result. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0027] In one embodiment, Figure 1 As shown, a data clustering method based on a large model is provided, and the method is applied to a computer device such as a client or a server as an example for explanation, including the following steps:

[0028] Step 101, obtaining data to be processed, and performing multiple random sampling on the data to be processed to obtain multiple groups of sample data.

[0029] The data to be processed refers to the data set to be clustered, which is usually a huge amount of data, that is, big data. The data to be processed may contain tens of thousands or even more entries, each of which may be text, numbers or other types of data. For example, in the field of financial technology, the data to be processed may include at least one type of data such as all transaction data, payment data, business data or purchase data within a period of time. In this scenario, the computer device can obtain transaction data in the transaction pool and obtain various types of data from other places such as databases and storage media as data to be processed.

[0030] Specifically, the computer device can collect or import the data to be processed at the data source, and use a variety of methods to perform multiple random sampling on the data to be processed. For example, multiple sampling methods such as repeated random sampling, simple random sampling, stratified sampling, and systematic sampling can be used to extract multiple groups of sample data. Among them, each random sampling can obtain a set of independent and representative sample data, and the number of samples extracted each time can be determined based on the processing capacity and context window limit of the large model and the total amount of data to be processed, so as to ensure that the sample data extracted each time can be within the tolerance range of the large model, and the representativeness of the sample data for the overall data can be ensured. For example, assuming that the total amount of data to be processed is 60,000, 3,000 data can be randomly extracted each time as a group of sample data, and extracted 3 times in total, then 3 groups of sample data can be obtained. In this example, the 3,000 randomly extracted data are not repeated, and the 3,000 data in each of the 3 groups of sample data can be crossed with each other.

[0031] In this embodiment, by performing multiple random sampling on the data to be processed, multiple groups of different sample data can be obtained, which will be used for subsequent clustering processing, and different clustering results can be obtained by using the randomness of the large model. In this way, the diversity and comprehensiveness of the clustering results can be improved, thereby helping to more accurately identify the various classification categories of the data to be processed in the subsequent clustering process.

[0032] Step 102, clustering each group of sample data is performed using a pre-trained large model to obtain multiple initial category names for each group of sample data.

[0033] Among them, the large model refers to a large language model (LLM), a computer vision model (CV), a multimodal model (LMMs), and other advanced artificial intelligence models that can handle complex tasks and large amounts of data. In this embodiment, the pre-trained large model can be a large model that has been adjusted with data from a preset field, or a large model in a general field that has been trained, and this embodiment does not make specific restrictions here.

[0034] Specifically, the large model can perform clustering processing on each group of sample data respectively to obtain multiple initial category names for each group of sample data, wherein the initial category names can represent the initial clustering results of the group of sample data and serve as the basis for subsequent dimensionality reduction processing and classification processing. In this embodiment, the clustering operation can be performed based on the similarity between the various data within the sample through the pre-trained large model and the pre-set prompt words. In the clustering process, the large model can identify the semantics of different data and classify the various data items in the sample into several categories. For each formed category, the large model can automatically generate a name that describes the characteristics of the category as the initial category name. This process can be performed once or repeatedly performed multiple times until the number of initial category names for each group of sample data is reduced to a preset number range, or the generated initial category names can no longer be clustered.

[0035] In this embodiment, by clustering multiple groups of sample data and obtaining clustering results corresponding to each group of sample data, multiple initial category names corresponding to each group of sample data can be obtained, thereby providing richer and more comprehensive category information for the data to be processed.

[0036] Step 103, using the large model, dimensionality reduction processing is performed on multiple initial category names of multiple groups of sample data to obtain multiple target category names of the data to be processed.

[0037] Specifically, for the multiple initial category names of each group of sample data obtained by clustering, some overlapping or similar categories may be generated. Based on this, the classification results can be further refined by the large model. In this embodiment, the large model can be called again to compare and merge the essentially identical or similar categories in the multiple initial category names of each group of sample data, and eliminate redundant categories, so as to obtain a more streamlined and more representative target category name. This process can be achieved through the semantic understanding and similarity calculation functions of the large model, wherein the large model can determine the semantic similarity between different category names, and merge or adjust multiple initial category names according to the degree of similarity. This process can be performed once or repeatedly performed multiple times until the number of merged category names is reduced to a preset number range, or the generated category names can no longer be clustered, and the target category name of the data to be processed can be obtained.

[0038] In this embodiment, by performing dimensionality reduction processing on multiple initial category names of multiple groups of sample data, the initial classification results of each group of sample data can be refined and optimized, thereby extracting more concise and comprehensive target category names, so as to better summarize the overall data structure of the original data set.

[0039] Step 104 , classify each data in the data to be processed one by one according to the multiple target category names to obtain a clustering result of the data to be processed.

[0040] Specifically, after obtaining multiple target category names, each data in the data to be processed can be classified one by one. In this embodiment, each data in the data to be processed can be matched with each target category name, and each data can be assigned to the category that best matches its characteristics based on the characteristics of the data and the semantic similarity of the target category name. The above process can be implemented through the classification function of the large model, or it can be implemented with the help of traditional rule-based methods or classification algorithms such as the minimum distance method. In the above manner, the clustering result of the data to be processed can be obtained, that is, the category information to which each data belongs can be obtained.

[0041] The method provided in this embodiment can significantly reduce the consumption of computing resources, reflecting a strategy of exchanging time for space. The above method can support data clustering of unlimited scale, and can directly output the group name of the cluster. In the prior art, a large amount of data is usually input into a large model at one time, and only a single processing is performed. This process consumes huge resources, not only the processing time is long, but also the generated clustering results are often not accurate enough, and are accompanied by lengthy output text. In contrast, the method provided in this embodiment is processed by multiple calls, and only a small amount of data is processed each time. The resource consumption of each call is low, the processing time is also shorter, and the accuracy of the classification can be gradually improved, and the clustering results obtained are more accurate and reliable.

[0042] Through the above embodiment, firstly, the data to be processed is obtained, and the data to be processed is randomly sampled multiple times to obtain multiple groups of sample data, and then each group of sample data is clustered through the pre-trained large model to obtain multiple initial category names of each group of sample data, wherein the initial category can represent the initial clustering result of each group of sample data, and then the multiple initial category names of the multiple groups of sample data are reduced in dimension through the large model to obtain multiple target category names of the data to be processed, wherein the target category name can represent the common clustering result of the multiple groups of sample data, and finally, each data in the data to be processed is classified one by one through the multiple target category names to obtain the clustering result of the data to be processed. The above method converts the clustering task of the large amount of data to be processed into the classification task of each data in the data to be processed. On the one hand, it can avoid the input data from exceeding the maximum context window limit supported by the large model, thereby reducing the requirements of the clustering task on the large model, and on the other hand, it can reduce the computing resource consumption of the large model through multiple simple calls, and improve the accuracy of the clustering results.

[0043] In one embodiment, step 102 can be specifically implemented by the following method: first, for each group of sample data, multiple data with similar semantics in the input sample data are merged through the prompt words set in the large model, and the merged data name is generated, and the merged data name is output. Then the following steps are executed in a loop: multiple data names with similar semantics in the merged data name are merged again through the prompt words set in the large model, and the merged data name is generated, and the merged data name is output. The above method stops looping when the following conditions are met: when the number of merged data names is less than or equal to the preset number of categories, and / or when the data names in the sample data can no longer be merged, the merged multiple data names are used as multiple initial category names of the sample data.

[0044] Specifically, each group of sample data can be clustered by the following method: for each group of sample data, first use the pre-set prompt words in the large model to preliminarily merge multiple data with similar semantics in the input sample data. In this process, the large model can identify and merge those semantically similar or related data entries according to the instructions of the prompt words, and generate a descriptive data name for the merged data. Subsequently, the large model and the prompt words can be used to further merge the data names obtained after the preliminary merger. The above process can be performed once or repeatedly in a loop. In this process, the large model can continuously identify and merge those data names that are still semantically similar or related until one of the following stop conditions is met: the number of merged data names is reduced to within the preset number of categories, or the data names in the sample data can no longer be effectively merged. At this time, the multiple merged data names output at the end can be output as multiple initial category names of the group of sample data. In this embodiment, the method of repeatedly merging the data in the sample data or further merging the merged data names can be performed through manual assistance in data input, or by setting a large model intelligent agent to automatically and cyclically execute the above data merging process, which is not specifically limited in this embodiment.

[0045] For example, assuming that the input sample data contains the following data: "Insufficient orders", "Not considering for the time being, currently plastic orders are decreasing", "Customer resistance is relatively strong, it is not suitable for promotion", "Just started renting", "Now cooperating with the bank", etc., then by clustering the above data, the following four initial category names can be obtained, namely: "Group 1: Poor business status / decreased orders", "Group 2: Unpleasant historical cooperation", "Group 3: No direct lease demand", "Group 4: Already have other cooperation / bank credit". In this embodiment, the large model can directly output the merged data name and its corresponding label, without the need to output the mapping relationship between each data and its corresponding group name. In this way, the generated initial category name can be made more concise and clear, and the amount of data processing for subsequent dimensionality reduction of the initial category name can be reduced. In a specific experiment, even in the face of 3,000 data, 20 reasonable and descriptive group names can be generated by the above method, which fully proves the effectiveness and practicality of the above method.

[0046] The above embodiment can avoid the context window limitation problem that may be caused by inputting a large amount of data at one time by performing multiple semantic merging processes on each group of sample data, and can gradually optimize the accuracy and rationality of clustering by merging data through multiple loop processes. In addition, by directly outputting the merged initial category name instead of outputting the direct relationship between the data and its corresponding group name, the complexity and redundancy of the data can be further reduced, and the efficiency of data processing and the readability of clustering results can be improved.

[0047] In one embodiment, step 103 can be specifically implemented by the following method: first, multiple initial category names of multiple groups of sample data are integrated to obtain multiple integrated initial category names, then, multiple initial category names with similar semantics in the integrated multiple initial category names are merged through the prompt words set in the large model, and a merged pre-selected category name is generated, and the merged pre-selected category name is output. Further, the following steps are executed in a loop: multiple pre-selected category names with similar semantics in the merged pre-selected category names are merged again through the prompt words set in the large model, and a merged pre-selected category name is generated, and the merged pre-selected category name is output. The above steps stop looping when the following conditions are met: when the number of merged pre-selected category names is less than or equal to the preset number of categories, and / or when the pre-selected category names can no longer be merged, the merged multiple pre-selected category names are used as multiple target category names for the data to be processed.

[0048] Specifically, the initial category names can be reduced in dimension by the following method: First, multiple initial category names obtained after preliminary clustering of multiple groups of sample data are integrated to form a set containing all initial category names. Then, the initial category names in this set are further merged using the pre-set prompt words in the large model. Specifically, the large model can identify and merge those semantically similar or related initial category names according to the instructions of the prompt words, generate new pre-selected category names, and then output these merged pre-selected category names. Next, a loop processing stage can be entered, in which the pre-selected category names can be iteratively merged using the large model and prompt word technology. In this loop, the large model can continuously identify and merge those semantically similar or related pre-selected category names until one of the following stop conditions is met: the number of merged pre-selected category names is reduced to within the preset number of categories, or the pre-selected category names can no longer be effectively merged. At this point, the multiple merged pre-selected category names obtained in the end can be output as multiple target category names of the data to be processed. In this embodiment, the method of repeatedly merging the pre-selected category names or further merging the merged pre-selected category names can be performed through manual assistance in data input, or by setting a large model intelligent agent to automatically and cyclically execute the above data merging process, which is not specifically limited in this embodiment.

[0049] In this embodiment, when clustering is performed using a large model, the input is no longer the original data entry, but the initial category name (or group name) obtained after preliminary clustering. The original data to be processed may contain tens of thousands of records. Directly inputting them into the large model for clustering processing may be limited by the context window, resulting in poor processing results. However, the number of initial category names after clustering is relatively small, so they can be input together into the large model for further merging processing, thereby improving the efficiency of data clustering.

[0050] This embodiment further merges the initial category names of multiple groups of sample data to obtain a reasonable number of target category names with clear semantics. The above method can avoid the context window limitation problem that may be caused by inputting a large amount of data at one time, and through multiple iterative processes, the accuracy and rationality of clustering can be gradually improved. At the same time, since only the initial category names after clustering need to be processed without processing the original data, the efficiency of clustering and the readability of clustering results can be significantly improved.

[0051] In one embodiment, the merged pre-selected category names can be output in the following manner: first, the data lengths of the merged multiple pre-selected category names are limited to a preset length range respectively through the prompt words set in the large model, and the multiple pre-selected category names are sorted. Then, according to the arrangement order of the multiple pre-selected category names, category labels are set for the multiple pre-selected category names, and the multiple pre-selected category names and their corresponding category labels are output according to the preset output format.

[0052] Specifically, when the merged pre-selected category names are output through the large model, the following operations can be implemented with the help of the preset prompt words: first, the preset prompt words in the large model are used to process the merged multiple pre-selected category names to limit the data length of the merged pre-selected category names to a preset range, thereby ensuring that the output pre-selected category names are both concise and descriptive. Next, these pre-processed pre-selected category names can be sorted, where the basis for sorting can be the alphabetical order of the category names, the frequency of occurrence of the categories, or other relevant indicators. Then, a unique category label can be set for each category name in the sorted order, and these labels can be expressed in the form of "Group 1", "Group 2", etc., or in the form of "Group A", "Group B", etc., so as to perform dimensionality reduction processing and reference later. Finally, the category name and its corresponding category label can be output according to the preset output format, that is, the json format corresponding to the key-value. This output process can be non-interactive, that is, the final json result can be directly generated and output. It can be understood that the above method can be applied to multiple links such as merging original data and outputting data names, outputting initial category names of each group of sample data, outputting pre-selected category names, and outputting target category names.

[0053] For example, the clustering prompt words may include the following content: "Our goal is to use a large model to cluster the given keywords and generate a json format output containing up to 20 categories. In this process, you need to carefully read and understand all the provided keywords, then analyze the semantic relationship between these keywords, and merge semantically related keywords into one category. For the merged similar keywords, you need to take a name that can summarize their common meaning as the clustering category. The entire processing process does not require interactive output, but directly generates a json result that meets the requirements. This json result will contain up to 20 categories, each category has a unique group number as a key (key), and a descriptive category name as a value (value)". Through the above prompt words, the original data entries and the pre-selected category names of each sample data can be merged, and the merged results can be output in a structured manner.

[0054] This embodiment can effectively cluster a large number of keywords by outputting the pre-selected category names and their corresponding category labels according to the preset structure, and generate a clearly structured and easy-to-understand category name output. This output not only meets the preset category quantity limit and output format requirements, but also can well summarize and cover the main meanings of the original keywords. By using large models and prompt word technology, the adaptability of clustering can be effectively improved to meet keyword clustering tasks of different fields and types.

[0055] At present, when using a large model combined with prompt word technology to perform classification tasks, most text classification needs are mainly met through the existing general knowledge base of the large model. However, when faced with many or complex classification categories, such as "unpleasant historical cooperation" and "poor business conditions", the model is prone to produce results with the same semantics but different expressions during the generation process, such as "unpleasant cooperation" and "poor user business conditions". These differences are difficult to accurately trace back and align in subsequent processing. Although they can be solved through additional post-processing methods, they will significantly increase the complexity of the overall process. In addition, there may be cases in the data set where the same paragraph corresponds to multiple categories, or interactive answers such as "Based on your input, my result is xxx" are generated. This type of response greatly increases the difficulty of subsequent analysis and category processing. Although the semantic similarity method can solve some problems, it is found in practice that the effect of this method is often not as ideal as directly using a large model for classification, so it does not fundamentally solve the problem.

[0056] In view of the problems that the classification results of the large model are not easy to trace back and the preprocessing process is complicated, in one embodiment, step 104 can be specifically implemented by the following method: First, for each data, the input data is semantically parsed through the prompt words set in the large model, and the target category name and category label corresponding to the data are determined according to the category label set in the prompt word and its corresponding target category name, as well as the category label corresponding to the output data. Then, based on the category label corresponding to the data, the target category name corresponding to the data is matched through the mapping relationship between the category label and the target category name. Finally, according to the target category name corresponding to each data in the data to be processed, the clustering result of the data to be processed is obtained.

[0057] Specifically, in view of the technical problems that the existing large model classification results are not easy to trace back and the preprocessing process is complicated, this embodiment proposes a data classification method based on large model output redirection. The method includes the following steps: First, by clearly specifying the category label (such as A, B, C, D, etc.) in the prompt word, the large model is guided to output a standardized category label instead of a specific category name. This method can reduce the possibility of the large model generating descriptive text with high randomness, thereby improving the standardization and consistency of the output results. In this case, the large model may still generate interactive answers or generate results containing additional text, for example, outputting "Based on your input, the category should be Category A". To this end, this embodiment further introduces regular expression matching technology to extract category labels that meet the expected format from the output of the large model, such as "[Class A]". If the large model output does not meet the expected format, such as "which category cannot be identified", no category label can be matched by the regular expression, and an empty array [] is returned. Finally, through the pre-established mapping relationship between the category label and the target category name, the extracted category label is converted into a specific target category name to complete the data classification task. This process not only ensures the accuracy and consistency of the classification results, but also greatly simplifies the subsequent data processing process.

[0058] For example, suppose the input data of the large model is "Mr. Tan 1377113xxxx, this is not needed", and the expected output is the "no demand" category. In the prior art, the large model may directly generate a sentence containing category information, such as outputting "The customer said it is not needed, so it is classified as Class G: no demand". However, although this output method is correctly classified, the format is not unified, making it difficult to perform subsequent processing. In contrast, the method proposed in this embodiment can directly extract "Class G" from the output through regular matching, and quickly map it to the "no demand" category. This process not only simplifies the processing flow, but also ensures the standardization of the output format, thereby facilitating subsequent data analysis and processing.

[0059] This embodiment can significantly improve the classification efficiency and classification accuracy of data classification tasks based on large models by introducing category labels and regular expression matching technology. Compared with the method of directly allowing the large model to generate specific category names, using category labels for output can effectively avoid classification errors or inconsistent results caused by the uncertainty of the content generated by the large model. In addition, through regular expression matching technology, category information can be accurately extracted from the interactive answers or other forms of output of the large model, ensuring the reliability of the classification results. Finally, the specific category names obtained after mapping processing are not only unified in format and easy to process later, but also greatly reduce the need for manual intervention and reduce labor costs.

[0060] In one embodiment, the target category name and category label corresponding to the data can be determined by the following method: first, the input data is semantically parsed through the prompt words set in the large model to obtain the semantic parsing results of the data. Then, the semantic parsing results of the data are matched with the target category names set in the prompt words one by one for semantic similarity to obtain at least one target category name corresponding to the data. Finally, based on the at least one target category name corresponding to the data, the target category name and category label corresponding to the data are determined, wherein the target category name and category label corresponding to the data are limited to the range of the category label set in the prompt word and its corresponding target category name.

[0061] Specifically, the preset prompt words in the large model can perform semantic analysis on the input data to accurately capture the core meaning of each piece of data. Subsequently, the large model can compare the semantic analysis results of the data with the multiple target category names preset in the prompt words one by one, and select the most matching target category name by calculating the semantic similarity. In this process, one data is allowed to correspond to multiple target category names at the same time, thereby improving the flexibility and accuracy of classification. Finally, based on the selected target category names, the final category name and category label corresponding to the data are determined within the category label range defined by the prompt words.

[0062] For example, the data classification prompt words may include the following content: "Assuming that the role is a marketing expert in the field of financial leasing, it is necessary to classify customers based on the account manager's visit follow-up records for subsequent customer stratification maintenance. The classification standards include eight categories from A to H, each corresponding to a specific customer situation or demand, such as Category A represents poor business conditions or reduced orders, and Category B represents unpleasant historical cooperation or aversion to leasing business. During the classification process, the preset classification standards must be strictly adhered to, and new categories must not be created or information that is not within the scope must not be generated. At the same time, it is required to directly return concise and clear classification results such as "Class A" and "Class B" to avoid the complexity of interactive answers." Through the above prompt words, each piece of data to be processed can be accurately classified, and the category label corresponding to the data can be output.

[0063] In the prior art, one data can usually only be classified into one category, which limits the flexibility and accuracy of classification to a certain extent. In this embodiment, by outputting the category label of the data and matching the target category name using regular matching technology, the same data can be divided into multiple categories at the same time. For example, when the input of the large model is "This is too expensive, I don't have any demand", after the classification processing of the large model, it can output "The customer said this is too expensive, so it is classified as Class D: High interest rate. In addition, the customer also pointed out that he has no demand, so it is classified as Class G: No demand". Through regular matching, [Class D, Class G] can be matched, and finally the data classification can be mapped to [High interest rate, no demand]. The above classification process not only improves the accuracy of classification, but also makes the classification results closer to the actual needs of customers.

[0064] This embodiment can achieve accurate classification and efficient processing of input data through a large model and pre-set prompt word technology. Compared with traditional methods, the classification method provided by this embodiment can effectively improve the accuracy of data classification and significantly enhance the flexibility and practicality of data classification. In addition, the above method allows the same data to be mapped to multiple target category names, successfully breaking the limitations of the classification function in the prior art, so that the classification results are more in line with the actual needs of customers.

[0065] In one embodiment, the classification result of the data can be optimized by the following method: first, verify whether the target category name and category label corresponding to the data are correct through the prompt words set in the large model. If it is verified that the target category name and category label corresponding to the data are incorrect, the target category name and category label corresponding to the data are replaced and / or added; if it is verified that the target category name and category label corresponding to the data are correct, the classification result of the data is output, wherein the classification result at least includes the category label corresponding to the data.

[0066] Specifically, the classification results of the data can be optimized with the help of prompt words as follows: First, use the preset prompt words in the large model to verify the target category name and category label corresponding to the input data to ensure the accuracy and reliability of the classification results. During the verification process, if errors are found in the target category name and category label corresponding to the data, replacement or addition operations can be performed to ensure the accuracy of the classification results. If the verification result is correct, that is, the target category name and category label corresponding to the data are consistent with the preset standard, the classification result of the data can be directly output. It should be noted that the classification result should at least contain the category label corresponding to the data to facilitate subsequent matching and mapping processing.

[0067] In this embodiment, by reasonably designing prompt words, the specific steps and requirements of the data classification task can be clarified. For example, the prompt words are as follows: "First, read the follow-up record content carefully, which is the basis for classification. Then, according to the record content, divide the customers into eight categories: A, B, C, D, E, F, G, and H. Each category represents the different characteristics or needs of the customers. After the classification is completed, the classification results need to be confirmed to ensure the accuracy of the classification. If errors are found in the classification results, the classification results are corrected, including operations such as adding new categories or replacing categories. The following are some cases and input content that needs to be processed. Through these examples, you can correctly understand the specific requirements and operation procedures of the classification task." Through the above prompt words, the classification results of the data can be accurately verified, thereby improving the accuracy of data classification.

[0068] The above embodiment can optimize the classification results of data by setting the classification result verification words in the prompt words, thereby ensuring the accuracy and reliability of the classification results.

[0069] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. In addition, the numbers corresponding to the various steps in the above embodiments only serve as identification and are not used as a limitation on the order in which the steps are executed. The order in which the steps are executed in each embodiment can be set according to actual conditions.

[0070] Further, as Figure 2 The specific implementation of the method, the embodiment of the present application provides a data clustering device based on a large model, such as Figure 3 As shown, the device comprises:

[0071] The data sampling module 31 can be used to obtain the data to be processed and perform multiple random sampling on the data to be processed to obtain multiple groups of sample data;

[0072] The data clustering module 32 can be used to perform clustering processing on each group of sample data through a pre-trained large model to obtain multiple initial category names for each group of sample data;

[0073] The data dimension reduction module 33 can be used to perform dimension reduction processing on multiple initial category names of multiple groups of sample data through the large model to obtain multiple target category names of the data to be processed;

[0074] The data classification module 34 can be used to classify each data in the data to be processed one by one according to a plurality of target category names to obtain a clustering result of the data to be processed.

[0075] In a specific application scenario, the data clustering module 32 can be specifically used to merge multiple data with similar semantics in the input sample data for each group of sample data through the prompt words set in the large model, generate a merged data name, and output the merged data name; loop the following steps: merge multiple data names with similar semantics in the merged data names again through the prompt words set in the large model, generate a merged data name, and output the merged data name; when the number of the merged data names is less than or equal to the preset number of categories, and / or when the data names in the sample data can no longer be merged, use the merged multiple data names as multiple initial category names of the sample data.

[0076] In a specific application scenario, the data dimension reduction module 33 can be specifically used to integrate multiple initial category names of multiple groups of sample data to obtain multiple integrated initial category names; merge multiple initial category names with similar semantics in the integrated multiple initial category names through the prompt words set in the large model, generate merged pre-selected category names, and output the merged pre-selected category names; loop the following steps: merge multiple pre-selected category names with similar semantics in the merged pre-selected category names again through the prompt words set in the large model, generate merged pre-selected category names, and output the merged pre-selected category names; when the number of the merged pre-selected category names is less than or equal to the preset number of categories, and / or the pre-selected category names can no longer be merged, use the merged multiple pre-selected category names as multiple target category names for the data to be processed.

[0077] In a specific application scenario, the data dimensionality reduction module 33 can also be used to limit the data length of the merged multiple pre-selected category names within a preset length range through the prompt words set in the large model, and sort the multiple pre-selected category names; set category labels for the multiple pre-selected category names according to the arrangement order of the multiple pre-selected category names, and output the multiple pre-selected category names and their corresponding category labels according to a preset output format.

[0078] In a specific application scenario, the data classification module 34 can be specifically used to perform semantic analysis on the input data for each data through the prompt words set in the large model, and determine the target category name and category label corresponding to the data according to the category label set in the prompt word and its corresponding target category name, and output the category label corresponding to the data; based on the category label corresponding to the data, the target category name corresponding to the data is matched through the mapping relationship between the category label and the target category name; and according to the target category name corresponding to each data in the data to be processed, the clustering result of the data to be processed is obtained.

[0079] In a specific application scenario, the data classification module 34 can also be used to perform semantic analysis on the input data through the prompt words set in the large model to obtain semantic analysis results of the data; perform semantic similarity matching on the semantic analysis results of the data with the target category names set in the prompt words one by one to obtain at least one target category name corresponding to the data; and determine the target category name and category label corresponding to the data based on the at least one target category name corresponding to the data, wherein the target category name and category label corresponding to the data are limited to the range of the category label set in the prompt word and its corresponding target category name.

[0080] In a specific application scenario, the data classification module 34 can also be used to verify whether the target category name and category label corresponding to the data are correct through the prompt words set in the large model; if it is verified that the target category name and category label corresponding to the data are incorrect, the target category name and category label corresponding to the data are replaced and / or added; if it is verified that the target category name and category label corresponding to the data are correct, the classification result of the data is output, wherein the classification result at least includes the category label corresponding to the data.

[0081] It should be noted that other corresponding descriptions of the functional units involved in the data clustering device based on a large model provided in the embodiment of the present application can be referred to Figure 2 The corresponding description in the method will not be repeated here.

[0082] The present application also provides a computer device, such as Figure 4As shown, the computer device can specifically be a personal computer, a server, a network device, etc. The computer device includes a system bus, a processor, a memory and a communication interface, and can also include an input and output interface and a display device. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store location information. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in each method embodiment are implemented.

[0083] Those skilled in the art will appreciate that the structure of the above-mentioned computer device is only a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components, or combine certain components, or have a different arrangement of components.

[0084] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile, and stores a computer program thereon. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0085] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0086] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0087] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0088] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data clustering method based on a large model, characterized in that: The method comprises: Acquire data to be processed, and perform multiple random sampling on the data to be processed to obtain multiple groups of sample data; Through the pre-trained large model, each group of sample data is clustered to obtain multiple initial category names for each group of sample data; Through the large model, multiple initial category names of multiple groups of sample data are subjected to dimensionality reduction processing to obtain multiple target category names of the data to be processed; By using multiple target category names, each data in the data to be processed is classified one by one to obtain a clustering result of the data to be processed.

2. The method according to claim 1, characterized in that The pre-trained large model is used to perform clustering processing on each group of sample data to obtain multiple initial category names for each group of sample data, including: For each group of sample data, multiple data with similar semantics in the input sample data are merged through the prompt words set in the large model, and a merged data name is generated, and the merged data name is output; The following steps are executed in a loop: using the prompt words set in the large model, multiple data names with similar semantics in the merged data names are merged again, and a merged data name is generated, and the merged data name is output; When the number of the merged data names is less than or equal to the preset number of categories, and / or the data names in the sample data cannot be merged any further, the merged data names are used as the initial category names of the sample data.

3. The method according to claim 1, characterized in that The large model is used to perform dimensionality reduction processing on multiple initial category names of multiple groups of sample data to obtain multiple target category names of the data to be processed, including: Integrate multiple initial category names of multiple groups of sample data to obtain multiple integrated initial category names; Merging multiple initial category names with similar semantics among the multiple integrated initial category names using the prompt words set in the large model, generating a merged pre-selected category name, and outputting the merged pre-selected category name; The following steps are executed in a loop: using the prompt words set in the large model, merging again a plurality of pre-selected category names with similar semantics in the merged pre-selected category names, generating a merged pre-selected category name, and outputting the merged pre-selected category name; When the number of the merged pre-selected category names is less than or equal to the preset number of categories, and / or the pre-selected category names cannot be merged any further, the merged multiple pre-selected category names are used as multiple target category names for the data to be processed.

4. The method according to claim 3, characterized in that The outputting the merged pre-selected category name includes: By using the prompt words set in the large model, the data lengths of the merged multiple pre-selected category names are respectively limited within a preset length range, and the multiple pre-selected category names are sorted; According to the arrangement order of the plurality of pre-selected category names, category labels are set for the plurality of pre-selected category names, and according to a preset output format, the plurality of pre-selected category names and their corresponding category labels are output.

5. The method according to claim 1, characterized in that The method of classifying each data in the data to be processed one by one through multiple target category names to obtain a clustering result of the data to be processed includes: For each data, semantic analysis is performed on the input data through the prompt words set in the large model, and the target category name and category label corresponding to the data are determined according to the category label set in the prompt words and the corresponding target category name, and the category label corresponding to the data is output; Based on the category label corresponding to the data, the target category name corresponding to the data is matched through a mapping relationship between the category label and the target category name; According to the target category name corresponding to each data in the data to be processed, a clustering result of the data to be processed is obtained.

6. The method according to claim 5, characterized in that The input data is semantically parsed using the prompt words set in the large model, and the target category name and category label corresponding to the data are determined according to the category label set in the prompt words and the corresponding target category name, including: Perform semantic analysis on the input data using the prompt words set in the large model to obtain semantic analysis results of the data; Perform semantic similarity matching on the semantic parsing results of the data with the target category names set in the prompt words one by one, to obtain at least one target category name corresponding to the data; Based on at least one target category name corresponding to the data, determine the target category name and category label corresponding to the data, wherein the target category name and category label corresponding to the data are limited to the range of the category label set in the prompt word and its corresponding target category name.

7. The method according to claim 5 or 6, characterized in that: After determining the target category name and category label corresponding to the data, the method further includes: Verify whether the target category name and category label corresponding to the data are correct by using the prompt words set in the large model; If it is verified that the target category name and category label corresponding to the data are incorrect, the target category name and category label corresponding to the data are replaced and / or added; If it is verified that the target category name and category label corresponding to the data are correct, the classification result of the data is output, wherein the classification result at least includes the category label corresponding to the data.

8. A data clustering device based on a large model, characterized in that: The device comprises: A data sampling module is used to obtain the data to be processed and perform multiple random sampling on the data to be processed to obtain multiple groups of sample data; The data clustering module is used to perform clustering processing on each group of sample data through a pre-trained large model to obtain multiple initial category names for each group of sample data; A data dimension reduction module, used to perform dimension reduction processing on multiple initial category names of multiple groups of sample data through the large model to obtain multiple target category names of the data to be processed; The data classification module is used to classify each data in the data to be processed one by one according to multiple target category names to obtain a clustering result of the data to be processed.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images