Data classification method and device

Abstract

The invention discloses a data classification method and device. A specific embodiment of the method comprises the steps of determining knowledge data under a target coarse classification to which to-be-classified data belongs; determining target historical data approximate to the knowledge data; in response to determining that the target historical data comprises unclassified data, determining the target fineness of a classification standard to be adopted by the classification standard knowledge base according to the accuracy rate of a classification result of the historical to-be-classified data; a classification result of the unclassified data is determined and stored through a large language model according to a classification standard knowledge base adopting a classification standard of target fineness, and the fineness of the classification standard of the classification standard knowledge base is improved in stages; and determining a classification result of the to-be-classified data according to the classification result of the target historical data. According to the method and the device, the classification result with relatively high precision and relatively high accuracy can be obtained by utilizing the knowledge data, the historical data, the classification standard knowledge base and other data resources which are relatively low in precision and easy to obtain under rough classification, and the fineness of the classification standard of the classification standard knowledge base is improved in stages; and the precision and accuracy of the classification result of the to-be-classified data are continuously improved as time goes on.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, specifically to the field of large models and natural language understanding technology, and in particular to a data classification method, device, computer-readable medium, electronic device and computer program product. Background Art

[0002] In the retail advertising business, there are a large number of customer complaint dialogues (i.e., VOB, voice of business) every year. In order to avoid the recurrence of customer complaints at the root and improve the merchant customer experience, identifying the complaint type in customer complaint dialogues is the primary task. In the existing solution of identifying complaint types based on large models, it is necessary to learn from a large amount of high-precision data resources to obtain a large model with high accuracy. However, large amounts of high-precision data resources are difficult to obtain, and large models cannot guarantee the accuracy of recognition results in the long term. Summary of the invention

[0003] The embodiments of the present application provide a data classification method, device, computer-readable medium, and electronic device.

[0004] In a first aspect, an embodiment of the present application provides a data classification method, including: determining knowledge data under a target coarse classification to which the data to be classified belongs; determining target historical data that is similar to the knowledge data; in response to determining that the target historical data includes unclassified data, determining a target fineness of a classification standard to be adopted by a classification standard knowledge base based on the accuracy of the classification results of the historical data to be classified, wherein the classification results of the historical data to be classified are determined based on a classification standard knowledge base that adopts a classification standard of a current fineness; determining and storing the classification results of the unclassified data through a large language model based on the classification standard knowledge base that adopts the classification standard of the target fineness; and determining the classification results of the data to be classified based on the classification results of the target historical data.

[0005] In some examples, the above-mentioned determining the classification results of the data to be classified based on the classification results of the target historical data includes: determining the weight of the classification results of the target historical data based on the determination method of the classification results of the target historical data, wherein the determination method includes: querying the classified data in the target historical data to determine the classification results, or determining the classification results of unclassified data based on the classification standard knowledge base; determining the classification results of the data to be classified based on the classification results and weights corresponding to each of the target historical data.

[0006] In some examples, the above-mentioned method of determining the classification result of the target historical data according to the classification result of the target historical data includes: in response to the classification result of the target historical data being determined based on querying the classified data, the weight of the classification result of the classified data is determined according to a strategy in which the weight increases step by step with the passage of time period; in response to the classification result of the target historical data being determined based on the classification standard knowledge base, the weight of the classification result of the unclassified data is determined according to a strategy in which the weight decreases step by step with the passage of time period.

[0007] In some examples, the above-mentioned determination of the knowledge data under the target coarse category to which the data to be classified belongs includes: determining a plurality of approximate knowledge data that are similar to the data to be classified from a coarse category knowledge base; determining the degree of aggregation of the plurality of approximate knowledge data based on the proportion of the approximate knowledge data belonging to the same coarse category among the plurality of approximate knowledge data; and determining the target coarse category and the knowledge data under the target coarse category based on the degree of aggregation of the plurality of approximate knowledge data.

[0008] In some examples, the above-mentioned determination of the target coarse classification based on the degree of aggregation of multiple approximate knowledge data includes: in response to determining that the degree of aggregation is less than a first preset threshold, determining new approximate knowledge data that is approximate to the data to be classified from the coarse classification knowledge base, until it is determined that the degree of aggregation of all the approximate knowledge data currently obtained under a coarse classification in the coarse classification knowledge base is greater than or equal to the first preset threshold, or, all the approximate knowledge data currently obtained includes all knowledge data under a coarse classification in the coarse classification knowledge base; and determining the coarse classification as the target coarse classification.

[0009] In some examples, before determining the target historical data that is similar to the knowledge data, the process also includes: determining whether the historical data includes data to be changed whose classification results need to be changed based on the fineness of the classification results corresponding to the historical data; and in response to determining that the data includes data to be changed, deleting the classification results corresponding to the data to be changed.

[0010] In some examples, the above-mentioned determination of whether the database includes data to be changed whose classification results need to be changed based on the fineness of the classification results corresponding to the historical data includes: determining whether the historical data includes the data to be changed based on the fineness of the classification results corresponding to the historical data and the fineness of the classification results of the data to be classified that can be currently obtained.

[0011] In some examples, the above-mentioned determination of target historical data that is similar to the knowledge data includes: determining the number of target historical data that need to be recalled, wherein the number increases stepwise with the passage of time periods; and determining a number of target historical data that are similar to the knowledge data from the historical data.

[0012] In some examples, the above-mentioned determining the classification results of the data to be classified based on the classification results and weights corresponding to each target historical data includes: in response to determining, based on the weights corresponding to a number of target historical data, that the degree of aggregation of the classification results corresponding to each of the number of target historical data in one category is less than a second preset threshold, determining new target historical data that is similar to the knowledge data from the historical data, until determining, based on the weights corresponding to each of all the target historical data currently obtained, that the degree of aggregation of the classification results corresponding to each of the target historical data in one category is greater than or equal to the second preset threshold, and determining the category with an aggregation degree greater than or equal to the second preset threshold as the classification result of the data to be classified.

[0013] In some examples, the above-mentioned determination of the target precision of the classification standard to be adopted by the classification standard knowledge base based on the accuracy of the classification results of the historical data to be classified includes: in response to the accuracy of the classification results of the historical data to be classified exceeding a preset accuracy threshold, increasing the current precision of the classification standard adopted by the classification standard knowledge base to determine the target precision.

[0014] In some examples, the above method further includes: using the data to be classified as historical data, and storing the data to be classified and the classification results of the data to be classified.

[0015] In a second aspect, an embodiment of the present application provides a data classification device, comprising: a first determination unit, configured to determine the knowledge data under the target coarse classification to which the data to be classified belongs; a second determination unit, configured to determine the target historical data that is similar to the knowledge data; a third determination unit, configured to determine, in response to determining that the target historical data includes unclassified data, a target fineness of the classification standard to be adopted by the classification standard knowledge base according to the accuracy of the classification results of the historical data to be classified, wherein the classification results of the historical data to be classified are determined based on the classification standard knowledge base that adopts the classification standard of the current fineness; a fourth determination unit, configured to determine and store the classification results of the unclassified data through a large language model according to the classification standard knowledge base that adopts the classification standard of the target fineness; and a fifth determination unit, configured to determine the classification results of the data to be classified according to the classification results of the target historical data.

[0016] In some examples, the fifth determination unit is further configured to: determine the weight of the classification result of the target historical data according to a method for determining the classification result of the target historical data, wherein the determination method includes: querying the classified data in the target historical data to determine the classification result, or determining the classification result of the unclassified data according to the classification standard knowledge base; determine the classification result of the data to be classified according to the classification results and weights corresponding to each of the target historical data.

[0017] In some examples, the fifth determination unit is further configured to: in response to the classification result of the target historical data being determined based on querying the classified data, determine the weight of the classification result of the classified data according to a strategy in which the weight increases step by step with the passage of time period; in response to the classification result of the target historical data being determined based on the classification standard knowledge base, determine the weight of the classification result of the unclassified data according to a strategy in which the weight decreases step by step with the passage of time period.

[0018] In some examples, the first determination unit is further configured to: determine a plurality of approximate knowledge data that are similar to the data to be classified from the coarse classification knowledge base; determine the degree of aggregation of the plurality of approximate knowledge data based on the proportion of the approximate knowledge data belonging to the same coarse classification among the plurality of approximate knowledge data; and determine the target coarse classification and the knowledge data under the target coarse classification according to the degree of aggregation of the plurality of approximate knowledge data.

[0019] In some examples, the first determination unit is further configured to: in response to determining that the degree of aggregation is less than a first preset threshold, determine new approximate knowledge data that is similar to the data to be classified from the coarse classification knowledge base, until it is determined that the degree of aggregation of all the approximate knowledge data currently obtained under a coarse category in the coarse classification knowledge base is greater than or equal to the first preset threshold, or, all the approximate knowledge data currently obtained includes all knowledge data under a coarse category in the coarse classification knowledge base; and determine the coarse category as the target coarse category.

[0020] In some examples, the above-mentioned device also includes: a change unit, which is configured to: before determining the target historical data that is similar to the knowledge data, determine whether the historical data includes data to be changed whose classification results need to be changed according to the fineness of the classification results corresponding to the historical data; in response to determining that it includes, delete the classification results corresponding to the data to be changed.

[0021] In some examples, the modification unit is further configured to determine whether the historical data includes the data to be modified according to the refinement of the classification result corresponding to the historical data and the refinement of the classification result of the data to be classified that is currently available.

[0022] In some examples, the second determination unit is further configured to: determine the number of target historical data that need to be recalled, wherein the number increases stepwise as the time period passes; and determine a number of target historical data that is approximately equal to the knowledge data from the historical data.

[0023] In some examples, the fifth determination unit is further configured to: in response to determining, based on the weights corresponding to a number of target historical data, that the degree of aggregation of classification results corresponding to a number of target historical data in one category is less than a second preset threshold, determine new target historical data that is similar to the knowledge data from the historical data, until determining, based on the weights corresponding to all the target historical data currently obtained, that the degree of aggregation of classification results corresponding to all target historical data in one category is greater than or equal to the second preset threshold, and determine the category with a degree of aggregation greater than or equal to the second preset threshold as the classification result of the data to be classified.

[0024] In some examples, the third determination unit is further configured to: in response to the accuracy of the classification results of the historical data to be classified exceeding a preset accuracy threshold, increase the current precision of the classification standard adopted by the classification standard knowledge base and determine the target precision.

[0025] In some examples, the apparatus further includes: a storage unit configured to: use the data to be classified as historical data, and store the data to be classified and a classification result of the data to be classified.

[0026] In a third aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0027] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.

[0028] In a fifth aspect, a computer program product is provided, comprising: a computer program, which implements the method described in any implementation manner of the first aspect when executed by a processor.

[0029] The data classification method and device provided by the embodiment of the present application determine the knowledge data under the target coarse classification to which the data to be classified belongs; determine the target historical data that is similar to the knowledge data; in response to determining that the target historical data includes unclassified data, determine the target precision of the classification standard to be adopted by the classification standard knowledge base according to the accuracy of the classification results of the historical data to be classified, wherein the classification results of the historical data to be classified are determined based on the classification standard knowledge base that adopts the classification standard of the current precision; through a large language model, determine and store the classification results of the unclassified data according to the classification standard knowledge base that adopts the classification standard of the target precision; determine the classification results of the data to be classified according to the classification results of the target historical data, thereby utilizing the knowledge data under the coarse classification, historical data, classification standard knowledge base and other data resources with lower precision and easy access to obtain, to obtain classification results with higher precision and higher accuracy, and the precision of the classification standard of the classification standard knowledge base is increased stepwise with the passage of time, so that the precision and accuracy of the classification results of the data to be classified continue to improve with the passage of time. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0031] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present application may be applied;

[0032] Figure 2 is a flow chart of an embodiment of a data classification method according to the present application;

[0033] Figure 3 is a schematic diagram of an application scenario of the data classification method according to this embodiment;

[0034] Figure 4 is a flow chart of another embodiment of the data classification method according to the present application;

[0035] Figure 5 is a flow chart of an embodiment of a VOB data classification method according to the present application;

[0036] Figure 6 A structural diagram of an embodiment of a data classification device according to the present application;

[0037] Figure 7 A schematic diagram of the structure of a computer system suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0038] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0039] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0040] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0041] Figure 1 An exemplary architecture 100 to which the data classification method and apparatus of the present application can be applied is shown.

[0042] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The communication connection between the terminal devices 101, 102, 103 constitutes a topological network, and the network 104 is used to provide a medium for the communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0043] The terminal devices 101, 102, 103 can interact with the server 105 through the network 104 to receive or send data, etc. The terminal devices 101, 102, 103 can be hardware devices or software that support network connection for data interaction and data processing. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices that support network connection, information acquisition, interaction, display, processing and other functions, including but not limited to smart phones, car computers, tablet computers, e-book readers, laptop portable computers and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0044] The server 105 may be a server that provides various services, for example, a background processing server that uses low-precision and easily accessible data resources such as knowledge data, historical data, and classification standard knowledge base under coarse classification to determine the classification results of the data to be classified provided by the terminal devices 101, 102, and 103. As an example, the server 105 may be a cloud server.

[0045] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0046] It should also be noted that the data classification method provided in the embodiments of the present application can be executed by a server, or by a terminal device, or by a server and a terminal device in cooperation with each other. Accordingly, the various parts (such as various units) included in the data classification device can be all set in the server, or all set in the terminal device, or can be set in the server and the terminal device respectively.

[0047] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the system is only illustrative. Any number of terminal devices, networks, and servers may be provided as required. When the electronic device on which the data classification method is running does not need to perform data transmission with other electronic devices, the system architecture may only include the electronic device (e.g., server or terminal device) on which the data classification method is running.

[0048] Continue to refer Figure 2 , shows a process 200 of an embodiment of a data classification method, comprising the following steps:

[0049] Step 201, determining the knowledge data under the target coarse classification to which the data to be classified belongs.

[0050] In this embodiment, the execution subject of the data classification method (for example Figure 1 The terminal device or server in the system can obtain the data to be classified remotely or locally through a wired network connection or a wireless network connection, and determine the knowledge data under the target coarse classification to which the data to be classified belongs.

[0051] The data to be classified can be various data represented in the form of text, voice, image, etc. For example, if the data to be classified is text data, specifically customer complaint dialogue data, it is necessary to determine the complaint type to which the customer complaint dialogue data belongs. For another example, if the data to be classified is image data, specifically image data uploaded by users through social platforms and short video platforms, it is necessary to determine the type of the image data to which it belongs in order to push targeted data to other interested users.

[0052] As an example, first, for different application fields, different coarse classification knowledge bases are specifically constructed, and the coarse classification knowledge base includes multiple coarse classifications, and each coarse classification includes corresponding knowledge data. The knowledge data under the coarse classification represents the data type included in the coarse classification.

[0053] The classification criteria for coarse classification of data in an application field include: 1. The classification granularity is significantly lower than that required by the classification business in the application field. For example, if the classification business requires 200 categories, the coarse classification is recommended to be around 20. 2. Each coarse classification has or exceeds a preset number threshold (for example, 3) of knowledge.

[0054] Continuing with the example of the customer complaint dialogue scenario, the knowledge data under the coarse classification is the types of complaints that may occur under this classification.

[0055] For example, the rough classification is: advertising-delivery end-CPS business (commission-paying advertising)-plan setting link-commission setting

[0056] Knowledge 1: The commission setting threshold does not meet the merchant's expectations

[0057] Knowledge 2: Error message when setting commission

[0058] Knowledge 3: Special products want to specify a fixed commission

[0059] Based on the above three pieces of knowledge, VOB under the business classification of "advertising-delivery end-CPS business (commission-paid advertising)-plan setting link-commission setting" is divided into the above three categories.

[0060] Then, the characteristic distance between the data to be classified and the coarse classification in the coarse classification knowledge base under the application field to which the data to be classified belongs is determined. Specifically, feature extraction can be performed on the data to be classified and the knowledge data under the coarse classification in the coarse classification set respectively to obtain the classification data vector and the knowledge data vector, and then the characteristic distance between the classification data vector and the knowledge data vector under each coarse classification is calculated.

[0061] Finally, the coarse category with the closest distance is determined as the target coarse category, and then the knowledge data under the target coarse category is determined.

[0062] In some optional implementations of this embodiment, the execution subject may perform step 201 in the following manner:

[0063] First, multiple approximate knowledge data similar to the data to be classified are determined from the coarse classification knowledge base.

[0064] As an example, the above-mentioned execution entity can determine the distance between the knowledge data under each coarse classification in the coarse classification knowledge base and the data to be classified on the feature vector, and sort the knowledge data in the coarse classification knowledge base in order of distance from small to large, and then determine the multiple knowledge data in the front sorting as multiple approximate knowledge data.

[0065] Second, based on the proportion of approximate knowledge data belonging to the same rough category in the multiple approximate knowledge data, the aggregation degree of the multiple approximate knowledge data is determined.

[0066] As an example, the execution subject may determine the rough categories to which the plurality of approximate knowledge data respectively belong, and determine the quantity and proportion of the approximate knowledge data under each rough category; and use the maximum proportion as the degree of aggregation of the plurality of approximate knowledge data. The proportion represents the proportion of the amount of the approximate knowledge data under each rough category to the total amount of the plurality of approximate knowledge data corresponding thereto.

[0067] Third, according to the aggregation degree of multiple approximate knowledge data, the target coarse classification and the knowledge data under the target coarse classification are determined.

[0068] In this embodiment, in response to the aggregation degree exceeding the first preset threshold, the coarse category corresponding to the aggregation degree is determined as the target coarse category, and then the knowledge data under the target coarse category in the coarse category knowledge base is determined.

[0069] The first preset threshold may be specifically set according to actual conditions, for example, 0.5 or 0.75.

[0070] In this implementation, a specific method for determining the target coarse category to which the data to be classified belongs and the knowledge data under the target coarse category is provided, and the accuracy of the determined knowledge data is improved based on the degree of aggregation of multiple approximate knowledge data similar to the data to be classified.

[0071] In some optional implementations of this embodiment, the execution subject may perform the third step in the following manner:

[0072] First, in response to determining that the degree of aggregation is less than a first preset threshold, new approximate knowledge data that is similar to the data to be classified is determined from the coarse classification knowledge base until it is determined that the degree of aggregation of all the approximate knowledge data currently obtained under a coarse classification in the coarse classification knowledge base is greater than or equal to the first preset threshold, or, all the approximate knowledge data currently obtained includes all knowledge data under a coarse classification in the coarse classification knowledge base.

[0073] Then, the coarse classification is determined as the target coarse classification.

[0074] As an example, in response to determining that the degree of aggregation is less than a first preset threshold, a specified number of new approximate knowledge data that are similar to the data to be classified are determined from the coarse classification knowledge base, and the degree of aggregation of all the approximate knowledge data currently obtained (including the multiple approximate knowledge data in the above-mentioned third step and the specified number of new approximate knowledge data) is determined; in response to the degree of aggregation of all the approximate knowledge data currently obtained under a coarse category in the coarse classification knowledge base being greater than or equal to the first preset threshold, or, all the approximate knowledge data currently obtained include all knowledge data under a coarse category in the coarse classification knowledge base, the above-mentioned coarse category is determined as the target coarse category.

[0075] In response to determining that the degree of aggregation of all the approximate knowledge data currently obtained under a coarse classification in the coarse classification knowledge base is still less than a first preset threshold, and all the approximate knowledge data currently obtained does not include all the knowledge data under a coarse classification in the coarse classification knowledge base, continue to determine a specified number of new approximate knowledge data that are similar to the data to be classified from the coarse classification knowledge base until the degree of aggregation of all the approximate knowledge data currently obtained under a coarse classification in the coarse classification knowledge base is greater than or equal to the first preset threshold, or, all the approximate knowledge data currently obtained includes all the knowledge data under a coarse classification in the coarse classification knowledge base.

[0076] In this implementation, there is no limit on the number of loops. The loop can be ended by satisfying one of the two ending conditions: the degree of aggregation under a coarse category in the coarse category knowledge base is greater than or equal to the first preset threshold, and all knowledge data under a coarse category in the coarse category knowledge base is included.

[0077] In this implementation, a method for determining the coarse classification of the target when the degree of aggregation is less than a first preset threshold is provided, which improves the flexibility of the determination process; since the data to be classified needs to use the knowledge data under the coarse classification of the target to determine its classification result, and the improvement of precision and accuracy is achieved based on the progress of stages, the flexible determination method in this implementation helps to improve the precision and accuracy of the classification results.

[0078] Step 202, determining target historical data that is similar to the knowledge data.

[0079] In this embodiment, the execution subject may determine target historical data that is similar to the knowledge data.

[0080] The execution subject may determine target historical data that is similar to the knowledge data from the historical data corresponding to the data to be classified.

[0081] As an example, for each knowledge data under the target coarse classification, the above-mentioned execution entity can determine at least one target historical data based on the distance between the historical data corresponding to the data to be classified and the knowledge data in the feature vector; combine the target historical data corresponding to each of the multiple knowledge data under the target coarse classification to obtain the final multiple target historical data.

[0082] As another example, the execution entity may determine the feature vector distance between each knowledge data and each historical data under the target coarse classification; sort the historical data in ascending order of feature vector distance, and select the multiple historical data in the front order as the target historical data.

[0083] In some optional implementations of this embodiment, the execution subject may perform step 202 in the following manner:

[0084] First, determine the amount of target historical data that needs to be recalled.

[0085] Among them, the number increases step by step as the time period goes by.

[0086] As an example, the execution entity may adopt a strategy of increasing the quantity in a step-by-step manner as time passes to determine the quantity of target historical data to be recalled.

[0087] Second, determine a number of target historical data that are similar to the knowledge data from the historical data.

[0088] In this implementation, the execution subject may determine a number of target historical data similar to the knowledge data from the historical data with reference to the two examples above.

[0089] It can be understood that, as the stages progress, using the classification results of a larger number of target historical data to determine the classification results of the data to be classified will help to further improve the accuracy of the classification results.

[0090] Step 203 , in response to determining that the target historical data includes unclassified data, a target precision of the classification standard to be adopted by the classification standard knowledge base is determined according to the accuracy of the classification results of the historical data to be classified.

[0091] In this embodiment, the execution subject may determine the target precision of the classification standard to be adopted by the classification standard knowledge base in response to determining that the target historical data includes unclassified data, according to the accuracy of the classification results of the historical data to be classified. The classification results of the historical data to be classified are determined based on the classification standard knowledge base using the classification standard of the current precision.

[0092] In this embodiment, the precision of the classification standard of the classification standard knowledge base is improved in stages. The staged improvement means that the precision of the classification standard of the classification standard knowledge base is different in different stages, and it is improved as the stages progress. As an example, there are two stages, an early stage and a late stage. In the early stage, the classification granularity of the classification standard adopted by the classification standard knowledge base is relatively large; when entering the late stage, the precision of the classification standard adopted by the classification standard knowledge base is improved, so that the classification granularity is reduced.

[0093] More specifically, the key feature of the classification standard knowledge base in the above-mentioned early stage is: a sufficiently coarse classification standard, so that the accuracy of the classification result of the unclassified data obtained in step 203 is not less than the first threshold (for example, 50%), and the accuracy of the classification result of the data to be classified obtained in the subsequent step 204 is not less than the second threshold (for example, 80%). If the above-mentioned first threshold and second threshold are not reached, it is necessary to continue to merge the classification standards in the classification standard knowledge base. The classification standards in the classification standard knowledge base can be at least binary classification of the data. Among them, binary classification refers to the judgment of whether the data is classified at the coarsest classification granularity, such as whether it is real customer complaint data.

[0094] If only the unclassified data in the target historical data is subjected to binary classification, the classification result of the unclassified data obtained in step 203 selects one of the two categories, so that the accuracy of the classification result of the unclassified data obtained in 203 is not lower than the first threshold, and the accuracy of the classification result of the data to be classified obtained in the subsequent step 204 is not lower than the second threshold.

[0095] The key feature of the classification standard knowledge base in the above-mentioned later stage is: the classification standard is fine enough so that its classification results can directly guide the optimization of classification business. "Directly guide the optimization of classification business" is a relatively vague concept, which depends on the requirements of the classification business system to which this application is applied. Its judgment standard can be defined as: the refinement of the classification standard of the classification standard knowledge base in the early stage.

[0096] It should be noted that the above example is not limited to setting only two stages, and can be set to multiple stages according to implementation needs.

[0097] Historical data includes unclassified data and classified data, wherein unclassified data does not include classification results, and classified data includes classification results. It can be understood that initially, historical data is mostly unclassified data, but as time goes by, more and more unclassified data are classified, making historical data mostly classified data.

[0098] As an example, based on the accuracy of the classification results of the historical data to be classified, it can be determined whether the classification standard knowledge base mounted on the large language model using the classification standard of the current fineness is competent for the classification task at the current stage. For example, if the accuracy of the classification result is high, it indicates that the classification standard knowledge base mounted on the large language model using the classification standard of the current fineness is competent for the classification task at the current stage, and the fineness corresponding to the next stage can be used as the target fineness; otherwise, the current fineness continues to be used as the target fineness.

[0099] In some optional implementations of this embodiment, the above-mentioned execution entity can perform the above-mentioned step 203 in the following manner: in response to the accuracy of the classification results of the historical data to be classified exceeding a preset accuracy threshold, improve the current precision of the classification standard adopted by the classification standard knowledge base, and determine the target precision.

[0100] As an example, the accuracy of the classification results of the historical data to be classified determined by the classification standard knowledge base using the current classification standard can be regularly counted, and in response to determining that the accuracy exceeds a preset accuracy threshold, the refinement of the classification standard of the classification standard knowledge base is improved in a preset manner.

[0101] For example, the precision of the classification standard is divided into different levels. In response to the accuracy of the classification results of the historical data to be classified determined by the classification standard knowledge base based on the classification standard of the current level exceeding a preset accuracy threshold, the level of precision of the classification standard of the classification standard knowledge base is increased from the current level to the next level higher than the current level.

[0102] In this implementation, a specific method for improving the refinement of the classification standard of a classification standard knowledge base is provided, which helps to ensure the continuous improvement of the refinement and accuracy of the classification results.

[0103] Step 204 , by using the large language model, according to a classification standard knowledge base using a classification standard of a target precision, a classification result of the unclassified data is determined and stored.

[0104] In this embodiment, the execution subject may determine and store the classification result of the unclassified data through a large language model according to a classification standard knowledge base using a classification standard of a target precision.

[0105] The core implementation of this embodiment is the natural language understanding capability based on a large (language) model. Compared with the traditional classification scheme based on a large model, this embodiment does not require the large model to have a high degree of accuracy. The large model disclosed by the industry can meet the requirements of this embodiment after mounting the classification standard knowledge base. The large model with the lowest classification effect can also achieve the effect of this solution, which is the core value of this embodiment, so this application does not need to perform specific optimization on the reasoning ability of the large model, and can use conventional methods.

[0106] Step 205: Determine the classification result of the data to be classified according to the classification result of the target historical data.

[0107] In this embodiment, the execution subject may determine the classification result of the data to be classified according to the classification result of the target historical data.

[0108] As an example, when the number of target historical data is one, the classification result of the target historical data may be directly determined as the classification result of the data to be classified.

[0109] As another example, when there are multiple target historical data, the classification results obtained by election based on the classification results of the multiple target historical data can be used as the classification results of the data to be classified. The election method includes but is not limited to a majority win method and a 2 / 3 win method.

[0110] In some optional implementations of this embodiment, the execution subject may perform step 205 in the following manner:

[0111] First, according to the determination method of the classification result of the target historical data, the weight of the classification result of the target historical data is determined.

[0112] The determination method includes: querying the classified data in the target historical data to determine the classification result, or determining the classification result of the unclassified data according to the classification standard knowledge base.

[0113] For the classified data in the target historical data, the classification result is determined by querying the classified data in the target historical data to determine the classification result; for the classified data in the target historical data, the classification result is determined by determining the classification result of the unclassified data according to the classification standard knowledge base.

[0114] In this implementation, different weights may be preset for different determination methods, and the weight of the classification result of the target historical data may be determined according to the determination method of the classification result of the target historical data.

[0115] Second, the classification results of the data to be classified are determined according to the classification results and weights corresponding to each of the target historical data.

[0116] The above-mentioned execution subject can select the classification results of the data to be classified from the classification results corresponding to each of the target historical data according to the weights.

[0117] As an example, the above-mentioned execution entity can merge the same classification results in the classification results corresponding to each target historical data, and determine the weight sum of each merged classification result, and then determine the classification result corresponding to the maximum weight sum as the classification result of the data to be classified.

[0118] In this implementation, different determination methods of the classification results of the target historical data correspond to different weights, and the weights determined based on the determination methods improve the accuracy of the classification results of the data to be classified.

[0119] In some optional implementations of this embodiment, the execution subject may perform the first step in the following manner:

[0120] In response to the classification result of the target historical data being determined based on querying the classified data, the weight of the classification result of the classified data is determined according to a strategy in which the weight increases stepwise as time passes.

[0121] Specifically, the classification result of the target historical data is determined based on the query of the classified data, and the weight of the classification result increases with the progress of the stages. For example, in the early stage, the weight of the classification result of the classified data is small; in the later stage, the weight of the classification result of the classified data is large.

[0122] In response to the classification result of the target historical data being determined according to the classification standard knowledge base, the weight of the classification result of the unclassified data is determined according to the strategy of stepwise reduction of the weight as time passes.

[0123] Specifically, in response to determining that the classification result of the target historical data is determined based on the classification standard knowledge base, the weight of the classification result decreases as the stage progresses. For example, in the early stage, the weight of the classification result of the unclassified data is large; in the later stage, the weight of the classification result of the unclassified data is small.

[0124] That is, in the early stage, the classification results of the target historical data determined according to the classification standard knowledge base are given priority; in the later stage, the classification results determined according to the classified data in the query target historical data are given priority. Among them, priority reference means that the weight accounts for more than 50%.

[0125] It is understandable that in the early stages of the development of the classification business, most of the historical data do not have classification results. At this time, the reasoning ability of the large model mounted with the classification standard knowledge base is more important, and the classification results of the target historical data determined according to the classification standard knowledge base should be given priority in the weight setting. In the later stages of the development of the classification business, most of the historical data have classification results. At this time, the existing information is more important, and the classification results determined according to the classified data in the query target historical data should be given priority in the weight setting.

[0126] In this implementation, the weights of the classification results determined by different determination methods corresponding to the target historical data change in a step-by-step manner over time, and the changing trends of the weights corresponding to different determination methods are different, which can make the reference focus in the early stage and the later stage different, and realize the automatic upgrade of the classification precision and accuracy of the classification business.

[0127] In some optional implementations of this embodiment, the execution subject may perform the second step in the following manner:

[0128] In response to determining, based on the weights corresponding to a number of target historical data, that the degree of aggregation of the classification results corresponding to the above number of target historical data in one category is less than a second preset threshold, new target historical data that is similar to the knowledge data is determined from the historical data until, based on the weights corresponding to all the target historical data currently obtained, it is determined that the degree of aggregation of the classification results corresponding to all the target historical data in one category is greater than or equal to the second preset threshold, and the category with a degree of aggregation greater than or equal to the second preset threshold is determined as the classification result of the data to be classified.

[0129] As an example, in response to determining, based on the weights corresponding to the above-mentioned number of target historical data, that the degree of aggregation of the classification results corresponding to the above-mentioned number of target historical data in one category is less than a second preset threshold, a specified number of new target historical data that are similar to the knowledge data are determined from the historical data, and based on the weights corresponding to all the target historical data currently obtained, it is determined whether the degree of aggregation of the classification results corresponding to all the target historical data in one category is greater than or equal to the second preset threshold; in response to the determination, the classification with a degree of aggregation greater than or equal to the second preset threshold is determined as the classification result of the data to be classified.

[0130] In response to a no determination, continue to determine a specified number of new target historical data that are similar to the knowledge data from the historical data until, based on the weights corresponding to all the target historical data currently obtained, it is determined that the degree of aggregation of the classification results corresponding to all the target historical data under one category is greater than or equal to a second preset threshold, and the classification with a degree of aggregation greater than or equal to the second preset threshold is determined as the classification result of the data to be classified.

[0131] In this implementation, a specific method for determining the classification result of the data to be classified is provided, which improves the flexibility of the determination process, ensures the determination of the classification result of the data to be classified, and helps to improve the accuracy of the classification result.

[0132] Continue to see Figure 3 , Figure 3 FIG. 3 is a schematic diagram 300 of an application scenario of the data classification method according to this embodiment. Figure 3 In the application scenario, user 301 sends data to be classified to server 303 through terminal device 302. After receiving the data to be classified, the server first determines the knowledge data under the target coarse classification to which the data to be classified belongs from the coarse classification knowledge base; then, determines the target historical data similar to the knowledge data from the historical database; then, in response to determining that the target historical data includes unclassified data, the server determines and stores the classification result of the unclassified data according to the classification standard knowledge base, wherein the refinement of the classification standard of the classification standard knowledge base increases stepwise as the time period passes; finally, the classification result of the data to be classified is determined according to the classification result of the target historical data.

[0133] The method provided by the above-mentioned embodiments of the present application determines the knowledge data under the target coarse classification to which the data to be classified belongs; determines the target historical data that is similar to the knowledge data; in response to determining that the target historical data includes unclassified data, determines the target fineness of the classification standard to be adopted by the classification standard knowledge base according to the accuracy of the classification results of the historical data to be classified, wherein the classification results of the historical data to be classified are determined based on the classification standard knowledge base that adopts the classification standard of the current fineness; through a large language model, determines and stores the classification results of the unclassified data according to the classification standard knowledge base that adopts the classification standard of the target fineness; determines the classification results of the data to be classified according to the classification results of the target historical data, thereby utilizing the knowledge data, historical data, classification standard knowledge base and other data resources with lower precision and easy access under the coarse classification to obtain classification results with higher precision and higher accuracy, and the fineness of the classification standard of the classification standard knowledge base increases step by step with the passage of time, so that the precision and accuracy of the classification results of the data to be classified continue to improve with the passage of time.

[0134] In some optional implementations of this embodiment, before executing the above step 202, the above execution subject may perform the following operations:

[0135] First, according to the precision of the classification results corresponding to the historical data, determine whether the historical data includes data to be changed for which the classification results need to be changed.

[0136] It is understandable that one of the purposes of this application is to achieve continuous improvement in the precision and accuracy of classification results. In order to achieve the above purpose, it is necessary to improve the precision and accuracy of classification results corresponding to historical data as the stages progress.

[0137] In this implementation, historical data to be changed whose classification result fineness does not meet the fineness of the current classification result of the data to be classified can be determined at regular intervals based on the fineness of the classification result corresponding to the historical data.

[0138] As an example, the classification standards are divided into primary, secondary and tertiary levels, and the precision of the tertiary classification standards gradually increases. At the current stage, the big model can achieve secondary classification for unclassified data in historical data, but the historical data still includes classification results using the primary classification standards. At this time, the historical data with classification results using the primary classification standards can be used as historical data to be changed.

[0139] Second, in response to the determination including, deleting the classification result corresponding to the data to be modified.

[0140] Delete the classification results corresponding to the data to be changed. This will change the data to be changed from classified data to unclassified data.

[0141] It can be understood that in the subsequent information processing process, in the execution process as shown in the above step 203, the data to be changed becomes unclassified data, which needs to be classified according to the classification standard knowledge base to obtain a classification result with higher precision.

[0142] In this implementation, a solution is provided for continuously improving the refinement of classification results of historical data, which helps to quickly improve the precision and accuracy of classification services.

[0143] In some optional implementations of this embodiment, the above-mentioned execution entity can perform the above-mentioned first step in the following manner: determine whether the historical data includes the data to be changed based on the fineness of the classification results corresponding to the historical data and the fineness of the classification results of the data to be classified that can be obtained currently.

[0144] As an example, the classification standards are divided into primary, secondary and tertiary levels, and the precision of the tertiary classification standards gradually increases. At the current stage, the secondary classification can be achieved for the data to be classified, but the historical data still includes the classification results using the primary classification standards. At this time, the historical data of the classification results using the primary classification standards can be used as the historical data to be changed.

[0145] It can be understood that the target historical data determined from the historical data is ultimately used to determine the classification results of the data to be classified. Therefore, the classification results of the historical data directly affect the classification results of the data to be classified. Based on the fineness of the classification results corresponding to the historical data and the fineness of the classification results of the data to be classified that can be obtained currently, it is determined whether the historical data includes the data to be changed, thereby further improving the accuracy of determining the data to be changed.

[0146] In some optional implementations of this embodiment, the above-mentioned execution subject may also perform the following operations: taking the data to be classified as historical data, and storing the data to be classified and the classification results of the data to be classified.

[0147] In this implementation, after obtaining the classification results of the data to be classified, the data to be classified is used as historical data, and the data to be classified and the classification results of the data to be classified are stored, thereby supplementing the historical data, improving the richness of the historical data, and helping to improve the accuracy of subsequent classification results of the data to be classified.

[0148] Continue to refer Figure 4 , shows a schematic process 400 of another embodiment of the data classification method according to the present application, comprising the following steps:

[0149] Step 401: determine a plurality of approximate knowledge data similar to the data to be classified from the coarse classification knowledge base.

[0150] Step 402 : determining the aggregation degree of the plurality of approximate knowledge data based on the proportion of the approximate knowledge data belonging to the same rough category in the plurality of approximate knowledge data.

[0151] The degree of aggregation of the multiple knowledge data is used to represent the proportion of the knowledge data belonging to the same rough category in the multiple knowledge data.

[0152] Step 403, in response to determining that the degree of aggregation is less than a first preset threshold, new approximate knowledge data that is similar to the data to be classified is determined from the coarse classification knowledge base, until it is determined that the degree of aggregation of all the approximate knowledge data currently obtained under a coarse classification in the coarse classification knowledge base is greater than or equal to the first preset threshold, or, all the approximate knowledge data currently obtained includes all the knowledge data under a coarse classification in the coarse classification knowledge base.

[0153] Step 404, determining the coarse category as the target coarse category, and determining the knowledge data under the target coarse category.

[0154] Step 405 : Determine, based on the precision of the classification result corresponding to the historical data, whether the historical data includes data to be changed whose classification result needs to be changed.

[0155] Step 406 , in response to the determination including, deleting the classification result corresponding to the data to be modified.

[0156] Step 407, determining the amount of target historical data to be recalled.

[0157] Among them, the number increases step by step as the time period goes by.

[0158] Step 408, determining a number of target historical data that are similar to the knowledge data from the historical data.

[0159] Step 409 , in response to determining that the target historical data includes unclassified data and the accuracy of the classification results of the historical data to be classified exceeds a preset accuracy threshold, the current precision of the classification standard adopted by the classification standard knowledge base is increased to determine the target precision.

[0160] The classification result of the historical data to be classified is determined based on a classification standard knowledge base that adopts the classification standard of the current precision.

[0161] Step 410 , determining and storing the classification result of the unclassified data by using the large language model according to the classification standard knowledge base using the classification standard of the target precision.

[0162] Step 411, in response to the classification result of the target historical data being determined based on querying the classified data, the weight of the classification result of the classified data is determined according to the strategy of increasing the weight step by step with the passage of time period; in response to the classification result of the target historical data being determined based on the classification standard knowledge base, the weight of the classification result of the unclassified data is determined according to the strategy of decreasing the weight step by step with the passage of time period.

[0163] Step 412, in response to determining, based on the weights corresponding to a number of target historical data, that the degree of aggregation of the classification results corresponding to a number of target historical data in one category is less than a second preset threshold, new target historical data that is similar to the knowledge data is determined from the historical data, until, based on the weights corresponding to all the target historical data currently obtained, it is determined that the degree of aggregation of the classification results corresponding to all the target historical data in one category is greater than or equal to the second preset threshold, and the category with a degree of aggregation greater than or equal to the second preset threshold is determined as the classification result of the data to be classified.

[0164] Step 413: taking the data to be classified as historical data, and storing the data to be classified and the classification results of the data to be classified.

[0165] It can be seen from this embodiment that Figure 2Compared with the corresponding embodiments, the process 400 of the data classification method in this embodiment specifically illustrates the process of determining the knowledge data and target historical data under the target coarse classification, the process of improving the refinement of the classification standard of the classification standard knowledge base, and the classification results of the data to be classified, thereby utilizing the knowledge data, historical data, classification standard knowledge base and other low-precision and easily accessible data resources under the coarse classification to obtain classification results with higher precision and higher accuracy, so that the precision and accuracy of the classification results of the data to be classified continue to improve with the passage of time.

[0166] Continue to refer Figure 5 , taking the customer complaint data to be classified as an example, the execution process of the data classification method is specifically described as follows:

[0167] Step 1: Historical VOB data storage and vectorization

[0168] Establish a historical VOB database and save all the conversations between merchants and customer service. Vectorize each historical VOB data in the VOB database, obtain and save the historical VOB vector, and establish a historical VOB vector library.

[0169] Step 1.1:

[0170] Build a database, define the VOB storage structure, and store historical VOB data.

[0171] The storage structure of historical VOB data must contain at least three fields: a unique VOB data identifier, historical VOB data, and classification results. If the historical VOB data does not include classification results, the classification result field is empty.

[0172] The database must include the function of persisting all historical VOB data, and may optionally include the function of web interface operation. The input link of historical VOB data can be data stream, file or other format, and the storage link of historical VOB data can be storage products such as MySQL or Redis.

[0173] As an example, first, build a MySQL database, use the create database command to create the database, and use the create table command to create the table; then, save the historical VOB data in a txt file; finally, use the MySQL import capability to import the file into the table created by the create table command.

[0174] The database must also include data maintenance capabilities, which must include "query", "add", "modify", and optionally "delete". Data maintenance capabilities must include the function of engineering execution operations. The reason for including engineering operations is that the automatic upgrade capability of this application relies on cross-system data modification and query operations.

[0175] Engineering execution operation refers to the ability to operate stored data through computer system server-side technology, and common methods include jdbc, etc. As an example, a jdbc connection is provided to the mysql database, which provides four operations: add, delete, modify, and query; a java service is set up to connect to the jdbc connection, and data maintenance can be achieved in the form of an interface; when an external modification instruction requests the interface, data modification is achieved; when an external query instruction requests the interface, data query is achieved.

[0176] Step 1.2: Vectorization of historical VOB data

[0177] 1.2.1: Choose a large model. This application can choose a large model with low parameters and low precision, as long as the classification accuracy rate is 50% (lower than the industry standard accuracy rate of 80%), which is the key feature of this application that distinguishes it from traditional high-precision operations.

[0178] 1.2.2: Generate historical VOB vectors based on the vectorization capability of the large model.

[0179] As an example, a large model with low parameters and low precision is selected, and its input and output capabilities are enabled; each historical VOB data in the historical VOB database in step 1.1 is input into the large language model, and the historical VOB vector corresponding to the historical VOB data is output.

[0180] 1.3: Storing historical VOB vectors

[0181] 1.3.1: Select open source vector libraries in the industry, which contain the necessary vector addition, deletion, modification and query capabilities.

[0182] 1.3.2: Use the preservation capability provided by the vector library to store historical VOB vectors.

[0183] The historical VOB vector in the historical VOB vector library must contain two fields: VOB data identifier and historical VOB vector.

[0184] The historical VOB vector library provides the ability to reverse query historical VOB data based on historical VOB vectors. As an example, the implementation steps of reverse querying historical VOB data based on historical VOB vectors include:

[0185] Based on the vector search operation, at least one historical VOB vector is confirmed; the VOB data identifier corresponding to each of the at least one historical VOB vector is extracted; and according to the VOB data identifier, based on the query capability of the historical VOB database, the historical VOB data corresponding to the VOB data identifier is determined.

[0186] Step 2: Identify domain knowledge

[0187] Customize the domain coarse classification knowledge base, vectorize the knowledge of the coarse classification knowledge base, provide knowledge election capabilities, provide election threshold setting capabilities, and provide automatic weight adjustment capabilities based on election results and threshold judgments. Finally, output the knowledge vector of the knowledge data under the target coarse classification to which the VOB data to be classified belongs, which is used to assist the subsequent step 3 to identify approximate VOB data.

[0188] Step 2.1: Create a coarse classification knowledge base and vectorize the knowledge data under coarse classification

[0189] 2.1.1: Domain experts manually perform rough classification of knowledge.

[0190] The criteria for coarse classification include: 1. The classification granularity is significantly lower than that required by the classification business. For example, if the business requires 200 VOB categories, the recommended coarse classification is about 20. 2. Each category has 3 or more pieces of knowledge (the necessary number to determine the degree of aggregation and approximate vector recall).

[0191] 2.1.2: The knowledge data under each coarse category is vectorized and stored through the large model. The storage structure contains at least three fields: knowledge data, the coarse category to which the knowledge data belongs, and knowledge vector.

[0192] Step 2.2: Approximate Knowledge Retrieval

[0193] An approximate knowledge vector that is similar to the VOB vector of the VOB data to be classified is retrieved from the coarse classification knowledge base.

[0194] The number of obtained approximate knowledge vectors is equal to the number of knowledge data under each coarse classification in the coarse classification knowledge base.

[0195] Step 2.3: Automatic adjustment

[0196] 2.3.1: Pre-set the clustering rate threshold. With 3 approximate knowledge vectors, the clustering rate threshold can be set to 2 / 3.

[0197] 2.3.2: Determine the degree of aggregation of the approximate knowledge vectors retrieved in step 2.2 and compare it with the aggregation rate threshold in step 2.3.1.

[0198] 2.3.3: If the degree of aggregation exceeds the aggregation rate threshold, proceed to step 2.4.

[0199] 2.3.4: If the degree of clustering is lower than the clustering rate threshold, it means that the large model cannot identify the coarse classification corresponding to the VOB data this time. Then the large model is even less likely to identify the approximate knowledge vector and needs to be flexibly adjusted.

[0200] The adjustment process is as follows: expand the number of recalled approximate knowledge vectors according to a predetermined ratio or quantity, for example, increase the number from 3 to 4; perform the first type of judgment: judge the degree of aggregation of the approximate knowledge vectors after increasing the number of recalls. If it meets the first preset threshold, the knowledge vectors under the coarse classification corresponding to the first preset threshold are taken as the result and go to step 2.4; if it does not meet the first preset threshold, perform the second type of judgment: whether the approximate knowledge vectors after increasing the number of recalls contain all the knowledge vectors under a coarse classification. If it contains all the knowledge vectors of a coarse classification, then go to step 2.4 with the coarse classification as the result. If it does not contain all the knowledge vectors of any coarse classification, the second type of judgment fails.

[0201] The above weighting process is executed cyclically until the aggregation degree of the approximate knowledge vector after increasing the number of recalls meets the first preset threshold or contains all the knowledge vectors of a certain coarse classification.

[0202] Step 2.4: Output the knowledge vector under the determined target coarse classification as the domain knowledge vector.

[0203] Step 3: Approximate VOB vector retrieval

[0204] Set a threshold N, and select N historical VOB vectors closest to the domain knowledge vector determined in step 2.4 from the historical VOB vectors generated in step 1.2 as approximate VOB vectors.

[0205] 1) Calculate the vector distance between the stored historical VOB vector and the domain knowledge vector;

[0206] 2) Sort the historical VOB vectors in ascending order of vector distance;

[0207] 3) Take the TopN of the sorting results;

[0208] 4) Output the TopN historical VOB vectors to the approximate vector list.

[0209] This step necessarily includes the capability of setting a threshold value, and optionally includes the capability of adjusting the threshold value. The threshold value may be set to a fixed value, such as setting n=5 (a preferred value in the business). If a fixed value is selected, the number of approximate VOB vectors retrieved in step 3 and the number of approximate VOB vectors involved in the selection process in the subsequent step 5 are both fixed values. Similarly, if the threshold value is adjustable, the number of approximate VOB vectors involved in the retrieval process in step 3 and the selection process in the subsequent step 5 are both adjusted.

[0210] Step 4: Retrieve classification results or model inference classification results

[0211] Determine whether the classification result of the approximate VOB vector already exists. If it does, search for the classification result. If it does not exist, infer its classification result through the large model.

[0212] 4.1: Retrieve classification results

[0213] 4.1.1: The VOB data identifier is taken from the approximate vector list generated in step 3.

[0214] 401.2: Based on the query function provided in step 1.1, use the VOB data identifier to query the historical VOB data.

[0215] 401.3: Retrieve the classification result fields in the historical VOB data and form an array.

[0216] The array examples are as follows: [

[0218] {vobId:1, identification result: "provides financial recharge query capabilities"},

[0219] {vobId:2, identification result: "provides express delivery query capability"},

[0220] {vobId:3, identification result: "provides plan to show query capability"},

[0221] {vobId:4,Recognition result:“”}, ]

[0223] 4.1.4: Determine the existence of classification results.

[0224] If the classification result field is empty, there is no classification result in the original historical VOB data stored in the database, and the classification result needs to be obtained through reasoning in the subsequent step 4.2.

[0225] 4.2: Model inference classification results

[0226] The core implementation of the process of reasoning historical VOB data based on the big model is the natural language understanding ability based on the big model. Compared with traditional solutions, this application does not require the big model to have high accuracy. Therefore, the industry's public big model can meet the requirements of this solution after mounting the classification standard knowledge base. The minimum effect of the big model can also achieve the effect of this solution. This is the core value of this application. Therefore, this application does not need to optimize the reasoning ability of the big model specifically, and can use the conventional method.

[0227] As an example, a large language model is selected; a classification standard knowledge base is set for the large language model; the large model is used to infer and recall candidate classification results of historical VOB data based on the classification standard knowledge base; and the recall results of the large model are used as classification results of historical VOB data.

[0228] The classification standard knowledge base used by the large model needs to be optimized to adapt to the VOB classification scenario. Specifically, the classification standard knowledge base is optimized by dividing the classification standard into at least two levels of precision, wherein at least one level of classification standard can achieve relatively accurate classification when adopted, and at least one level of classification standard can achieve high-precision classification when adopted.

[0229] The key feature of the classification standard that can achieve relatively accurate classification is: a sufficiently coarse classification granularity, so that the accuracy of the classification result obtained based on the large model (inference accuracy) is not less than 50%, and the accuracy of the classification result after the election in the subsequent step 5 (election accuracy) is not less than 80%. If this standard is not met, the classification granularity must be further merged, and at least only binary classification recognition can be performed on the historical VOB data.

[0230] If only binary classification is performed on historical VOB data, the classification standard knowledge base will have only two records: system optimization required and system optimization not required. When the large model infers the results, one of the two categories will be selected, which will inevitably achieve an inference accuracy of no less than 50% and a selection accuracy of no less than 80%.

[0231] The key feature of a classification standard that can achieve high-precision classification is: a sufficiently detailed classification standard so that its classification results can directly guide business optimization. "Directly guide business optimization" is a relatively vague concept, depending on the demands of the business system to which this application is applied. Its judgment standard can be defined as: a result that exceeds the second classification (i.e., higher than the minimum value of the first classification).

[0232] It should be noted that at least two levels of classification standards in the classification standard knowledge base are not effective at the same time. Taking the two-level classification standard as an example, the first level classification standard is selected in the initial stage of implementation, and the second level classification standard is selected in the later stage of implementation.

[0233] The sign of the transition between the early stage of execution and the late stage of execution is that the election accuracy using the first-level classification standard exceeds a preset accuracy threshold (eg, 92%).

[0234] 4.3: Generate classification result array

[0235] After the classification results are obtained by inferring the historical VOB data that does not include the classification results through the large model, a classification result array corresponding to the approximate VOB vector is generated. The data structure used by the classification result array must include three fields: whether it is the VOB data to be identified this time, the classification result, and the method of determining the classification result.

[0236] The value range of "Whether it is the VOB to be identified this time" is: true, false (or other fields that can represent true / false, such as 0 / 1). For each approximate VOB vector in the approximate vector list, if the classification result corresponding to the approximate VOB vector is obtained through the current inference of the large model, then this field is true, otherwise it is false.

[0237] The value range of "Identification method" is: model reasoning, result query (or other methods that can represent the binary classification, such as search / reasoning)

[0238] If the value of the "Whether this is the VOB to be identified" field is true, the value of the "Identification method" must be model reasoning.

[0239] The classification result array example is as follows: [

[0241] {isOrigin:true, identification result:"provides financial recharge query capability", source:"result query"},

[0242] {isOrigin:false, identification result: "provides financial invoice query capabilities", source: "result query"},

[0243] {isOrigin:false, identification result: "provides express consumption data query capability", source: "result query"},

[0244] {isOrigin:false, identification result: "provides a plan to show query capabilities", source: "model reasoning"} ]

[0246] Step 5: Select the classification results of the VOB data to be classified and store them in the database

[0247] The election rules are set and the election capability is provided. The classification result of the VOB data to be classified is determined based on the election process. The VOB data to be classified is used as the historical VOB data, and the VOB data to be classified and its classification result are stored in the historical VOB database.

[0248] 5.1: Elections

[0249] 5.1.1: Weight setting

[0250] This step is used to weight the classification results of the approximate VOB vectors determined in step 4, so as to generate more accurate election results, that is, the classification results of the VOB data to be classified.

[0251] According to the judgment of the business development stage described in step 4 above, different weight modes need to be selected, namely: the classification results recorded in the historical VOB database are prioritized, and the classification results obtained by model reasoning are prioritized. Among them, priority means that the weight accounts for more than 50%.

[0252] Regarding the setting method of giving priority to the classification results obtained by model reasoning, in the early stage of business development, most of the historical VOB data have no classification results. At this time, the reasoning ability of the large model for the classification results is more important. The classification results obtained by model reasoning should be given priority in the weight setting.

[0253] Regarding the setting mode of giving priority to the classification results recorded in the historical VOB database, in the later stage of business development, most of the historical VOB data have classification results. At this time, the existing information in the historical VOB database is more important, and the classification results recorded in the historical VOB database should be given priority in the weight setting. This is also the core logic that the classification results of the VOB data to be classified in this embodiment can be automatically upgraded to truly high-precision.

[0254] The data example after setting the weights is: [

[0256] {isOrigin:true, identification result: "provides financial recharge query capability", weight:"5"},

[0257] {isOrigin:false, identification result: "provides financial invoice query capabilities", weight:"3"},

[0258] {isOrigin:false, identification result: "provides express consumption data query capability", weight:"2"},

[0259] {isOrigin:false, identification result: "provides plan to show query capabilities", weight:"2"} ]

[0261] 5.1.2: Election process settings

[0262] According to the set weight, the weight score of each record in the classification result array obtained in step 4.3 can be obtained, and the score can be voted and counted; the dimension of statistical aggregation is the "recognition result" field. As an example, the election method is such as the majority win method and the 2 / 3 win method.

[0263] Optionally, the election fails if the degree of aggregation does not exceed the second preset threshold. If this method is selected, it is necessary to set an additional expansion threshold and re-search capability for step 3 to determine a new approximate historical vector from the historical VOB database until the degree of aggregation of the classification results corresponding to all the approximate historical vectors in one category is determined to be greater than or equal to the second preset threshold based on the weights corresponding to all the currently obtained approximate historical vectors.

[0264] 5.1.3: Election Results

[0265] Regardless of the election method selected, two results can be obtained in the end: determining the winner and not determining the winner. If the winner is determined, the classification result of the winner is determined as the classification result of the VOB data to be classified.

[0266] If the winner is not determined, the following two strategies can be selected: 1. Abandon the model inference results of this time, mark the model inference process of this time as "inaccurate", and change it to manual recognition. 2. Use the classification result obtained by inferring the historical VOB data with the large model as the final result. This step will inevitably generate a final result.

[0267] 5.2: Save the classification results of the VOB data to be classified

[0268] Call the "modify" capability provided in step 1.1, retrieve the corresponding historical VOB data in the historical VOB database according to the VOB data identifier, and modify its "classification result" field to the classification result determined by the election in step 5.1.3.

[0269] 5.3: Tuning solution

[0270] Based on the judgment of the business development stage described in step 4 above, there are two different tuning solutions:

[0271] In the early stage of business development, the classification results saved in step 5.2 are regularly counted, the accuracy of the results is manually judged, and the classification results that are manually determined to be "inaccurate" and the classification results of the data of election failure in step 5.1.3 are revised to correct values.

[0272] In the later stage of business development, as more historical VOB data have accurate identification results, the weight setting of step 5.1.1 tends to give priority to the classification results recorded in the historical VOB database, and then adjusts the threshold of step 3. By increasing the threshold, more historical VOB data with accurate classification results can be recalled, thereby achieving automatic upgrade of the classification results in terms of progress and accuracy.

[0273] Continue to refer Figure 6 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a data classification device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0274] like Figure 6 As shown, a data classification device includes: a first determination unit 601, configured to determine the knowledge data under the target coarse classification to which the data to be classified belongs; a second determination unit 602, configured to determine the target historical data that is similar to the knowledge data; a third determination unit 603, configured to determine the target fineness of the classification standard to be adopted by the classification standard knowledge base in response to determining that the target historical data includes unclassified data, according to the accuracy of the classification results of the historical data to be classified, wherein the classification results of the historical data to be classified are determined based on the classification standard knowledge base that adopts the classification standard of the current fineness; a fourth determination unit 604, configured to determine and store the classification results of the unclassified data through a large language model according to the classification standard knowledge base that adopts the classification standard of the target fineness; a fifth determination unit 605, configured to determine the classification results of the data to be classified according to the classification results of the target historical data.

[0275] In some examples, the fifth determination unit 605 is further configured to: determine the weight of the classification result of the target historical data according to a method for determining the classification result of the target historical data, wherein the determination method includes: querying the classified data in the target historical data to determine the classification result, or determining the classification result of the unclassified data according to the classification standard knowledge base; determine the classification result of the data to be classified according to the classification results and weights corresponding to each of the target historical data.

[0276] In some examples, the fifth determination unit 605 is further configured to: in response to the classification result of the target historical data being determined based on querying the classified data, determine the weight of the classification result of the classified data according to a strategy in which the weight increases step by step with the passage of time period; in response to the classification result of the target historical data being determined based on the classification standard knowledge base, determine the weight of the classification result of the unclassified data according to a strategy in which the weight decreases step by step with the passage of time period.

[0277] In some examples, the first determination unit 601 is further configured to: determine a plurality of approximate knowledge data that are similar to the data to be classified from the coarse classification knowledge base; determine the degree of aggregation of the plurality of approximate knowledge data based on the proportion of the approximate knowledge data belonging to the same coarse classification among the plurality of approximate knowledge data; and determine the target coarse classification and the knowledge data under the target coarse classification according to the degree of aggregation of the plurality of approximate knowledge data.

[0278] In some examples, the first determination unit 601 is further configured to: in response to determining that the degree of aggregation is less than a first preset threshold, determine new approximate knowledge data that is similar to the data to be classified from the coarse classification knowledge base, until it is determined that the degree of aggregation of all the approximate knowledge data currently obtained under a coarse category in the coarse classification knowledge base is greater than or equal to the first preset threshold, or, all the approximate knowledge data currently obtained includes all knowledge data under a coarse category in the coarse classification knowledge base; and determine the coarse category as the target coarse category.

[0279] In some examples, the above-mentioned device also includes: a change unit (not shown in the figure), which is configured to: before determining the target historical data that is approximate to the knowledge data, determine whether the historical data includes data to be changed whose classification results need to be changed according to the fineness of the classification results corresponding to the historical data; in response to determining that it includes, delete the classification results corresponding to the data to be changed.

[0280] In some examples, the above-mentioned change unit (not shown in the figure) is further configured to: determine whether the historical data includes the data to be changed based on the fineness of the classification results corresponding to the historical data and the fineness of the classification results of the data to be classified that can be obtained currently.

[0281] In some examples, the second determination unit 602 is further configured to: determine the number of target historical data to be recalled, wherein the number increases stepwise as the time period passes; and determine a number of target historical data approximate to the knowledge data from the historical data.

[0282] In some examples, the fifth determination unit 605 is further configured to: in response to determining, based on the weights corresponding to a number of target historical data, that the degree of aggregation of the classification results corresponding to a number of target historical data in one category is less than a second preset threshold, determine new target historical data that is similar to the knowledge data from the historical data, until determining, based on the weights corresponding to all the target historical data currently obtained, that the degree of aggregation of the classification results corresponding to all the target historical data in one category is greater than or equal to the second preset threshold, and determine the category with a degree of aggregation greater than or equal to the second preset threshold as the classification result of the data to be classified.

[0283] In some examples, the third determination unit 603 is further configured to: in response to the accuracy of the classification results of the historical data to be classified exceeding a preset accuracy threshold, increase the current precision of the classification standard adopted by the classification standard knowledge base and determine the target precision.

[0284] In some examples, the apparatus further includes: a storage unit (not shown in the figure), configured to: use the data to be classified as historical data, and store the data to be classified and the classification results of the data to be classified.

[0285] In this embodiment, the first determination unit in the data classification device determines the knowledge data under the target coarse classification to which the data to be classified belongs; the second determination unit determines the target historical data that is similar to the knowledge data; the third determination unit determines the target fineness of the classification standard to be adopted by the classification standard knowledge base in response to determining that the target historical data includes unclassified data, according to the accuracy of the classification results of the historical data to be classified, wherein the classification results of the historical data to be classified are determined based on the classification standard knowledge base that adopts the classification standard of the current fineness; the fourth determination unit determines and stores the classification results of the unclassified data through a large language model according to the classification standard knowledge base that adopts the classification standard of the target fineness; the fifth determination unit determines the classification results of the data to be classified according to the classification results of the target historical data, thereby using the knowledge data, historical data, classification standard knowledge base and other data resources with lower precision and easy access under the coarse classification to obtain a classification result with higher precision and higher accuracy, and the fineness of the classification standard of the classification standard knowledge base is increased stepwise with the passage of time, so that the precision and accuracy of the classification results of the data to be classified continue to improve with the passage of time.

[0286] Reference below Figure 7 , which shows a device suitable for implementing the embodiments of the present application (eg Figure 1 Schematic diagram of the structure of a computer system 700 of devices 101, 102, 103, 105) shown. Figure 7 The device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0287] like Figure 7 As shown, the computer system 700 includes a processor (e.g., CPU, central processing unit) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The processor 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0288] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0289] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 709, and / or installed from a removable medium 711. When the computer program is executed by the processor 701, the above-mentioned functions defined in the method of the present application are executed.

[0290] It should be noted that the computer-readable medium of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0291] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the client computer, partially on the client computer, as a separate software package, partially on the client computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the client computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0292] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the device, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0293] The units involved in the embodiments described in the present application may be implemented by software or by hardware. The units described may also be set in a processor, for example, may be described as: a processor comprising a first determination unit, a second determination unit, a third determination unit, a fourth determination unit, and a fifth determination unit. Among them, the names of these units do not constitute a limitation on the unit itself under certain circumstances. For example, the third determination unit may also be described as "a unit that determines the target fineness of the classification standard to be adopted by the classification standard knowledge base in response to determining that the target historical data includes unclassified data, according to the accuracy of the classification results of the historical data to be classified".

[0294] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the computer device: determines the knowledge data under the target coarse classification to which the data to be classified belongs; determines the target historical data similar to the knowledge data; in response to determining that the target historical data includes unclassified data, determines the target fineness of the classification standard to be adopted by the classification standard knowledge base according to the accuracy of the classification results of the historical data to be classified, wherein the classification results of the historical data to be classified are determined based on the classification standard knowledge base that adopts the classification standard of the current fineness; through the large language model, the classification results of the unclassified data are determined and stored according to the classification standard knowledge base that adopts the classification standard of the target fineness; and the classification results of the data to be classified are determined according to the classification results of the target historical data.

[0295] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A data classification method, comprising: Determine the knowledge data under the target coarse classification to which the data to be classified belongs; determining target historical data that is similar to the knowledge data; In response to determining that the target historical data includes unclassified data, determining a target fineness of a classification standard to be adopted by a classification standard knowledge base according to the accuracy of a classification result of the historical data to be classified, wherein the classification result of the historical data to be classified is determined based on a classification standard knowledge base adopting a classification standard of a current fineness; Determining and storing the classification result of the unclassified data according to the classification standard knowledge base using the classification standard of the target fineness through the large language model; According to the classification result of the target historical data, the classification result of the data to be classified is determined.

2. The method according to claim 1, wherein: Determining the classification result of the data to be classified according to the classification result of the target historical data includes: Determining the weight of the classification result of the target historical data according to a determination method of the classification result of the target historical data, wherein the determination method includes: querying the classified data in the target historical data to determine the classification result, or determining the classification result of the unclassified data according to the classification standard knowledge base; The classification results of the data to be classified are determined according to the classification results and weights corresponding to each of the target historical data.

3. The method according to claim 2, wherein: The determining of the weight of the classification result of the target historical data according to the determination method of the classification result of the target historical data includes: In response to the classification result of the target historical data being determined based on querying the classified data, determining the weight of the classification result of the classified data according to a strategy of increasing the weight in a stepwise manner as time passes; In response to the classification result of the target historical data being determined according to the classification standard knowledge base, the weight of the classification result of the unclassified data is determined according to a strategy in which the weight decreases stepwise as time passes.

4. The method according to claim 1, wherein: The step of determining the knowledge data under the target rough classification to which the data to be classified belongs includes: Determine a plurality of approximate knowledge data similar to the data to be classified from the coarse classification knowledge base; Determining the degree of aggregation of the plurality of approximate knowledge data based on the proportion of the approximate knowledge data belonging to the same rough category among the plurality of approximate knowledge data; The target rough category and the knowledge data under the target rough category are determined according to the aggregation degree of the plurality of approximate knowledge data.

5. The method according to claim 4, wherein: The determining the target rough classification according to the aggregation degree of the plurality of approximate knowledge data comprises: In response to determining that the degree of aggregation is less than a first preset threshold, new approximate knowledge data that is similar to the data to be classified is determined from the coarse classification knowledge base, until it is determined that the degree of aggregation of all the currently obtained approximate knowledge data under a coarse classification in the coarse classification knowledge base is greater than or equal to the first preset threshold, or that all the currently obtained approximate knowledge data includes all the knowledge data under a coarse classification in the coarse classification knowledge base; The coarse category is determined as the target coarse category.

6. The method according to claim 1, wherein: Before determining the target historical data that is similar to the knowledge data, the method further includes: Determining, based on the precision of the classification results corresponding to the historical data, whether the historical data includes data to be changed for which the classification results need to be changed; In response to the determination including, deleting the classification result corresponding to the data to be modified.

7. The method according to claim 6, wherein: The determining, based on the precision of the classification results corresponding to the historical data, whether the database includes data to be changed whose classification results need to be changed comprises: Whether the historical data includes the data to be modified is determined according to the fineness of the classification result corresponding to the historical data and the fineness of the classification result of the currently available data to be classified.

8. The method according to claim 2, wherein: The determining of the target historical data that is similar to the knowledge data includes: Determining the amount of target historical data to be recalled, wherein the amount increases in a step-by-step manner as time passes; The number of target historical data similar to the knowledge data is determined from the historical data.

9. The method according to claim 8, wherein: Determining the classification results of the data to be classified according to the classification results and weights corresponding to the target historical data includes: In response to determining, based on the weights corresponding to the respective target historical data, that the degree of aggregation of the classification results corresponding to the respective target historical data in one category is less than a second preset threshold, new target historical data that is similar to the knowledge data is determined from the historical data, until determining, based on the weights corresponding to all the target historical data currently obtained, that the degree of aggregation of the classification results corresponding to all the target historical data in one category is greater than or equal to the second preset threshold, and determining the category with a degree of aggregation greater than or equal to the second preset threshold as the classification result of the data to be classified.

10. The method according to claim 1, wherein: Determining the target precision of the classification standard to be adopted by the classification standard knowledge base according to the accuracy of the classification results of the historical data to be classified includes: In response to the accuracy of the classification result of the historical data to be classified exceeding a preset accuracy threshold, the current precision of the classification standard adopted by the classification standard knowledge base is increased, and the target precision is determined.

11. The method according to claim 1, wherein: Also includes: The data to be classified is used as historical data, and the data to be classified and the classification result of the data to be classified are stored.

12. A method for classifying customer complaint data, comprising: A first determining unit is configured to determine the knowledge data under the target coarse classification to which the data to be classified belongs; a second determining unit configured to determine target historical data that is similar to the knowledge data; a third determining unit, configured to determine, in response to determining that the target historical data includes unclassified data, a target fineness of a classification standard to be adopted by the classification standard knowledge base according to the accuracy of the classification result of the historical data to be classified, wherein the classification result of the historical data to be classified is determined based on the classification standard knowledge base adopting the classification standard of the current fineness; A fourth determining unit is configured to determine and store a classification result of the unclassified data according to a classification standard knowledge base using a classification standard of the target fineness through a large language model; The fifth determining unit is configured to determine the classification result of the data to be classified according to the classification result of the target historical data.

13. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

14. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 11.

15. A computer program product comprising: A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images