Financial data grading method and device based on domain knowledge

Abstract

The invention provides a financial data grading method and device based on domain knowledge, relates to the field of artificial intelligence, can also be used in the financial field, and comprises the following steps: inputting graded financial data in an annotation data set into each language model, determining a global field keyword embedding expression and a specialty field keyword embedding expression corresponding to the graded financial data; determining a large language model matched with the financial data to be graded according to the global field keyword embedding representation and the specialty field keyword embedding representation; and generating a grading result of the financial data to be graded by using the matched large language model. According to the method and the device, prediction can be performed by applying the model with better performance in the subdivided field to which the to-be-tested data belongs, the advantages of the model in each subdivided field are fully exerted, and the grading effect of financial data is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and can be used in the financial field. Specifically, it is a method and device for grading financial data based on domain knowledge. Background Art

[0002] Financial data has a high degree of confidentiality, and the practical process based on its data also has certain confidentiality requirements. Therefore, data security and controllability are particularly critical. Financial data security classification requires not only sufficient domain knowledge as a basis to understand the business meaning of the data, but also the ability to evaluate the degree of impact and the objects of impact contained in the data. This work involves a wide range of areas and has strict requirements. At present, it is mainly completed through qualitative work methods, and the reliability of qualitative work is improved by referring to the completed security classification results. However, different researchers cannot reach a consensus on many results, and even the results given by the same researcher at different stages are biased.

[0003] As research on large language models (LLMs) continues to deepen, more and more work is no longer limited to improving the performance of the model itself, but is more focused on how to achieve more efficient and reliable performance in tasks. Even general-purpose offline large language models have different expertise in various fields and tasks. Therefore, how to integrate multiple large language models to achieve more consistent performance improvement has become an urgent problem to be solved.

[0004] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by inclusion in this section. Summary of the invention

[0005] In response to the problems in the prior art, the present application provides a method and device for financial data classification based on domain knowledge, which can use a model with better performance in the field to make predictions for the sub-field to which the data to be tested belongs, give full play to the advantages of the model in each sub-field, and improve the classification effect of financial data.

[0006] In order to solve the above technical problems, this application provides the following technical solutions:

[0007] In a first aspect, the present application provides a method for grading financial data based on domain knowledge, comprising:

[0008] Input the classified financial data in the annotated data set into various language models to determine the global domain keyword embedding representation and the expertise domain keyword embedding representation corresponding to the classified financial data;

[0009] Determine a large language model matching the financial data to be classified according to the global domain keyword embedding representation and the expertise domain keyword embedding representation;

[0010] The matched large language model is used to generate a classification result of the financial data to be classified.

[0011] Furthermore, the classified financial data in the annotated data set is input into various language models to determine the global domain keyword embedding representation and the expertise domain keyword embedding representation corresponding to the classified financial data, including:

[0012] Input the domain keywords in the annotated data set into various language models to obtain the corresponding prediction results;

[0013] Dividing the labeled data set into a completely correct set, a partially correct set, and a completely wrong set according to the corresponding prediction results;

[0014] Extracting a global domain keyword embedding representation and an expertise domain keyword embedding representation of the domain keyword from the all correct set and the partially correct set respectively;

[0015] For each domain keyword in the partially correct set, multiple large language models with the best performance for the domain keyword are determined one by one.

[0016] Furthermore, the determining of the large language model matching the to-be-classified financial data according to the global domain keyword embedding representation and the expertise domain keyword embedding representation includes:

[0017] Calculating a first similarity between the financial data to be classified and the global domain keyword embedding representation;

[0018] Calculating a second similarity between the financial data to be classified and the embedded representation of the expertise domain keyword;

[0019] If the first similarity is greater than the second similarity, the pre-selected K large language models are determined as large language models matching the financial data to be classified; wherein K is a positive integer;

[0020] Otherwise, according to the field to which the financial data to be classified belongs, a pre-selected large language model with the best performance in the field is determined as the large language model matching the financial data to be classified.

[0021] Furthermore, the large language model with the best performance in the field is the large language model matched with the financial data to be classified; and the step of generating the classification result of the financial data to be classified by using the matched large language model includes:

[0022] Calculating a third similarity between the financial data to be classified and the keywords in each field in the partial correct set;

[0023] Sort the keywords in each field according to the third similarity to obtain the first k keywords in the field that are most similar to the financial data to be classified; wherein k is a positive integer;

[0024] The financial data to be classified is classified using the large language models corresponding to the first k domain keywords to obtain a final prediction result.

[0025] In a second aspect, the present application provides a financial data classification device based on domain knowledge, comprising:

[0026] An embedding representation generating unit, used for inputting the classified financial data in the annotated data set into various language models to determine the global domain keyword embedding representation and the expertise domain keyword embedding representation corresponding to the classified financial data;

[0027] A language model determination unit, configured to determine a large language model matching the to-be-classified financial data according to the global domain keyword embedding representation and the expertise domain keyword embedding representation;

[0028] A grading result generating unit is used to generate a grading result of the financial data to be graded by using the matched large language model.

[0029] Furthermore, the embedding representation generating unit comprises:

[0030] The prediction result generation module is used to input the domain keywords in the annotated data set into various language models to obtain the corresponding prediction results;

[0031] A correct and incorrect set division module, used to divide the labeled data set into a completely correct set, a partially correct set and a completely incorrect set according to the corresponding prediction results;

[0032] An embedding representation generating module, used for extracting a global domain keyword embedding representation and an expertise domain keyword embedding representation of the domain keyword from the all correct set and the partially correct set respectively;

[0033] The domain model determination module is used to determine, for each domain keyword in the partial correct set, a plurality of large language models that perform best for the domain keyword one by one.

[0034] Furthermore, the language model determination unit includes:

[0035] A first similarity determination module, configured to calculate a first similarity between the financial data to be classified and the global domain keyword embedding representation;

[0036] A second similarity determination module, configured to calculate a second similarity between the financial data to be classified and the embedded representation of the domain-of-expertise keyword;

[0037] A first model determination module, configured to determine K pre-selected large language models as large language models matching the financial data to be classified if the first similarity is greater than the second similarity; wherein K is a positive integer;

[0038] The second model determination module is used to determine, according to the field to which the to-be-classified financial data belongs, a pre-selected large language model with the best performance in the field as the large language model matching the to-be-classified financial data.

[0039] Furthermore, the grading result generating unit includes:

[0040] A third similarity determination module, used to calculate a third similarity between the financial data to be classified and the keywords in each field in the partial correct set;

[0041] A domain key ranking module, used to sort the domain keywords according to the third similarity to obtain the top k domain keywords that are most similar to the financial data to be classified; wherein k is a positive integer;

[0042] The final prediction module is used to classify the financial data to be classified by using the large language models corresponding to the first k domain keywords to obtain the final prediction result.

[0043] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the domain knowledge-based financial data classification method when executing the program.

[0044] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the financial data classification method based on domain knowledge.

[0045] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the financial data classification method based on domain knowledge.

[0046] In response to the problems in the prior art, the domain knowledge-based financial data classification method and device provided in the present application can find the large language model with the best performance in various fields through the performance of the large language model on the labeled data set; extract the semantic features of each sub-field by extracting the semantic information representation of domain keywords in combination with the context; determine the sub-field to which the data to be predicted belongs by calculating the weighted similarity; and integrate the prediction results of each large language model by weighted voting to obtain the final prediction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 This is a flowchart of a method for grading financial data based on domain knowledge in an embodiment of the present application;

[0049] Figure 2 A flowchart for generating a keyword embedding representation in an embodiment of the present application;

[0050] Figure 3 A flowchart for determining a large language model in an embodiment of the present application;

[0051] Figure 4 A flowchart for generating classification results in an embodiment of the present application;

[0052] Figure 5 This is a structural diagram of a financial data classification device based on domain knowledge in an embodiment of the present application;

[0053] Figure 6 This is a structural diagram of an embedded representation generation unit in an embodiment of the present application;

[0054] Figure 7 This is a structural diagram of a language model determination unit in an embodiment of the present application;

[0055] Figure 8 This is a structural diagram of a grading result generating unit in an embodiment of the present application;

[0056] Fig. 9 A schematic diagram of the structure of an electronic device in an embodiment of the present application;

[0057] Fig.10 This is one of the system block diagrams of the training phase in the embodiment of the present application;

[0058] Fig.11This is the second system block diagram of the training phase in the embodiment of the present application;

[0059] Fig.12 This is a system block diagram of the prediction stage in an embodiment of the present application. DETAILED DESCRIPTION

[0060] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0061] The information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0062] Provide users with corresponding operation entrances for them to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0063] In one embodiment, see Figure 1 In order to be able to predict the sub-field to which the data to be tested belongs, apply the model with better performance in the field to make full use of the advantages of the model in each sub-field and improve the classification effect of financial data, this application can provide a financial data classification method based on domain knowledge, including:

[0064] S101: Input the classified financial data in the annotated data set into various language models to determine the global domain keyword embedding representation (KeyEmb_SA) and the expertise domain keyword embedding representation (KeyEmb_SP) corresponding to the classified financial data;

[0065] S102: Determine a large language model matching the financial data to be classified according to the global domain keyword embedding representation and the expertise domain keyword embedding representation;

[0066] S103: Generate a classification result of the financial data to be classified by using the matched large language model.

[0067] It is understandable that the large language model has a certain general ability through learning through massive data in the pre-training stage. Compared with traditional machine learning or deep learning models, less training data is needed to achieve strong generalization ability and provide explanations for the prediction results. In the scenario of financial data security classification, multiple large language models can be applied to perform security classification on financial data, such as payment and clearing data. While it effectively saves the cost of manpower labeling, it also brings some problems: the prediction results of the major language models are not completely consistent, and how to integrate the results to achieve the best effect is an urgent problem to be solved.

[0068] To this end, the present application provides a method that integrates the results of multiple large language models after prediction to give a final financial data rating, wherein the rating may refer to a data security level rating.

[0069] See also Fig.10 , Fig.11 , which gives the system block diagram of the training phase.

[0070] The specific steps of training are as follows:

[0071] (1) Input: labeled data set, which contains classified financial data, i.e., each financial data and its corresponding classification.

[0072] (2) Output: two sets of embedding representations of domain keywords, KeyEmb_SA and KeyEmb_SP; and a list of optimal models for each record (classified financial data).

[0073] Among them, KeyEmb_SA and KeyEmb_SP are used to compare the similarity with the financial data to be classified in the future, so as to determine which large language model should be used to predict the financial data to be classified (also known as classification). The optimal model list records some large language models that are most accurate in predicting the classification of each classified financial data, so that when the financial data to be classified is similar to the subdivision field of the classified financial data, these large language models can be used to predict the financial data to be classified (also known as classification).

[0074] See also Fig.12 , which gives the system block diagram of the prediction stage.

[0075] The main prediction process is as follows:

[0076] (1) Input: data to be predicted (also called financial data to be classified).

[0077] (2) Output: prediction result (also called classification result).

[0078] From the above description, it can be seen that the domain knowledge-based financial data classification method and device provided in the present application can find the large language model with the best performance in each field through the performance of the large language model on the labeled data set; extract the semantic features of each sub-field by extracting the semantic information representation of domain keywords in combination with the context; determine the sub-field to which the data to be predicted belongs by calculating the weighted similarity; integrate the prediction results of each large language model by weighted voting to obtain the final prediction result.

[0079] In one embodiment, see Figure 2 , the classified financial data in the annotated data set is input into various language models to determine the global domain keyword embedding representation (KeyEmb_SA) and the expertise domain keyword embedding representation (KeyEmb_SP) corresponding to the classified financial data, including:

[0080] S201: Input the domain keywords in the annotated data set into various language models to obtain corresponding prediction results;

[0081] S202: Dividing the labeled data set into a completely correct set, a partially correct set, and a completely wrong set according to the corresponding prediction results;

[0082] S203: extracting the global domain keyword embedding representation and the expertise domain keyword embedding representation of the domain keyword from the all correct set and the partially correct set respectively;

[0083] S204: For each domain keyword in the partially correct set, determine a plurality of large language models that perform best for the domain keyword one by one.

[0084] It can be understood that the steps of determining the global domain keyword embedding representation (KeyEmb_SA) and the expertise domain keyword embedding representation (KeyEmb_SP) corresponding to the classified financial data are as follows:

[0085] T1: N large language models make predictions on the labeled datasets respectively, and obtain N groups of prediction results.

[0086] T2: Perform a preliminary screening of the models and remove models with poor performance. Specifically, compare the prediction results of the major language models and select the top K models with the highest accuracy. The prediction results are compared with the manual annotations (grading) in the annotated dataset to analyze whether the prediction is accurate.

[0087] T3: Based on the prediction results of K models on each record (classified financial data), the labeled data is divided into three sets, and domain embedding representations of keywords are extracted from two of the sets, the weight of domain keywords is increased, and domain keyword sets are formed. The specific method is as follows:

[0088] For each record r_i, there are K prediction values ​​(p1, p2, ..., pK) from K models, and a groundtruth value. According to the prediction results of each model, the records are divided into three sets: the set of records predicted correctly by all K large language models, denoted as S_A; the set of records predicted correctly by some large language models, denoted as S_P; the set of records predicted incorrectly by all K large language models, denoted as S_N. The S_A set represents the areas where all models perform well, while for the S_P set, each model has its own areas of expertise, which will be recorded in the T4 step later.

[0089] ·according to Fig.11 In the way shown, the embedding representation of domain keywords is extracted on the record sets S_A and S_P. The same keyword has different meanings in different contexts. Therefore, when extracting domain features, the contextual information of the keywords needs to be fully considered. The specific steps are:

[0090] a. Based on the Linear Discriminant Analysis (LDA) algorithm, perform topic modeling on S_A and S_P to obtain keywords and corresponding scores (if the keywords appear in multiple topics, the scores are averaged). For example:

[0091] ('Participating institutions', 0.0448379), ('Liquidation', 0.04245793)

[0092] b. For each keyword, calculate the domain embedding representation as follows:

[0093]

[0094] Where n represents the number of data items containing the keyword in the annotated data; m represents the total number of words in the keyword; e ij The embedding representation corresponding to the word at position j of the keyword after the i-th annotation data is encoded by BERT.

[0095] Increase the weight of domain keywords, search the industry glossary, and increase the corresponding weight of keywords that appear in the industry glossary. The weight value is adjusted according to the scenario and actual situation.

[0096] For a keyword in the record set S_P, if the keyword does not exist in S_A at the same time, its corresponding weight is increased. The weight value is adjusted according to the scenario and actual situation.

[0097] Use the weights obtained in the above two steps to correct the weight value of LDA obtained in step a.

[0098] After the above steps, the embedding representation sets of the keywords in the two fields are finally formed, corresponding to "the set of records S_A predicted correctly by all K large language models" and "the set of records S_P predicted correctly by some large language models", and the corresponding relationship is:

[0099] S_A==>KeyEmb_SA

[0100] S_P==>KeyEmb_SP

[0101] Each element in the set is in the form of (keyEmb, w), which is the domain embedding representation and weight of the keyword respectively.

[0102] For example, the weight determination process can be as follows:

[0103] 1. When extracting topics and keywords based on the LDA model, a score can be obtained for each keyword, which can be used as the initial value of the weight.

[0104] 2. Search the industry glossary. If the keyword appears in the industry glossary, increase the weight. For example, add 0.1 to the LDA score. This value needs to be adjusted according to the scenario and actual situation.

[0105] 3. For the keywords in S_P, if the keyword does not exist in S_A at the same time, increase the weight of the keyword. For example, based on the previous step, add 0.1. This value also needs to be adjusted according to the scenario and actual situation.

[0106] –T4: For each record in the S_P set, find the list of models that perform best. For each record record_i, record the list of models that can correctly predict the results: m_1, m_2...m_n.

[0107] From the above description, it can be seen that the domain knowledge-based financial data classification method and device provided in the present application can input the classified financial data in the labeled data set into various language models to determine the global domain keyword embedding representation (KeyEmb_SA) and the expertise domain keyword embedding representation (KeyEmb_SP) corresponding to the classified financial data.

[0108] In one embodiment, see Figure 3 The method of determining a large language model for matching the financial data to be classified according to the global domain keyword embedding representation and the expertise domain keyword embedding representation includes:

[0109] S301: Calculating a first similarity between the financial data to be classified and the global domain keyword embedding representation;

[0110] S302: Calculating a second similarity between the financial data to be classified and the embedded representation of the expertise domain keyword;

[0111] S303: If the first similarity is greater than the second similarity, determining the pre-selected K large language models as large language models matching the financial data to be classified; wherein K is a positive integer;

[0112] S304: Otherwise, according to the field to which the financial data to be classified belongs, a pre-selected large language model with the best performance in the field is determined as the large language model matching the financial data to be classified.

[0113] It can be understood that the process of determining the large language model that matches the financial data to be graded is as follows:

[0114] –P1: Use the similarity calculation method to find a set of records that are more similar to the data to be tested (financial data to be classified). Calculate the weighted average similarity between the financial data to be classified and each keyword in KeyEmb_SA and KeyEmb_SP. The weighted average similarity calculation method is as follows:

[0115]

[0116] in:

[0117] e p The embedding representation of the keyword P to be predicted obtained by BERT;

[0118] Cos stands for CosineSimilarity, which is the cosine similarity of vectors;

[0119] n represents the number of keyword embedding representations in KeyEmb_SA or KeyEmb_SP;

[0120] w i represents the weight of the i-th keyword obtained in step T3;

[0121] j represents the jth keyword in KeyEmb_SA or KeyEmb_SP;

[0122] ω j represents the weight of the jth keyword;

[0123] ·Represents the multiplication sign in multiplication operations.

[0124] –P2: If the test data is more similar to S_A, the K models selected in T2 are used for prediction, and the final result is determined by voting.

[0125] According to the similarity weighted average S calculated by the above formula, compare which of the test data is more similar to S_A and S_P. If it is more similar to S_A, use the K models selected in T2 for prediction. The final result is

[0126] Determined by vote:

[0127]

[0128] Among them, y j represents the prediction result of the jth model, K is the total number of j; i represents the level of security classification; I is an indicator function, which returns 1 if the condition is met, otherwise it returns 0; argmax represents the input corresponding to the maximum value of the function.

[0129] – P3: If the data to be tested is more similar to S_P, it is necessary to determine the field to which the data to be tested belongs, and use the model with the best performance in this field for prediction. For details, please refer to the description of steps S401 to S403.

[0130] From the above description, it can be seen that the domain knowledge-based financial data classification method and device provided in this application can determine the large language model matching the financial data to be classified based on the global domain keyword embedding representation and the expertise domain keyword embedding representation.

[0131] In one embodiment, see Figure 4 , the large language model with the best performance in the field is the large language model matched with the financial data to be classified; and the step of using the matched large language model to generate the classification result of the financial data to be classified includes:

[0132] S401: Calculating the third similarity between the financial data to be classified and the keywords in each field in the partial correct set;

[0133] S402: Sort the keywords in each field according to the third similarity to obtain the first k keywords in the field that are most similar to the financial data to be classified; wherein k is a positive integer;

[0134] S403: Using the large language models corresponding to the first k domain keywords, the financial data to be classified is classified to obtain a final prediction result.

[0135] It can be understood that the steps of generating the classification results of the financial data to be classified are as follows:

[0136] 1. Calculate the similarity between the financial data to be classified and each record in the S_P set, and take the first k most similar data and arrange them in descending order according to similarity: q_1, q_2, ... q_i ... q_k. These k data represent the field to which the data to be tested belongs. The similarity calculation method is as follows:

[0137] s i =Sim ST (SP i ,P)+α·DomainSim i

[0138] in:

[0139] Sims r Indicates the similarity calculated based on SentenceTransformer;

[0140] SP i Represents the i-th record in SP;

[0141] α is a hyperparameter, which can be determined by the user according to the actual application;

[0142] DomainSim i Represents the weights related to domain knowledge.

[0143] In similarity calculation, in order to pay more attention to domain knowledge, the DomainSim part is added. The calculation method is as follows:

[0144] If SP i Contains the field keyword Key_SP j ,but:

[0145] DomainSim i =Sim ST (Key-SP i , P)

[0146] Such as SP i Contains multiple field keywords Key-SP i , then take the average value;

[0147] Such as SP i The domain keyword Key-SP is not included i , then: DomainSim i =0.

[0148] Among them, P is the financial data to be classified.

[0149] 2. For each similar record, use the optimal model obtained in T4 to make predictions and obtain a set of prediction values:

[0150] Similar records Similarity with the record to be tested Prediction results of the optimal model (example) q_1 s_1 a, b, a, c q_2 s_2 a, b, b ... q_i s_i b, c, b, d ... q_k s_k a, c

[0151] 3. For each prediction result, use the similarity coefficient to perform weighted voting to obtain the final prediction result.

[0152] For example, the voting results for prediction result a are:

[0153] (s_1×(1+1)) / 4+(s_2×1) / 3+...+s_i×0+...+(s_k×1) / 2

[0154] It can be seen from the above description that the domain knowledge-based financial data classification method and device provided in the present application can generate the classification results of the financial data to be classified by using the matched large language model.

[0155] It should be noted that the financial data classification method based on domain knowledge provided in the embodiment of the present invention can be used in the financial field, and can also be used in any technical field other than the financial field. The embodiment of the present invention does not limit the application field of the financial data classification method based on domain knowledge.

[0156] Based on the same inventive concept, the embodiments of the present application also provide a financial data grading device based on domain knowledge, which can be used to implement the method described in the above embodiments, as described in the following embodiments. Since the principle of solving the problem by the financial data grading device based on domain knowledge is similar to that of the financial data grading method based on domain knowledge, the implementation of the financial data grading device based on domain knowledge can refer to the implementation of the method based on software performance benchmark determination, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0157] In one embodiment, see Figure 5 In order to be able to predict the sub-field to which the data to be tested belongs, apply a model that performs better in the field to make predictions, give full play to the advantages of the model in each sub-field, and improve the classification effect of financial data, this application provides a financial data classification device based on domain knowledge, including:

[0158] The embedding representation generating unit 501 is used to input the classified financial data in the annotated data set into various language models to determine the global domain keyword embedding representation and the expertise domain keyword embedding representation corresponding to the classified financial data;

[0159] A language model determination unit 502, configured to determine a large language model for matching the to-be-classified financial data according to the global domain keyword embedding representation and the expertise domain keyword embedding representation;

[0160] The classification result generating unit 503 is used to generate a classification result of the financial data to be classified by using the matched large language model.

[0161] In one embodiment, see Figure 6, the embedding representation generating unit 501 comprises:

[0162] Prediction result generation module 601, used to input domain keywords in the annotated data set into various language models to obtain corresponding prediction results;

[0163] A correct and incorrect set division module 602 is used to divide the labeled data set into a completely correct set, a partially correct set and a completely incorrect set according to the corresponding prediction results;

[0164] An embedding representation generating module 603 is used to extract a global domain keyword embedding representation and an expertise domain keyword embedding representation of the domain keyword from the all correct set and the partial correct set respectively;

[0165] The domain model determination module 604 is used to determine, for each domain keyword in the partial correct set, a plurality of large language models that perform best for the domain keyword.

[0166] In one embodiment, see Figure 7 , the language model determination unit 502 comprises:

[0167] A first similarity determination module 701, configured to calculate a first similarity between the financial data to be classified and the global domain keyword embedding representation;

[0168] A second similarity determination module 702, configured to calculate a second similarity between the financial data to be classified and the embedded representation of the expertise domain keyword;

[0169] A first model determination module 703 is configured to determine, if the first similarity is greater than the second similarity, K pre-selected large language models as large language models matching the financial data to be classified; wherein K is a positive integer;

[0170] The second model determination module 704 is used to determine, otherwise, according to the field to which the to-be-classified financial data belongs, a pre-selected large language model with the best performance in the field as the large language model matching the to-be-classified financial data.

[0171] In one embodiment, see Figure 8 , the grading result generating unit 503 comprises:

[0172] A third similarity determination module 801 is used to calculate a third similarity between the financial data to be classified and the keywords in each field in the partial correct set;

[0173] A domain keyword sorting module 802 is used to sort the domain keywords according to the third similarity to obtain the first k domain keywords that are most similar to the financial data to be classified; wherein k is a positive integer;

[0174] The final prediction module 803 is used to classify the financial data to be classified by using the large language models corresponding to the first k domain keywords to obtain the final prediction result.

[0175] From the hardware level, in order to be able to predict the sub-field to which the data to be tested belongs, apply a model that performs better in the field to make predictions, give full play to the advantages of the model in each sub-field, and improve the classification effect of financial data, the present application provides an embodiment of an electronic device for implementing all or part of the content of the financial data classification method based on domain knowledge, and the electronic device specifically includes the following content:

[0176] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the financial data classification device based on domain knowledge and related equipment such as core business system, user terminal and related database; the logic controller can be a desktop computer, tablet computer and mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the financial data classification method based on domain knowledge and the embodiment of the financial data classification device based on domain knowledge in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0177] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0178] In practical applications, part of the financial data classification method based on domain knowledge can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0179] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0180] Fig. 9 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig. 9 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig. 9 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0181] In one embodiment, the function of the financial data classification method based on domain knowledge can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0182] S101: Inputting the classified financial data in the annotated data set into various language models to determine the global domain keyword embedding representation and the expertise domain keyword embedding representation corresponding to the classified financial data;

[0183] S102: Determine a large language model matching the financial data to be classified according to the global domain keyword embedding representation and the expertise domain keyword embedding representation;

[0184] S103: Generate a classification result of the financial data to be classified by using the matched large language model.

[0185] From the above description, it can be seen that the domain knowledge-based financial data classification method and device provided in the present application can find the large language model with the best performance in each field through the performance of the large language model on the labeled data set; extract the semantic features of each sub-field by extracting the semantic information representation of domain keywords in combination with the context; determine the sub-field to which the data to be predicted belongs by calculating the weighted similarity; integrate the prediction results of each large language model by weighted voting to obtain the final prediction result.

[0186] In another embodiment, the domain knowledge-based financial data classification device can be configured separately from the central processing unit 9100. For example, the data composite transmission device based on domain knowledge of the financial data classification device can be configured as a chip connected to the central processing unit 9100, and the functions of the domain knowledge-based financial data classification method can be implemented through the control of the central processing unit.

[0187] like Fig. 9 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig. 9 In addition, the electronic device 9600 may also include Fig. 9 For components not shown, reference may be made to the prior art.

[0188] like Fig. 9 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0189] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0190] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0191] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0192] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0193] The communication module 9110 is a transmitter / receiver 9110 that sends and receives signals via an antenna 9111. The communication module (transmitter / receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0194] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0195] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the method for grading financial data based on domain knowledge in the above embodiments, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps of the method for grading financial data based on domain knowledge in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0196] S101: Inputting the classified financial data in the annotated data set into various language models to determine the global domain keyword embedding representation and the expertise domain keyword embedding representation corresponding to the classified financial data;

[0197] S102: Determine a large language model matching the financial data to be classified according to the global domain keyword embedding representation and the expertise domain keyword embedding representation;

[0198] S103: Generate a classification result of the financial data to be classified by using the matched large language model.

[0199] From the above description, it can be seen that the domain knowledge-based financial data classification method and device provided in the present application can find the large language model with the best performance in each field through the performance of the large language model on the labeled data set; extract the semantic features of each sub-field by extracting the semantic information representation of domain keywords in combination with the context; determine the sub-field to which the data to be predicted belongs by calculating the weighted similarity; integrate the prediction results of each large language model by weighted voting to obtain the final prediction result.

[0200] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0201] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0202] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0203] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0204] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for classifying financial data based on domain knowledge, characterized in that: include: Input the classified financial data in the annotated data set into various language models to determine the global domain keyword embedding representation and the expertise domain keyword embedding representation corresponding to the classified financial data; Determine a large language model matching the financial data to be classified according to the global domain keyword embedding representation and the expertise domain keyword embedding representation; The matched large language model is used to generate a classification result of the financial data to be classified.

2. The method for classifying financial data based on domain knowledge according to claim 1, characterized in that: The step of inputting the classified financial data in the annotated data set into various language models to determine the global domain keyword embedding representation and the expertise domain keyword embedding representation corresponding to the classified financial data includes: Input the domain keywords in the annotated data set into various language models to obtain the corresponding prediction results; Dividing the labeled data set into a completely correct set, a partially correct set, and a completely wrong set according to the corresponding prediction results; Extracting a global domain keyword embedding representation and an expertise domain keyword embedding representation of the domain keyword from the all correct set and the partially correct set respectively; For each domain keyword in the partially correct set, multiple large language models with the best performance for the domain keyword are determined one by one.

3. The method for classifying financial data based on domain knowledge according to claim 1, characterized in that: The method of determining a large language model matching the financial data to be classified according to the global domain keyword embedding representation and the expertise domain keyword embedding representation includes: Calculating a first similarity between the financial data to be classified and the global domain keyword embedding representation; Calculating a second similarity between the financial data to be classified and the embedded representation of the expertise domain keyword; If the first similarity is greater than the second similarity, the pre-selected K large language models are determined as large language models matching the financial data to be classified; wherein K is a positive integer; Otherwise, according to the field to which the financial data to be classified belongs, a pre-selected large language model with the best performance in the field is determined as the large language model matching the financial data to be classified.

4. The method for classifying financial data based on domain knowledge according to claim 2, characterized in that: The large language model with the best performance in the field is the large language model matched with the financial data to be classified; and the step of using the matched large language model to generate the classification result of the financial data to be classified includes: Calculating a third similarity between the financial data to be classified and the keywords in each field in the partial correct set; Sort the keywords in each field according to the third similarity to obtain the first k keywords in the field that are most similar to the financial data to be classified; wherein k is a positive integer; The financial data to be classified is classified using the large language models corresponding to the first k domain keywords to obtain a final prediction result.

5. A financial data classification device based on domain knowledge, characterized in that: include: An embedding representation generating unit, used for inputting the classified financial data in the annotated data set into various language models to determine the global domain keyword embedding representation and the expertise domain keyword embedding representation corresponding to the classified financial data; A language model determination unit, configured to determine a large language model matching the to-be-classified financial data according to the global domain keyword embedding representation and the expertise domain keyword embedding representation; A grading result generating unit is used to generate a grading result of the financial data to be graded by using the matched large language model.

6. The financial data classification device based on domain knowledge according to claim 5, characterized in that: The embedding representation generating unit comprises: The prediction result generation module is used to input the domain keywords in the annotated data set into various language models to obtain the corresponding prediction results; A correct and incorrect set division module, used to divide the labeled data set into a completely correct set, a partially correct set and a completely incorrect set according to the corresponding prediction results; An embedding representation generating module, used for extracting a global domain keyword embedding representation and an expertise domain keyword embedding representation of the domain keyword from the all correct set and the partially correct set respectively; The domain model determination module is used to determine, for each domain keyword in the partial correct set, a plurality of large language models that perform best for the domain keyword one by one.

7. The financial data classification device based on domain knowledge according to claim 5, characterized in that: The language model determination unit includes: A first similarity determination module, configured to calculate a first similarity between the financial data to be classified and the global domain keyword embedding representation; A second similarity determination module, used to calculate a second similarity between the financial data to be classified and the embedded representation of the expertise field keyword; A first model determination module, configured to determine K pre-selected large language models as large language models matching the financial data to be classified if the first similarity is greater than the second similarity; wherein K is a positive integer; The second model determination module is used to determine, according to the field to which the to-be-classified financial data belongs, a pre-selected large language model with the best performance in the field as the large language model matching the to-be-classified financial data.

8. The financial data classification device based on domain knowledge according to claim 6, characterized in that: The grading result generating unit comprises: A third similarity determination module, used to calculate a third similarity between the financial data to be classified and the keywords in each field in the partial correct set; A domain key ranking module, used to sort the domain keywords according to the third similarity to obtain the top k domain keywords that are most similar to the financial data to be classified; wherein k is a positive integer; The final prediction module is used to classify the financial data to be classified by using the large language models corresponding to the first k domain keywords to obtain the final prediction result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the financial data classification method based on domain knowledge described in any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for grading financial data based on domain knowledge as described in any one of claims 1 to 4 are implemented.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method for grading financial data based on domain knowledge as described in any one of claims 1 to 4 are implemented.

Images