Data processing method and device based on large model

Abstract

The invention discloses a data processing method and device based on a large model, and the method comprises the steps: obtaining structured information based on to-be-processed data uploaded by a terminal object; generating a to-be-tested data set based on each header name and the corresponding header position; calling the large model, and matching the header information in the to-be-tested data set with the standard header information in the template data set to obtain a matching result; according to the matching relationship, filling the data corresponding to each piece of header information into a target primary key of the target data table; determining a data deviation index based on the filling data of all the primary keys in the target data table; and when the data deviation index does not meet the requirement, sending an early warning notification to the terminal object. According to the method, matching between the standard header information and the standard header information is achieved through the large model, the target data table is filled with the data corresponding to the header information on the basis of the matching relation output by the large model, effective arrangement of the to-be-processed data is completed, and the processing efficiency of multi-source data integration is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular to a data processing method and device based on a large model. BACKGROUND

[0002] Data collection and data arrangement are the basis of all data analysis. Traditional data arrangement methods are first to collect various basic data of different formats by a large number of manual operations, and then to complete the integration of the basic data by manual operations through copying and pasting or various formulas. If the data sources exceed three, the data entries exceed a specified number of entries, and the data format is not fixed, a skilled manual operation is required to process and integrate the data in one working day. The accuracy of the data obtained by complete manual arrangement is completely dependent on the carefulness of the operator, and cannot solve the data deviation caused by accidental events in the operation process.

[0003] For the scenario of financial enterprises needing to integrate financial data, how to solve the problem of the application requirement of long process, low efficiency, and easy to make human errors in the process of multi-source data integration with a large number of human intervention has become a research hotspot. SUMMARY

[0004] The present application provides a data processing method and device based on a large model, which aims to improve the processing efficiency of multi-source data integration.

[0005] In order to achieve the above purpose, the present application provides the following technical solutions:

[0006] A data processing method based on a large model, comprising:

[0007] Based on the terminal object uploaded data to be processed, the corresponding structured information is obtained; the data to be processed includes a data table participating in financial data integration; the structured information includes a table header area and a data area; the table header area includes a plurality of table header information, the table header information includes a table header name and a corresponding table header position, and the data area includes a plurality of table header positions and corresponding data;

[0008] Based on each table header name and corresponding table header position, a corresponding test data set is generated;

[0009] A pre-trained large model is called to match the table header information in the test data set with the standard table header information in the template data set to obtain a corresponding matching result; the matching result includes the matching relationship between the table header information and the standard table header information;

[0010] According to the matching relationship, data corresponding to each table header information is filled into a target primary key of a target data table; the target primary key is a primary key corresponding to the terminal object;

[0011] Based on the filled data of all primary keys in the target data table, a data deviation index of the to-be-processed data is determined;

[0012] When the data deviation index does not meet the requirement, a warning notification is sent to the terminal object; the warning notification is used to prompt that the to-be-processed data has an error.

[0013] Optionally, based on the to-be-processed data uploaded by the terminal object, corresponding structured information is obtained, including:

[0014] Using a data segmentation model, the to-be-processed data uploaded by the terminal object is subjected to data segmentation, so as to obtain the structured information corresponding to the to-be-processed data; wherein the data segmentation model is , representing an upper limit of the number of table header information pre-calibrated, representing a table header name of the last row data in the data table, representing the number of each row in the data table, is a function for judging whether the row data in the range belonging to the first column in the data table is empty, is a function for determining the number of the smallest row with continuous data in the data table, is a function for determining the number of the largest row with a merged cell in the data table.

[0015] Optionally, based on each table header name and corresponding table header position, a corresponding to-be-tested data set is generated, including:

[0016] Based on the semantics of each table header name, a hierarchy of each table header name is determined in combination with a preset semantic hierarchy relationship table; the semantic hierarchy relationship table includes multiple hierarchies and corresponding semantics;

[0017] In order of high to low hierarchy, each table header name is sequentially sorted to obtain a table header name sequence;

[0018] A corresponding table header position is added to the table header name in the table header name sequence to obtain a corresponding to-be-tested data set.

[0019] Optionally, the training process of the large model includes:

[0020] After deploying the large model locally, the large model is pre-trained using the SQuAD corpus;

[0021] using a designated Chinese word segmentation system, performing word segmentation on the pre-acquired professional term information related to the financial field to obtain a vector set; the vector set includes keyword vectors corresponding to multiple professional terms;

[0022] calculating the similarity between each keyword vector in the vector set;

[0023] processing each keyword vector and the similarity between each keyword vector and other keyword vectors according to the corpus format of the SQuAD corpus to obtain a corresponding professional corpus;

[0024] training the pre-trained large model using the professional corpus to obtain a trained large model.

[0025] Optionally, after calling the pre-trained large model and matching the table header information in the to-be-tested data set with the standard table header information in the template data set to obtain a corresponding matching result, the method further includes:

[0026] if the matching result further includes unmatched information, sending the unmatched information to the terminal object to enable the terminal object to supplement corresponding target data; the unmatched information includes unmatched standard table header information; and the target data includes table header information and data corresponding to the unmatched standard table header information.

[0027] Optionally, based on the filling data of other primary keys in the target data table, a data deviation index of the to-be-processed data is determined, including:

[0028] based on the filling data of the target primary key and other primary keys in the target data table, as input of a data dispersion model, to obtain a dispersion result output by the data dispersion model; the data dispersion model includes wherein represents the dispersion result, represents the weight under the same condition for any primary key in the target data table, represents the weight of all data in the target data table, and , represents the filling data of any column in the target primary key, represents the average value of the filling data of all columns in the same primary key, represents the average value of the filling data of all columns in the target data table; the dispersion result includes the dispersion of the filling data of the target primary key;

[0029] based on the dispersion result, a data deviation index of the to-be-processed data is determined.

[0030] Optionally, the method further comprises:

[0031] In a case where the matching relationship is saved as a target rule without modification, table header information in the to-be-tested data set is verified according to the target rule to obtain a corresponding verification result.

[0032] When the verification result is qualified, data corresponding to each of the table header information is filled into the target primary key.

[0033] A data processing apparatus based on a large model comprises:

[0034] An information determination unit is configured to obtain corresponding structured information based on to-be-processed data uploaded by a terminal object; the to-be-processed data comprises a data table participating in financial data integration; the structured information comprises a table header area and a data area; the table header area comprises a plurality of table header information, the table header information comprises a table header name and a corresponding table header position, and the data area comprises a plurality of the table header positions and corresponding data.

[0035] A data set determination unit is configured to generate a corresponding to-be-tested data set based on each of the table header names and the corresponding table header positions.

[0036] A model calling unit is configured to call a pre-trained large model, match table header information in the to-be-tested data set with standard table header information in a template data set, and obtain a corresponding matching result; the matching result comprises a matching relationship between the table header information and the standard table header information.

[0037] A data filling unit is configured to fill data corresponding to each of the table header information into a target primary key of a target data table according to the matching relationship; the target primary key is a primary key corresponding to the terminal object.

[0038] An index determination unit is configured to determine a data deviation index of the to-be-processed data based on filled data of all primary keys in the target data table.

[0039] An early warning notification unit is configured to send an early warning notification to the terminal object when the data deviation index does not meet a requirement; the early warning notification is used to prompt that the to-be-processed data has an error.

[0040] A storage medium comprises a stored program, wherein the program is run by a processor to perform the data processing method based on a large model.

[0041] An electronic device comprises a processor, a memory and a bus; the processor is connected with the memory through the bus.

[0042] The memory is configured to store a program, and the processor is configured to execute the program, wherein the program performs the large model-based data processing method when executed by the processor.

[0043] The technical scheme provided in the present application obtains structured information based on the to-be-processed data uploaded by the terminal object. Based on each table header name and the corresponding table header position, a to-be-tested data set is generated. A large model is called to match the table header information in the to-be-tested data set with the standard table header information in the template data set to obtain a matching result. According to the matching relationship, the data corresponding to each table header information is filled into the target primary key of the target data table. Based on the filled data of all primary keys in the target data table, a data deviation index is determined. When the data deviation index does not meet the requirements, a warning notice is sent to the terminal object. The present application uses a large model to realize the matching of standard table header information and standard table header information, fills the data corresponding to the table header information into the target data table based on the matching relationship output by the large model, effectively organizes the to-be-processed data, and improves the processing efficiency of multi-source data integration. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 A flowchart of a large model-based data processing method provided by an embodiment of the present application;

[0046] Figure 2 A flowchart of another large model-based data processing method provided by an embodiment of the present application;

[0047] Figure 3 A flowchart of another large model-based data processing method provided by an embodiment of the present application;

[0048] Figure 4 An architectural diagram of a large model-based data processing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0050] In this application, the relational terms such as first and second and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any such actual relationship or order between such entities or actions, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0051] As shown in Figure 1 A flowchart of a data processing method based on a large model provided by an embodiment of the present application is shown, which includes the following steps.

[0052] S101: Obtain the corresponding structured information based on the terminal object uploaded data to be processed.

[0053] Among them, the data to be processed includes a data table participating in financial data integration, the structured information includes a table header area and a data area, the table header area includes a plurality of table header information, the table header information includes a table header name and a corresponding table header position, and the data area includes a plurality of table header positions and corresponding data.

[0054] In some examples, the table header can be understood as the column header of the data column in the data table, and the data table includes a plurality of data columns, and the column header of each data column can be regarded as the corresponding table header.

[0055] In some examples, the table header name can be understood as the column name of the data column corresponding to the table header, and the table header position can be understood as the position (for example, the column number) of the data column corresponding to the table header in the data table. In addition, the data corresponding to the table header position can be understood as the field in the data row under the data column corresponding to the table header.

[0056] It should be noted that the terminal object can be understood as a data source, and the data structures of the data tables uploaded by different data sources are different. Therefore, based on the data to be processed uploaded by different terminal objects, the corresponding structured information is obtained, so that the data tables with different data structures can be integrated into structured information with a unified format.

[0057] Optionally, the implementation process of obtaining the corresponding structured information based on the terminal object uploaded data to be processed can be: using a data segmentation model to perform data segmentation on the terminal object uploaded data to be processed to obtain the structured information corresponding to the data to be processed; wherein the data segmentation model is , This represents the predefined maximum number of header information entries. This represents the header name of the last non-empty row in the data table. This represents the row number in the data table. To determine the category of a data table A function to check if a range of rows in a column is empty. This is a function used to determine the smallest consecutive non-empty row number in a data table. This is used to identify the largest row in the data table that contains merged cells.

[0058] In some examples, It can be set to 10, which can be understood as the header area having a maximum of 10 rows, that is, limiting the total number of data rows under the header to no more than 10.

[0059] It should be noted that there is interfering data in the data to be processed. By using a data segmentation model to segment the data, the interfering data can be effectively removed and accurate structured information can be obtained.

[0060] S102: Generate the corresponding test dataset based on each header name and its corresponding header position.

[0061] The expression for the dataset to be tested can be: K={K1:COL1,K2:COL2,...KN:COLN}, where K1, K2...KN represent the corresponding header names, and COL1, COL2...COLN represent the corresponding header positions.

[0062] Optionally, for the implementation process of generating the corresponding test dataset based on each header name and its corresponding header position, please refer to [link / reference]. Figure 2 The steps shown are accompanied by corresponding explanations.

[0063] S103: Call the pre-trained large model to match the header information in the dataset to be tested with the standard header information in the template dataset to obtain the corresponding matching results.

[0064] The matching results include the matching relationship between the table header information and the standard table header information.

[0065] In some examples, large models include, but are not limited to, BERT-Base large models, Chinese large models, etc.

[0066] Optionally, the training process for large models can be found in [link to documentation]. Figure 3 The steps shown are accompanied by corresponding explanations.

[0067] In some examples, the matching relationship includes a complete matching relationship and a fuzzy matching relationship, the complete matching relationship represents that the similarity between the table header information and the standard table header information is greater than or equal to 0.9, and the fuzzy matching relationship represents that the similarity between the table header information and the standard table header information is in the range of (0.7, 0.9).

[0068] It should be noted that the template data set is determined based on reference data pre-uploaded by the user, and after obtaining the reference data, the reference data needs to be arranged to form a structured template data set S={S1:COL1,S2:COL2,...SN:COLN}, wherein S1,...,SN represents the standard table header name of the reference data, and COL1,...,COLN represents the position of the standard table header name in the data table shown by the reference data (i.e., the standard table header position).

[0069] In some examples, the large model matches the table header information in the to-be-tested data set with the standard table header information in the template data set, and the obtained matching result further includes unmatched information, and the unmatched information includes unmatched standard table header information (i.e., standard table header information that is not matched to corresponding table header information in the to-be-tested data set).

[0070] In some examples, the matching result output by the large model can be: T={complete matching relationship {S1-K1:COL1-COL1,...,SN-KN:COLN-COLN}, fuzzy matching relationship {S1-K1:COL1-COL1,...,SN-KN:COLN-COLN}, and unmatched information {ST:COLT}}.

[0071] In possible implementations, the matching result output by the large model can be: T={complete matching relationship {company code-A:company code:A}, fuzzy matching relationship {accounting period-B:period-B, operating income-C:operating income-C}, and unmatched information {non-operating income-D}}.

[0072] Optionally, if the matching result further includes unmatched information, the unmatched information is sent to the terminal object to enable the terminal object to supplement the corresponding target data, the unmatched information includes unmatched standard table header information, and the target data includes table header information and data corresponding to the unmatched standard table header information.

[0073] In possible implementations, the target data can be {non-operating income-D:other income-F}.

[0074] It should be noted that after the user feeds back the target data, the final mapping relationship can be determined based on the matching relationship and the target data, and the final mapping relationship is saved as a target rule for subsequent direct calling.

[0075] In some examples, the target rule can be {company code-A: company code: A, accounting period-B: period-B, operating income-C: operating income-C, other income-D: other income-F}.

[0076] Optionally, in the case that the matching relationship is saved as the target rule and the target rule is not modified, the table header information in the to-be-tested data set is verified according to the target rule to obtain a corresponding verification result; when the verification result is qualified, the data corresponding to each table header information is filled into the target primary key.

[0077] In some examples, the matching relationship can also be used as new training corpus to retrain the large model, so that the large model performs accurate matching when facing subsequent similar semantics, and the understanding ability of the large model is improved.

[0078] S104: Fill the data corresponding to each table header information into the target primary key of the target data table according to the matching relationship.

[0079] The target primary key is a primary key corresponding to a terminal object. The target data table includes multiple primary keys, and each primary key corresponds to a different terminal object and represents a different data source.

[0080] S105: Determine the data deviation index of the to-be-processed data based on the filled data of all primary keys in the target data table.

[0081] The filled data of the other primary keys in the target data table can be understood as historical financial data that has been processed.

[0082] Optionally, the implementation process of filling the data corresponding to each table header information into the target primary key of the target data table according to the matching relationship can be: based on the filled data of the target primary key and the other primary keys in the target data table, as the input of a data dispersion model, to obtain a dispersion result output by the data dispersion model; the data dispersion model includes wherein represents the dispersion result, represents the weight under the same condition for any primary key in the target data table, represents the weight of all data in the target data table, and , represents the filled data of any column in the target primary key, represents the average value of the filled data of all columns in the same primary key, represents the average value of the filled data of all columns in the target data table, and the dispersion result includes the dispersion of the filled data of the target primary key; based on the dispersion result, the data deviation index of the to-be-processed data is determined.

[0083] In some examples, assuming that there is historical data for the target primary key "A001-2023Q1" (i.e., there is fill-in data for other primary keys), and the "current sales expense" (which belongs to a column in the target primary key) in the target primary key suddenly increases by 200%, the dispersion degree f of the target primary key is calculated according to the data dispersion model, and the dispersion degree f of the target primary key is greater than 0.3. Since f>0.3, the data deviation index does not meet the requirements, and a corresponding warning notification needs to be sent to the corresponding terminal object.

[0084] S106: When the data deviation index does not meet the requirements, send a warning notification to the terminal object.

[0085] The warning notification is used to prompt that the to-be-processed data has errors.

[0086] It can be understood that the fill-in data whose data deviation index meets the requirements will be considered as successfully arranged and saved to the database.

[0087] It should be noted that the to-be-processed data of the plurality of terminal objects is processed by the large model to realize the data arrangement and induction method, respectively from the corpus pre-training, data column header segmentation and assembly, data warning, to ensure the accuracy of data arrangement. Not only can the data arrangement efficiency be greatly improved, but also the data accuracy during data arrangement can be improved.

[0088] The above-mentioned process of S101-S106 uses a large model to realize the matching of the table header information in the to-be-processed data of the terminal object and the standard table header information in the template data set. Based on the matching relationship output by the large model, the data corresponding to the table header information is filled into the target data table, the effective arrangement of the to-be-processed data is completed, the processing efficiency of multi-source data integration is improved, and based on the fill-in data of all primary keys in the target data table, the data deviation index of the to-be-processed data is analyzed to make a corresponding warning notification in the case of data anomaly, thereby improving the accuracy of data processing.

[0089] As shown in Figure 2 , it is a flowchart of another large model-based data processing method provided by the embodiment of the present application, which includes the following steps.

[0090] S201: Based on the semantics of each table header name, and combined with a preset semantic hierarchy table, the hierarchy of each table header name is determined.

[0091] The semantic hierarchy table includes a plurality of hierarchies and corresponding semantics.

[0092] S202: In order from high to low, each table header name is sequentially sorted to obtain a table header name sequence.

[0093] Wherein, each table header information in the table header area is sorted and combined according to the hierarchical relationship to form a table header name sequence K = {K1, K2, … KN}.

[0094] S203: Adding a corresponding table header position to the table header name in the table header name sequence to obtain a corresponding to-be-tested data set.

[0095] Wherein, if there is a merged cell in the table header name, the content of the merged cell is spliced with a designated symbol to a lower level to form a to-be-tested data set K = {K1: COL1, K2: COL2, … KN: COLN}.

[0096] In some examples, after it is calculated according to the data segmentation model that the table header area is the first row and the data area is the second row, the table header area is processed, if there is a merged column header, the column header is merged according to the level, and finally the relationship between the column header and the column name is K = {company code: A, period: B, revenue: C, …}, and the data is {{A: “A0001”, B: “2025Q1”, C: “500000”, …}}.

[0097] The above-mentioned processes S201-S203 can generate a corresponding to-be-tested data set according to each table header information in the table header area, so as to participate in the subsequent data matching process.

[0098] As shown in Figure 3 , another process schematic diagram of the data processing method based on a large model provided by the embodiment of the present application is shown, which includes the following steps.

[0099] S301: After deploying the large model locally, the large model is pre-trained using the SQuAD corpus.

[0100] Wherein, the large model is pre-trained using the SQuAD corpus, so that the pre-trained large model has basic data analysis and reasoning capabilities.

[0101] S302: Using a specified Chinese word segmentation system, the pre-acquired professional term information related to the financial field is segmented to obtain a vector set.

[0102] Wherein, the vector set includes keyword vectors corresponding to multiple professional terms.

[0103] In some examples, the specified Chinese word segmentation system can be the SharpICTCLAS word segmentation system.

[0104] It should be noted that the vector set can be identified as T i = {w1, w2, …, w n}, w1, w2, …, w n are corresponding keyword vectors respectively.

[0105] In some examples, the professional term information is split by the Sharp ICT CLAS word segmentation system to generate a vector set, such as the term balance sheet-current assets-partner funds, which can be split into {“assets”, “liabilities”, “sheet”, “current”, “currency”, “funds”}, and the vectorized result is T i = {w1(assets), w2(liabilities),..., w n (funds)}.

[0106] S303: Calculate the similarity between each keyword vector in the vector set.

[0107] Where W i and W j corresponding keywords constitute the keyword vectors of W i and W j , and then consider shorter keyword vectors, assuming Len(W i ) <= Len(W j ), calculate the initial weight value vector W i = {b1, b2, … bn} of W Bi , W i and W j represent any two keyword vectors, and Len represents the length of the corresponding keyword vector. For each keyword mi in W i , the following processing is performed in turn: if mi exists in W j , consider the previous keywords of mi in W i and W j , if the previous keywords are the same or synonyms, then increase the weight of mi in W Bi by σ times, and the same processing is performed for the keywords after mi, if mi does not exist in W j , then the corresponding weight of mi in W Bi is unchanged, then calculate W i based on the existence vector E j of W i,j , for each keyword in E i,j , take the weight value of the corresponding keyword from W Bi to constitute the existence value vector TE i,j = {v1, v2, …, v p}, and finally calculate the similarity Sim(T i , T j ) between the two keyword vectors, the specific calculation formula is shown in formula (1).

[0108] (1)

[0109] In formula (1), W Bi The value of the i-th term in the middle, TE i,j The value of the i-th term in the middle, and σ is set to 1.3, and the calculation result of Sim(T i , T j ) is greater than or equal to 0.9, it is determined that W i and W j are completely matched, the calculation result is in the range of (0.7, 0.9), it is determined that W i and W j are fuzzy matched, and the calculation result is less than or equal to 0.7, it is determined that W i and W j are not matched.

[0110] S304: Process each keyword vector and the similarity between each keyword vector and other keyword vectors according to the corpus format of the SQuAD corpus to obtain the corresponding professional corpus.

[0111] Among them, processing keyword vectors and corresponding similarities according to the specified corpus format can ensure that all keyword vectors maintain a unified format and that the professional corpus can be recognized by the pre-trained large model.

[0112] S305: Train the pre-trained large model using the professional corpus to obtain a trained large model.

[0113] Among them, training the pre-trained large model using the professional corpus to obtain a trained large model can effectively improve the data processing and analysis capabilities of the large model in the financial field.

[0114] The above-mentioned processes S301-S305 use the SQuAD corpus and the professional corpus to train the large model, thereby effectively improving the data processing and analysis capabilities of the large model in the financial field.

[0115] As shown in Figure 4 , it is an architecture schematic diagram of a data processing device based on a large model provided by an embodiment of the present application, which includes the following units.

[0116] The information determination unit 100 is configured to obtain corresponding structured information based on the terminal object uploaded to-be-processed data; the to-be-processed data includes a data table participating in financial data integration; the structured information includes a table header area and a data area; the table header area includes a plurality of table header information, the table header information includes a table header name and a corresponding table header position, and the data area includes a plurality of table header positions and corresponding data.

[0117] Optionally, the information determining unit 100 is specifically configured to: use a data segmentation model to perform data segmentation on the to-be-processed data uploaded by the terminal object to obtain structured information corresponding to the to-be-processed data; and the data segmentation model is a data segmentation model for ,represents an upper limit of the number of table header information pre-labeled, represents a table header name of the last row data in the data table, represents the number of each row in the data table, is a function for judging whether the row data in the range belonging to the column in the data table is empty, is a function for determining the number of the smallest row with continuous data that is not empty in the data table, is a function for determining the number of the largest row with a merged cell in the data table.

[0118] The data set determining unit 200 is configured to generate a corresponding to-be-tested data set based on each table header name and a corresponding table header position.

[0119] Optionally, the data set determining unit 200 is specifically configured to: determine the level of each table header name based on the semantics of each table header name and in combination with a preset semantic level relationship table; the semantic level relationship table includes multiple levels and corresponding semantics; sort each table header name in order from high to low according to the level to obtain a table header name sequence; and add a corresponding table header position to the table header name in the table header name sequence to obtain the corresponding to-be-tested data set.

[0120] The model calling unit 300 is configured to call a pre-trained large model, match the table header information in the to-be-tested data set with standard table header information in a template data set to obtain a corresponding matching result; and the matching result includes a matching relationship between the table header information and the standard table header information.

[0121] Optionally, the model calling unit 300 is specifically configured to: after deploying the large model locally, pre-train the large model using a SQuAD corpus; perform word segmentation on pre-acquired professional term information related to the financial field using a designated Chinese word segmentation system to obtain a vector set; the vector set includes keyword vectors corresponding to multiple professional terms; calculate the similarity between each keyword vector in the vector set; process each keyword vector and the similarity between each keyword vector and other keyword vectors according to the corpus format of the SQuAD corpus to obtain corresponding professional corpus; and train the pre-trained large model using the professional corpus to obtain a trained large model.

[0122] The data filling unit 400 is configured to fill data corresponding to each table header information into a target primary key of the target data table according to the matching relationship; the target primary key is a primary key corresponding to the terminal object.

[0123] Optionally, the data filling unit 400 is further configured to send unmatching information to the terminal object to enable the terminal object to supplement corresponding target data, if the matching result further includes the unmatching information; the unmatching information includes unmatching standard table header information; and the target data includes table header information and data corresponding to the unmatching standard table header information.

[0124] Optionally, the data filling unit 400 is further configured to, if the matching relationship is saved as a target rule and the target rule is not modified, verify table header information in the to-be-tested data set according to the target rule to obtain a corresponding verification result; and fill data corresponding to each table header information into the target primary key when the verification result is qualified.

[0125] The index determining unit 500 is configured to determine a data deviation index of the to-be-processed data based on the filled data of all primary keys in the target data table.

[0126] Optionally, the index determining unit 500 is specifically configured to take the filled data of the target primary key and the other primary keys in the target data table as inputs of a data dispersion model to obtain a dispersion result output by the data dispersion model; the data dispersion model includes wherein represents the dispersion result, represents a weight for a same case of any primary key in the target data table, represents a weight of all data in the target data table, and , represents filled data of any column in the target primary key, represents an average value of filled data of all columns in the same primary key, represents an average value of filled data of all columns in the target data table; the dispersion result includes a dispersion of the filled data of the target primary key; and the data deviation index of the to-be-processed data is determined based on the dispersion result.

[0127] The early warning notification unit 600 is configured to send an early warning notification to the terminal object when the data deviation index does not meet a requirement; the early warning notification is used to prompt that the to-be-processed data has an error.

[0128] The above-mentioned various units realize matching of header information in terminal object to-be-processed data and standard header information in template data set by using a large model, fill data corresponding to the header information into a target data table based on the matching relationship output by the large model, complete effective arrangement of the to-be-processed data, improve the processing efficiency of multi-source data integration, and analyze a data deviation index of the to-be-processed data based on the filled data of all primary keys in the target data table, so as to make corresponding early warning notification in the case of data exception, thereby improving the accuracy of data processing.

[0129] The application further provides a computer-readable storage medium, which comprises a stored program, wherein the program executes the large model-based data processing method provided in the application.

[0130] The application further provides an electronic device, which comprises a processor, a memory and a bus. The processor is connected with the memory through the bus, the memory is used for storing a program, and the processor is used for running the program, wherein the program executes the large model-based data processing method provided in the application when running.

[0131] Although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the application. Some features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0132] The above description is only the preferred embodiment of the application and the explanation of the applied technical principles. Those skilled in the art should understand that the disclosed range of the application is not limited to the technical solution formed by the specific combination of the above technical features, and also covers other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the disclosed concept. For example, the above features are replaced with the technical features disclosed in the application (but not limited to) having similar functions to form a technical solution.

Claims

1. A data processing method based on a large model, characterized in that, include: Based on the data to be processed uploaded by the terminal object, obtain the corresponding structured information; The data to be processed includes data tables involved in the integration of financial data; The structured information includes a header area and a data area; the header area includes multiple header information, each header information including a header name and a corresponding header position; the data area includes multiple header positions and their corresponding data. Based on each header name and its corresponding header position, a corresponding dataset to be tested is generated; A pre-trained large model is invoked to match the header information in the dataset to be tested with the standard header information in the template dataset to obtain the corresponding matching results; the matching results include the matching relationship between the header information and the standard header information; Based on the matching relationship, the data corresponding to each of the header information is filled into the target primary key of the target data table; the target primary key is the primary key corresponding to the terminal object. Based on the populated data of all primary keys in the target data table, determine the data deviation index of the data to be processed; When the data deviation index does not meet the requirements, a warning notification is sent to the terminal object; the warning notification is used to indicate that there is an error in the data to be processed.

2. The method according to claim 1, characterized in that, Based on the data to be processed uploaded by the terminal object, obtain the corresponding structured information, including: A data segmentation model is used to segment the data to be processed uploaded by the terminal object to obtain the structured information corresponding to the data to be processed; wherein, the data segmentation model is... , This represents the predefined maximum number of header information entries. This represents the header name of the last non-empty row in the data table. This represents the number of each row in the data table. To determine the category of the data table A function to check if a range of rows in a column is empty. This is a function used to determine the number of the smallest consecutive non-empty row in the data table. This is the number used to determine the largest row in the data table that contains merged cells.

3. The method according to claim 1, characterized in that, Based on each header name and its corresponding header position, a corresponding dataset to be tested is generated, including: Based on the semantics of each header name and in conjunction with a preset semantic hierarchy table, the hierarchy of each header name is determined; the semantic hierarchy table includes multiple levels and their corresponding semantics. Sort each header name in descending order of hierarchy to obtain a header name sequence; Add corresponding header positions to the header names in the header name sequence to obtain the corresponding dataset to be tested.

4. The method according to claim 1, characterized in that, The training process of the large model includes: After deploying the large model locally, the large model is pre-trained using the SQuAD corpus; Using a specified Chinese word segmentation system, the pre-acquired financial field-related professional terminology information is segmented to obtain a vector set; the vector set includes keyword vectors corresponding to multiple professional terms; Calculate the similarity between the keyword vectors in the vector set; According to the corpus format of the SQuAD corpus, each keyword vector and the similarity between each keyword vector and other keyword vectors are processed to obtain the corresponding professional corpus; Using the aforementioned specialized corpus, the pre-trained large model is trained to obtain a well-trained large model.

5. The method according to claim 1, characterized in that, After calling a pre-trained large model to match the header information in the test dataset with the standard header information in the template dataset to obtain the corresponding matching results, the method further includes: If the matching result also includes unmatched information, the unmatched information is sent to the terminal object so that the terminal object can supplement the corresponding target data; the unmatched information includes unmatched standard header information; the target data includes header information and data corresponding to the unmatched standard header information.

6. The method according to claim 1, characterized in that, Based on the populated data of other primary keys in the target data table, determine the data deviation index of the data to be processed, including: The data, based on the target primary key and other primary keys in the target data table, are used as input to the data discretization model to obtain the dispersion result output by the data discretization model; the data discretization model includes... ,in This represents the dispersion result. This represents the weight given that any primary key in the target data table is the same. The weights represent the weights of all data in the target data table, and , This represents the fill data for any column in the target primary key. This represents the average of the populated data across all columns within the same primary key. The average value of the fill data in all columns of the target data table; the dispersion result includes the dispersion of the fill data of the target primary key; Based on the discrete results, a data deviation index is determined for the data to be processed.

7. The method according to claim 1, characterized in that, The method further includes: If the matching relationship is saved as a target rule and the target rule is not modified, the header information in the dataset to be tested is verified according to the target rule to obtain the corresponding verification result. When the verification result is qualified, the data corresponding to each of the header information is filled into the target primary key.

8. A data processing device based on a large model, characterized in that, include: The information determination unit is used to obtain the corresponding structured information based on the data to be processed uploaded by the terminal object; The data to be processed includes data tables involved in the integration of financial data; The structured information includes a header area and a data area; the header area includes multiple header information, each header information including a header name and a corresponding header position; the data area includes multiple header positions and their corresponding data. The dataset determination unit is used to generate a corresponding test dataset based on each of the header names and the corresponding header positions; The model invocation unit is used to invoke a pre-trained large model to match the header information in the dataset to be tested with the standard header information in the template dataset to obtain the corresponding matching results; the matching results include the matching relationship between the header information and the standard header information. A data filling unit is used to fill the data corresponding to each of the header information into the target primary key of the target data table according to the matching relationship; The target primary key is the primary key corresponding to the terminal object; The indicator determination unit is used to determine the data deviation indicator of the data to be processed based on the fill data of all primary keys in the target data table. The early warning notification unit is used to send an early warning notification to the terminal object when the data deviation index does not meet the requirements; the early warning notification is used to indicate that there is an error in the data to be processed.

9. A storage medium, characterized in that, The storage medium includes a stored program, wherein the program is executed by a processor to perform the data processing method based on any one of claims 1-7.

10. An electronic device, characterized in that, include: Processor, memory, and bus; The processor and the memory are connected via the bus; The memory is used to store a program, and the processor is used to run the program, wherein the program is executed by the processor to perform the data processing method based on any one of claims 1-7.

Images