Data processing method, apparatus and electronic device
By automating the identification of source tables and field values associated with target null values in financial institutions' data reports, the problem of low efficiency in filling missing fields in data reports is solved, achieving efficient data processing and quality assurance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- AGRICULTURAL BANK OF CHINA
- Filing Date
- 2022-09-28
- Publication Date
- 2026-06-02
AI Technical Summary
Financial institutions are inefficient at filling in missing fields in data reports, relying mainly on manual communication and queries, which leads to inefficiency and waste of resources.
By obtaining the target null value from the form to be processed, and using multiple target source tables in the database that are related to it, the system automatically determines the target field value that has the same related information as the target null value and fills it into the target null value, thus realizing an automated data filling process.
It improved the data quality and processing efficiency of data reports, realized automated data retrieval, querying and population, reduced the input of human and material resources, and met the quality requirements of regulatory agencies.
Smart Images
Figure CN115470861B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus and electronic device. Background Technology
[0002] With the development of the financial industry, the China Banking and Insurance Regulatory Commission (CBIRC) has increased the intensity of on-site inspections of financial institutions, such as banks. It requires financial institutions to implement data standardization and submit data reports according to stipulated timelines and requirements, thereby achieving unified data quality monitoring. For financial institutions, due to the large number and high dimensionality of the data reports to be submitted, and the short processing time, missing data report fields frequently occur.
[0003] Currently, filling missing fields in data reports mainly relies on extensive communication between personnel and manual queries by different departments to fill in the missing fields. This method results in low efficiency in filling missing fields in data reports. Summary of the Invention
[0004] This application provides a data processing method, apparatus, and electronic device to address the problem of low filling efficiency for missing fields in current data reports.
[0005] The first aspect of this application provides a data processing method, comprising: obtaining a form to be processed, the form having target null values; in a database, determining multiple target source tables that match the form to be processed, wherein a first table name of the form to be processed is associated with a second table name of the target source tables; in the multiple target source tables, determining target field values that have the same association information as the target null values; and filling the target null values in the form to be processed with the target field values to obtain the target form.
[0006] In one embodiment of this application, determining multiple target source tables that match the form to be processed includes: determining a table name whose string similarity to a first table name is greater than a similarity threshold, which is designated as a second table name; and determining a source table with the second table name as a target source table.
[0007] In one embodiment of this application, before determining the source table with the second table name as the target source table in the absence of a table name whose string similarity to the first table name is greater than a similarity threshold, the method further includes: obtaining the target label of the first table name; in the database, determining multiple source tables whose table names belong to the target label; and among the table names of the multiple source tables, determining the table name whose Hamming distance to the first table name is less than a preset distance threshold as the second table name.
[0008] In one embodiment of this application, before determining the source table with the second table name as the target source table in the absence of a table name whose Hamming distance to the first table name is less than a preset distance threshold, the method further includes: inputting the table name of the source table and the first table name in the database into a pre-trained natural language processing model for text processing to obtain the probability value corresponding to the table name of the source table, wherein the similarity between the first table name and the table name is positively correlated with the probability value; and determining the table name with the probability value greater than the probability threshold as the second table name.
[0009] In one embodiment of this application, the method further includes: determining the third table name of the target source table to which the target field value belongs; and using the third table name and the first table name as positive example samples to optimize the training of the natural language processing model.
[0010] In one embodiment of this application, the associated information includes: field name and related information. In multiple target source tables, determining the target field value that has the same associated information as the target null value includes: in the form to be processed, determining the field name and related information of the target null value; in multiple target source tables, determining the field value corresponding to the field name; and when there are multiple field values, determining the field value with related information among the multiple field values as the target field value.
[0011] In one embodiment of this application, after filling the target null value in the form to be processed with the target field value to obtain the target form, the method further includes: validating the format of the target form and the strings in the target form to obtain the valid target form.
[0012] In one embodiment of this application, the target field value is used to fill the target null value in the form to be processed. After obtaining the target form, the method further includes sending the target form to a regulatory agency for supervision.
[0013] A second aspect of this application provides a data processing apparatus, comprising:
[0014] The acquisition module is used to acquire the form to be processed, which contains the target null value.
[0015] The first determination module is used to determine multiple target source tables in the database that match the form to be processed, and the first table name of the form to be processed is related to the second table name of the target source table;
[0016] The second determination module is used to determine the target field value that has the same association information as the target null value in multiple target source tables;
[0017] The fill module is used to fill the target empty values in the form to be processed with the target field values to obtain the target form.
[0018] In one embodiment of this application, the first determining module is specifically used to: determine a table name whose string similarity to the first table name is greater than a similarity threshold, which is the second table name; and determine the source table with the second table name as the target source table.
[0019] In one embodiment of this application, if no table name with a string similarity greater than a similarity threshold to the first table name is determined, the first determining module is further configured to: obtain the target label of the first table name; determine multiple source tables in the database whose table names belong to the target label; and determine, among the table names of the multiple source tables, the table name whose Hamming distance to the first table name is less than a preset distance threshold as the second table name.
[0020] In one embodiment of this application, if no table name with a Hamming distance less than a preset distance threshold is determined to be the first table name, the first determining module is further configured to: input the table name of the source table in the database and the first table name into a pre-trained natural language processing model for text processing to obtain the probability value corresponding to the table name of the source table, wherein the similarity between the first table name and the table name is positively correlated with the probability value; and determine the table name with a probability value greater than the probability threshold as the second table name.
[0021] In one embodiment of this application, it further includes an optimization training module, used to determine the third table name of the target source table to which the target field value belongs; and to optimize the training of the natural language processing model using the third table name and the first table name as positive example samples.
[0022] In one embodiment of this application, the associated information includes: field name and related information. The second determining module is specifically used to: determine the field name and related information of the target null value in the form to be processed; determine the field value corresponding to the field name in multiple target source tables; and determine the field value with related information as the target field value when there are multiple field values.
[0023] In one embodiment of this application, it further includes: a verification module, used to verify the format of the target form and the strings in the target form after filling the target null value in the form to be processed with the target field value to obtain the target form, so as to obtain the target form that passes the verification.
[0024] In one embodiment of this application, it further includes: a sending module for sending the target form to a regulatory agency for oversight.
[0025] A third aspect of this application provides an electronic device, including a memory and a processor; wherein,
[0026] The memory is used to store program code;
[0027] The processor is used to call program code to implement any of the above data processing methods.
[0028] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it causes an electronic device to perform any of the data processing methods described above.
[0029] The fifth aspect of this application provides a computer program product, comprising: a computer program stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, and the at least one processor executing the computer program causing the electronic device to perform the data processing method of any of the first aspects.
[0030] As can be seen from the above technical solution, the embodiments of this application obtain a form to be processed, which contains target null values; in the database, multiple target source tables that match the form to be processed are determined, and the first table name of the form to be processed is related to the second table name of the target source tables; in the multiple target source tables, target field values with the same related information as the target null values are determined; the target field values are used to fill the target null values in the form to be processed, thus obtaining the target form. This application accurately and quickly fills the null values in the form to be processed in an automated manner, obtaining a target form that meets regulatory quality requirements. Attached Figure Description
[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0032] Figure 1 This is a schematic diagram illustrating an application scenario of the data processing method provided in this application;
[0033] Figure 2 This is a flowchart of the steps of a data processing method provided in an embodiment of this application;
[0034] Figure 3 This is a flowchart of the steps of a data processing method provided in another embodiment of this application;
[0035] Figure 4 This is a schematic diagram of a target form provided in an embodiment of this application;
[0036] Figure 5 This is a structural block diagram of a data processing apparatus provided in an embodiment of this application;
[0037] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0038] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0039] For data reports with missing fields, existing technologies typically use database join statements to query the source table after the missing field is identified. However, for bank data reports, especially those involving specific business needs, table relationships are complex, making join statements inefficient. Because big data platforms or data middleware systems contain different data sources, query fields in joins may have inconsistent naming conventions, different naming rules, or lack of correlation for some important information. Therefore, simply using simple joins is inefficient and immature, and there's a risk of missing fields not being found, requiring consultation with relevant business departments. Furthermore, the retrieved source tables and fields require manual formatting and filling, which cannot be automated for integrated review and submission, resulting in wasted manpower and resources and low efficiency.
[0040] To address the aforementioned issues, this application provides a data processing method. This method involves obtaining a form to be processed, which contains target null values. In a database, multiple target source tables matching the form to be processed are identified, where the first table name of the form to be processed is related to the second table name of the target source tables. Within these target source tables, target field values with the same association information as the target null values are determined. The target field values are then used to fill the target null values in the form to be processed, resulting in the target form. This method automates the retrieval, querying, organization, and filling steps for the form to be processed, facilitating users' understanding and processing of bank data reports, quickly identifying and automating data quality issues in data reports, and improving the data quality and efficiency of data reports.
[0041] For example, Figure 1 This is a schematic diagram illustrating an application scenario of the data processing method provided in this application. For example... Figure 1As shown, financial institutions need to submit data reporting forms to regulatory agencies. After obtaining the form to be processed 11, the financial institution needs to review the form to be processed 11 and determine that there are target empty values in the form to be processed 11. Then, it fills these target empty values according to the data in the database to obtain the target form 12. The target form is then sent to the regulatory agency as a data reporting form for supervision.
[0042] The technical solution of this application will now be described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.
[0043] Reference Figure 2 The flowchart below shows the steps of the data processing method provided in this application embodiment, which specifically includes the following steps:
[0044] S201, Get the form to be processed.
[0045] The form to be processed contains target null values. In this embodiment, the form to be processed has a first table name, such as... Figure 1 The first table in Form 11 to be processed is named "Customer Deposit Information Table". Furthermore, the content in the form to be processed is stored in a key-value pair format. Here, "key" corresponds to the field name, and "value" corresponds to the field value.
[0046] Furthermore, for the forms to be processed, a Python or shell script is first used to search for missing field values. If no field values are missing, a format review is performed, and the format review report includes information such as character type and character size. Forms that pass the format review are then sent to the regulatory agency as data submission forms. If field values are missing, subsequent steps are performed. The location of the missing field value in the form to be processed is designated as the target null value.
[0047] For example, refer to Figure 1 In Form 11 to be processed: the target null value is the missing field value corresponding to "Name" in the first row; the missing field value corresponding to "ID Number" in the second row; the missing field value corresponding to "Phone Number" in the fourth row; in addition, "Name", "ID Number", "Phone Number" and "Deposit" are field names, and these field names correspond to field values.
[0048] In this embodiment, a Python script (a computer programming language), such as the `isnan()` method in the NumPy library, can be used to perform key-value pair checks and record target null values, or the form to be processed can be treated as a dictionary data structure. Alternatively, a Shell script can be used to determine the target null values in the form to be processed; this embodiment does not limit the method for determining the target null values.
[0049] S202, in the database, identify multiple target source tables that match the form to be processed.
[0050] In this embodiment, the first table name of the form to be processed is related to the second table name of the target source table. This relatedness may include: the first table name and the second table name being the same; the first table name and the second table name representing the same meaning; the first table name and the second table name having a certain similarity; or the first table name and the second table name having the same label.
[0051] In addition, the target source table is determined from multiple source tables based on whether there is a correlation between the first table name of the form to be processed and the table name of the source table in the database.
[0052] Furthermore, given the large number of source tables in the database, comparing the data content of each source table to determine if it contains the target field value to fill the target null value would be extremely labor-intensive and inefficient. However, this embodiment compares the first table name with the names of the source tables in the database to determine the target source table as the second table name associated with the first table name. Subsequent searching for the target null value in the target source table significantly improves the efficiency of filling the target field value.
[0053] S203, in multiple target source tables, determine the target field value that has the same association information as the target null value.
[0054] The relationship includes the corresponding field names and other relevant information. For example... Figure 1 In this context, if the target null value is the field value corresponding to "Name", then the field name corresponding to the target null value is "Name". Other relevant information includes at least one of "ID Number: 0000 0000 0000 00" and "Phone Number: 1340000000". This relevant information can be unique.
[0055] In this embodiment of the application, the target source table can be searched for fields that have the same association information as the target null value and whose field names correspond to the target field values.
[0056] S204, fill the target empty value in the form to be processed with the target field value to obtain the target form.
[0057] Reference Figure 1 Fill the target field values “Zhao Liu”, “1111 1111 1111 1111 1111” and “15800000000” into the corresponding target empty values to obtain target form 12.
[0058] In this embodiment, the format of the target form is also reviewed. Once the review is passed, the form can be sent to the regulatory authority for submission. If the review fails, the field values that fail the review can be used as target null values, and steps S203 to 204 can be executed.
[0059] In this embodiment, a form to be processed is obtained, which contains target null values. In a database, multiple target source tables matching the form to be processed are identified, with the first table name of the form to be processed and the second table name of the target source tables having a correlation. In the multiple target source tables, target field values with the same correlation information as the target null values are identified. The target field values are then used to fill the target null values in the form to be processed, resulting in a target form. This automated process filters target source tables based on table name correlation, determines target field values in the target source tables, and automatically and accurately fills the target null values in the form to be processed, thereby obtaining a target form that meets regulatory quality requirements.
[0060] Based on the above embodiments, this application provides another data processing method, as detailed below. Figure 3 Specifically, it includes the following steps:
[0061] S301, Get the form to be processed.
[0062] The specific implementation process of this step is described in S201 and will not be repeated here.
[0063] S302, determine the table name whose string similarity to the first table name is greater than the similarity threshold, and designate it as the second table name.
[0064] In querying a target field value, it is necessary to locate the target source table containing that value. Therefore, identifying the target source table is crucial for retrieving the target field value. This embodiment of the application searches the target source table sequentially from easiest to most difficult, which improves the efficiency of the search.
[0065] Furthermore, due to the interconnectedness of financial institution operations, the creation of source tables in the database is based on business commands. While source tables belonging to different databases may have different naming conventions, the data storage format is the same—key-value format—and the stored data content remains unchanged. For example, Zhao Liu's ID number and phone number should store the same information in different source tables. Therefore, for the same data content, the corresponding source table names have a certain degree of similarity; for example, a customer deposit information table and a customer loan information table have a certain degree of similarity.
[0066] In this embodiment, the KMP (a string matching algorithm) algorithm can be used to calculate the similarity between the first table name and the table names of each source table using string similarity. The table name with a similarity greater than a similarity threshold is used as the second table name. The similarity threshold can be preset.
[0067] S303, if no table name with a string similarity greater than the similarity threshold with the first table name is found, obtain the target label of the first table name.
[0068] Furthermore, if the above method fails to determine the second table name, then the logical relationships between the source tables from multiple perspectives are considered. For example, the target tags for the first table name are obtained. These target tags could include, for example, loan-related, customer information-related, wealth management product-related, corporate customer-related, or institution-related. In this embodiment, the table names of each source table can be pre-classified.
[0069] S304, in the database, identify multiple source tables whose table names belong to the target label.
[0070] Each tag contains multiple source tables, and multiple source tables are identified that correspond to the table names belonging to the same target tag as the first table name. This application determines the target source table from these multiple source tables.
[0071] S305, among the table names of multiple source tables, determine the table name whose Hamming distance to the first table name is less than a preset distance threshold, and designate it as the second table name.
[0072] The Hamming distance refers to the number of characters that need to be replaced to transform one string (the first table name) into another string (the other table name). Hamming distance is used in data transmission error control coding. It's a concept representing the number of corresponding bits that differ between two words (of the same length). We use d(x,y) to represent the Hamming distance between two words x and y. The Hamming distance is calculated by performing an XOR operation on the two strings and counting the number of 1s in the result.
[0073] In this embodiment of the application, the preset distance threshold is pre-set.
[0074] S306, if no table name is found whose Hamming distance to the first table name is less than a preset distance threshold, input the source table name and the first table name in the database into a pre-trained natural language processing model for text processing to obtain the probability value corresponding to the source table name.
[0075] In cases where both methods for querying the second table name fail, it indicates a significant difference between the table to be processed and the source table, making tracing the source difficult. Therefore, in this step, the first table name and the table names of each source table in the database are input into a Natural Language Processing (NLP) model for text processing. The model outputs a probability value for each source table name; the higher the probability value, the greater the similarity between the table name and the first table name. In other words, the similarity between the first table name and the source table names is positively correlated with the probability value.
[0076] In addition, the natural language processing model can output the table names of each source table in descending order of probability value.
[0077] In this embodiment, the natural language processing model is pre-trained. Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies various theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science integrating linguistics, computer science, and mathematics. Therefore, research in this field involves natural language, i.e., the language people use in daily life, and thus it is closely related to linguistics, but also has important differences. NLP does not simply study natural language in general, but rather focuses on developing computer systems, especially the software systems, that can effectively achieve natural language communication. Therefore, it is a part of computer science. NLP is mainly applied to machine translation, public opinion monitoring, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, speech recognition, and Chinese OCR.
[0078] Specifically, the natural language processing model includes an embedding layer, an interaction layer, and a presentation layer. The embedding layer calculates the vector representations of the first table name and the source table name by accumulating word vectors bit by bit. The interaction layer calculates the similarity between two vectors using cosine similarity, and the presentation layer normalizes the similarity and outputs a probability value.
[0079] S307, determine the table name for the probability value that is greater than the probability threshold as the second table name.
[0080] S308, determine the source table with the second table name as the target source table.
[0081] S309, in the form to be processed, determine the field name and related information of the target null value.
[0082] The associated information includes: field names and related information.
[0083] Reference Figure 1For the information in the first row of Form 11 to be processed, the target null value field name is "Name", and the relevant information is "ID Number: 0000 0000 0000 00" and / or "Phone Number: 13400000000". For the information in the second row, the target null value field name is "ID Number", and the relevant information is "Name: Zhang San" and / or "Phone Number: 15500000000".
[0084] S310, in multiple target source tables, determine the field value corresponding to the field name.
[0085] In this embodiment of the application, the field value corresponding to the field name is first determined, such as the field value corresponding to "name" and the field value corresponding to "ID number".
[0086] S311, when there are multiple field values, determine the field value with relevant information as the target field value among the multiple field values.
[0087] If a field value is determined to be singular, that field value is used as the target field value. If there are multiple field values, the field value with the same related information as the target null value is used as the target field value. For example, the target field value may also have the same ID number and phone number.
[0088] S312, determine the third table name of the target source table to which the target field value belongs.
[0089] In this embodiment, the target field value is searched in multiple target source tables, such as target source table A, target source table B, and target source table C. Once the target field value is determined in target source table C, the table name of target source table C is designated as the third table name.
[0090] S313 uses the third table name and the first table name as positive examples to optimize the training of the natural language processing model.
[0091] This application embodiment uses the third table name and the first table name as positive examples to optimize the training of the natural language processing model, which can improve the accuracy of the natural language processing model, realize the iterative optimization of the natural language processing model, and facilitate the use of the natural language processing model in the future.
[0092] S314, fill the target empty value in the form to be processed with the target field value to obtain the target form.
[0093] S315 validates the format of the target form and the strings in the target form, and obtains the target form that has passed the validation.
[0094] Reference Figure 4After the above steps, the form to be processed yields target form 41, which is an intermediate form. This intermediate form is then validated. The validation format includes the type of the validation string. For example, if the ID number in the third row contains the letters "XXXX", the format validation will fail because the ID number should be numeric. The validation string also includes its length. For instance, if the ID number in the fourth row of target form 41 is 12 digits, it does not meet the requirement of an 18-digit ID number and is therefore also invalid.
[0095] In this embodiment of the application, for unqualified target forms, the unqualified field values are extracted as target null values, and the above steps are continued until the target form has no null values and the validation passes.
[0096] S316, Send the target form to the regulatory authority for oversight.
[0097] The embodiments of this application can automatically retrieve target null values in the form to be processed, and then automatically query the target source table and the target field value corresponding to the target null value, and automatically fill it in. It can also automatically perform validation, thereby improving the quality of reports submitted to regulatory agencies and having high processing efficiency for the form to be processed.
[0098] First, this application is easy to implement, employing algorithms with high compatibility, high intelligence, and strong engineering practicality. It automates data retrieval, format review, and data filling in data reports, significantly reducing user complexity, time costs, and manpower and material resources, making it easier to submit and deploy in production applications. Furthermore, this invention is based on a lightweight algorithm framework written in Python, which is simple and easy to use. A wealth of third-party open-source components are available, and other programming languages such as Java, C, or R can also be used for the algorithm. It also offers portability across operating systems and platform environments. Finally, this invention boasts a high degree of automation and strong engineering practicality, showing promising application prospects for improving data quality in the field of regulatory reporting.
[0099] In summary, this application constructs an automated model for reviewing and filling out submitted data reports. It designs and implements an overall framework for field retrieval, field supplementation, field search, format review, and field entry. This framework, from data retrieval to the search algorithm and report retrieval algorithm for final report presentation, helps submitters to more conveniently and intuitively understand and investigate data quality issues, and automates the source table field search and entry, assisting submitters in reviewing data and processing information.
[0100] Reference Figure 5 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. The data processing apparatus 50 includes:
[0101] Module 501 is used to retrieve the form to be processed, which contains the target null value.
[0102] The first determining module 502 is used to determine multiple target source tables in the database that match the form to be processed, wherein the first table name of the form to be processed is associated with the second table name of the target source table;
[0103] The second determining module 503 is used to determine the target field value that has the same association information as the target null value in multiple target source tables;
[0104] The fill module 504 is used to fill the target empty values in the form to be processed with the target field values to obtain the target form.
[0105] In one embodiment of this application, the first determining module 502 is specifically used to: determine a table name whose string similarity to the first table name is greater than a similarity threshold, which is the second table name; and determine the source table with the second table name as the target source table.
[0106] In one embodiment of this application, if no table name with a string similarity greater than a similarity threshold to the first table name is determined, the first determining module 502 is further configured to: obtain the target label of the first table name; determine multiple source tables in the database whose table names belong to the target label; and determine, among the table names of the multiple source tables, the table name whose Hamming distance to the first table name is less than a preset distance threshold as the second table name.
[0107] In one embodiment of this application, if no table name with a Hamming distance less than a preset distance threshold is determined, the first determining module 502 is further configured to: input the table name of the source table in the database and the first table name into a pre-trained natural language processing model for text processing to obtain the probability value corresponding to the table name of the source table, wherein the similarity between the first table name and the table name is positively correlated with the probability value; and determine that the table name with a probability value greater than the probability threshold is the second table name.
[0108] In one embodiment of this application, it further includes an optimization training module (not shown) for determining the third table name of the target source table to which the target field value belongs; and using the third table name and the first table name as positive example samples to optimize the training of the natural language processing model.
[0109] In one embodiment of this application, the associated information includes: field name and related information. The second determining module 503 is specifically used to: determine the field name and related information of the target null value in the form to be processed; determine the field value corresponding to the field name in multiple target source tables; and determine the field value with related information as the target field value when there are multiple field values.
[0110] In one embodiment of this application, it further includes: a verification module (not shown), used to verify the format of the target form and the strings in the target form after filling the target null value in the form to be processed with the target field value to obtain the target form, so as to obtain the target form that passes the verification.
[0111] In one embodiment of this application, it further includes: a sending module (not shown) for sending the target form to a regulatory agency for oversight.
[0112] For details on the specific working content of each module of the data processing device, please refer to the embodiments of the data processing method described above, which will not be repeated here.
[0113] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, a processing module can be a separate processing element, or it can be integrated into a chip within the device. Alternatively, it can be stored as program code in the device's memory, and its functions can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.
[0114] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to implement a system-on-a-chip (SOC).
[0115] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).
[0116] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specifically, this electronic device can be a bank server or a blockchain platform. Figure 6As shown, the electronic device may include: a processor 61, a memory 62, a communication interface 63, and a system bus 64. The memory 62 and the communication interface 63 are connected to the processor 61 via the system bus 64 and communicate with each other. The memory 62 stores computer-executed instructions, and the communication interface 63 communicates with other devices. When the processor 61 executes the aforementioned computer-executed instructions, it implements the scheme described in the above embodiment.
[0117] Should Figure 6 The system bus mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.
[0118] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0119] Optionally, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the electronic device to perform the above-described actions. Figures 2 to 3 The method of the illustrated embodiment.
[0120] Optionally, embodiments of this application also provide a computer program product, comprising: a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the above-described... Figures 2 to 3 The method of the illustrated embodiment.
[0121] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates an "or" relationship between the preceding and following related objects; in formulas, the character " / " indicates a "division" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.
[0122] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. In the embodiments of this application, the order of the above-mentioned process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A data processing method, characterized in that, include: Obtain the form to be processed, which contains the target null value; In the database, multiple target source tables that match the form to be processed are identified, and the first table name of the form to be processed is associated with the second table name of the target source table; In the plurality of target source tables, determine the target field value that has the same association information as the target null value; The target empty value in the form to be processed is filled with the target field value to obtain the target form; The determination of multiple target source tables that match the form to be processed includes: The table name whose string similarity to the first table name is greater than a similarity threshold is determined as the second table name; The source table with the second table name is identified as the target source table; Before determining the source table with the second table name as the target source table, in the absence of a table name whose string similarity to the first table name is greater than a similarity threshold, the method further includes: Obtain the target label for the first table name; In the database, multiple source tables whose table names belong to the target tag are identified; Among the table names of the multiple source tables, the table name whose Hamming distance from the first table name is less than a preset distance threshold is determined to be the second table name.
2. The data processing method according to claim 1, characterized in that, Before determining the source table with the second table name as the target source table, in the absence of a table name whose Hamming distance to the first table name is less than a preset distance threshold, the method further includes: The table name of the source table in the database and the first table name are input into a pre-trained natural language processing model for text processing to obtain the probability value corresponding to the table name of the source table. The similarity between the first table name and the table name of the source table is positively correlated with the probability value. The table name of the source table whose probability value is greater than the probability threshold is determined to be the second table name.
3. The data processing method according to claim 2, characterized in that, Also includes: Determine the name of the third table in the target source table to which the target field value belongs; The natural language processing model is trained by using the third table name and the first table name as positive examples.
4. The data processing method according to any one of claims 1 to 3, characterized in that, The association information includes: field name and related information. Determining the target field value with the same association information as the target null value in the multiple target source tables includes: In the form to be processed, determine the field name and related information of the target null value; In the multiple target source tables, determine the field value corresponding to the field name; When there are multiple field values, the field value containing the relevant information is determined to be the target field value.
5. The data processing method according to any one of claims 1 to 3, characterized in that, After filling the target empty value in the form to be processed with the target field value to obtain the target form, the process further includes: The format of the target form and the strings in the target form are verified to obtain a target form that passes the verification.
6. The data processing method according to any one of claims 1 to 3, characterized in that, After filling the target empty value in the form to be processed with the target field value to obtain the target form, the process further includes: The target form is sent to the regulatory agency for oversight.
7. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire a form to be processed, wherein the form to be processed contains a target null value; The first determining module is used to determine multiple target source tables in the database that match the form to be processed, wherein the first table name of the form to be processed is associated with the second table name of the target source table; The second determining module is used to determine, in the plurality of target source tables, the target field value that has the same association information as the target null value; A fill module is used to fill the target empty value in the form to be processed with the target field value to obtain the target form; The first determining module is specifically used for: The table name whose string similarity to the first table name is greater than a similarity threshold is determined as the second table name; The source table with the second table name is identified as the target source table; In the absence of a table name whose string similarity to the first table name is greater than a similarity threshold, the first determining module, before determining the source table with the second table name as the target source table, is further specifically used for: Obtain the target label for the first table name; In the database, multiple source tables whose table names belong to the target tag are identified; Among the table names of the multiple source tables, the table name whose Hamming distance from the first table name is less than a preset distance threshold is determined to be the second table name.
8. An electronic device, characterized in that, Includes memory and processor; among which, The memory is used to store program code; The processor is used to call the program code to implement the data processing method as described in any one of claims 1 to 6.