Data table storage method and device and data storage system

Abstract

The invention provides a data table storage method and device and a data storage system.The method comprises the steps that related information of a data table is obtained, and the related information comprises one or more of the number of calling times, the urgency degree of service requirements, the proposing time of the service requirements and the data size; the comprehensive score is the score of the importance of the data table, the calling frequency and the comprehensive score are in positive correlation, the emergency degree of the service demand and the comprehensive score are in positive correlation, and the distance between the proposing time of the service demand and the current time and the comprehensive score are in negative correlation; the data volume and the comprehensive score are in negative correlation; sorting all the data tables according to the comprehensive scores to obtain sorted data tables; and storing all the sorted data tables into a database according to the sequence of the sorted data tables. According to the method, the problem that in the prior art, when data is stored in a database, due to the fact that the data size is large, the efficiency is poor in the whole database storage process is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data warehousing, in particular to a data table warehousing method, a data table warehousing device, a computer program product and a data warehousing system. BACKGROUND

[0002] In the era of big data, financial institutions are facing unprecedented challenges in data management and data analysis. Among them, temporary data extraction is a common and crucial task that serves various demand scenarios such as business analysis, management decision-making, information disclosure, auditing, judicial investigation, etc.

[0003] In high-frequency business scenarios, the rapid growth of data volume makes data recovery the most time-consuming part of the entire extraction process. Especially in the case of processing large-capacity data or recovering multiple time point data, the time-consuming of data recovery may reach several hours, which seriously affects the timeliness and business continuity of data extraction. Therefore, some financial institutions want to restore and store the library table in advance, i.e. so-called "normal warehousing". However, during data warehousing, due to the large amount of data, the overall warehousing process is inefficient. SUMMARY

[0004] The main purpose of the present application is to provide a data table warehousing method, a data table warehousing device, a computer program product and a data warehousing system to at least solve the problem that the overall warehousing process is inefficient due to the large amount of data during data warehousing in the prior art.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a data table warehousing method is provided, comprising: obtaining the related information of the data table, wherein the related information includes one or more of the number of calls, the urgency of business demand, the time of business demand and the data volume; calculating the comprehensive score of the data table according to the related information, wherein the comprehensive score is the score of the importance of the data table, the number of calls and the comprehensive score are positively correlated, the urgency of business demand and the comprehensive score are positively correlated, the distance between the time of business demand and the current time and the comprehensive score are negatively correlated, and the data volume and the comprehensive score are negatively correlated; sorting all the data tables according to the comprehensive score to obtain sorted data tables; and warehousing all the sorted data tables in the order of the sorted data tables.

[0006] Optionally, the comprehensive score of the data table is calculated according to the related information, comprising: evaluating the related information to obtain an evaluation score corresponding to each of the related information, wherein the calling frequency and the evaluation score are in a positive correlation relationship, the urgency of the business requirement and the evaluation score are in a positive correlation relationship, the distance between the proposing time of the business requirement and the current time and the evaluation score are in a negative correlation relationship, and the data volume and the evaluation score are in a negative correlation relationship; calculating a weighted average value of all the evaluation scores to obtain a pre-score; and determining the pre-score as the comprehensive score.

[0007] Optionally, the comprehensive score of the data table is calculated according to the related information, further comprising: calculating a difference value between a preset value and a damping coefficient to obtain a first calculation result, wherein the preset value is 1; calculating a quotient of the first calculation result and a total number of the data table to obtain a second calculation result; calculating a quotient of the pre-score and a data volume of an associated table of the data table to obtain a third calculation result; calculating a sum of the third calculation results of all associated tables of the data table to obtain a fourth calculation result; and calculating a sum of the second calculation result and the fourth calculation result to obtain the comprehensive score.

[0008] Optionally, before the related information of the data table is obtained, the method further comprises: extracting all entity information, wherein the entity information comprises one or more of the business requirement, a database system, the data table, and a field of the data table; extracting all relationship information, wherein the relationship information comprises one or more of a calling relationship between the business requirement and the database system, an ownership relationship between the database system and the data table, a calling relationship between the business requirement and the data table, an association relationship between the data tables, and an ownership relationship between the data table and the field; and constructing a knowledge graph by taking the entity information as nodes and the relationship information as edges, wherein the knowledge graph is used for data association analysis.

[0009] Optionally, the related information of the data table is obtained, comprising: extracting a number of times that all the business requirements in the knowledge graph connect the same data table; and determining the number of times as the calling frequency of the data table.

[0010] Optionally, all the sorted data tables are stored in a database in the order of the sorted data tables, comprising: storing the first M sorted data tables in the order, wherein M=1 / N, and N is a total number of the sorted data tables; and in the case that a new business requirement is proposed, storing the sorted data tables that have not been stored in the database in the order.

[0011] Optionally, after sorting all the data tables according to the comprehensive scores to obtain sorted data tables, the method further comprises: compressing all the sorted data tables according to the comprehensive scores of the sorted data tables to obtain compressed data tables, wherein the comprehensive score and the compression rate are in a negative correlation; and storing the compressed data tables in sequence.

[0012] According to another aspect of the present application, a data table storage device is provided, comprising: an acquisition unit configured to acquire relevant information of data tables, wherein the relevant information comprises one or more of a call frequency, an urgency of a service requirement, a time of proposing the service requirement, and a data volume; a calculation unit configured to calculate a comprehensive score of the data tables according to the relevant information, wherein the comprehensive score is a score of importance of the data tables, the call frequency and the comprehensive score are in a positive correlation, the urgency of the service requirement and the comprehensive score are in a positive correlation, a distance between the time of proposing the service requirement and a current time and the comprehensive score are in a negative correlation, and the data volume and the comprehensive score are in a negative correlation; a sorting unit configured to sort all the data tables according to the comprehensive scores to obtain sorted data tables; and a first storage unit configured to store all the sorted data tables in sequence.

[0013] According to still another aspect of the present application, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of any of the data table storage methods.

[0014] According to yet another aspect of the present application, a data storage system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs comprise instructions for implementing any of the data table storage methods.

[0015] By applying the technical solution of the present application, the call frequency, the urgency of the service requirement, the distance between the time of proposing the service requirement and the current time, and the data volume are introduced to comprehensively score the data tables and sort the data tables according to the comprehensive scores, which, compared with the direct data storage in the prior art, comprehensively considers multiple parameters of the data tables, is equivalent to comprehensively sorting the data tables according to priorities, and thus improves the efficiency of data storage. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application and are incorporated herein for illustrative purposes. The illustrative embodiments of the present application and their description serve to explain the present application. In the drawings:

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for performing a data table warehousing Z method is shown according to an embodiment of the present application;

[0018] Figure 2 A flowchart of a data table warehousing method is shown according to an embodiment of the present application;

[0019] Figure 3 A temporary data extraction flowchart based on a knowledge graph is shown;

[0020] Figure 4 A structure block diagram of a data table warehousing device is shown according to an embodiment of the present application.

[0021] Among the above drawings, the following reference signs are included:

[0022] 102, processor; 104, memory; 106, transmission device; 108, input and output device. DETAILED DESCRIPTION

[0023] It should be noted that the embodiments and features in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units need not be limited to those clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] For the convenience of description, the following describes some nouns or terms related to the embodiments of the present application:

[0027] Temporary data extraction: Generally includes processes such as demand application, application analysis, and data extraction implementation. Among them, the data extraction implementation process requires operators to restore data at specified time points according to the SQL scripts obtained through application analysis, and then extract data.

[0028] Knowledge graph: A knowledge graph is a knowledge base represented by a graph structure. Its core elements include:

[0029] 1) Entity: Represents a specific thing or abstract concept, and is the basic element (i.e. node) of the knowledge graph;

[0030] 2) Relation: Describes the connection between entities (i.e. edges) that interact or are related;

[0031] 3) Attribute: Characteristic information of an entity, represented in the form of key-value pairs.

[0032] It integrates multi-dimensional data, links entities, concepts, and relationships using semantic information and context, providing prior knowledge support for intelligent search, deep question answering, decision reasoning, and other scenarios. In addition, knowledge graphs also support reasoning (deriving new knowledge from known information) and dynamic updating (real-time or periodic adjustment of data) to enhance the logicality and accuracy of decision-making.

[0033] Temporary data extraction refers to a one-time data extraction task implemented temporarily due to the inability of existing application systems to meet business operation, management decision-making, information disclosure, external supervision, external audit, judicial inquiry, and other needs. Temporary data extraction usually includes processes such as demand application, application analysis, and data extraction implementation. Among them, the data extraction implementation process requires operators to restore data at specified time points according to the SQL scripts obtained through application analysis, and then extract data.

[0034] The main pain points faced by current temporary data extraction scenarios are concentrated in time efficiency: for tables with large data volume or multiple time points to be restored, data restoration after demand arrival often consumes a lot of time, affecting business efficiency; although some methods such as normalizing data into the database are used to store tables in advance, the selection of normalizing data into the database often depends on past experience, lacks accurate quantitative evaluation methods, and may also cause storage performance bottlenecks, further prolonging the processing cycle.

[0035] Based on the above problems, the present scheme proposes a temporary data extraction optimization method based on knowledge graph. First, through the analysis of past demand and extraction script, the "business demand", "data source system", "database table", "data table field" and other entity / entity relationship and entity attributes are analyzed; on this basis, a knowledge graph analysis method based on graph algorithm is proposed, which can help the operator to locate the data table with higher importance, so as to prepare the data in advance through the normal storage and other ways, and improve the extraction efficiency.

[0036] As introduced in the background, in the prior art, when data is stored, the efficiency is poor due to the large amount of data. To solve the above problems, the embodiments of the present application provide a data table storage method, a data table storage device, a computer program product and a data storage system.

[0037] The core requirement of the current temporary data extraction scene is to quickly respond to the sudden and non-standardized data extraction requirements in business operation, management decision, external supervision, audit and other scenes. When dealing with temporary data extraction business scenes, operators need to complete a series of processes such as business analysis, script writing and database recovery to complete data extraction. However, the existing process has the pain point of low time efficiency.

[0038] In view of the problem of low time efficiency, the current operator will improve efficiency by normalizing the storage of the table. However, the selection of the table for normal storage currently depends on the operator's past experience, and lacks an accurate quantitative evaluation method; at the same time, business requirements and data access patterns may change over time, and experience-based selection may not be able to adapt to these changes in time, and may not be able to accurately identify the table that causes performance bottlenecks, resulting in no obvious optimization effect; at the same time, some rarely used or unimportant tables may also be included in the normal storage, occupying additional storage space.

[0039] Therefore, the present scheme proposes an analysis method based on knowledge graph, which can dynamically and quantitatively analyze the more important tables according to the demand, and improve the data extraction efficiency.

[0040] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application.

[0041] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking the running on a mobile terminal as an example, Figure 1 is a hardware structure block diagram of a mobile terminal of a data table storage method of the embodiments of the present application. As Figure 1 shown, the mobile terminal can include one or more (Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0042] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data table entry method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one instance, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0043] This embodiment provides a method for storing data tables that run on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0044] Figure 2 This is a flowchart illustrating the data table insertion method according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0045] In step S201, relevant information of the data table is obtained, wherein the relevant information includes one or more of the number of invocations, the urgency of the business requirement, the time of the business requirement, and the data volume.

[0046] Specifically, various data related to the data table are collected and analyzed, including but not limited to the number of invocations of the data table in the past period of time, the urgency of the recent business requirement, the distance between the time of the business requirement and the current time, and the average data volume of the data table. These information can be obtained through log analysis, business system interface call statistics, database performance monitoring, and other ways.

[0047] In the above embodiment, it is helpful to identify key data tables and provide a basis for subsequent comprehensive score calculation. By quantifying the number of invocations, the urgency, the distance between the time of the business requirement and the current time, and the data volume, a preliminary judgment of the importance of the data table is realized. This step improves the accuracy and responsiveness of data management.

[0048] In step S202, a comprehensive score of the data table is calculated according to the relevant information, wherein the comprehensive score is a score of the importance of the data table, the number of invocations and the comprehensive score are positively correlated, the urgency of the business requirement and the comprehensive score are positively correlated, the distance between the time of the business requirement and the current time and the comprehensive score are negatively correlated, and the data volume and the comprehensive score are negatively correlated.

[0049] Specifically, the calculation of the comprehensive score adopts a complex quantitative evaluation system, which includes weighted processing of the number of invocations, the urgency of the business requirement, the distance between the time of the business requirement and the current time, and the data volume. Specifically, the more the number of invocations, the more urgent the business requirement, the closer the time of the business requirement, and the smaller the data volume, the higher the comprehensive score of the corresponding data table.

[0050] In the above embodiment, the calculation of the comprehensive score ensures that those data tables that are crucial to the business can be identified and processed first. By combining the number of invocations, the urgency, the distance between the time of the business requirement and the current time, and the data volume, the system can automatically identify the relative importance of the data table, thereby guiding the subsequent sorting and storage operation, and improving the efficiency and response speed of the overall data management.

[0051] In step S203, all the data tables are sorted according to the comprehensive score to obtain sorted data tables.

[0052] Specifically, after the comprehensive score calculation of all data tables, the system sorts the data tables in descending order of scores to form an ordered data table list. This process enables the most important data table to appear at the front of the list, providing a basis for the subsequent warehousing operation.

[0053] In the above embodiment, the sorted data table list provides a clear priority guide for the subsequent warehousing operation, ensuring that important data tables can be processed quickly, while relatively less important data tables can be processed later or on demand, thereby optimizing the allocation of database resources, reducing the waiting time for high-priority data extraction, and improving business response speed.

[0054] Step S204, according to the order of the above sorted data tables, all the above sorted data tables are warehoused.

[0055] Specifically, according to the above sorting result, the system will sequentially perform data warehousing according to the importance of the data table from high to low. This includes data backup, recovery and storage operations to ensure that high-score data tables are ready in the first time for subsequent data extraction tasks.

[0056] In the above embodiment, warehousing according to the importance of the sorted data table can minimize the preparation time of critical data, thereby speeding up the overall process of data extraction. At the same time, this method also ensures the rational use of storage resources, avoids excessive storage of large data tables with low frequency access, reduces unnecessary storage overhead, and improves the operating efficiency and resource utilization of the database.

[0057] Through the above embodiment, the number of invocations, business demand urgency, time proximity and data volume can be introduced as parameters to comprehensively score the data table, and the data table can be sorted according to the comprehensive score. Compared with the direct data warehousing method in the prior art, the present scheme comprehensively considers multiple parameters of the data table, which is equivalent to a comprehensive priority sorting of the data table, thereby improving the efficiency of data warehousing.

[0058] The full analysis temporary number of single extraction extracts business demand / number of system / query script. Get the number of single table information from the data extraction platform, extract the demand name, demand number, demand type (business operation, management decision, information disclosure, external supervision, external audit, judicial inquiry, etc.), involved secret level, required number of system, etc. for business demand; for the number of system, get the Chinese name, English name, current state (online / offline), subordinate query script; get the specific content of the query script.

[0059] Based on the SQL script parsing method designed according to the scheme, the syntax analysis of the SQL statements in the script is realized, so as to determine the data table name and its relationship required to be extracted by each script. The specific method can be divided into the following two small steps:

[0060] a. Lexical analysis: the SQL statement is disassembled into an independent symbol sequence (Token), including keywords, identifiers, constants, etc. For example, through a tool such as Flex or a custom rule, the SQL statement is disassembled into keywords (such as SELECT, FROM), field names, table names, etc.

[0061] b. Syntax analysis: based on the SQL syntax rules, the Token sequence generated by the lexical analysis is organized into an abstract syntax tree (AST). This process includes top-down or bottom-up analysis methods, which are completed through the tool Yacc. For example, through the Yacc tool to generate a syntax analysis program yparse, the SQL statement is checked for syntax and a syntax tree is generated.

[0062] On this basis, the data table name, the used field, and the data table nested query of each query can be abstracted.

[0063] As shown in Figure 3 , the present scheme is divided into five steps:

[0064] 1. Batch analysis of temporary data extraction requirements: first, the temporary data extraction requirements are analyzed in batches;

[0065] 2. Analyzing SQL scripts based on syntax analysis tree: using SQL statement script expansion analysis;

[0066] 3. Building knowledge graph based on entity / relationship / attribute: building a knowledge graph based on entity, relationship and attribute;

[0067] 4. Calculating the key degree score of the data table based on the graph: using the built graph to calculate the key degree score of each data table;

[0068] 5. Sending the list of data tables with high ranking to the operator: finally, the list of data tables with high key degree score is sent to the operator.

[0069] The technical solution aims to solve the business efficiency problem caused by low time efficiency, dependence on experience to select library tables and storage performance bottleneck in the current temporary data extraction scene. By constructing an optimization method based on a knowledge graph, the present solution first analyzes past demands and extraction scripts to parse entities such as "business demand", "data source system", "database table", "data table field" and their relationships and attributes (as described above in the solution content). On this basis, the knowledge graph is quantitatively analyzed using graph algorithms to accurately locate data tables with higher importance, assisting operators in preparing data in advance through normalization storage and other methods. This method not only avoids the subjectivity of traditional experience-dependent library table selection, but also balances data extraction efficiency and storage resource utilization through deep mining of entity relationships in the knowledge graph, ultimately shortening the data extraction processing period and improving script running efficiency.

[0070] The collected information is authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws and regulations and standards of the country and region, necessary security measures are taken, and the public order is not violated, and appropriate operation entrances are provided for users to choose authorization or refusal; provide the user with the corresponding operation entrance to choose to agree or refuse the automatic decision result; if the user chooses to refuse, enter the expert decision-making process to avoid relevant legal risks and public opinion risks.

[0071] In the specific implementation process, the comprehensive score of the above data table can be calculated according to the above related information through the following steps: evaluating the above related information to obtain an evaluation score corresponding to each of the above related information, wherein the above call frequency and the above evaluation score are positively correlated, the above urgency of the above business demand and the above evaluation score are positively correlated, the distance between the above time of the above business demand and the current time and the above evaluation score are negatively correlated, and the above data volume and the above evaluation score are negatively correlated; calculate the weighted average of all the above evaluation scores to obtain a pre-score; and determine the above pre-score as the above comprehensive score.

[0072] In the present solution, each parameter is independently evaluated to ensure that the contribution of each dimension is fairly and accurately reflected in the final comprehensive score, improving the comprehensiveness and accuracy of the evaluation. The calculation of the weighted average ensures that key factors have a larger proportion in the comprehensive score, thereby improving the decision-making quality and efficiency of the method. The pre-score is directly converted into the comprehensive score, simplifying the calculation process while ensuring the continuity and consistency of the score, and improving the overall efficiency of data table storage management.

[0073] The factors are evaluated separately, the characteristics and influence of each parameter are independent, and the mutual interference or covering between different indicators is avoided, so as to more accurately quantify the overall importance of the data table and guide the subsequent storage optimization strategy. The weighted average value can dynamically adjust the influence of the indicators, such as the weights of the call frequency and the urgency, which can be set higher, because they directly affect the immediate value of the data table and the response speed of the business demand; and the weight of the data volume can be set to negative, so as to suppress the immediate importance of the large data volume table and prevent it from occupying too much storage resources. In this way, the system can more intelligently identify those data tables that should be prioritized for storage, reduce the processing priority of irrelevant or low-frequency tables, and thus improve resource utilization efficiency and response speed. After completing the evaluation and weighted average calculation of each parameter, the pre-score has fully considered all relevant factors, so it is directly determined as the comprehensive score, which can avoid introducing unnecessary complex calculation process again, save computing resources, and at the same time ensure the timeliness and effectiveness of the storage decision, thereby speeding up the preparation and extraction process of the key data table and improving the business processing speed.

[0074] According to the call history of the data table, the urgency and submission time of the business demand, and the data volume, each indicator is assigned a value separately to form an initial quantitative evaluation score. The above evaluation scores are weighted, and the weight setting reflects the influence of each parameter on the comprehensive importance score. Then the weighted average value is calculated to obtain the pre-score. The pre-score obtained after weighting is directly identified as the comprehensive score for subsequent data table sorting.

[0075] Suppose in a financial enterprise's data analysis center, a large amount of transaction data and customer information is processed every day. In order to respond to temporary data extraction requests from various departments, especially those with high urgency and large data volume report requests, the company needs to optimize its data table storage process to ensure that high-frequency access and high-urgency data tables can be prepared faster. At this time, the system first evaluates each data table, focusing on those that have been accessed frequently (>10 times) in the past week, have a very high business demand urgency (>8 / 10), have a demand submission time within the past two days, and have a data volume less than 1GB. The evaluation score of such data tables will naturally be higher, so they will be given a higher weight when calculating the weighted average value. Suppose the weight of the call frequency is 0.4, the weight of the urgency is 0.3, the weight of the submission time is 0.2, and the weight of the data volume is -0.1. Then, all data tables are sorted according to the comprehensive score from high to low, and data tables with a comprehensive score higher than a predetermined threshold (for example, comprehensive score > 60 / 100) are prioritized for storage. In this way, the system can quickly respond to high-frequency and high-urgency data extraction requests, while avoiding unreasonable occupation of storage and computing resources by data tables with large data volume or low access frequency, improving the efficiency and resource utilization of the entire process.

[0076] According to the atlas information, the importance score of each data table is calculated. The score of the influence of the importance of the data table is calculated according to four parts of the upstream business demand on the data table, the upstream business system, the data table itself, and the association relationship with other data tables, respectively, wherein:

[0077] a. For each upstream business demand of the data table, if the database table is called multiple times, it indicates that the importance of the database table is higher; if the demand urgency is higher, the data table should be given a higher importance. The demand urgency can be measured by the demand type and the demand business party; if the demand time is far from the current time, there may be demand changes, therefore, the demand with a long time distance should have a weak influence on the importance of the data table.

[0078] b. For each upstream data system of the data table, if the system has an associated data mart, the data can be directly obtained based on the data mart when querying, therefore, the importance of the database table with the associated data mart of the data system can be reduced.

[0079] c. For the database table itself, if the data volume is too large (more than 100 million per day), normalizing the data into the database will seriously affect the database efficiency, therefore, the data table with a large data volume has a negative influence on the importance of the data table.

[0080] d. For the library table associated with the current library table, the influence of the library table on the current library table is calculated according to the importance score, and the final importance score is obtained through multiple iterations.

[0081] In some embodiments, the comprehensive score of the data table is calculated according to the above-mentioned related information, further comprising the following steps: calculating the difference between a preset value and a damping coefficient to obtain a first calculation result, wherein the preset value is 1; calculating the quotient of the first calculation result and the total number of the data tables to obtain a second calculation result; calculating the quotient of the preset score and the data volume of the associated table of the data table to obtain a third calculation result; calculating the sum of the third calculation results of all associated tables of the data table to obtain a fourth calculation result; calculating the sum of the second calculation result and the fourth calculation result to obtain the comprehensive score.

[0082] In the scheme, by setting the damping coefficient, the dynamic calculation of the weighting factor can adjust the transmission strength of the data table importance score in the network, avoid the phenomenon of excessive concentration of weights that may occur in network calculation, ensure the balance and rationality of evaluation, and enhance the robustness and generalization ability of the method. The balanced allocation of the basic score ensures that each data table has an equal opportunity to participate in the overall score calculation during the iterative calculation of the importance score of the association relationship, preventing the risk of important data tables being underestimated due to data imbalance in the initial stage. By introducing the influence factor of data volume, the weight of the data table is dynamically adjusted in the comprehensive score, especially for those data tables with huge data volume but low usage frequency, their comprehensive score will be suppressed to a certain extent, so that in the case of limited storage and computing resources, important and moderately data volume data tables can be prepared for storage in time. Through the summary of the influence factor of the associated table, the position perception of the data table in the network is strengthened, ensuring that those data tables that play a pivotal role in multiple business processes can obtain a higher comprehensive score, thereby occupying a favorable position in the storage sorting and accelerating the preparation and execution speed of business data extraction. Through the final determination of the comprehensive score, the accuracy of the data table importance evaluation is realized, ensuring the efficient use of resources, reducing unnecessary storage occupation and computing overhead, and improving the timeliness and accuracy of business demand response.

[0083] In weighted network calculation, the role of damping coefficient is similar to the resistance in thermodynamics, which limits the diffusion speed of importance score in the network, prevents the score of key nodes (such as high-frequency called data tables) from being too high, and causes the unreasonable phenomenon of excessive concentration of resources in a few tables. The default value is 1, which means that when there is no damping effect, the importance scores of all data tables are considered equal, which provides a balanced starting point for subsequent calculations. By dividing the first calculation result by the total number of data tables, each data table obtains the same basic score, which is particularly important in graph algorithms, as it provides the starting condition for weight transfer between data tables, ensuring that each data table can be treated fairly and participate in subsequent score iteration calculations, and finally form a more balanced and reasonable comprehensive score distribution. It is obvious that too large data volume has a significant impact on database performance, especially in resource-limited situations, too large data volume will cause I / O bottleneck, calculation delay and other problems, and thus affect data extraction efficiency. By calculating the quotient of the pre-score and the data volume of the associated table, the system can identify and appropriately reduce the importance score of these large data volume tables, thus prioritizing the processing of data tables with large business impact but controllable data volume in actual data loading operations, and achieving optimal resource allocation. In actual business scenarios, some data tables may become the common dependence of multiple business requirements due to their key position in the data flow, and these tables have a direct and far-reaching impact on the overall efficiency of the business. The fourth calculation result is calculated by accumulating the third calculation results of all associated tables, which reflects this influence and gives appropriate weighting in the comprehensive score, making the score of key tables more competitive and ultimately prioritizing in the data loading order, providing fast response for subsequent data extraction requests. The comprehensive score is the result of multiple rounds of calculation, which takes into account the balanced score of data tables in the initial state and their role in the associated network. Through this final score, the system can accurately locate those data tables that are both crucial to the business and have good data volume control, ensuring the priority of these tables in data loading, thus maximizing data extraction efficiency and business processing speed in resource-limited situations, reducing operating costs, and improving user experience and enterprise competitiveness.

[0084] The damping coefficient is a number between 0 and 1 that controls the decay rate in the weight transfer process during network calculation, with a default value of 1. The difference between the two is calculated to provide a basis for the weighting factor in the subsequent steps, embodying the initial balance idea in the algorithm, ensuring that the importance score of each data table tends to the same level without other information interference. The result of this calculation is equivalent to assigning a basic score to each data table at the initial stage, which is a balanced starting point before network calculation, i.e., the second calculation result. Based on the comparison of the pre-score and the associated table data volume of the data table, the third calculation result is obtained. This step focuses on the management of data volume, especially for large-scale data tables, by calculating the proportion of their pre-score to their associated table data volume, quantifying the negative impact of data volume on the comprehensive score. The sum of the third calculation results of all associated tables for each data table is obtained, which is the fourth calculation result. This step further refines the assessment of the association between data tables, adjusting the importance score of the original data table by quantifying the influence of the associated table, reflecting the importance of the interaction between data tables in the network. The sum of the second calculation result and the fourth calculation result is the final comprehensive score. This sum calculation takes into account the initial balanced score of the data table and its association score in the network, forming a comprehensive evaluation of the overall importance of the data table.

[0085] Suppose in the information technology department of a large financial institution, thousands of data extraction requests need to be processed every day, covering various aspects such as customer transaction record analysis, risk assessment, financial report generation, and regulatory compliance review. In order to improve data extraction efficiency and reduce storage pressure, the technical team adopts the above optimization method to comprehensively score and optimize the storage of all data tables. In specific implementation, assume that the total number of data tables is 1000, and the damping coefficient is 0.9, which means that in the initial state, each data table will obtain a basic score (1000 (1 - 0.9) = 100), which reflects the balanced starting point without other factors. Subsequently, for those core data tables associated with a large number of other tables and with moderate data volume (such as customer account information table), when calculating the pre-score, it will consider its close association with multiple business processes and its data volume is large but within an acceptable range (for example, data volume < 1GB), so the comprehensive score will be higher. Conversely, for those small data tables with extremely large data volume (data volume > 5GB) but low association, their comprehensive score will be negatively affected by the data volume factor, with a lower pre-score, which may not be prioritized for storage or deep compression processing to save storage space. In actual storage operation, the system will sort the data tables according to the comprehensive score, prioritizing those with a score higher than 80, while postponing the storage or on-demand processing of those with a score lower than 50.

[0086] The final algorithm flow is:

[0087] (1) According to the upstream business requirements, the data system, the data table itself data volume calculation importance pre-score , the calculation formula is: Among them, is a normalization function, , , is the weight of the three, is the score given according to the demand urgency, is the urgency score given according to the nature of the business, is given according to whether there is a data mart score, taking 1 or -1 (1 for -1, no for 1), is the daily data volume of the data table; is a function to measure the time distance from the current demand, the specific calculation method is: is a Sigmoid function, which controls the interval of the function in the range (acts on the value of this section to be normalized to the interval (0, 1)), where and are parameters obtained by training, which are adjusted according to actual conditions, that is, the parameters that control the decay of the function, which changes according to the specific situation, is the current time distance, is the time corresponding to the demand. is a function to measure the data volume, the specific calculation method is: , where so that the data volume can be negative and decay rapidly after exceeding the value of k, k is determined according to the database capacity and other specific business scenarios, and a and b are obtained according to k.

[0088] (2) For the association relationship network composed of all library tables, use PR algorithm to calculate the final score Q_final of each node iteration ), for database table The calculation formula is: , where, is the damping coefficient, is the total number of data tables, is all associated data tables associated data tables, represents the number of data tables associated data tables; The core idea of the PR algorithm is that the importance of a node depends on the number and quality of other nodes connected to it. This algorithm can be well applied to library table relationships to mine higher key library tables

[0089] (3) The importance scores of all library tables are sorted, and the data tables with higher importance are obtained.

[0090] In the specific implementation process, before obtaining the related information of the data table, the above method further includes the following steps: extracting all entity information, wherein the entity information includes one or more of the business requirement, the database system, the data table, and the fields of the data table; extracting all relationship information, wherein the relationship information includes one or more of the calling relationship between the business requirement and the database system, the belonging relationship between the database system and the data table, the calling relationship between the business requirement and the data table, the association relationship between the data tables, and the belonging relationship between the data table and the fields; constructing a knowledge graph by taking the entity information as nodes and the relationship information as edges, wherein the knowledge graph is used for data correlation analysis.

[0091] In the present scheme, comprehensive entity information collection is the basis for forming the knowledge graph, which can ensure the depth and breadth of data correlation analysis, provide rich background information for the importance of data tables and the optimization strategy of data storage, and improve the scientificity and accuracy of decision-making. Fine relationship information construction strengthens the network structure of the knowledge graph, which can more accurately reflect the real correlation between entities and the complexity of business processes, and improve the precision and efficiency of data table importance evaluation. The construction of the knowledge graph greatly improves the data visibility and management efficiency. Through graphical display, the operator can more intuitively understand the position and value of the data table in the network, provide intuitive basis for decision-making, and facilitate data correlation analysis and potential business insight mining.

[0092] Entity information is the first step in building a knowledge graph. Collecting all entity information, including business requirements, database system status, data table structure, and field attributes, provides a complete data perspective for subsequent processes. This step is crucial as it directly impacts the quality and practicality of the knowledge graph. Only with accurate entity information can we build a graph that reflects real business relationships and data structures, providing strong support for data table importance scoring and storage strategy development. Relationship information extraction is the soul of knowledge graph construction. It not only reveals direct connections between entities but also discovers implicit indirect relationships through graph analysis, forming a comprehensive entity network. For example, the calling relationship between business requirements and database systems helps identify which systems and tables are most critical to specific businesses. The association between data tables reveals data flow paths and dependencies, which is valuable for predicting future data extraction patterns and demand trends. Therefore, through detailed relationship information construction, we can obtain more accurate data table importance scores, providing stronger evidence for subsequent storage optimization. Knowledge graph is not only a high-level form of data storage and retrieval, but also a powerful tool for data relationship understanding and business process analysis. By mapping entity information and relationship information to a graph structure, we can more directly identify the propagation path of business requirements, the association depth of data tables, and the frequency of field use. This not only simplifies the complexity of data table importance assessment, but also allows operators to more easily understand the position and value of data tables in the network through a graphical interface, making more accurate storage decisions. The visualization feature of the knowledge graph also helps us quickly locate problems, such as identifying abnormal data access patterns or potential data quality defects, so that we can adjust data management strategies in a timely manner to improve overall business continuity and data security.

[0093] Before assessing the importance of data tables, extract all relevant entity information from historical temporary number sheets, database logs, and system metadata, including business requirements, database systems, data tables, and data table fields. This is the prerequisite for building a knowledge graph, ensuring that the data basis for subsequent analysis is complete and accurate. Extract the calling, belonging, and association relationships between entities, including the calling relationship between business requirements and database systems, the membership of database systems and data tables, the calling relationship between business requirements and data tables, the reference or nested query relationship between data tables, and the membership relationship between data tables and fields. These are important components of building a knowledge graph. Use the extracted entity information as nodes in the graph and the relationship information between entities as edges connecting the nodes to build a knowledge graph. This graph can visually present the multi-level association between entities and the flow path of business requirements, providing a visual tool for data management and optimization.

[0094] Assume that in a global retail enterprise's data management platform, there are numerous sales data, inventory information, market research results, and supply chain management records processed every day. To meet the rapidly changing market demand and decision support, the data management department adopts the technology of this solution - a temporary data extraction optimization method based on knowledge graph. Specifically, they first extract all entity information from historical extraction orders and database logs, including various types of business requirements (such as sales analysis, inventory check, market prediction, etc.), related database systems (such as ERP, CRM, BI system, etc.), relevant data tables (such as product sales table, customer information table, supply chain order table, etc.), and key fields in data tables (such as product ID, sales date, customer feedback, etc.). Next, the system starts extracting relationship information, which includes the calling relationship between business requirements and database systems, the association between database systems and data tables, the direct calling between business requirements and data tables, the reference relationship between data tables, and the ownership of data tables and fields. For example, the sales analysis business requirement will frequently call the product sales table and the customer information table, and these two tables will have a high calling frequency and business importance score in the knowledge graph. During the construction of the knowledge graph, each entity is regarded as a node, and the relationship between entities is regarded as an edge. Through the construction of the knowledge graph, the system can clearly show which data tables are the core of the entire data ecosystem, which fields are frequently used, and how business requirements are transmitted between different databases and data tables.

[0095] According to the business requirements / extraction system / data table entities and their attributes / relationships, a knowledge graph is constructed, wherein the entities and attributes include:

[0096] Entity 1: business requirement, including attribute information: requirement name, requirement number, requirement type, requirement business party, requirement submission time, and urgency.

[0097] Entity 2: data system, including attribute information: system name, system number, system status, and whether there is an external data mart association.

[0098] Entity 3: database table, including attribute information: data table name, data table type (main file / detail), and data table single-day data volume.

[0099] Entity 4: data table field, including attribute information: data table field name and data table field type.

[0100] Relationships include:

[0101] Relationship 1: business requirement - database system: one-to-many relationship, describing the corresponding database system of each business requirement.

[0102] Relationship 2: Database system - data table: one-to-many relationship, describing which data tables are under each database system;

[0103] Relationship 3: Business requirement - database table: one-to-many relationship, describing which data tables are needed for each business requirement;

[0104] Relationship 4: Database table - database table: many-to-many relationship, describing tables associated with each other / nested queries, while recording the association time as a relationship attribute;

[0105] Relationship 5: Database table - field inclusion relationship: one-to-many relationship, describing the relationship between database tables and fields;

[0106] Relationship 6: Database table - field association relationship: many-to-many relationship, describing the association relationship between database tables and fields (records and data table nested / associated fields).

[0107] In some embodiments, obtaining the relevant information of the data table can be achieved by the following steps: extracting the number of times that the above business requirements connect the same above data table in the above knowledge graph; and determining the above number of times as the above call number of the above data table.

[0108] In this scheme, the quantitative statistics of the call number can effectively reveal the activity degree and value proportion of the data table in enterprise operation, providing an empirical basis for subsequent comprehensive score calculation, ensuring that the setting of the storage priority is more reasonable and objective. The formal confirmation of the call number as an important indicator means that the data management strategy will focus more on high-frequency data tables, which not only accelerates the data extraction process and reduces the waiting time of high-value data, but also promotes the rational allocation of resources and avoids excessive attention to low-frequency data tables and ineffective storage, thereby improving the overall efficiency of the database and the utilization efficiency of data resources.

[0109] Based on the knowledge graph, the calling relationships between data tables and business needs become clearly visible. By quantifying the number of times each data table is called, we can intuitively understand which data tables are most frequently accessed and relied upon in enterprise operations, thus assigning them higher weight when calculating their importance scores. This step is a key component of obtaining relevant information about data tables. It ensures that the subsequent positive correlation between the number of calls and the overall score is based on actual usage data, rather than subjective guesses or outdated experience, thereby improving the scientific nature and efficiency of the overall data management process. In the field of data processing, the number of times a data table is called is a direct indicator for assessing its usage frequency and business importance. By identifying it as one of the important attributes of a data table, we can assign appropriate weight to the number of calls in subsequent comprehensive score calculations, ensuring that data tables that are crucial to business operations are prioritized. This not only reduces waiting time for critical business processes and improves overall work efficiency, but also promotes the effective use of storage resources, avoids blindly processing infrequently used data tables, saves valuable storage space, and reduces operating costs.

[0110] By traversing the constructed knowledge graph, the total number of times each data table is accessed by different business needs is counted. This number reflects the frequency of use of the data table in various business processes and is one of the key indicators for assessing the importance of the data table. The number of accesses is explicitly identified as an important related attribute of the data table, and this attribute will play a crucial role in the subsequent comprehensive evaluation, affecting the sorting and database entry priority of the data table.

[0111] Imagine an electronic medical record management system in a comprehensive hospital. Every day, a large amount of medical data is entered, queried, and analyzed to support various business needs, including diagnosis, treatment, research, and administration. To optimize the data extraction process and improve the efficiency of medical services, the hospital's IT department decided to adopt a temporary data extraction optimization method based on knowledge graphs. In practice, they first constructed a knowledge graph containing entities such as patient information, medical records, prescriptions, and departmental resources, as well as the various relationships between these entities, including their calls, ownership, and associations. Based on this, the system began to count the number of connections between business needs and data tables. For example, the research department's research needs for specific diseases frequently call patient medical records and prescriptions, while the administrative department's resource auditing focuses more on the usage of departmental resources. By extracting the connection counts, the system can accurately calculate the number of times data tables are called and adjust the data table's entry priority accordingly.

[0112] Suppose the system records that the patient treatment record table was accessed 1000 times within a week for various business needs, while the department resource usage table was accessed only 200 times. Based on this information, the access count for the patient treatment record table is determined to be 1000, and the access count for the department resource usage table is determined to be 200. In subsequent data entry operations, the system will prioritize processing the patient treatment record table to ensure its timely availability, while the department resource usage table will be processed later based on resource occupancy. This approach ensures immediate response to critical business needs while avoiding waste of storage resources.

[0113] In the specific implementation process, according to the order of the sorted data tables, all the sorted data tables are entered into the database. This can be achieved through the following steps: Enter the first M sorted data tables into the database in order, where M = 1 / N, and N is the total number of sorted data tables; if new business requirements are raised, enter the sorted data tables that have not yet been entered into the database in order.

[0114] In this solution, the priority-based data entry strategy significantly improves the response speed of critical data and the utilization efficiency of storage resources, reduces overall processing time, and enhances business continuity and data management flexibility. Dynamically updating the data entry form enhances the flexibility and responsiveness of data management, ensuring that the data entry strategy always meets current business needs, avoiding data extraction delays and resource mismatches caused by changes in requirements, and improving the overall level of data governance.

[0115] By setting M to the reciprocal of N, meaning the number of tables to be added to the database is 1 / N of the total, the system can still ensure the timely addition of a certain proportion of the most critical tables even with limited processing resources and storage space. This not only reduces the burden on the database and avoids resource waste, but also ensures that the system can respond quickly when business demands suddenly increase, reducing data retrieval waiting time and improving business continuity and user experience. The constantly changing business demands require data management strategies to be highly dynamic and adaptable. By recalculating the comprehensive score of data tables each time a new business demand is received, the system can promptly identify and add data tables that become important due to the new demand, even if they were previously ranked lower. This not only ensures rapid response in data retrieval but also avoids resource waste caused by static strategies, improving storage space utilization and database operating efficiency.

[0116] After determining the overall score and sorting of the data tables, the top M data tables are selected for priority entry into the database. Here, M is set as the reciprocal of the total number of data tables N, i.e., M = 1 / N. This strategy ensures that, with limited resources, the most important data tables with the greatest impact on business are processed first, achieving efficient resource utilization. When the system receives new business requirements, it reassesses the importance of the data tables and enters those that have not yet been entered into the database but are assessed as important based on the latest requirements. This dynamic update mechanism ensures that the data management strategy can quickly adapt to changes in the business environment, maintaining the timeliness and effectiveness of the entry forms.

[0117] Suppose a medium-sized e-commerce company needs to process a large amount of user orders, product information, inventory status, and market analysis data. The company decides to adopt a knowledge graph-based temporary data extraction optimization method to improve the efficiency and accuracy of data extraction. In specific implementation, assume the total number of sorted data tables, N, is 200. According to the above method, M = 1 / N = 1 / 200 = 0.005, meaning the system will prioritize storing the first 1% (i.e., 2) of data tables in the database. These tables are typically user order and product information tables, as they have the highest importance and frequency of access in the business process. In subsequent daily operations, whenever new business needs arise, such as in-depth market trend analysis, the system will re-evaluate the overall score of the data tables. If the importance of the inventory status and market analysis data tables increases, even if they are not initially in the top 2 tables, the system will store them in the database according to their importance based on the new needs, ensuring timely data availability while avoiding unnecessary processing of low-frequency access tables, saving valuable storage space and computing resources.

[0118] In some embodiments, after sorting all the above-mentioned data tables according to the above-mentioned comprehensive score to obtain a sorted data table, the above-mentioned method further includes the following steps: compressing all the above-mentioned sorted data tables according to the above-mentioned comprehensive score of the sorted data tables to obtain a compressed data table, wherein the above-mentioned comprehensive score and compression rate are negatively correlated; and storing the above-mentioned compressed data tables into the database in order.

[0119] In this solution, the dynamic compression strategy significantly improves storage space utilization while maintaining the access efficiency of critical business data, reducing the total cost of data processing, and enhancing the overall efficiency of data management. The combination of sequential data entry and compression not only optimizes the data storage strategy but also accelerates the data extraction process. By first entering important data tables and using a lower compression rate, it ensures rapid access and accurate use of these tables in subsequent data extraction processes, thereby improving overall data processing efficiency and user experience.

[0120] By setting a negative correlation between the overall score and compression rate of data tables, the system can apply a higher degree of compression to less important data tables without sacrificing access speed and data quality for important tables. This reduces storage space usage and avoids impacting the performance of critical data. This approach is particularly important when processing large amounts of data, ensuring efficient database operation while effectively controlling storage costs, thus improving overall data management efficiency and business continuity. Combining data table compression with sequential data entry allows enterprises to achieve dual optimization of storage and access. Compression ensures effective storage space saving, while sequential data entry prioritizes tables with the greatest impact on business operations based on their importance and compression rate, reducing data retrieval waiting time and improving business response speed. This combined strategy can significantly reduce the total cost of data processing in practical applications, while improving data access efficiency and quality, enhancing the enterprise's data management capabilities and competitiveness.

[0121] After sorting the data tables, each table is compressed based on its overall score; a higher score results in a lower compression rate, and vice versa. This strategy ensures that important data tables maintain high data integrity and access speed during operations, while less important tables are compressed more aggressively to save storage space. After compression, the data tables are stored in the database in the sorted order. This ensures that highly important tables are processed first, and an optimized storage solution is used to further reduce storage space requirements.

[0122] Imagine an IT department at a large financial institution that processes tens of thousands of data extraction requests daily, covering everything from customer transaction analysis and risk assessment to financial report generation and regulatory compliance review. To improve data extraction efficiency and reduce storage pressure, the technical team adopted the optimization method described above, compressing and storing all data tables based on a comprehensive score. Specifically, assuming a total of 500 sorted data tables, the top 100 tables (approximately 20% of the total) have higher comprehensive scores, while the remaining 400 tables have relatively lower scores. For the top 100 tables, the team decided to use a lower compression rate (e.g., no more than 20%) to ensure fast access to critical business data and data integrity; while for the remaining 400 tables, a higher compression rate (e.g., up to 60%) was adopted to maximize storage space savings. When processing the data inbound process, the technical team prioritizes the processing of the first 100 data tables in the sorted order to ensure that the data in these tables is quickly available. Then, the remaining data tables are processed step by step according to the resource situation. This strategy not only ensures business continuity and data security, but also significantly reduces storage costs and improves the efficiency of data extraction and business processing.

[0123] The innovative aspects of this solution are:

[0124] 1) A knowledge graph construction method for temporary data extraction scenarios: This paper proposes a knowledge graph construction method for temporary data extraction scenarios, which can extract entities, relationships, and attributes from temporary data (such as logs, caches, temporary tables, etc.) in near real-time and quickly construct a domain knowledge graph. For unstructured (such as SQL files) and semi-structured (such as JSON application forms) data commonly found in temporary data, multimodal data parsing rules are designed to identify entities and implicit relationships.

[0125] 2) Graph-based key data table mining method: Based on the constructed knowledge graph, a composite weight calculation model was designed to evaluate the importance of data tables from multiple dimensions such as the urgency of business needs, the frequency of database table usage, and the frequency of cross-references of database tables.

[0126] The effects that this application can achieve are:

[0127] 1) Helps operators more accurately locate the database tables for routine data entry. The graph-based key data table mining method can significantly improve the efficiency and accuracy of data management. By accurately locating important data tables, routine database entry can be performed more effectively, improving the efficiency of temporary data extraction.

[0128] 2) Operators can gain a more holistic perspective. Through the visualization of the knowledge graph, operators can intuitively see the relationships between all temporary data tables and their position in the entire data ecosystem.

[0129] Current temporary data extraction solutions primarily rely on manual experience to pre-select commonly used database tables for routine storage, which suffers from the following problems: a lack of dynamic adaptability leads to the database table list failing to respond promptly to changes in business needs; redundant storage consumes excessive disk space and causes performance bottlenecks; furthermore, the lack of knowledge graph technology for quantitative analysis of the business value, data quality, and relationships of database tables results in strong subjectivity and low accuracy in identifying key database tables. This solution constructs a dynamic knowledge graph and combines it with a multi-dimensional composite weight model (business value, data quality, and relationship complexity) to achieve intelligent identification and optimized resource allocation of key database tables, significantly improving storage efficiency and data extraction response speed. Table 1 shows a comparison of expected results.

[0130] Table 1

[0131]

[0132] This application also provides a data table insertion device. It should be noted that the data table insertion device of this application embodiment can be used to execute the data table insertion method provided in this application embodiment. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0133] The following describes the data table entry device provided in the embodiments of this application.

[0134] Figure 4 This is a structural block diagram of a data table entry device according to an embodiment of this application. Figure 4 As shown, the device includes:

[0135] The acquisition unit 10 is used to acquire relevant information of the data table, wherein the aforementioned relevant information includes one or more of the following: the number of calls, the urgency of the business requirement, the time when the business requirement was proposed, and the amount of data.

[0136] The calculation unit 20 is used to calculate the comprehensive score of the data table based on the above-mentioned relevant information. The comprehensive score is the score of the importance of the data table. The number of calls is positively correlated with the comprehensive score. The urgency of the business requirement is positively correlated with the comprehensive score. The proximity of the time when the business requirement was submitted to the current time is negatively correlated with the comprehensive score. The amount of data is negatively correlated with the comprehensive score.

[0137] The sorting unit 30 is used to sort all the above data tables according to the above comprehensive scores to obtain the sorted data table.

[0138] The first data entry unit 40 is used to enter all the sorted data tables into the database according to the order of the sorted data tables.

[0139] This embodiment allows for the introduction of parameters such as the number of calls, the urgency of business needs, the time of submission, and the amount of data to comprehensively score the data table and sort it according to the comprehensive score. Compared with the existing technology of directly inserting data into the database, this solution comprehensively considers multiple parameters of the data table, which is equivalent to prioritizing and sorting the data table, thereby improving the efficiency of data insertion.

[0140] In the specific implementation process, the calculation unit includes an evaluation module, a first calculation module, and a first determination module. The evaluation module is used to evaluate the above-mentioned relevant information and obtain an evaluation score corresponding to each of the above-mentioned relevant information. Among them, the number of calls and the evaluation score are positively correlated, the urgency of the business requirement and the evaluation score are positively correlated, the proximity of the time the business requirement was submitted to the current time and the evaluation score are negatively correlated, and the amount of data and the evaluation score are negatively correlated. The first calculation module is used to calculate the weighted average of all the above-mentioned evaluation scores to obtain a pre-score. The first determination module is used to determine the pre-score as the comprehensive score.

[0141] This approach ensures that the contribution of each dimension is fairly and accurately reflected in the final composite score by independently evaluating each parameter, thus improving the comprehensiveness and accuracy of the evaluation. The calculation of the weighted average ensures that key factors have a greater weight in the composite score, thereby improving the decision-making quality and efficiency of the method. Pre-scored scores are directly converted into composite scores, simplifying the calculation process while ensuring the consistency and coherence of the scores, improving the overall efficiency of data entry and management.

[0142] In some embodiments, the calculation unit includes a second calculation module, a third calculation module, a fourth calculation module, a fifth calculation module, and a sixth calculation module. The second calculation module is used to calculate the difference between a preset value and a damping coefficient to obtain a first calculation result, wherein the preset value is 1. The third calculation module is used to calculate the quotient of the first calculation result and the total number of the data table to obtain a second calculation result. The fourth calculation module is used to calculate the quotient of the pre-score and the data volume of the associated tables of the data table to obtain a third calculation result. The fifth calculation module is used to calculate the sum of the third calculation results of all associated tables of the data table to obtain a fourth calculation result. The sixth calculation module is used to calculate the sum of the second calculation result and the fourth calculation result to obtain the comprehensive score.

[0143] This scheme, by setting a damping coefficient and dynamically calculating weighting factors, can adjust the transmission strength of the importance scores of data tables in the network, avoiding the excessive concentration of weights that may occur in network calculations, ensuring the balance and rationality of the evaluation, and enhancing the robustness and generalization ability of the method. The balanced distribution of basic scores ensures that each data table has an equal opportunity to participate in the overall score calculation when iteratively calculating the importance scores of association relationships, preventing the risk of important data tables being underestimated due to data imbalance in the initial stage. By introducing the influence factor of data volume, the weight of data tables is dynamically adjusted in the comprehensive score. In particular, for data tables with huge data volumes but low usage frequency, their comprehensive scores will be suppressed to a certain extent. Thus, under limited storage and computing resources, it prioritizes ensuring that important data tables with moderate data volumes are prepared for timely data entry. By summarizing the influence factors of association tables, the position awareness of data tables in the network is strengthened, ensuring that data tables that play a pivotal role in multiple business processes can obtain higher comprehensive scores, thereby occupying a favorable position in the data entry ranking and accelerating the preparation and execution speed of business data extraction. The final determination through comprehensive scoring achieves precise assessment of data table importance, ensures efficient resource utilization, reduces unnecessary storage and computational overhead, and improves the timeliness and accuracy of responding to business needs.

[0144] In the specific implementation process, the above-mentioned device further includes a first extraction unit, a second extraction unit, and a construction unit. The first extraction unit is used to extract all entity information before obtaining relevant information of the data table. The entity information includes one or more of the business requirements, the database system, the data table, and the fields of the data table. The second extraction unit is used to extract all relationship information. The relationship information includes one or more of the calling relationship between the business requirements and the database system, the ownership relationship between the database system and the data table, the calling relationship between the business requirements and the data table, the association relationship between the data tables, and the ownership relationship between the data table and the fields. The construction unit is used to construct a knowledge graph by using the entity information as nodes and the relationship information as edges. The knowledge graph is used for data association analysis.

[0145] In this solution, comprehensive entity information collection forms the foundation for the knowledge graph. It ensures the depth and breadth of data association analysis, providing rich background information for the importance of data tables and optimization strategies for data entry, thus improving the scientific rigor and accuracy of decision-making. The detailed construction of relationship information strengthens the network structure of the knowledge graph, enabling it to more accurately reflect the real relationships between entities and the complexity of business processes, improving the accuracy and efficiency of data table importance assessment. The construction of the knowledge graph significantly improves data visibility and management efficiency. Through graphical displays, operators can more intuitively understand the position and value of data tables within the network, providing a clear basis for decision-making and facilitating data association analysis to uncover potential business insights.

[0146] In some embodiments, the acquisition unit includes an extraction module and a second determination module. The extraction module is used to extract the number of times that all the above-mentioned business requirements in the above-mentioned knowledge graph are connected to the same above-mentioned data table; the second determination module is used to determine the above-mentioned number of times as the above-mentioned number of calls to the above-mentioned data table.

[0147] In this solution, the quantitative statistics of call counts effectively reveal the activity level of data tables and their value proportion in enterprise operations, providing an empirical basis for subsequent comprehensive score calculations and ensuring that the setting of data entry priorities is more reasonable and objective. The formal confirmation of call counts as an important indicator means that data management strategies will focus more on frequently used data tables. This not only accelerates the data extraction process and reduces waiting time for high-value data, but also promotes the rational allocation of resources, avoiding excessive attention to and ineffective storage of low-frequency data tables, thereby improving the overall operating efficiency of the database and the utilization efficiency of data resources.

[0148] In the specific implementation process, the first data entry unit includes a first data entry module and a second data entry module. The first data entry module is used to enter the first M data tables of the above sorted data tables in order, where M=1 / N and N is the total number of the above sorted data tables. The second data entry module is used to enter the above sorted data tables that have not yet been entered into the database in order when new business requirements are raised.

[0149] In this solution, the priority-based data entry strategy significantly improves the response speed of critical data and the utilization efficiency of storage resources, reduces overall processing time, and enhances business continuity and data management flexibility. Dynamically updating the data entry form enhances the flexibility and responsiveness of data management, ensuring that the data entry strategy always meets current business needs, avoiding data extraction delays and resource mismatches caused by changes in requirements, and improving the overall level of data governance.

[0150] In some embodiments, the above-described apparatus further includes a compression unit and a second storage unit. The compression unit is used to compress all the sorted data tables according to the comprehensive score to obtain a sorted data table, and then compress the sorted data tables according to the comprehensive score of the sorted data tables to obtain a compressed data table, wherein the comprehensive score and the compression rate are negatively correlated. The second storage unit is used to store the compressed data tables in the database in order.

[0151] In this solution, the dynamic compression strategy significantly improves storage space utilization while maintaining the access efficiency of critical business data, reducing the total cost of data processing, and enhancing the overall efficiency of data management. The combination of sequential data entry and compression not only optimizes the data storage strategy but also accelerates the data extraction process. By first entering important data tables and using a lower compression rate, it ensures rapid access and accurate use of these tables in subsequent data extraction processes, thereby improving overall data processing efficiency and user experience.

[0152] The data entry device for the aforementioned data table includes a processor and a memory. The acquisition unit, calculation unit, sorting unit, and first entry unit are all stored as program units in the memory, and the processor executes the program units stored in the memory to achieve the corresponding functions. All of the above modules are located in the same processor; alternatively, the modules may be located in different processors in any combination.

[0153] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can address the problem of poor overall efficiency during data import in existing technologies, particularly when dealing with large amounts of data.

[0154] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0155] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the data table insertion method.

[0156] This invention provides a processor for running a program, wherein the program executes the data table insertion method during runtime.

[0157] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements at least the steps of a data table insertion method. The device described herein can be a server, PC, PAD, mobile phone, etc.

[0158] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform a program that initializes a database entry method step having at least a data table.

[0159] This application also provides a data entry system, including: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for executing any of the above-described data table entry methods.

[0160] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0161] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0166] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0167] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0168] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0169] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0170] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for inserting data into a database, characterized in that, include: Obtain relevant information from the data table, wherein the relevant information includes one or more of the following: the number of calls, the urgency of the business requirement, the time when the business requirement was proposed, and the amount of data. The comprehensive score of the data table is calculated based on the relevant information. The comprehensive score is a score indicating the importance of the data table. The number of calls is positively correlated with the comprehensive score. The urgency of the business requirement is positively correlated with the comprehensive score. The proximity of the time the business requirement was submitted to the current time is negatively correlated with the comprehensive score. The amount of data is negatively correlated with the comprehensive score. Sort all the data tables according to the comprehensive score to obtain the sorted data table; According to the order of the sorted data tables, all the sorted data tables are entered into the database.

2. The method according to claim 1, characterized in that, Calculate the overall score of the data table based on the relevant information, including: The relevant information is evaluated to obtain an evaluation score for each piece of information. The number of calls is positively correlated with the evaluation score, the urgency of the business requirement is positively correlated with the evaluation score, the proximity of the time the business requirement was submitted to the current time is negatively correlated with the evaluation score, and the amount of data is negatively correlated with the evaluation score. Calculate the weighted average of all the assessment scores to obtain the preliminary score; The pre-score is determined as the comprehensive score.

3. The method according to claim 2, characterized in that, Calculating the overall score of the data table based on the relevant information also includes: The difference between the preset value and the damping coefficient is calculated to obtain the first calculation result, wherein the preset value is 1; Calculate the quotient of the first calculation result and the total number in the data table to obtain the second calculation result; The third calculation result is obtained by calculating the quotient of the pre-score and the data volume of the associated table of the data table; The sum of the third calculation results of all related tables in the data table is calculated to obtain the fourth calculation result; The sum of the second calculation result and the fourth calculation result is calculated to obtain the comprehensive score.

4. The method according to claim 1, characterized in that, Before retrieving relevant information from the data table, the method further includes: Extract all entity information, wherein the entity information includes one or more of the business requirements, the database system, the data table, and the fields of the data table; Extract all relationship information, wherein the relationship information includes one or more of the following: the call relationship between the business requirement and the database system, the ownership relationship between the database system and the data table, the call relationship between the business requirement and the data table, the association relationship between the data tables, and the ownership relationship between the data table and the field. The entity information is used as nodes and the relationship information is used as edges to construct a knowledge graph, which is used for data association analysis.

5. The method according to claim 4, characterized in that, Retrieve relevant information from the data table, including: Extract the number of times all the business requirements in the knowledge graph are connected to the same data table; The number is determined as the number of times the data table is called.

6. The method according to claim 1, characterized in that, According to the order of the sorted data tables, all the sorted data tables are entered into the database, including: The first M data tables after sorting are stored in the database in order, where M = 1 / N and N is the total number of data tables after sorting. If new business requirements arise, the sorted data tables that have not yet been entered into the database will be entered into the database in sequence.

7. The method according to any one of claims 1 to 6, characterized in that, After sorting all the data tables according to the comprehensive score to obtain the sorted data table, the method further includes: Based on the comprehensive score of the sorted data tables, all the sorted data tables are compressed to obtain compressed data tables, wherein the comprehensive score and the compression rate are negatively correlated. The compressed data tables are then stored in the database in sequence.

8. A data table entry device, characterized in that, include: The acquisition unit is used to acquire relevant information of the data table, wherein the relevant information includes one or more of the following: the number of calls, the urgency of the business requirement, the time when the business requirement was proposed, and the amount of data. A calculation unit is used to calculate a comprehensive score of the data table based on the relevant information, wherein the comprehensive score is a score indicating the importance of the data table, the number of calls is positively correlated with the comprehensive score, the urgency of the business requirement is positively correlated with the comprehensive score, the proximity of the time the business requirement was submitted to the current time is negatively correlated with the comprehensive score, and the amount of data is negatively correlated with the comprehensive score; A sorting unit is used to sort all the data tables according to the comprehensive score to obtain a sorted data table; The first data entry unit is used to enter all the sorted data tables into the database according to the order of the sorted data tables.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the data table insertion method according to any one of claims 1 to 7.

10. A data entry system, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a method for performing an input method for a data table as described in any one of claims 1 to 7.

Images