Data table processing method and device and computing equipment
By extracting the condition fields in the database operation statement and splitting the data table as the main and slave tables, the problems of low efficiency and high labor cost in the existing technology are solved, and automated splitting and efficient data operations are realized.
Patent Information
- Application Number
- CN202411150800.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-22
AI Technical Summary
In the prior art, data table splitting is inefficient and labor cost, which is mainly due to the reliance on manual experience for data table splitting processing.
By extracting database operation statements from the code file, identifying and recording condition fields, splitting the data table into the master and slave tables according to the condition fields, and generating corrected database operation statements to narrow the scope of data processing.
Automatic splitting of data tables is realized, the efficiency of data table splitting is improved, labor costs are saved, and data operation efficiency is improved.
Smart Images

Figure CN119003532A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data table processing method, device, computing equipment, computer storage medium and computer program product. Background Art
[0002] At present, databases usually store data in the form of data tables. As data continues to increase, data tables containing a large amount of data will appear in the database, referred to as large tables. Due to the low efficiency of reading and writing such data tables, it is usually necessary to split such data tables.
[0003] However, the inventors found that the prior art has at least the following defects during implementation: the prior art is to manually split the data table by relevant personnel based on their own experience. This data table processing method is inefficient and has high labor costs. Summary of the invention
[0004] In view of the above problems, the present application is proposed to provide a data table processing method, apparatus, computing device, computer storage medium and computer program product that overcome the above problems or at least partially solve the above problems.
[0005] According to a first aspect of the present application, a data table processing method is provided, comprising:
[0006] Extracting database operation statements for a target database from at least one code file;
[0007] Extracting condition fields from each of the database operation statements, and recording the extracted condition fields in a condition field set corresponding to the target database;
[0008] For any preset data table in the target database, a first field is identified from the fields included in the preset data table according to the condition field set, and fields other than the first field in the preset data table are used as second fields;
[0009] The field data corresponding to the first field in the preset data table is split into the main table corresponding to the preset data table, and the field data corresponding to the second field in the preset data table is split into the sub-table corresponding to the preset data table, so as to convert the original database operation statement for the preset data table into the modified database operation statement for the main table and / or the sub-table.
[0010] In an optional implementation, after recording the extracted condition fields in the condition field set corresponding to the target database, the method further includes: recording the occurrence frequency corresponding to each condition field;
[0011] The step of identifying the first field from the fields included in the preset data table according to the conditional field set includes:
[0012] Identify, from the preset data table, a candidate field that is consistent with a condition field in the condition field set;
[0013] The occurrence frequency corresponding to the candidate field of the record is searched, and the first field is screened out from the candidate fields according to the occurrence frequency.
[0014] In an optional implementation, the searching for the occurrence frequency corresponding to the candidate field of the record, and filtering out the first field from the candidate fields according to the occurrence frequency includes:
[0015] If the number of fields in the candidate fields exceeds a preset number threshold, filtering out a first field from the candidate fields according to the occurrence frequency;
[0016] If the field quantity of the candidate fields does not exceed the preset quantity threshold, each candidate field is used as the first field.
[0017] In an optional implementation, the method further includes: storing the master table in a memory and a disk; and storing the slave table in the disk.
[0018] In an optional implementation, after splitting the field data corresponding to the first field in the preset data table into the main table corresponding to the preset data table, and splitting the field data corresponding to the second field in the preset data table into the sub-table corresponding to the preset data table, the method further includes:
[0019] Obtaining the original table name of the preset data table, and generating the master table name of the master table and the slave table name of the slave table corresponding to the preset data table according to the original table name;
[0020] Creating index information of the main table according to the main table name of the main table and field information contained in the main table;
[0021] The index information of the slave table is created according to the slave table name of the slave table and the field information contained in the slave table.
[0022] In an optional implementation, the index information is recorded in an index component outside the target database.
[0023] In an optional implementation, converting the original database operation statement for the preset data table into the modified database operation statement for the master table and / or the slave table includes: receiving the sent original database operation statement;
[0024] Parsing the original database operation statement to obtain the original table name and condition field corresponding to the original database operation statement;
[0025] Searching for the generated index information, and determining the master table and the slave table that match the original table name;
[0026] Filtering out a target master table and / or a target slave table containing the conditional field from the matched master table and slave table;
[0027] A modified database operation statement is generated according to the target master table and / or target slave table and the condition field corresponding to the original database operation statement.
[0028] According to a second aspect of the present application, a data table processing device is provided, comprising:
[0029] A statement extraction module, used to extract database operation statements for a target database from at least one code file;
[0030] A field extraction module, used to extract condition fields from each of the database operation statements, and record the extracted condition fields in a condition field set corresponding to the target database;
[0031] an identification module, configured to identify, for any preset data table in the target database, a first field from the fields included in the preset data table according to the conditional field set, and use the fields other than the first field in the preset data table as the second field;
[0032] A splitting module is used to split the field data corresponding to the first field in the preset data table into the main table corresponding to the preset data table, and split the field data corresponding to the second field in the preset data table into the sub-table corresponding to the preset data table, so as to convert the original database operation statement for the preset data table into the modified database operation statement for the main table and / or the sub-table.
[0033] In an optional implementation, the field extraction module is further used to: record the occurrence frequency corresponding to each condition field;
[0034] The identification module is used to: identify candidate fields consistent with the condition fields in the condition field set from the preset data table;
[0035] The occurrence frequency corresponding to the candidate field of the record is searched, and the first field is screened out from the candidate fields according to the occurrence frequency.
[0036] In an optional implementation, the identification module is used to: if the number of fields of the candidate fields exceeds a preset number threshold, filter out the first field from the candidate fields according to the occurrence frequency;
[0037] If the field quantity of the candidate fields does not exceed the preset quantity threshold, each candidate field is used as the first field.
[0038] In an optional implementation, the master table is stored in a memory and a disk; and the slave table is stored in the disk.
[0039] In an optional embodiment, the device also includes: an index component, used to obtain the original table name of the preset data table, generate the main table name of the main table and the slave table name of the slave table corresponding to the preset data table based on the original table name; create the index information of the main table according to the main table name of the main table and the field information contained in the main table; create the index information of the slave table according to the slave table name of the slave table and the field information contained in the slave table.
[0040] In an optional implementation, the index component is used to: record the index information in an index component outside the target database.
[0041] In an optional embodiment, the device also includes: a statement parsing module, which is used to receive the original database operation statement sent; parse the original database operation statement to obtain the original table name and condition field corresponding to the original database operation statement; search the generated index information to determine the main table and the slave table that match the original table name; from the matching main table and slave table, filter out the target main table and / or target slave table containing the condition field; generate a modified database operation statement based on the target main table and / or target slave table, and the condition field corresponding to the original database operation statement.
[0042] According to a third aspect of the present application, there is provided a computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;
[0043] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned data table processing method.
[0044] According to a fourth aspect of the present application, a computer storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned data table processing method.
[0045] According to a fifth aspect of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned data table processing method.
[0046] The embodiment of the present application automatically splits the preset data table into a main table and a sub-table, converts the original database operation statement for the preset data table into a modified database operation statement for the main table and / or the sub-table, and reduces the data processing scope of the operation statement while ensuring the accuracy of the operation statement processing result, improves the statement processing efficiency, and can realize automatic splitting of the preset data table, improve the data table splitting efficiency and save labor costs; moreover, when splitting, the embodiment of the present application extracts the database operation statement from the code file associated with the target database, and extracts the condition field from the database operation statement, determines the first field in the preset data table according to the condition field, and splits the first field into the main table, thereby splitting the frequently operated fields into the same main table, avoiding frequent aggregation of the main table and the sub-table, and further improving the data operation efficiency.
[0047] The embodiment of the present application filters out the first field according to the frequency of occurrence of the field, so that the fields with high operation frequency can be split into the main table, further avoiding frequent aggregation of the main table and the sub-table, and improving data operation efficiency.
[0048] The embodiment of the present application uses a preset quantity threshold to regulate the number of first fields, ensuring that the fields included in the split sub-table do not exceed the preset quantity threshold, avoiding excessive data volume in the split sub-table, and improving the operation efficiency of the data table.
[0049] The embodiment of the present application stores the master table in the memory and stores the slave table in the disk. On the one hand, the operation efficiency of the data in the master table is improved, and on the other hand, storing the slave table in the disk can save memory resources.
[0050] After the preset data table is split into a master table and a slave table, the embodiment of the present application assigns corresponding table names to the split master table and slave table according to the original table name of the preset data table, thereby facilitating subsequent data operations on the master table and the slave table. The embodiment of the present application creates corresponding index information according to the table name and the field information contained therein, thereby being able to quickly operate the data in the data table according to the index information.
[0051] The embodiment of the present application records the index information in an index component outside the database to implement external indexing and improve the overall data operation efficiency.
[0052] The embodiment of the present application determines the target master table and / or target slave table involved in the original database operation statement by parsing the original database operation statement, and obtains a modified database operation statement based on the target master table and / or target slave table and the corresponding conditional field, so that the modified database operation statement only operates the corresponding target master table and / or target slave table, and other sub-tables not related to the original database operation statement do not participate in this operation, thereby improving database operation efficiency and saving system resources.
[0053] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0055] Figure 1 A schematic diagram of an operating environment provided for implementing at least one embodiment of the present application is shown;
[0056] Figure 2 A schematic diagram of a process flow of a data table processing method provided in Embodiment 1 of the present application is shown;
[0057] Figure 3 A schematic diagram of a process flow of a data table processing method provided in Embodiment 2 of the present application is shown;
[0058] Figure 4 A schematic diagram of a process flow of a data table processing method provided in Embodiment 3 of the present application is shown;
[0059] Figure 5 A schematic diagram of a request processing method provided in Embodiment 3 of the present application is shown;
[0060] Figure 6 A schematic diagram of the structure of a data table processing device provided in Embodiment 4 of the present application is shown;
[0061] Figure 7 A schematic diagram of the structure of a computing device provided in Embodiment 5 of the present application is shown. DETAILED DESCRIPTION
[0062] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.
[0063] Figure 1 The present invention is applicable to an application environment including, but not limited to, a client 2 , a server 4 , and a network 6 .
[0064] in:
[0065] The server 4 may be composed of a single or multiple computing devices. The multiple computing devices may include virtualized computing instances. Virtualized computing instances may include virtual machines, such as simulations of computer systems, operating systems, servers, etc. The computing device may load a virtual machine based on a virtual image and / or other data defining specific software (e.g., operating system, dedicated application, server) for simulation. As the demand for different types of processing services changes, different virtual machines may be loaded and / or terminated on one or more computing devices. A hypervisor may be implemented to manage the use of different virtual machines on the same computing device.
[0066] The server 4 may be configured to communicate with the client 2, etc., via a network 6. The network 6 includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or the like. The network 6 may include physical links, such as coaxial cable links, twisted pair cable links, optical fiber links, combinations thereof, etc., or wireless links, such as cellular links, satellite links, Wi-Fi links, etc.
[0067] The server 4 can provide storage, reading, downloading, writing, querying, deleting and other services, such as providing static resource download services to clients through multiple domain names.
[0068] Client 2 can be running Windows, Android TM ) or IOS and other operating systems, such as smart phones, tablet devices, laptop computers, virtual reality devices, gaming devices, set-top boxes, car terminals, smart TVs. Based on the above operating systems, various applications, such as browsers, can be run.
[0069] Embodiment 1
[0070] Figure 2 The flowchart of a data table processing method provided in the first embodiment of the present application is shown. The data table processing method provided in the embodiment of the present application is executed on the server side.
[0071] like Figure 2 As shown, the method specifically comprises the following steps:
[0072] Step S210: extracting database operation statements for a target database from at least one code file.
[0073] The target database is the database to be optimized for data table splitting. Obtain the code file associated with the target database, and the associated code file involves operations on the target database. Further extract database operation statements for the target database from the code file, and the database operation statements include but are not limited to: query statements, modification statements, new statements, deletion statements, etc. For example, in the actual implementation process, the project source code associated with the target database can be extracted, and various database operation files (such as mapper files) can be extracted from the project source code, and database operation statements such as SQL statements can be extracted from the database operation files.
[0074] Step S220, extracting condition fields from each database operation statement, and recording the extracted condition fields in a condition field set corresponding to the target database.
[0075] For any extracted database operation statement, the database operation statement is parsed to extract the conditional fields contained in the database operation statement. The present application does not limit the specific extraction method. For example, a regular expression can be used to extract the various conditional fields in the database operation statement. Taking the database operation statement as an SQL statement as an example, a regular expression (such as a / / where / clause, etc.) can be used to extract the conditional fields in the SQL statement according to the grammatical rules of the SQL statement.
[0076] Each condition field extracted from the database operation statement corresponding to the target database is recorded in the condition field set corresponding to the target database. For example, for any code file, each condition field is extracted through steps S210-S220 to generate a condition field list corresponding to the code file, and the condition field lists of each code file associated with the target database are merged to generate a condition field summary table corresponding to the target database. The condition field summary table is the condition field set corresponding to the target database. In the actual implementation process, the field identifier (such as field name or field ID, etc.) of each condition field can be recorded in the condition field set.
[0077] In an optional implementation, the occurrence frequency of each condition field is further counted, and the occurrence frequency is the number of times the condition field appears in each database operation statement, and the occurrence frequency of each condition field is recorded. It is also possible to add each condition field to the condition field set in order of the occurrence frequency of the condition field from high to low, or assign a bit sequence identifier to each condition field.
[0078] In an optional implementation, corresponding weights can also be assigned to each database operation statement according to the priority of the business or function to which the database operation statement belongs. Wherein, if the database operation statement is extracted from the code file of a certain business or function, then the business or function is the business or function to which the database operation statement belongs. Specifically, the priority of the business or function can be determined according to the business or function level (for example, the business can be divided into multiple levels such as primary, secondary, and tertiary levels according to whether the business is a core business), the degree of dependence of the business or function (for example, if a certain business or function is called by multiple other businesses or functions, then its degree of dependence is high), etc., and the priority is positively correlated to the function level and the degree of dependence. After determining the priority of the business or function, the weight of the database operation statement extracted from the code file of the business or function is determined according to the priority, and the weight is positively correlated to the priority of the business or function to which it belongs.
[0079] Further, the importance coefficient of the condition field extracted from the database operation statement is determined according to the weight of the database operation statement, and the importance coefficient of the condition field is positively correlated with the weight of the database operation statement from which it comes. In the specific implementation process, each time a condition field is extracted from a database operation statement, an importance coefficient is assigned to the extracted condition field according to the weight of the database operation statement. Since the same condition field can exist in different database operation statements, the same condition field can have multiple important coefficients, and the number of important coefficients assigned to the condition field is consistent with the frequency of occurrence of the condition field. Moreover, the weights of multiple database operation statements from the same condition field source may be different. When the condition field corresponds to multiple important coefficients, the multiple important coefficients may be the same or different. After obtaining the important coefficients of each condition field, for any condition field, all the important coefficients of the condition field are accumulated to obtain the weight of the condition field.
[0080] Step S230: for any preset data table in the target database, a first field is identified from the fields included in the preset data table according to the conditional field set, and fields other than the first field in the preset data table are used as second fields.
[0081] At least one data table is stored in the target database, and a preset data table is screened out from the data table. The amount of data in the preset data table exceeds the preset data amount threshold, so that the preset data table is a data table with a large amount of data. In order to improve the read and write rate of the preset data table, the present application splits the preset data table.
[0082] In the splitting process, firstly, the first field is identified from the fields included in the preset data table according to the conditional field set of the target database. For any first field in the preset data table, a conditional field consistent with the field identifier of the first field can be found in the conditional field set. After the first field is determined, the other fields in the preset data table other than the first field are used as the second field.
[0083] In an optional implementation, in order to improve the rationality of data table splitting and improve the efficiency of data operations after data table splitting, the first field can be determined in the following manner: identify candidate fields that are consistent with the conditional fields in the conditional field set from the preset data table, that is, the candidate fields are the fields in the preset data table that are consistent with the field identifiers of the conditional fields. Further search for the occurrence frequency corresponding to the candidate fields of the record, and filter out the first field from the candidate fields based on the occurrence frequency. For example, the candidate fields can be sorted in descending order according to the occurrence frequency corresponding to the candidate fields, and the first M candidate fields can be selected as the first field, that is, the first field is the field with the highest occurrence frequency among the candidate fields.
[0084] Further optionally, in order to ensure data reading and writing efficiency, the present application sets a corresponding quantity threshold (i.e., a preset quantity threshold) for the main table. After screening out the candidate fields, it is determined whether the number of fields of the candidate fields exceeds the preset quantity threshold. If the number of fields of the candidate fields exceeds the preset quantity threshold, the first field is screened out from the candidate fields according to the frequency of occurrence, that is, the candidate fields are sorted in order from high to low according to the corresponding frequency of occurrence of the candidate fields, and the first M (M is the preset quantity threshold) candidate fields are selected as the first field; if the number of fields of the candidate fields does not exceed the preset quantity threshold, each candidate field is used as the first field.
[0085] In another optional implementation, after the candidate fields are determined, the weights of the candidate fields are determined according to the weights of the condition fields of the record. The first field is further selected from the candidate fields according to the weights of the candidate fields. For example, the candidate fields may be sorted in descending order according to the weights of the candidate fields, and the first M candidate fields are selected as the first field.
[0086] Step S240, split the field data corresponding to the first field in the preset data table into the main table corresponding to the preset data table, and split the field data corresponding to the second field in the preset data table into the sub-table corresponding to the preset data table, so as to convert the original database operation statement for the preset data table into the modified database operation statement for the main table and / or the sub-table.
[0087] The preset data table is split into a main table and at least one slave table, so as to realize the splitting of the preset data table and improve the data operation efficiency. Moreover, since the first field is derived from the database operation statement in the code file, the first field is a frequently operated field in the preset data table, and the frequently operated field is uniformly split into the main table, and there is no need to frequently aggregate the main table and the slave table when the data table is subsequently operated, thereby improving the data operation efficiency. After the field data corresponding to the first field is split into the main table, the second field is sequentially split into at least one slave table. If the number of the second field exceeds the threshold of the number of slave table fields, the field that has not been split is split into another slave table, and so on, until all the second fields are split into the corresponding slave tables. The main table and the sub-table can both be called the sub-table of the preset data table. Thus, the original database operation statement originally for the preset data table can be converted into a modified database operation statement for the main table and / or the slave table. Since the main table and the slave table are obtained by splitting from the preset data table, the modified database operation statement and the original database operation statement can obtain the same processing result. Furthermore, since the modified database operation statement is for the master table and / or slave table, and the master table and the slave table are only a part of the preset data table, the data processing scope of the modified database operation statement can be narrowed and the statement processing efficiency can be improved.
[0088] In an optional implementation, in order to further improve the data operation efficiency, the master table is stored in the memory and the disk, and the slave table is stored in the disk. Storing the master table in the memory can improve the operation efficiency of the master table, storing the master table in the disk can also ensure the accuracy of data storage, and storing the slave table in the disk can reduce memory overhead.
[0089] It can be seen that the data table processing method provided in the embodiment of the present application can split the preset data table into a main table and a sub-table, and convert the original database operation statement for the preset data table into a modified database operation statement for the main table and / or the sub-table. While ensuring the accuracy of the operation statement processing results, the data processing scope of the operation statement is narrowed, the statement processing efficiency is improved, and the automatic splitting of the preset data table can be realized, thereby improving the data table splitting efficiency and saving labor costs. Moreover, when splitting, the embodiment of the present application extracts the database operation statement from the code file associated with the target database, and extracts the conditional field from the database operation statement, determines the first field in the preset data table according to the conditional field, and splits the first field into the main table, thereby splitting the frequently operated fields into the same main table, avoiding frequent aggregation of the main table and the sub-table, and further improving the data operation efficiency.
[0090] Embodiment 2
[0091] Figure 3The flowchart of a data table processing method provided in the second embodiment of the present application is shown. The data table processing method provided in the embodiment of the present application is executed on the server side.
[0092] like Figure 3 As shown, the method specifically comprises the following steps:
[0093] Step S310: for any preset data table of the target database, split the preset data table into a main table and at least one subordinate table.
[0094] The specific splitting process can be referred to the description in other embodiments and will not be described in detail here.
[0095] Step S320: for any preset data table, obtain the original table name of the preset data table, and generate the main table name of the main table and the slave table name of the slave table corresponding to the preset data table according to the original table name.
[0096] The table name of the preset data table is the original table name. In order to improve the accuracy of data operations, the master table name of the master table corresponding to the preset data table and the slave table name of the slave table are generated according to the original table name in this embodiment. That is, the master table name of the master table and the slave table name of the slave table obtained after the preset data table is split carry the original table name information of the preset data table. For example, the master table identifier can be added after the original table name to obtain the master table name, and the slave table identifier can be added after the original table name to obtain the slave table name. For example, after the preset data table Table1 is split, a master table and two slave tables are obtained. The master table name of the master table can be Table1_P, and the slave table names of the two slave tables are Table1_F1 and Table1_F2 respectively. Therefore, the original table from which the master table and the slave table come can be quickly determined through the master table name and the slave table name after the split.
[0097] Step S330: create index information of the main table according to the main table name and field information contained in the main table.
[0098] Create index information of the main table, the index information includes the main table name of the main table and field information of each field included in the main table. In addition, the index information may also include information such as the storage address of the main table.
[0099] Step S340: Create index information of the slave table according to the slave table name and field information contained in the slave table.
[0100] Create index information of the slave table, the index information includes the slave table name of the slave table and field information of each field included in the slave table. In addition, the index information may also include information such as the storage address of the slave table.
[0101] In an optional implementation, in order to improve data operation performance, each piece of index information may be recorded in an index component outside the target database. For example, each piece of generated index information may be recorded in an ES (ElasticSearch) component to implement an external index of the target database.
[0102] In an optional implementation, a change operation on a master table and / or a slave table is detected, where the change operation includes a table name modification operation, a field identifier modification operation, a field deletion operation, a field addition operation, etc. After the change operation is executed in the database, the field information and / or the table name in the index information corresponding to the master table and / or the slave table is modified to ensure synchronization of the index data with the data in the database.
[0103] In an optional implementation, for an unsplit non-preset data table, index information corresponding to the non-preset data table can also be created based on the table name of the non-preset data table and the field information contained therein, so that the corresponding index information can be uniformly queried when a database operation statement is subsequently executed.
[0104] It can be seen that the data table processing method provided in the embodiment of the present application, after splitting the preset data table into a main table and a sub-table, assigns corresponding table names to the split main table and sub-table according to the original table name of the preset data table, thereby facilitating subsequent data operations on the main table and the sub-table; and, creates corresponding index information based on the table name and the field information contained therein, so that the data in the data table can be quickly operated according to the index information.
[0105] Embodiment 3
[0106] Figure 4 The flowchart of a data table processing method provided in the third embodiment of the present application is shown. The data table processing method provided in the embodiment of the present application is executed on the server side.
[0107] like Figure 4 As shown, the method specifically comprises the following steps:
[0108] Step S410: receiving the sent original database operation statement.
[0109] The database operation statement generated according to the request is the original database operation statement.
[0110] Step S420, parsing the original database operation statement to obtain the original table name and condition field corresponding to the original database operation statement.
[0111] The original database operation statement is parsed, and the table name of the data table corresponding to the original database operation statement is extracted from the original database operation statement, and the table name is the original table name. And the corresponding condition field is extracted from the original database operation statement.
[0112] Step S430, searching the generated index information, and determining the main table and the subordinate table that match the original table name.
[0113] As can be seen from the second embodiment, the master table name and the slave table name are generated based on the original table name before splitting, and the index information of the master table and the slave table includes the master table name and the slave table name, so the master table and the slave table matching the original table name can be determined by searching the generated index information. For example, if a certain index information includes the original table name, the data table corresponding to the index information is used as the master table or slave table matching the original table name.
[0114] In addition, if the original table name corresponds to a non-preset data table, there is no need to execute this step and the corresponding data table can be directly operated in the target database according to the original database operation statement.
[0115] Step S440: Filter out the target master table and / or target slave table containing the condition field from the matched master table and slave table.
[0116] The matched master table and slave table are further screened, and only the master table or slave table containing the condition field corresponding to the original database operation statement is screened out. The screened out master table is called the target master table, and the screened out slave table is called the target slave table. The sub-tables that do not contain the condition field parsed in step S420 are filtered out, and the target master table and / or target slave table thus screened out are subsets of the preset data table corresponding to the original table name.
[0117] In an optional implementation, in order to improve search efficiency, in this implementation, the conditional fields parsed in step S420 are preferentially matched with the fields included in the target master table. If all the parsed conditional fields are located in the fields included in the target master table, there is no need to compare the slave table fields. If not all the parsed conditional fields are located in the fields included in the target master table, then the slave table fields are compared.
[0118] Step S450: Generate a modified database operation statement according to the target master table and / or the target slave table and the condition field of the original database operation statement.
[0119] In the specific implementation process, if step S440 only screens out one sub-table (i.e., one target master table or one target slave table), the original table name in the original database operation statement is replaced with the target master table or the target slave table to generate the corresponding modified database operation statement; if step S440 screens out multiple sub-tables, for each screened sub-table, the conditional field parsed by step S420 contained in the sub-table is determined, and the modified database operation statement is obtained by using an aggregate query, for example, for each screened sub-table, an operation sub-statement is generated according to the sub-table name and the conditional field in step S420 contained in the sub-table, and each operation sub-statement is aggregated to obtain the modified database operation statement. For example, the operation sub-statements can be connected by JOIN during the aggregation process. After obtaining the modified database operation statement, the modified database operation statement is used to perform database operations.
[0120] The following Figure 5 For example, the processing process of the user request of this application is explained in an exemplary manner: the client sends the user request to the server, the business processing module in the server processes the user request and generates the original SQL statement (original database operation statement), the statement parsing module in the server parses the original SQL statement to determine the original table name and condition field corresponding to the original SQL statement, and queries the index information in the index component to determine the sub-table and field (i.e., the target main table and / or the target sub-table) that matches the original table name and condition field, generates a modified SQL statement (modified database operation statement) based on the determined sub-table and field, and uses the modified SQL statement to operate the database. And obtain the return data, the return data passes through the database → statement parsing module → business processing module → client, and then is returned to the client.
[0121] It can be seen that the data table processing method provided in the embodiment of the present application determines the target main table and / or target sub-table involved in the original database operation statement by parsing the original database operation statement, and obtains the modified database operation statement according to the target main table and / or target sub-table and the corresponding conditional field, so that the modified database operation statement only operates the corresponding target main table and / or target sub-table, and other sub-tables not related to the original database operation statement do not participate in this operation, thereby improving database operation efficiency and saving system resources.
[0122] Embodiment 4
[0123] Figure 6 FIG. 4 shows a schematic diagram of the structure of a data table processing device provided in Embodiment 4 of the present application. Figure 6 As shown, the device 600 includes: a sentence extraction module 610, a field extraction module 620, a recognition module 630, and a splitting module 640.
[0124] A statement extraction module 610 is used to extract database operation statements for a target database from at least one code file;
[0125] A field extraction module 620 is used to extract condition fields from each of the database operation statements, and record the extracted condition fields in a condition field set corresponding to the target database;
[0126] An identification module 630 is used to identify, for any preset data table in the target database, a first field from the fields included in the preset data table according to the condition field set, and use the fields other than the first field in the preset data table as the second field;
[0127] The splitting module 640 is used to split the field data corresponding to the first field in the preset data table into the main table corresponding to the preset data table, and split the field data corresponding to the second field in the preset data table into the sub-table corresponding to the preset data table, so as to convert the original database operation statement for the preset data table into the modified database operation statement for the main table and / or the sub-table.
[0128] In an optional implementation, the field extraction module 620 is further used to: record the occurrence frequency corresponding to each condition field;
[0129] The identification module 630 is used to: identify a candidate field consistent with a condition field in the condition field set from the preset data table;
[0130] The occurrence frequency corresponding to the candidate field of the record is searched, and the first field is screened out from the candidate fields according to the occurrence frequency.
[0131] In an optional implementation, the identification module 630 is configured to: if the number of fields of the candidate fields exceeds a preset number threshold, filter out the first field from the candidate fields according to the occurrence frequency;
[0132] If the field quantity of the candidate fields does not exceed the preset quantity threshold, each candidate field is used as the first field.
[0133] In an optional implementation, the master table is stored in a memory and a disk; and the slave table is stored in the disk.
[0134] In an optional embodiment, the device also includes: an index component (not shown in the figure), which is used to obtain the original table name of the preset data table, and generate the main table name of the main table and the slave table name of the slave table corresponding to the preset data table based on the original table name; create the index information of the main table according to the main table name of the main table and the field information contained in the main table; create the index information of the slave table according to the slave table name of the slave table and the field information contained in the slave table.
[0135] In an optional implementation, the index component is used to: record the index information in an index component outside the target database.
[0136] In an optional embodiment, the device also includes: a statement parsing module (not shown in the figure), which is used to receive the original database operation statement sent; parse the original database operation statement to obtain the original table name and condition field corresponding to the original database operation statement; search the generated index information to determine the main table and the slave table that match the original table name; from the matching main table and slave table, filter out the target main table and / or target slave table containing the condition field; generate a modified database operation statement based on the target main table and / or target slave table, and the condition field corresponding to the original database operation statement.
[0137] It can be seen that the data table processing device provided by the embodiment of the present application can split the preset data table into a main table and a slave table, and convert the original database operation statement for the preset data table into a modified database operation statement for the main table and / or the slave table. While ensuring the accuracy of the operation statement processing results, the data processing scope of the operation statement is reduced, the statement processing efficiency is improved, and the automatic splitting of the preset data table can be realized, thereby improving the data table splitting efficiency and saving labor costs. Moreover, when splitting, the embodiment of the present application extracts the database operation statement from the code file associated with the target database, and extracts the conditional field from the database operation statement, determines the first field in the preset data table according to the conditional field, and splits the first field into the main table, thereby splitting the frequently operated fields into the same main table, avoiding frequent aggregation of the main table and the slave table, and further improving the data operation efficiency.
[0138] Embodiment 5
[0139] Figure 7 A schematic diagram of the structure of an embodiment of a computing device of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the computing device.
[0140] like Figure 7As shown, the computing device may include: a processor (processor) 702 , a communications interface (Communications Interface) 704 , a memory (memory) 706 , and a communication bus 708 .
[0141] The processor 702, the communication interface 704, and the memory 706 communicate with each other via a communication bus 708. The communication interface 704 is used to communicate with other devices such as a client or other server network elements. The processor 702 is used to execute a program 710, which can specifically execute the relevant steps in the above-mentioned data table processing method embodiment for a computing device.
[0142] Specifically, the program 710 may include program codes, which include computer operation instructions.
[0143] The processor 702 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.
[0144] The memory 706 is used to store the program 710. The memory 706 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The program 710 may be specifically used to enable the processor 702 to perform the operations in the above method embodiment.
[0145] Embodiment 6
[0146] Embodiment 6 of the present application provides a non-volatile computer storage medium, wherein the computer storage medium stores at least one executable instruction or computer program, and the executable instruction or computer program enables a processor to execute operations corresponding to the data table processing method in any of the above method embodiments.
[0147] Embodiment 7
[0148] Embodiment 7 of the present application provides a computer program product, which includes at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to execute operations corresponding to the data table processing method in any of the above method embodiments.
[0149] In summary, according to the computing device, computer storage medium and computer program product provided by the present embodiment, the preset data table is split into a main table and a sub-table, and the original database operation statement for the preset data table is converted into a modified database operation statement for the main table and / or the sub-table. While ensuring the accuracy of the operation statement processing result, the data processing scope of the operation statement is narrowed, the statement processing efficiency is improved, and the preset data table can be automatically split, the data table splitting efficiency is improved, and labor costs are saved; moreover, when splitting, the embodiment of the present application extracts the database operation statement from the code file associated with the target database, and extracts the conditional field from the database operation statement, determines the first field in the preset data table according to the conditional field, and splits the first field into the main table, thereby the frequently operated fields can be split into the same main table, avoiding frequent aggregation of the main table and the sub-table, and further improving the data operation efficiency.
[0150] The algorithm or display provided here are not inherently related to any specific computer, virtual system or other equipment. Various general systems can also be used together with the teaching based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages can be utilized to realize the content of the present application described here, and the above description of specific languages is to disclose the best mode of implementation of the present application.
[0151] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.
[0152] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.
[0153] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.
[0154] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.
[0155] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components according to the embodiments of the present application. The application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing a part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.
[0156] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be constructed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.
Claims
1. A data table processing method, characterized in that: include: Extracting database operation statements for a target database from at least one code file; Extracting condition fields from each of the database operation statements, and recording the extracted condition fields in a condition field set corresponding to the target database; For any preset data table in the target database, a first field is identified from the fields included in the preset data table according to the condition field set, and the fields other than the first field in the preset data table are used as second fields; The field data corresponding to the first field in the preset data table is split into the main table corresponding to the preset data table, and the field data corresponding to the second field in the preset data table is split into the sub-table corresponding to the preset data table, so as to convert the original database operation statement for the preset data table into the modified database operation statement for the main table and / or the sub-table.
2. The method according to claim 1, characterized in that After recording the extracted condition fields in the condition field set corresponding to the target database, the method further includes: recording the occurrence frequency corresponding to each condition field; The step of identifying the first field from the fields included in the preset data table according to the conditional field set includes: Identify, from the preset data table, a candidate field that is consistent with a condition field in the condition field set; The occurrence frequency corresponding to the candidate field of the record is searched, and the first field is screened out from the candidate fields according to the occurrence frequency.
3. The method according to claim 2, characterized in that The searching for the occurrence frequency corresponding to the candidate field of the record, and filtering out the first field from the candidate fields according to the occurrence frequency comprises: If the number of fields in the candidate fields exceeds a preset number threshold, filtering out a first field from the candidate fields according to the occurrence frequency; If the field quantity of the candidate fields does not exceed the preset quantity threshold, each candidate field is used as the first field.
4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: The master table is stored in memory and on disk; and the slave table is stored on the disk.
5. The method according to any one of claims 1 to 4, characterized in that After the field data corresponding to the first field in the preset data table is split into the main table corresponding to the preset data table, and the field data corresponding to the second field in the preset data table is split into the sub-table corresponding to the preset data table, the method further includes: Obtaining the original table name of the preset data table, and generating the master table name of the master table and the slave table name of the slave table corresponding to the preset data table according to the original table name; Creating index information of the main table according to the main table name of the main table and field information contained in the main table; The index information of the slave table is created according to the slave table name of the slave table and the field information contained in the slave table.
6. The method according to claim 5, characterized in that The index information is recorded in an index component outside the target database.
7. The method according to claim 5 or 6, characterized in that: The converting of the original database operation statement for the preset data table into the modified database operation statement for the master table and / or the slave table comprises: Receive the original database operation statement sent; Parsing the original database operation statement to obtain the original table name and condition field corresponding to the original database operation statement; Searching for the generated index information, and determining the master table and the slave table that match the original table name; Filtering out a target master table and / or a target slave table containing the conditional field from the matched master table and slave table; A modified database operation statement is generated according to the target master table and / or target slave table and the condition field corresponding to the original database operation statement.
8. A data table processing device, characterized in that: include: A statement extraction module, used to extract database operation statements for a target database from at least one code file; A field extraction module, used to extract condition fields from each of the database operation statements, and record the extracted condition fields in a condition field set corresponding to the target database; an identification module, configured to identify, for any preset data table in the target database, a first field from the fields included in the preset data table according to the conditional field set, and use the fields other than the first field in the preset data table as the second field; A splitting module is used to split the field data corresponding to the first field in the preset data table into the main table corresponding to the preset data table, and split the field data corresponding to the second field in the preset data table into the sub-table corresponding to the preset data table, so as to convert the original database operation statement for the preset data table into the modified database operation statement for the main table and / or the sub-table.
9. A computing device, characterized in that include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the data table processing method according to any one of claims 1 to 7.
10. A computer storage medium, characterized in that: The storage medium stores at least one executable instruction, and the executable instruction enables the processor to execute an operation corresponding to the data table processing method according to any one of claims 1 to 7.
11. A computer program product, characterized in that It includes at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the data table processing method according to any one of claims 1 to 7.
