A data table processing method and device, computer equipment and a storage medium

Abstract

The application provides a data table processing method and device, computer equipment and a storage medium. The processing method is applied to a data table processing system. The data table processing system comprises an application program, a database and a mapping layer arranged between the application program and the database. The mapping layer is used for mutual access of the application program and the database. The data table processing method comprises the following steps: when the application program receives an access request, a mapping model corresponding to the access request is queried in the mapping layer; according to the query result, the database is accessed, data operation is performed, and data information is obtained. The mapping model is used for storing a mapping relationship between a data table of the application program and an entity table of the database. The preset splitting standard and the data information returned by the database are obtained in the mapping layer. The data table operation is performed according to the preset splitting standard and the data information returned by the database. Automatic splitting is realized, and time-consuming data table request is avoided.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, and more specifically to a data table processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the continuous development of internet technology, almost every project needs to utilize databases to store relevant business data tables. As projects grow and the number of users increases, the access volume of certain popular functions also increases, leading to a daily increase in the amount of data in the corresponding business tables.

[0003] However, when the data volume in the business form reaches a bottleneck, related database operations will slow down. Typically, database sharding and table partitioning are required at this point, but these operations necessitate changes to the application, making them impossible to automate or respond in real time. Summary of the Invention

[0004] The problem solved by this invention is that the request time is slow due to the increase in data volume in the business table in the database.

[0005] To address the above problems, this invention provides a data table processing method applied to a data table processing system. The data table processing system includes: an application program, a database, and a mapping layer configured between the application program and the database. The mapping layer is used to access both the application program and the database. The data table processing method includes:

[0006] When the application receives an access request, it queries the mapping layer for the mapping model corresponding to the access request.

[0007] Access the database based on the query results, perform data operations, and obtain data information. The mapping model is used to store the mapping relationship between the data tables of the application and the entity tables of the database.

[0008] In the mapping layer, a preset splitting standard and the data information returned by the database are obtained, and a table splitting operation is performed according to the preset splitting standard and the data information returned by the database.

[0009] Preferably, the step of accessing the database based on the query results, performing data operations, and obtaining data information includes:

[0010] If the mapping model corresponding to the access request is not found, an initial operation statement is obtained, the database is accessed according to the initial operation statement, the data operation is performed, and the data information is obtained.

[0011] If the mapping model corresponding to the access request is found, the mapping relationship between the data table and the entity table is obtained, the current operation statement is obtained according to the mapping relationship, the database is accessed according to the current operation statement, the data operation is performed, and the data information is obtained.

[0012] Preferably, the step of performing table partitioning based on the preset partitioning criteria and the data information returned by the database includes:

[0013] When the data information returned by the database meets the preset splitting criteria,

[0014] In the mapping layer, the splitting conditions are obtained, and the table splitting operation is performed according to the splitting conditions to obtain the current mapping model. The current mapping model is used to store the mapping relationship between the data tables of the application and the entity tables of the database after the table splitting operation.

[0015] Obtain the current mapping relationship between the data table and the entity table in the current mapping model, obtain the table partitioning operation statement based on the current mapping relationship, access the database based on the table partitioning operation statement, perform the data operation, and obtain the data information.

[0016] Preferably, the data information returned by the database includes the amount of data returned and the time taken to return the data, and the data information returned by the database satisfies the preset splitting criteria, including:

[0017] When the ratio of the return time to the return data volume is greater than the preset splitting standard, it is determined that the data information returned by the database meets the preset splitting standard.

[0018] Preferably, the splitting conditions include a splitting statement and a splitting time point. The table partitioning operation is performed according to the splitting conditions to obtain the current mapping model, including:

[0019] The current mapping model is obtained by performing table partitioning operations based on the partitioning statement and the partitioning time point.

[0020] Preferably, the data information returned by the database also includes business information, and the data table processing method further includes: returning the business information from the database to the application through the mapping layer.

[0021] Preferably, the data table processing method further includes:

[0022] Obtain the preset merging table standard in the mapping layer;

[0023] Initiate a table merging operation request in the application according to the preset table merging criteria;

[0024] Based on the table merging operation request, retrieve the mapping relationship between the application's data tables and the database's entity tables after the table splitting operation in the current mapping model;

[0025] The table merge operation statement and the split table after the table split operation are obtained in the mapping layer, and the table merge operation is performed according to the split table and the table merge operation statement.

[0026] Compared to existing technologies, this invention establishes a mapping layer between the database and the application that allows mutual access. Upon receiving a data access request, the mapping layer first queries whether a mapping model corresponding to the data access request exists. Based on the query result, the database is accessed, thus accelerating data access speed. When the mapping layer receives data information returned by the database, it performs table partitioning operations according to the preset partitioning criteria and the data information returned by the database. This achieves automatic table partitioning, avoiding the problem of slow request times caused by increased data volume in the database. Furthermore, this invention does not require modification of the database application, making operation simpler.

[0027] To address the above problems, the present invention also provides a data table processing apparatus applied to a data table processing system. The data table processing system includes: an application program, a database, and a mapping layer disposed between the application program and the database. The mapping layer is used to access both the application program and the database. The data table processing apparatus includes:

[0028] A query module is used when the application receives an access request.

[0029] In the mapping layer, the mapping model corresponding to the access request is queried. Based on the query result, the database is accessed, data operations are performed, and data information is obtained. The mapping model is used to store the mapping relationship between the data tables of the application and the entity tables of the database.

[0030] The acquisition module is used to acquire preset splitting criteria and data information returned by the database in the mapping layer.

[0031] The table partitioning module is used to perform table partitioning operations based on the preset partitioning criteria and the data information returned by the database.

[0032] The data table processing device and the data table processing method described in this invention have the same advantages over the prior art, and will not be repeated here.

[0033] To address the above problems, the present invention also provides a computer device, including a memory and a processor:

[0034] The memory is used to store computer programs;

[0035] The processor is configured to implement the data table processing method described above when executing the computer program.

[0036] The data table processing system and method described in this invention have the same advantages over the prior art, and will not be repeated here.

[0037] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when read and executed by a processor, implements the data table processing method described above.

[0038] The computer-readable storage medium and the data table processing method described in this invention have the same advantages over the prior art, and will not be repeated here. Attached Figure Description

[0039] Figure 1 This is a structural diagram of the data table processing system in an embodiment of the present invention;

[0040] Figure 2 This is a flowchart of the data table processing method in an embodiment of the present invention;

[0041] Figure 3 This is a structural diagram of the data table processing device in an embodiment of the present invention;

[0042] Figure 4 This is an internal structural diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0043] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0044] The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of embodiments of the invention includes the entire scope of the claims, as well as all available equivalents of the claims. In this document, each embodiment may be referred to individually or collectively with the term "invention," which is merely for convenience and is not intended to automatically limit the scope of the application to any single invention or inventive concept if more than one invention is disclosed.

[0045] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without requiring or implying any actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For structures, products, etc., disclosed in the embodiments, the descriptions are relatively simple because they correspond to the disclosed parts; relevant details can be found in the method section.

[0046] See Figure 1 and Figure 2 As shown, this embodiment of the invention provides a data table processing method applied to a data table processing system. The data table processing system includes: an application program 110, a database 120, and a mapping layer 130 disposed between the application program 110 and the database 120. The mapping layer 130 is used to access the application program 110 and the database 120 respectively. The data table processing method includes:

[0047] Step S1: When the application receives an access request,

[0048] In the mapping layer, the mapping model corresponding to the access request is queried. Based on the query results, the database is accessed, data operations are performed, and data information is obtained. The mapping model is used to store the mapping relationship between the application's data tables and the database's entity tables.

[0049] Step S2: Obtain the preset splitting criteria and the data information returned by the database in the mapping layer, and perform table splitting operation according to the preset splitting criteria and the data information returned by the database.

[0050] In some specific embodiments, the mapping layer also includes a user-defined management table. The management table is used to record the table name of the split entity table, the preset split criteria, the split conditions, the table splitting operation statement, the table merging operation statement, the preset table merging criteria, the amount of data returned, and the return time.

[0051] This embodiment establishes a mapping layer between the database and the application that allows mutual access. Upon receiving a data access request, the mapping layer first checks if a mapping model corresponding to the data access request exists. Based on the query result, it accesses the database, thus accelerating data access speed. The mapping layer receives the data information returned by the database, sends the entity tables returned by the database to the application, and simultaneously performs table partitioning operations according to the preset partitioning criteria and the data information returned by the database. This achieves automatic table partitioning, avoiding the problem of slow request times caused by the increase in data volume in the database. Furthermore, it eliminates the need to modify the database application, making the operation simpler.

[0052] In some preferred embodiments, based on the query results, the database is accessed, data operations are performed, and data information is obtained, including: if no mapping model corresponding to the access request is found, an initial operation statement is obtained, the database is accessed based on the initial operation statement, data operations are performed, and data information is obtained;

[0053] If a mapping model corresponding to the access request is found, the mapping relationship between the data table and the entity table in the mapping model is obtained. Based on the mapping relationship, the current operation statement is obtained. Based on the current operation statement, the database is accessed, data operations are performed, and data information is obtained.

[0054] In some specific embodiments, in step S1, the application receives a data table access request, searches for the data model corresponding to the access request in the mapping layer. If no mapping model corresponding to the access request is found, an initial operation statement is obtained from the management table. Based on the initial operation statement, the application accesses the database, performs data operations in the database, and obtains data information. If a mapping model corresponding to the access request is found, the mapping model is parsed, the mapping relationship between the data table and the entity table is obtained from the mapping model, the operation statement is rewritten based on the mapping relationship to obtain the current operation statement, and the application accesses the database based on the current operation statement, performs data operations in the database, and obtains data information.

[0055] This embodiment speeds up data access by first querying the mapping layer to see if there is a mapping model corresponding to the data access request when a data access request is received, and then accessing the database based on the query results.

[0056] Some preferred embodiments involve performing table partitioning operations based on preset partitioning criteria and data information returned by the database, including: when the data information returned by the database meets the preset partitioning criteria, obtaining partitioning conditions in the mapping layer, performing table partitioning operations based on the partitioning conditions, and obtaining the current mapping model. The current mapping model is used to store the mapping relationship between the application's data tables and the database's entity tables after the table partitioning operation.

[0057] Retrieve the current mapping relationship between data tables and entity tables in the current mapping model, obtain the table partitioning operation statements based on the current mapping relationship, access the database based on the table partitioning operation statements, perform data operations, and obtain data information.

[0058] In some specific embodiments, in step S2, the mapping layer obtains the data information returned by the database and determines whether the returned information meets the preset splitting criteria. If it does, a table splitting operation is performed. The table splitting operation includes: obtaining the splitting conditions in the management table, sending an entity table splitting request to the database according to the splitting conditions, and performing table splitting. Simultaneously, the mapping relationship between the split entity tables and data tables is obtained, and a current mapping model is established based on the mapping relationship. When the application initiates a data access request again, the current mapping model is queried in the mapping layer, parsed, and the mapping relationship between the data tables and entity tables in the current mapping model is obtained. The operation statement is rewritten according to the mapping relationship to obtain the current operation statement. The database is accessed according to the current operation statement, data operations are performed in the database, and data information is obtained.

[0059] In this embodiment, the mapping layer receives data information returned by the database. The mapping layer performs table splitting operations according to the preset splitting criteria and the data information returned by the database, thereby realizing automatic splitting of the data table and avoiding the problem of slow request time caused by the increase of data volume in the database.

[0060] In some preferred embodiments, the data information returned by the database includes the amount of returned data and the time taken to return the data. The data information returned by the database meets a preset splitting standard, including: when the ratio of the time taken to return the data to the amount of returned data is greater than the preset splitting standard, it is determined that the data information returned by the database meets the preset splitting standard.

[0061] In some specific embodiments, in step S2, the mapping layer receives data information returned by the database. First, it obtains a preset splitting standard from the management table. This preset splitting standard is user-defined. Then, it calculates the ratio of the time taken to return data from the database to the amount of data returned. This ratio is compared with the splitting standard. If the ratio is greater than the preset splitting standard, it is determined that the data information returned by the database meets the preset splitting standard, meaning there are too many data tables and table splitting is required. Otherwise, table splitting is not required.

[0062] This embodiment compares the ratio of database return time to the amount of returned data with the splitting criteria, and determines whether table splitting is necessary based on the comparison result, thereby achieving automatic table splitting and avoiding the problem of slow request time caused by the increase in data volume in the database.

[0063] In some preferred embodiments, the splitting conditions include splitting statements and splitting time points. Table partitioning is performed based on these conditions to obtain the current mapping model, including:

[0064] The current mapping model is obtained by performing table partitioning operations based on the splitting statement and the splitting time point.

[0065] In some specific embodiments, in step S2, when the returned data information meets the splitting criteria, firstly, the splitting conditions in the management table are obtained, including the splitting statement and the splitting time point. The splitting time point can be preset by the user. If the user sets a splitting time point, the set time point is used; if the user does not set one, the current time is obtained as the splitting time point. A table splitting request is sent to the database according to the splitting statement and the splitting time point to perform the table splitting operation. Simultaneously, the mapping relationship between the split entity tables and data tables is obtained, and the current mapping model is established based on the mapping relationship.

[0066] This embodiment performs table partitioning based on partitioning conditions and establishes a current mapping model based on the mapping relationship between the partitioned entity tables and data tables. This enables automated table partitioning while ensuring the efficiency and accuracy of data access.

[0067] In some preferred embodiments, the data information returned by the database also includes business information, and the data table processing method further includes: returning the business information from the database to the application through a mapping layer. This allows users to obtain data information and improves the efficiency of data access.

[0068] In some specific embodiments, the data table processing method includes:

[0069] When an application programmer receives a data access request, they look up the mapping model corresponding to the access request in the mapping layer.

[0070] If no mapping model corresponding to the access request is found, the operation statement is obtained from the management table. Based on the initial operation statement, the database is accessed to perform data operations and obtain data information.

[0071] If a mapping model corresponding to the access request is found, the mapping model is parsed, the mapping relationship between the data table and the entity table is obtained from the mapping model, the operation statement is rewritten according to the mapping relationship, the current operation statement is obtained, the database is accessed according to the current operation statement, data operations are performed, and data information is obtained.

[0072] At this point, the mapping layer retrieves data information returned from the database, including business information, the amount of data returned, and the return time. The business information is returned to the application. A preset splitting standard is obtained from the management table, and the ratio of the database return time to the amount of data returned is calculated. This ratio is compared with the preset splitting standard. If the ratio is greater than the splitting standard, table partitioning is performed. The splitting statement and splitting time point in the management table are retrieved, and table partitioning is performed based on these statements and time points. The mapping relationship between the entity tables and data tables after the partitioning is obtained, and the current mapping model is established based on this mapping relationship. If the ratio is less than the splitting standard, table partitioning is not performed.

[0073] In some preferred embodiments, the data table processing method further includes:

[0074] Retrieve the preset merging criteria from the mapping layer, and initiate a merging operation request in the application based on the preset merging criteria.

[0075] Based on the table merge operation request, retrieve the mapping relationship between the application's data tables and the database's entity tables after the table partitioning operation in the current mapping model.

[0076] In the mapping layer, obtain the merge operation statement and the split table after the split operation, and perform the merge operation based on the split table and the merge operation statement.

[0077] In some specific embodiments, when the mapping layer receives a data table merging request, it extracts the mapping relationship between the application's data tables and the database's entity tables after the table splitting operation in the mapping model, obtains the merging operation statement in the management table, and performs the merging operation according to the merging operation statement.

[0078] This embodiment prevents resource waste caused by the unlimited expansion of data tables through table merging operations.

[0079] like Figure 3 As shown, this embodiment also provides a data table processing device applied to a data table processing system. The data table processing system includes: an application program, a database, and a mapping layer set between the application program and the database. The mapping layer is used to access the application program and the database respectively. The data table processing device includes: a query module, which is used to, when the application program receives an access request,

[0080] In the mapping layer, the mapping model corresponding to the access request is queried. Based on the query results, the database is accessed, data operations are performed, and data information is obtained. The mapping model is used to store the mapping relationship between the application's data tables and the database's entity tables.

[0081] The acquisition module is used to retrieve data information from the database and the preset splitting criteria in the mapping layer.

[0082] The table partitioning module is used to perform table partitioning operations based on preset partitioning criteria and data information returned by the database.

[0083] In some preferred embodiments, the acquisition module is also used to acquire a preset merge table standard in the mapping layer.

[0084] The data table processing device and the data table processing method described in this embodiment have the same advantages over the prior art, and will not be repeated here.

[0085] like Figure 4 As shown, this embodiment also provides a computer device, including a memory and a processor:

[0086] Memory, used to store computer programs;

[0087] A processor, used to implement the data table processing method described above when executing a computer program.

[0088] Figure 4 An internal structural diagram of a computer device in one embodiment is shown. The computer device includes a processor, memory, network interface, input device, and display screen connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program that, when executed by the processor, enables the processor to perform data table processing methods. The internal memory may also store a computer program that, when executed by the processor, enables the processor to perform data table processing methods. The display screen of the computer device may be a liquid crystal display (LCD) or an e-ink display. The input device may be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0089] This invention also provides a computer-readable storage medium storing a computer program, which is read and executed by a processor to implement the above-described data table processing method.

[0090] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0091] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A data table processing method characterized by, The application is applied to a data table processing system, the data table processing system comprises an application program, a database and a mapping layer arranged between the application program and the database, and the mapping layer is used for mutual access of the application program and the database respectively, and the data table processing method comprises: When the application program receives an access request, a mapping model corresponding to the access request is queried in the mapping layer, According to the query result, the database is accessed, data operation is performed, and data information is obtained, wherein the mapping model is used for storing the mapping relationship between the data table of the application program and the entity table of the database, and comprises: If the mapping model corresponding to the access request is queried, the mapping relationship between the data table and the entity table is obtained, the current operation statement is obtained according to the mapping relationship, the database is accessed according to the current operation statement, the data operation is performed, and the data information is obtained; The preset splitting standard and the data information returned by the database are obtained in the mapping layer, and the table splitting operation is performed according to the preset splitting standard and the data information returned by the database, comprising: When the data information returned by the database meets the preset splitting standard, The splitting condition is obtained in the mapping layer, the table splitting operation is performed according to the splitting condition, and the current mapping model is obtained, wherein the current mapping model is used for storing the mapping relationship between the data table of the application program and the entity table of the database after the table splitting operation; The current mapping relationship between the data table and the entity table in the current mapping model is obtained, the table splitting operation statement is obtained according to the current mapping relationship, the database is accessed according to the table splitting operation statement, the data operation is performed, and the data information is obtained.

2. The data table processing method according to claim 1, characterized by, According to the query result, the database is accessed, data operation is performed, and data information is obtained, comprising: If the mapping model corresponding to the access request is not queried, an initial operation statement is obtained, the database is accessed according to the initial operation statement, the data operation is performed, and the data information is obtained.

3. The data table processing method of claim 1, wherein, The data information returned by the database comprises return data volume and return time consumption, and the data information returned by the database meets the preset splitting standard, comprising: When the ratio of the return time consumption to the return data volume is greater than the preset splitting standard, it is determined that the data information returned by the database meets the preset splitting standard.

4. The data table processing method of claim 1, wherein, The splitting condition comprises a splitting statement and a splitting time point, and the table splitting operation is performed according to the splitting condition to obtain the current mapping model, comprising: The table splitting operation is performed according to the splitting statement and the splitting time point to obtain the current mapping model.

5. The data table processing method of claim 1, wherein, The data information returned by the database further comprises business information, and the data table processing method further comprises returning the business information from the database to the application program through the mapping layer.

6. The data table processing method of claim 1, wherein, Further comprising: Obtaining a preset table merging standard in the mapping layer; Initiating a table merging operation request in the application program according to the preset table merging standard; According to the combined table operation request, a mapping relationship between a data table of the application program and an entity table of the database after the split table operation is taken out in the current mapping model; In the mapping layer, a combined table operation statement and the split table after the split table operation are obtained, and the combined table operation is performed according to the split table and the combined table operation statement.

7. A data table processing apparatus characterized by comprising: The data table processing device is applied to a data table processing system, and the data table processing system comprises an application program, a database and a mapping layer arranged between the application program and the database, and the mapping layer is used to access each other with the application program and the database. When the application program receives an access request, In the mapping layer, a mapping model corresponding to the access request is queried, and according to the query result, the database is accessed to perform data operation to obtain data information, wherein the mapping model is used to store a mapping relationship between a data table of the application program and an entity table of the database, and comprises: If the mapping model corresponding to the access request is queried, the mapping relationship between the data table and the entity table is obtained, the current operation statement is obtained according to the mapping relationship, the database is accessed according to the current operation statement, the data operation is performed, and the data information is obtained; In the mapping layer, a preset split standard and data information returned by the database are obtained, The split table module is used to perform split table operation according to the preset split standard and the data information returned by the database, and comprises: When the data information returned by the database meets the preset split standard, In the mapping layer, a split condition is obtained, the split table operation is performed according to the split condition, and a current mapping model is obtained, wherein the current mapping model is used to store a mapping relationship between a data table of the application program and an entity table of the database after the split table operation; The current mapping relationship between the data table and the entity table in the current mapping model is obtained, a split table operation statement is obtained according to the current mapping relationship, the database is accessed according to the split table operation statement, the data operation is performed, and the data information is obtained.

8. A computer device, comprising: The memory is used to store a computer program; The processor is used to implement the data table processing method in any one of claims 1-6 when the computer program is executed. The computer readable storage medium stores a computer program, and when the computer program is read and run by the processor, the data table processing method in any one of claims 1-6 is implemented.

9. A computer-readable storage medium, characterized in that, ​

Images