Method and device for improving high-capacity data management efficiency and storage medium

Abstract

The invention discloses a method, a device, equipment and a storage medium for improving high-capacity data management efficiency, and the method comprises the steps: obtaining a name, a mode, a Partial period and a Partial field of a large data volume table in a parameter form, using a compiled automatic script, generating statements for generating and creating Partial according to a year mode, a month mode and the like, and completing the management of the large-capacity data. And storing the automatically generated and created new Partial statement in the temporary executable file, executing the new Partial statement according to an allowable mode of the database, judging whether the execution is successful or not, giving a prompt if the execution is not successful, and deleting the temporary executable file if the execution is successful. The data management efficiency of the large data volume table can be improved, excessive extra space is not occupied, meanwhile, a mechanism for automatically creating the Partition is established, and the management efficiency and the economic cost are both considered.

Description

Technical Field

[0001] The present invention relates to the field of data management technology, in particular to the field of being able to improve efficiency through data management, and more particularly to a method, device and storage medium for improving the efficiency of large-capacity data management. Background Art

[0002] Databases are warehouses for storing data. Usually, databases store a large amount of data tables, stored procedures, functions, views and other information. They are an indispensable and important part of the bank's information system. As a storage location for the bank's massive data, databases have great business value and are the basis for the construction of other information systems. Therefore, ensuring the query efficiency of the database is an important issue that banks need to solve.

[0003] The following problems arise during the query of the existing database: As the database usage time increases, the amount of data in the transaction detail table will become extremely large. For example, a table with a data volume of hundreds of millions will cause data query to slow down, and it will become extremely difficult to delete and modify data, which will lead to a decrease in the efficiency of the bank's information system.

[0004] The problem of slow query can be partially solved by creating indexes for the table, but this method cannot solve the problem of slow data deletion and modification. At the same time, creating a new index will take up a large space and affect the efficiency of data addition, deletion and modification.

[0005] Therefore, how to propose a method to solve the problems of slow data deletion and modification and large space occupied by newly created indexes, while improving the efficiency of data addition, deletion and modification, is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0006] To this end, the present invention divides the table into partitions of different granularities by using the most commonly used fields to implement partition management. When querying, only the partitions involved in the query conditions are queried, which improves the data query efficiency of large data tables without taking up too much additional space. A mechanism for automatically creating partitions is established to automatically create new partitions to meet the needs of data growth, taking into account both management efficiency and usage efficiency.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a method for improving the efficiency of large-capacity data management, comprising:

[0008] Establish a Partition configuration table for a large data volume table, wherein the Partition configuration table includes: a data table name, a data table mode, a Partition period, and a Partition field;

[0009] Get the name of the data table, the mode of the data table, the period of the partition, and the fields of the partition and convert them into parameter form;

[0010] According to the data information converted into parameter form, a Partition statement is automatically generated in a chronological manner using an automatic script;

[0011] The automatically generated Partition statement is stored in an executable file and executed according to the method set by the database.

[0012] Furthermore, the automatically generated Partition statement is stored in an executable file, and executed according to the method set by the database, including: judging whether the execution is successful, and giving an unsuccessful prompt if it is judged to be unsuccessful; and deleting the temporary executable file if it is judged to be successful.

[0013] Furthermore, the automatic generation of Partition statements in a chronological order by using an automatic script includes: the chronological order is a method of sorting in the order of year, month, and day.

[0014] Furthermore, the Partition configuration table for establishing a large data table includes: data table name, data table mode, Partition period and Partition fields include: using the most commonly used fields to divide the table into Partitions of different granularities, only querying the Partitions involved in the query conditions, while improving the data query efficiency of the large data table without taking up too much additional space, and establishing a mechanism for automatically creating Partitions.

[0015] In a second aspect, the present invention further provides a device for improving the efficiency of large-capacity data management, the device comprising: a configuration module, a first processing module, a second processing module, and a verification execution module;

[0016] The configuration module is used to establish a Partition configuration table for a large data volume table, wherein the Partition configuration table includes: a data table name, a data table mode, a Partition period, and a Partition field;

[0017] The first processing module is used to obtain the name of the data table, the mode of the data table, the period of the partition and the field of the partition and convert them into parameter form;

[0018] The second processing module is used to automatically generate a Partition statement in a chronological manner using an automatic script according to the data information converted into parameter form;

[0019] The verification execution module is used to store the automatically generated Partition statement into an executable file and execute it according to the method set by the database.

[0020] Furthermore, the automatically generated Partition statement is stored in an executable file, and executed according to the method set by the database, including: judging whether the execution is successful, and giving an unsuccessful prompt if it is judged to be unsuccessful; and deleting the temporary executable file if it is judged to be successful.

[0021] Furthermore, the automatic generation of Partition statements in a chronological order by using an automatic script includes: the chronological order is a method of sorting in the order of year, month, and day.

[0022] Furthermore, the Partition configuration table for establishing a large data table includes: data table name, data table mode, Partition period and Partition fields include: using the most commonly used fields to divide the table into Partitions of different granularities, only querying the Partitions involved in the query conditions, while improving the data query efficiency of the large data table without taking up too much additional space, and establishing a mechanism for automatically creating Partitions.

[0023] In a third aspect, the present invention provides a computing device, comprising:

[0024] A memory for storing program instructions;

[0025] The processor is used to call the program instructions stored in the memory and execute the above-mentioned method for improving the efficiency of large-capacity data management according to the obtained program instructions.

[0026] In a fourth aspect, the present invention provides a computer-readable storage medium, including computer-readable instructions. When a computer reads and executes the computer-readable instructions, the above-mentioned method for improving the efficiency of large-capacity data management is implemented. The readable medium may be multiple, and the multiple readable media can operate independently of each other.

[0027] Compared with the existing technology, this solution obtains the name, mode, partition period and partition field of the large data table in the form of parameters, uses the written automatic script to generate statements for creating partitions by year, month, etc., saves the automatically generated new partition statements to a temporary executable file, and executes them in the way allowed by the database, determines whether the execution is successful, and gives a prompt if it is unsuccessful, and deletes the temporary executable file if it is successful. This can improve the data management efficiency of large data tables without taking up too much extra space, and at the same time establish a mechanism for automatically creating partitions, taking into account management efficiency and economic costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 A flow chart of an embodiment of a method for improving large-capacity data management efficiency provided by the present invention;

[0030] Figure 2 A structural block diagram of an embodiment of a device for improving large-capacity data management efficiency provided by the present invention;

[0031] Figure 3 The present invention provides a schematic structural diagram of an electronic device. DETAILED DESCRIPTION

[0032] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0033] It will be understood by those skilled in the art that, unless otherwise stated, the singular forms "a", "an" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0034] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0035] Partition technology refers to the technology of dividing the whole into parts or regions. This technology achieves more efficient management, processing and optimization through segmentation and reorganization.

[0036] Secondly, in algorithm design, partition technology refers to dividing an array or data structure into several parts for more efficient processing. For example, the partition process in the quick sort algorithm is to divide the array into sub-arrays less than a certain value and sub-arrays greater than the value.

[0037] In order to make the objectives, technical solutions and advantages of the present application more clear, the specific implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0038] The core of the present invention is to provide a method for improving the efficiency of large-capacity data management. Figure 1 The flowchart of the method for improving the efficiency of large-capacity data management provided by this embodiment is shown in FIG. Figure 1 The technical solution of this embodiment is described in detail:

[0039] Step S1: Establish a Partition configuration table for a large data volume table, wherein the Partition configuration table includes: a data table name, a data table mode, a Partition period, and a Partition field.

[0040] Step S2: Obtain the name of the data table, the mode of the data table, the period of the partition, and the field of the partition and convert them into parameter form.

[0041] In a preferred embodiment, the name, mode, desired partition period and partition fields of the large data table are obtained in the form of parameters to achieve automatic management.

[0042] Step S3: Based on the data information converted into parameter form, an automatic script is used to automatically generate a Partition statement in a chronological manner.

[0043] In a preferred embodiment, the creation of Partitions is automatically generated, and a written automatic script is used to generate statements for creating Partitions by year, month, etc.

[0044] Step S4: Store the automatically generated Partition statement into an executable file, and execute it according to the method set by the database.

[0045] In a preferred embodiment, the automatically generated Partition statement is stored in an executable file, and executed according to the method set by the database, including: judging whether the execution is successful, and giving an unsuccessful prompt if it is judged to be unsuccessful; and deleting the temporary executable file if it is judged to be successful.

[0046] In a preferred embodiment, the automatic script is used to automatically generate the Partition statement in a chronological order, including: the chronological order is a method of sorting in the order of year, month, and day.

[0047] In a preferred embodiment, the Partition configuration table for establishing a large data table includes: data table name, data table mode, Partition period and Partition fields include: using the most commonly used fields to divide the table into Partitions of different granularities, only querying the Partitions involved in the query conditions, while improving the data query efficiency of the large data table without taking up too much additional space, and establishing a mechanism for automatically creating Partitions.

[0048] In the scheme of this embodiment, by obtaining the name, mode, partition period and partition field of the large data table in the form of parameters, using the written automatic script, generating statements for creating partitions by year, month, etc., saving the automatically generated new partition statements to a temporary executable file, and executing them in a manner allowed by the database, judging whether the execution is successful, giving a prompt if unsuccessful, and deleting the temporary executable file if successful. This can improve the data management efficiency of large data tables without taking up too much extra space, and at the same time establish a mechanism for automatically creating partitions, taking into account both management efficiency and economic cost.

[0049] In one embodiment, the present invention provides a device for improving the efficiency of large-capacity data management. Figure 2 The structure block diagram of a device for improving the efficiency of large-capacity data management provided by this embodiment is shown, and the device includes: a configuration module 10, a first processing module 20, a second processing module 30, and a verification execution module 40;

[0050] Configuration module 10: used to establish a Partition configuration table for a large data volume table, wherein the Partition configuration table includes: a data table name, a data table mode, a Partition period, and a Partition field.

[0051] The first processing module 20 is used to obtain the name of the data table, the mode of the data table, the period of the Partition and the field of the Partition and convert them into parameter form.

[0052] In a preferred embodiment, the name, mode, desired partition period and partition fields of the large data table are obtained in the form of parameters to achieve automatic management.

[0053] The second processing module 30 is used to automatically generate a Partition statement in a chronological manner using an automatic script according to the data information converted into parameter form.

[0054] In a preferred embodiment, the creation of Partitions is automatically generated, and a written automatic script is used to generate statements for creating Partitions by year, month, etc.

[0055] Verification execution module 40: used to store the automatically generated Partition statement into an executable file and execute it according to the method set in the database.

[0056] In a preferred embodiment, the automatically generated Partition statement is stored in an executable file, and executed according to the method set by the database, including: judging whether the execution is successful, and giving an unsuccessful prompt if it is judged to be unsuccessful; and deleting the temporary executable file if it is judged to be successful.

[0057] In a preferred embodiment, the automatic script is used to automatically generate the Partition statement in a chronological order, including: the chronological order is a method of sorting in the order of year, month, and day.

[0058] In a preferred embodiment, the Partition configuration table for establishing a large data table includes: data table name, data table mode, Partition period and Partition fields include: using the most commonly used fields to divide the table into Partitions of different granularities, only querying the Partitions involved in the query conditions, while improving the data query efficiency of the large data table without taking up too much additional space, and establishing a mechanism for automatically creating Partitions.

[0059] In the scheme of this embodiment, by obtaining the name, mode, partition period and partition field of the large data table in the form of parameters, using the written automatic script, generating statements for creating partitions by year, month, etc., saving the automatically generated new partition statements to a temporary executable file, and executing them in a manner allowed by the database, judging whether the execution is successful, giving a prompt if unsuccessful, and deleting the temporary executable file if successful. This can improve the data management efficiency of large data tables without taking up too much extra space, and at the same time establish a mechanism for automatically creating partitions, taking into account both management efficiency and economic cost.

[0060] The device for improving bulk data management efficiency of this embodiment is used to implement the aforementioned method for improving bulk data management efficiency. Therefore, the specific implementation method of the device for improving bulk data management efficiency can be seen in the specific implementation method of the method for improving bulk data management efficiency in the previous text. For example, the configuration module 10 is used to implement step S1 in the aforementioned method for improving bulk data management efficiency, the first processing module 20 is used to implement step S2 in the aforementioned method for improving bulk data management efficiency, the second processing module 30 is used to implement step S3 in the aforementioned method for improving bulk data management efficiency, and the verification execution module 40 is used to implement step S4 in the aforementioned method for improving bulk data management efficiency. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part. The device provided by the present invention is attached to the method provided by the present invention, and its specific effects can refer to the above method, which will not be repeated here.

[0061] In one embodiment, a computing device is provided, comprising:

[0062] A memory for storing program instructions;

[0063] A processor is used to implement the steps of any of the above-mentioned methods for improving the efficiency of large-capacity data management when executing the computer program.

[0064] As an example, Figure 3 A schematic diagram of the structure of an electronic device applicable to the embodiment of the present application is shown. Figure 3 As shown, the electronic device 2000 includes: a processor 2001 and a memory 2003. The processor 2001 and the memory 2003 are connected, such as through a bus 2002. Optionally, the electronic device 2000 may also include a transceiver 2004. It should be noted that in actual applications, the transceiver 2004 is not limited to one, and the structure of the electronic device 2000 does not constitute a limitation on the embodiments of the present application.

[0065] The processor 2001 is applied in the embodiment of the present application to implement the method shown in the above method embodiment. The transceiver 2004 may include a receiver and a transmitter. The transceiver 2004 is applied in the embodiment of the present application to implement the function of the electronic device of the embodiment of the present application communicating with other devices when executed.

[0066] Processor 2001 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 2001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0067] The bus 2002 may include a path to transmit information between the above components. The bus 2002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 2002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0068] The memory 2003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0069] Optionally, the memory 2003 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 2001. The processor 2001 is used to execute the application code stored in the memory 2003 to implement the investment and financing project recommendation method provided in any embodiment of the present application.

[0070] The electronic device provided in the embodiment of the present application is applicable to any embodiment of the above method and will not be described in detail here.

[0071] The method for improving the efficiency of large-capacity data management provided by the present invention has obvious advantages and beneficial effects compared with the prior art. By obtaining the name, mode, partition cycle and partition field of a large data table in the form of parameters, using the written automatic script, generating statements for creating partitions by year, month, etc., saving the automatically generated new partition statements to a temporary executable file, and executing them in a manner allowed by the database, judging whether the execution is successful, giving a prompt if unsuccessful, and deleting the temporary executable file if successful. The method can improve the data management efficiency of large data tables without taking up too much extra space, and at the same time establish a mechanism for automatically creating partitions, taking into account both management efficiency and economic cost.

[0072] In one embodiment, a computer-readable storage medium is provided, including computer-readable instructions. When a computer reads and executes the computer-readable instructions, the steps of the method for improving the management efficiency of large-capacity data described above are implemented. The method for improving the management efficiency of large-capacity data provided by the present invention has obvious advantages and beneficial effects compared with the prior art. By obtaining the name, mode, partition cycle and partition field of a large data table in the form of parameters, using the written automatic script, generating statements for creating partitions by year, month, etc., saving the automatically generated new partition statement to a temporary executable file, and executing it in a manner allowed by the database, judging whether the execution is successful, giving a prompt if unsuccessful, and deleting the temporary executable file if successful. It is achieved that the data management efficiency of large data tables can be improved without taking up too much additional space, and at the same time, a mechanism for automatically creating partitions is established, taking into account management efficiency and economic cost.

[0073] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the specific device can be divided into different functional modules to complete all or part of the functions described above.

[0074] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0075] In several embodiments covered by the present application, it should be understood that the disclosed devices, equipment and methods can be implemented in other ways. For example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, or it can be an electrical, mechanical or other form of connection.

[0076] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0077] In addition, each functional module in each embodiment of the present application can be implemented in the form of hardware or in the form of software functional modules. If these functional modules are implemented in the form of software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or part of the prior art that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a computer program product, which includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer program product is stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, a computer instruction can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)).

[0078] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A method for improving the efficiency of large-capacity data management, characterized in that: include: Establish a Partition configuration table for a large data volume table, wherein the Partition configuration table includes: a data table name, a data table mode, a Partition period, and a Partition field; Get the name of the data table, the mode of the data table, the period of the partition, and the fields of the partition and convert them into parameter form; According to the data information converted into parameter form, a Partition statement is automatically generated in a chronological manner using an automatic script; The automatically generated Partition statement is stored in an executable file and executed according to the method set by the database.

2. The method for improving large-capacity data management efficiency according to claim 1, characterized in that: The storing of the automatically generated Partition statement into an executable file and executing it according to the method set by the database includes: judging whether the execution is successful, and giving an unsuccessful prompt if it is judged to be unsuccessful; and deleting the temporary executable file if it is judged to be successful.

3. The method for improving large-capacity data management efficiency according to claim 1, characterized in that: The automatic script is used to automatically generate the Partition statement in a chronological order, including: the chronological order is a method of sorting in the order of year, month, and day.

4. The method for improving large-capacity data management efficiency according to claim 2, characterized in that: The partition configuration table for establishing a large data table includes: data table name, data table mode, partition period and partition fields including: using the most commonly used fields to divide the table into partitions of different granularities, only querying the partitions involved in the query conditions, while improving the data query efficiency of the large data table without taking up too much additional space, and establishing a mechanism for automatically creating partitions.

5. A device for improving the efficiency of large-capacity data management, characterized in that: The device comprises: a configuration module, a first processing module, a second processing module, and a verification execution module; The configuration module is used to establish a Partition configuration table for a large data volume table, wherein the Partition configuration table includes: a data table name, a data table mode, a Partition period, and a Partition field; The first processing module is used to obtain the name of the data table, the mode of the data table, the period of the partition and the field of the partition and convert them into parameter form; The second processing module is used to automatically generate a Partition statement in a chronological manner using an automatic script according to the data information converted into parameter form; The verification execution module is used to store the automatically generated Partition statement into an executable file and execute it according to the method set by the database.

6. The device for improving large-capacity data management efficiency according to claim 5, characterized in that: The storing of the automatically generated Partition statement into an executable file and executing it according to the method set by the database includes: judging whether the execution is successful, and giving an unsuccessful prompt if it is judged to be unsuccessful; and deleting the temporary executable file if it is judged to be successful.

7. The device for improving large-capacity data management efficiency according to claim 5, characterized in that: The automatic script is used to automatically generate the Partition statement in a chronological order, including: the chronological order is a method of sorting in the order of year, month, and day.

8. The device for improving large-capacity data management efficiency according to claim 6, characterized in that: The partition configuration table for establishing a large data table includes: data table name, data table mode, partition period and partition fields including: using the most commonly used fields to divide the table into partitions of different granularities, only querying the partitions involved in the query conditions, while improving the data query efficiency of the large data table without taking up too much additional space, and establishing a mechanism for automatically creating partitions.

9. A computing device, characterized in that include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 4 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that: It includes computer-readable instructions. When a computer reads and executes the computer-readable instructions, it implements the method as claimed in any one of claims 1 to 4. There can be multiple readable media, and the multiple readable media can operate independently of each other.

Images