Data storage method, device and equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of data storage, is applied to a financial business big data storage scene, and relates to a data storage method, device and equipment and a storage medium thereof. Performing storage category verification on all to-be-stored contents contained in the target storage data by querying a historical form; according to the method, different storage processing modes are adopted to store and process the newly added incremental data and the hot data and the non-hot data in the historical stock data, so that excessive data volume in the historical form is avoided, the hot data and the non-hot data can be stored in different tables, and data maintenance is facilitated for data maintenance personnel. Especially in the field of financial businesses with more stored data, the data storage method provided by the invention is adopted, so that the data storage mode is more scientific and reasonable.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and is applied to big data storage scenarios of financial services, and in particular to a data storage method, device, equipment and storage medium thereof. Background Art

[0002] With the rapid development of Internet technology, more and more financial services are handled online. The popularity of online processing also requires the limited number of salesmen in financial companies to face more unspecified business objects. This is of great benefit to the company's business growth. However, it will also bring more business data processing problems, for example: the business storage data will increase significantly.

[0003] At present, in order to cope with the massive increase in business storage data, some financial companies adopt cloud storage and entrust data management and maintenance to third-party cloud platforms. However, this method is not conducive to data security. Some financial companies adopt self-built databases for independent management and maintenance. Although this provides a high degree of guarantee for data security, the large amount of data often leads to the self-built database storage mode being not scientific and reasonable enough, which makes it difficult for data maintenance personnel to manage and maintain it. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a data storage method, device, equipment and storage medium thereof to solve the problem that the storage mode of the existing self-built database is not scientific and rational enough, which makes it difficult for data maintenance personnel to manage and maintain it.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a data storage method, which adopts the following technical solution:

[0006] A data storage method comprises the following steps:

[0007] Get the target storage data;

[0008] Performing storage category verification on all the to-be-stored contents contained in the target storage data by querying the historical form, wherein the storage category verification includes verifying whether the current to-be-stored contents are historical stock data or newly added incremental data;

[0009] If the current content to be stored contained in the target storage data can be queried in the history table, the current content to be stored is historical stock data, and according to a preset identification strategy, it is determined whether the current content to be stored is hot data;

[0010] If the current content to be stored is hot data, the current content to be stored is written into a preset new sharding table from the historical table by data migration, and the current content to be stored is deleted from the historical table;

[0011] If the current content to be stored contained in the target storage data cannot be found in the historical table, the current content to be stored is newly added incremental data, and the current content to be stored is written into the preset new shard table and the historical table.

[0012] Furthermore, before executing the step of verifying the storage category of all the to-be-stored contents contained in the target storage data by querying the history table, the method further includes:

[0013] Generate a new shard table for data storage according to a preset first parameter, a second parameter and a preset sharding condition, wherein the first parameter includes an estimated total storage data volume, and the second parameter includes a maximum storage data volume of a single table.

[0014] The preset fragmentation conditions include:

[0015] X∈2 N , and X is the distance The smallest power of 2,

[0016] In the formula, X represents the number of shards, N is a positive integer, m is the first parameter, and n is the second parameter. Indicates that the ratio of the first parameter to the second parameter is rounded up by adding 1;

[0017] Obtain storage address information of all shards in the new shard table.

[0018] Furthermore, after executing the step of generating a new shard table for data storage, the method further includes:

[0019] Each shard in the new shard table is numbered using a binary method, wherein the decimal number corresponding to the number is an integer value in the interval [0, X), and X represents the number of shards.

[0020] Furthermore, the step of verifying the storage category of all the to-be-stored contents contained in the target storage data by querying the history table specifically includes:

[0021] Selecting different contents to be stored in sequence from all contents to be stored contained in the target storage data as current storage contents;

[0022] Inputting the current storage content as an index item into a preset search engine Elasticsearch, and querying the current storage content from the historical table through a query interface of the search engine Elasticsearch;

[0023] If the current storage content can be queried in the history table, the current storage content is marked with a preset first distinguishing mark, and a pair of data consisting of the first distinguishing mark and the current storage content is output through the output interface of the search engine Elasticsearch;

[0024] If the current stored content cannot be found in the history table, the current stored content is marked with a preset second distinguishing identifier, and a prompt content including the second distinguishing identifier and the current stored content is sent to the target prompt terminal;

[0025] Based on the distinguishing identifier corresponding to the current storage content, the storage category of the current storage content is verified.

[0026] Furthermore, the step of verifying the storage category of the current storage content based on the distinguishing identifier corresponding to the current storage content specifically includes:

[0027] If the distinguishing identifier corresponding to the current storage content is the first distinguishing identifier, then the current storage content is historical stock data;

[0028] If the distinguishing identifier corresponding to the currently stored content is the second distinguishing identifier, then the currently stored content is newly added incremental data.

[0029] Furthermore, the step of writing the current content to be stored into a preset new sharding table specifically includes:

[0030] By parsing the target storage data, identifying a unique user number corresponding to the current content to be stored, wherein the user number is a random integer;

[0031] According to the preset algorithm formula:

[0032]

[0033] Perform a remainder calculation and perform binary conversion on the remainder to obtain a binary conversion result, where I represents the user number, X represents the number of shards, and Y represents the ratio of the user number to the number of shards;

[0034] Compare the binary conversion result with the distinguishing number of each shard in the new shard table, and select a shard whose distinguishing number is consistent with the binary conversion result as the storage shard of the current content to be stored;

[0035] Identify storage address information corresponding to the storage slice of the current content to be stored;

[0036] The current content to be stored is stored based on the storage address information corresponding to the storage slice of the current content to be stored.

[0037] Furthermore, the method further comprises:

[0038] Using a preset call monitoring component to continuously monitor the number of times all historical storage contents in the historical form are called;

[0039] If the number of calls corresponding to the target historical storage content reaches the preset hotspot number threshold, the target historical storage content is written from the historical form to the preset new shard table using data migration, and the target historical storage content is deleted from the historical form.

[0040] In order to solve the above technical problems, the embodiment of the present application also provides a data storage device, which adopts the following technical solution:

[0041] A data storage device, comprising:

[0042] A target storage data acquisition module is used to acquire target storage data;

[0043] A storage category verification query module, used to verify the storage category of all the to-be-stored contents contained in the target storage data by querying the history form, wherein the storage category verification includes verifying whether the current to-be-stored contents are historical stock data or newly added incremental data;

[0044] A hot data judgment module, used for judging whether the current content to be stored contained in the target storage data is hot data according to a preset identification strategy if the current content to be stored contained in the target storage data can be queried in the historical form and the current content to be stored is historical stock data;

[0045] A first storage content writing module is used for, if the current content to be stored is hot data, writing the current content to be stored from the historical table into a preset new sharding table by using a data migration method, and deleting the current content to be stored in the historical table;

[0046] The second storage content writing module is used to write the current content to be stored into the preset new shard table and the historical form if the current content to be stored contained in the target storage data cannot be queried in the historical form and the current content to be stored is the newly added incremental data.

[0047] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0048] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned data storage method when executing the computer-readable instructions.

[0049] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0050] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data storage method described above.

[0051] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0052] The data storage method described in the embodiment of the present application obtains the target storage data; verifies the storage category of all the contents to be stored contained in the target storage data by querying the historical form; writes the newly added incremental data to the preset new sharding table and the historical form at the same time, and writes the current contents to be stored from the historical form to the preset new sharding table by data migration for the hot data in the historical stock data, and deletes the current contents to be stored in the historical form, which not only avoids excessive data in the historical form, but also can store the hot data and non-hot data in separate tables, which is convenient for data maintenance personnel to perform data maintenance. Especially in the field of financial business with a large amount of stored data, the data storage method described in the present application is adopted to make the data storage mode more scientific and reasonable. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0055] Figure 2 is a flow chart of an embodiment of a data storage method according to the present application;

[0056] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 202 is shown;

[0057] Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 305 is shown;

[0058] Figure 5 is a schematic structural diagram of an embodiment of a data storage device according to the present application;

[0059] Figure 6 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0061] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0062] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0063] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0064] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0065] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0066] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0067] It should be noted that the data storage method provided in the embodiment of the present application is generally executed by a terminal device, and accordingly, the data storage device is generally arranged in the terminal device.

[0068] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0069] Continue to refer Figure 2 , shows a flow chart of an embodiment of a data storage method according to the present application. The data storage method comprises the following steps:

[0070] Step 201, obtaining target storage data;

[0071] In this embodiment, the target storage data includes target business data and / or target customer data to be stored that have been pre-organized in the financial business field. The target business data, for example, insurance processing business summary data to be stored after organization this quarter; the target customer data, for example, identity information data of users who open credit cards this month. By acquiring the target storage data, the target storage data can be stored scientifically and rationally in the future.

[0072] Step 202, verifying the storage category of all the to-be-stored contents contained in the target storage data by querying the historical table, wherein the storage category verification includes verifying whether the current to-be-stored contents are historical stock data or newly added incremental data;

[0073] In this embodiment, the historical stock data refers to the data that has been cached historically, and the newly added incremental data refers to the data that has been cached historically.

[0074] Specifically, the historical form refers to a data storage form that has been put into use in the past. The number of forms is determined by the number of forms that have been put into use in the past. It may also specifically refer to or not specifically refer to a certain form.

[0075] The storage category of all the to-be-stored contents contained in the target storage data is verified by querying the historical form, so that different storage methods can be used to perform data storage processing on the historical stock data and the newly added incremental data in the future.

[0076] Step 203: If the current content to be stored contained in the target storage data can be queried in the history table, the current content to be stored is historical stock data, and according to a preset identification strategy, it is determined whether the current content to be stored is hot data;

[0077] Wherein, the preset identification strategy includes counting the number of historical calls of the current content to be stored, and comparing the size relationship between the number of historical calls and a preset hotspot number threshold. If the number of historical calls exceeds the preset hotspot number threshold, the current content to be stored is hotspot data; if the number of historical calls does not exceed the preset hotspot number threshold, the current content to be stored is non-hotspot data.

[0078] In combination with the historical call times of the current content to be stored and the preset hotspot number threshold, it is determined whether the current content to be stored is hotspot data, so as to process the hotspot data and non-hotspot data separately.

[0079] Step 204: if the current content to be stored is hot data, the current content to be stored is written from the history table to a preset new shard table by using a data migration method, and the current content to be stored is deleted from the history table;

[0080] Specifically, after identifying that the current content to be stored is hot data, the data migration method is used to write the current content to be stored from the historical form into a preset new shard table, and the current content to be stored is deleted from the historical form, thereby optimizing the historical form and avoiding storing data with too many call times in the historical form. The data with too many historical call times is stored separately in a new shard table, thereby avoiding excessive data volume in the historical form and performing shard transfer in a scientific and reasonable manner.

[0081] If the current content to be stored is non-hotspot data, the current content to be stored is continued to be retained in the history table;

[0082] Specifically, after identifying that the current content to be stored is non-hotspot data, the current content to be stored will continue to be retained in the historical form. This can keep the non-hotspot data in the original table without occupying the resources consumed by the transfer, and can also separate the hotspot data and non-hotspot data well, so that the hotspot data can be concentratedly transferred to the newly generated form, which is more conducive to data maintenance personnel to perform data maintenance.

[0083] Step 205: If the current content to be stored contained in the target storage data cannot be found in the historical table, the current content to be stored is newly added incremental data, and the current content to be stored is written into the preset new shard table and the historical table.

[0084] By writing the newly added incremental data into the preset new shard table and the historical table, the data is backed up to ensure the storage correctness comparison of the newly added incremental data.

[0085] In this embodiment, by acquiring the target storage data; verifying the storage category of all the contents to be stored contained in the target storage data by querying the historical form; writing the newly added incremental data into the preset new sharding table and the historical form at the same time; using the data migration method for the hot data in the historical stock data to write the current contents to be stored from the historical form into the preset new sharding table, and deleting the current contents to be stored in the historical form; monitoring the number of calls of non-hot data, so that when the number of calls is too many, timely transfer and storage processing is performed, which not only avoids excessive data in the historical form, but also can store hot data and non-hot data in separate tables, which is convenient for data maintenance personnel to perform data maintenance. Especially in the field of financial business with a large amount of stored data, the data storage method described in this application is adopted to make the data storage mode more scientific and reasonable.

[0086] In some optional implementations, before step 202, the method further includes a method for generating a new shard table, including the following steps:

[0087] Generate a new shard table for data storage according to a preset first parameter, a second parameter and a preset sharding condition, wherein the first parameter includes an estimated total storage data volume, and the second parameter includes a maximum storage data volume of a single table.

[0088] The preset fragmentation conditions include:

[0089] X∈2 N , and X is the distance The smallest power of 2,

[0090] In the formula, X represents the number of shards, N is a positive integer, m is the first parameter, and n is the second parameter. Indicates that the ratio of the first parameter to the second parameter is rounded up by adding 1;

[0091] Obtain storage address information of all shards in the new shard table.

[0092] In this embodiment, a new shard table for data storage is generated by using the first parameter, the second parameter and the preset shard condition, so that the number of new shard tables generated can not only meet the storage requirements of the estimated total storage data volume, but also avoid the excessive number of new shard tables generated, which is not conducive to comprehensive maintenance. The storage address information of all shards in the new shard table is obtained to facilitate subsequent storage and processing of data. Specifically, the storage address information of all shards in the new shard table can be expressed in the form of "new shard table name + shard distinction number".

[0093] In some optional implementations, after performing the step of generating a new shard table for data storage, the method further includes:

[0094] Each shard in the new shard table is numbered using a binary method, wherein the decimal number corresponding to the number is an integer value in the interval [0, X), and X represents the number of shards.

[0095] Specifically, for example: there are a total of 8 fragments in the new fragment table, and the corresponding distinction numbers of the 8 fragments are "000", "001", "010", "011", "100", "101", "110" and "111" respectively; for another example, there are a total of 16 fragments in the new fragment table, and the corresponding distinction numbers of the 16 fragments are "0000", "0001", "0010", "0011", "0100", "0101", "0110", "0111", "1000", "1001", "1010", "1011", "1100", "1101", "1110" and "1111".

[0096] By using the binary method to distinguish and number all the slices in the new slice table, the computer can quickly identify the target slice.

[0097] Continue to refer Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 202 shown includes the following steps:

[0098] Step 301, selecting different contents to be stored from all contents to be stored contained in the target storage data in turn as current storage contents;

[0099] Step 302: input the current storage content as an index item into a preset search engine Elasticsearch, and query the current storage content from the historical table through the query interface of the search engine Elasticsearch;

[0100] Using the query interface of the search engine Elasticsearch to query the current stored content from the historical table can further improve the search efficiency when the amount of data stored in the historical table is large.

[0101] Step 303: if the current storage content can be queried in the history table, the current storage content is marked with a preset first distinguishing mark, and a pair of data consisting of the first distinguishing mark and the current storage content is output through the output interface of the search engine Elasticsearch;

[0102] Step 304: if the current stored content cannot be found in the history table, the current stored content is marked with a preset second distinguishing identifier, and a prompt content including the second distinguishing identifier and the current stored content is sent to a target prompt terminal;

[0103] Through step 303 and step 304, different distinguishing identifiers are used to distinguish and mark the storage content that can be queried and the storage content that cannot be queried, so as to facilitate the subsequent identification of the storage categories corresponding to the storage content, that is, to facilitate the identification of historical stock data and new incremental data, and to facilitate the sending of historical stock data and new incremental data to different data storage and processing modules.

[0104] Step 305: verify the storage category of the current storage content based on the distinguishing identifier corresponding to the current storage content.

[0105] Continue to refer Figure 4 , Figure 4 yes Figure 3The flowchart of a specific embodiment of step 305 shown includes the following steps:

[0106] Step 401: If the distinguishing identifier corresponding to the current storage content is the first distinguishing identifier, the current storage content is historical stock data;

[0107] Step 402: If the distinguishing identifier corresponding to the currently stored content is the second distinguishing identifier, then the currently stored content is newly added incremental data.

[0108] In some optional implementations, the step of writing the current content to be stored into a preset new sharding table in step 204 and step 205 specifically includes:

[0109] By parsing the target storage data, identifying a unique user number corresponding to the current content to be stored, wherein the user number is a random integer;

[0110] According to the preset algorithm formula:

[0111]

[0112] Perform a remainder calculation and perform binary conversion on the remainder to obtain a binary conversion result, where I represents the user number, X represents the number of shards, and Y represents the ratio of the user number to the number of shards;

[0113] Specifically, assuming that the user number is 209 and the number of shards is 8, Y is 26, and the remainder obtained by the modulo calculation is 1, then the binary conversion result corresponding to 1 is "001"; similarly, assuming that the user number is 211 and the number of shards is 16, then Y is 13, and the remainder obtained by the modulo calculation is 3, then the binary conversion result corresponding to 3 is "0011".

[0114] Compare the binary conversion result with the distinguishing number of each shard in the new shard table, and select a shard whose distinguishing number is consistent with the binary conversion result as the storage shard of the current content to be stored;

[0115] Since, when the number of shards is 8, the distinction numbers corresponding to the 8 shards are "000", "001", "010", "011", "100", "101", "110" and "111" respectively, therefore, the shard with the distinction number "001" is used as the storage shard of the current content to be stored; similarly, when the number of shards is 16, the distinction numbers corresponding to the 16 shards are "0000", "0001", "0010", "0011", "0100", "0101", "0110", "0111", "1000", "1001", "1010", "1011", "1100", "1101", "1110" and "1111", and the shard with the distinction number "0011" is used as the storage shard of the current content to be stored. The above storage shard selection method ensures that the shard where the user's related storage data is stored can be quickly determined by knowing only the user ID and the number of shards in the new shard table, which facilitates subsequent calls to the user's related storage data.

[0116] Identify storage address information corresponding to the storage slice of the current content to be stored;

[0117] The current content to be stored is stored based on the storage address information corresponding to the storage slice of the current content to be stored.

[0118] In some optional implementations, the method further includes: using a preset call monitoring component to continuously monitor the number of times all historical storage contents in the historical form are called; if the number of calls corresponding to the target historical storage content reaches the preset hotspot number threshold, then using data migration to write the target historical storage content from the historical form into a preset new sharding table, and deleting the target historical storage content in the historical form.

[0119] By adopting a preset call monitoring component, the number of times all historical storage contents in the historical form are called is continuously monitored, so that when the number of data calls in the historical form is too large and exceeds the number of calls required by the hot data, the transfer processing can be carried out in time.

[0120] The present application obtains the target storage data; verifies the storage category of all the contents to be stored contained in the target storage data by querying the historical form; writes the newly added incremental data into the preset new sharding table and the historical form at the same time; uses data migration to write the current contents to be stored from the historical form into the preset new sharding table for the hot data in the historical stock data; deletes the current contents to be stored from the historical form; monitors the number of times the non-hot data is called, so that when the number of calls is too many, it can be transferred and processed in time, which not only avoids excessive data in the historical form, but also can store the hot data and non-hot data in separate tables, which is convenient for data maintenance personnel to perform data maintenance. Especially in the field of financial business with a large amount of stored data, the data storage method described in the present application is adopted to make the data storage mode more scientific and reasonable.

[0121] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0122] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0123] In the embodiment of the present application, by acquiring the target storage data; verifying the storage category of all the contents to be stored contained in the target storage data by querying the historical form; writing the newly added incremental data into the preset new sharding table and the historical form at the same time, using the data migration method for the hot data in the historical stock data to write the current contents to be stored from the historical form into the preset new sharding table, and deleting the current contents to be stored in the historical form, monitoring the number of calls of non-hot data, so that when the number of calls is too many, timely transfer and storage processing is performed, which not only avoids excessive data in the historical form, but also can store hot data and non-hot data in separate tables, which is convenient for data maintenance personnel to perform data maintenance. Especially in the field of financial business with a large amount of stored data, the data storage method described in the present application is adopted to make the data storage mode more scientific and reasonable.

[0124] Further references Figure 5 , as a response to the above Figure 2In order to realize the method shown in the figure, the present application provides an embodiment of a data storage device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0125] like Figure 5 As shown, the data storage device 500 described in this embodiment includes: a target storage data acquisition module 501, a storage category verification query module 502, a hot data judgment module 503, a storage content first writing module 504 and a storage content second writing module 505. Among them:

[0126] The target storage data acquisition module 501 is used to acquire the target storage data;

[0127] The storage category verification query module 502 is used to verify the storage category of all the to-be-stored contents contained in the target storage data by querying the history form, wherein the storage category verification includes verifying whether the current to-be-stored contents are historical stock data or newly added incremental data;

[0128] The hot data judgment module 503 is used to judge whether the current content to be stored contained in the target storage data is hot data according to a preset identification strategy if the current content to be stored contained in the target storage data can be queried in the historical form and the current content to be stored is historical stock data;

[0129] The first storage content writing module 504 is used for writing the current content to be stored from the history table to a preset new sharding table by adopting a data migration method if the current content to be stored is hot data, and deleting the current content to be stored in the history table;

[0130] The second storage content writing module 505 is used to write the current content to be stored into the preset new shard table and the historical form if the current content to be stored contained in the target storage data cannot be queried in the historical form and the current content to be stored is the newly added incremental data.

[0131] The present application obtains the target storage data; verifies the storage category of all the contents to be stored contained in the target storage data by querying the historical form; writes the newly added incremental data into the preset new sharding table and the historical form at the same time; uses data migration to write the current contents to be stored from the historical form into the preset new sharding table for the hot data in the historical stock data; deletes the current contents to be stored from the historical form; monitors the number of times the non-hot data is called, so that when the number of calls is too many, it can be transferred and processed in time, which not only avoids excessive data in the historical form, but also can store the hot data and non-hot data in separate tables, which is convenient for data maintenance personnel to perform data maintenance. Especially in the field of financial business with a large amount of stored data, the data storage method described in the present application is adopted to make the data storage mode more scientific and reasonable.

[0132] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0133] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0134] To solve the above technical problems, the present application also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0135] The computer device 6 includes a memory 6a, a processor 6b, and a network interface 6c which are interconnected through a system bus. It should be noted that: Figure 6Only a computer device 6 having a component memory 6a, a processor 6b, and a network interface 6c is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0136] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0137] The memory 6a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 6a can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 6a can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (Smart Med i aCard, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 6a can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 6a is generally used to store the operating system and various application software installed on the computer device 6, such as a computer-readable instruction of a data storage method, etc. In addition, the memory 6a can also be used to temporarily store various types of data that have been output or are to be output.

[0138] The processor 6b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 6b is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 6b is used to run the computer-readable instructions stored in the memory 6a or process data, such as computer-readable instructions for running the data storage method.

[0139] The network interface 6c may include a wireless network interface or a wired network interface. The network interface 6c is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0140] The computer device proposed in this embodiment belongs to the field of data storage technology and is applied to big data storage scenarios in financial services. This application obtains target storage data; verifies the storage category of all the content to be stored contained in the target storage data by querying the historical form; writes the newly added incremental data to the preset new sharding table and the historical form at the same time, and uses data migration to write the current content to be stored from the historical form to the preset new sharding table for the hot data in the historical stock data, and deletes the current content to be stored in the historical form, and monitors the number of calls of non-hot data, so that when the number of calls is too many, it can be transferred and processed in time, which not only avoids excessive data in the historical form, but also can store hot data and non-hot data in separate tables, which is convenient for data maintenance personnel to perform data maintenance. Especially in the field of financial services with a large amount of stored data, the data storage method described in this application is adopted to make the data storage mode more scientific and reasonable.

[0141] The present application also provides another embodiment, that is, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the data storage method as described above.

[0142] The computer-readable storage medium proposed in this embodiment belongs to the field of data storage technology and is applied to big data storage scenarios in financial services. This application obtains target storage data; verifies the storage category of all the content to be stored contained in the target storage data by querying the historical form; writes the newly added incremental data to the preset new sharding table and the historical form at the same time, and uses data migration to write the current content to be stored from the historical form to the preset new sharding table for the hot data in the historical stock data, and deletes the current content to be stored in the historical form, and monitors the number of calls of non-hot data, so that when the number of calls is too many, it can be transferred and processed in time, which not only avoids excessive data in the historical form, but also can store hot data and non-hot data in separate tables, which is convenient for data maintenance personnel to perform data maintenance. Especially in the field of financial services with a large amount of stored data, the data storage method described in this application is adopted to make the data storage mode more scientific and reasonable.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0144] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data storage method, characterized in that: The steps include: Get the target storage data; Performing storage category verification on all the to-be-stored contents contained in the target storage data by querying the historical form, wherein the storage category verification includes verifying whether the current to-be-stored contents are historical stock data or newly added incremental data; If the current content to be stored contained in the target storage data can be queried in the history table, the current content to be stored is historical stock data, and according to a preset identification strategy, it is determined whether the current content to be stored is hot data; If the current content to be stored is hot data, the current content to be stored is written into a preset new sharding table from the historical table by data migration, and the current content to be stored is deleted from the historical table; If the current content to be stored contained in the target storage data cannot be found in the historical table, the current content to be stored is newly added incremental data, and the current content to be stored is written into the preset new shard table and the historical table.

2. The data storage method according to claim 1, characterized in that: Before executing the step of verifying the storage category of all the to-be-stored contents contained in the target storage data by querying the history table, the method further includes: Generate a new shard table for data storage according to a preset first parameter, a second parameter and a preset sharding condition, wherein the first parameter includes an estimated total storage data volume, and the second parameter includes a maximum storage data volume of a single table. The preset fragmentation conditions include: X∈2 N , and X is the distance The smallest power of 2, In the formula, X represents the number of shards, N is a positive integer, m is the first parameter, and n is the second parameter. Indicates that the ratio of the first parameter to the second parameter is rounded up by adding 1; Obtain storage address information of all shards in the new shard table.

3. The data storage method according to claim 2, characterized in that: After executing the step of generating a new shard table for data storage, the method further includes: Each shard in the new shard table is numbered using a binary method, wherein the decimal number corresponding to the number is an integer value in the interval [0, X), and X represents the number of shards.

4. The data storage method according to claim 1, characterized in that: The step of verifying the storage category of all the to-be-stored contents contained in the target storage data by querying the history table specifically includes: Selecting different contents to be stored in sequence from all contents to be stored contained in the target storage data as current storage contents; Input the current stored content as an index item into a preset search engine Elasticsearch, and perform a current stored content query from the historical table through a query interface of the search engine Elasticsearch; If the current stored content can be queried in the historical form, the current stored content is marked with a preset first distinguishing identifier, and a pair of data consisting of the first distinguishing identifier and the current stored content is output through the output interface of the search engine Elasticsearch; If the current stored content cannot be found in the history table, the current stored content is marked with a preset second distinguishing identifier, and a prompt content including the second distinguishing identifier and the current stored content is sent to the target prompt terminal; Based on the distinguishing identifier corresponding to the current storage content, the storage category of the current storage content is verified.

5. The data storage method according to claim 4, characterized in that: The step of verifying the storage category of the current storage content based on the distinguishing identifier corresponding to the current storage content specifically includes: If the distinguishing identifier corresponding to the current storage content is the first distinguishing identifier, then the current storage content is historical stock data; If the distinguishing identifier corresponding to the currently stored content is the second distinguishing identifier, then the currently stored content is newly added incremental data.

6. The data storage method according to claim 1 or 3, characterized in that: The step of writing the current content to be stored into a preset new sharding table specifically includes: By parsing the target storage data, identifying a unique user number corresponding to the current content to be stored, wherein the user number is a random integer; According to the preset algorithm formula: Perform a remainder calculation and perform binary conversion on the remainder to obtain a binary conversion result, where I represents the user number, X represents the number of shards, and Y represents the ratio of the user number to the number of shards; Compare the binary conversion result with the distinguishing number of each shard in the new shard table, and select a shard whose distinguishing number is consistent with the binary conversion result as the storage shard of the current content to be stored; Identify storage address information corresponding to the storage slice of the current content to be stored; The current content to be stored is stored based on the storage address information corresponding to the storage slice of the current content to be stored.

7. The data storage method according to claim 1, characterized in that: The method further comprises: Using a preset call monitoring component to continuously monitor the number of times all historical storage contents in the historical form are called; If the number of calls corresponding to the target historical storage content reaches the preset hotspot number threshold, the target historical storage content is written from the historical form to the preset new shard table using data migration, and the target historical storage content is deleted from the historical form.

8. A data storage device, characterized in that: include: A target storage data acquisition module is used to acquire target storage data; A storage category verification query module, used to verify the storage category of all the to-be-stored contents contained in the target storage data by querying the history form, wherein the storage category verification includes verifying whether the current to-be-stored contents are historical stock data or newly added incremental data; A hot data judgment module, used for judging whether the current content to be stored contained in the target storage data is hot data according to a preset identification strategy if the current content to be stored contained in the target storage data can be queried in the historical form and the current content to be stored is historical stock data; A first storage content writing module is used for, if the current content to be stored is hot data, writing the current content to be stored from the historical table into a preset new sharding table by using a data migration method, and deleting the current content to be stored in the historical table; The second storage content writing module is used to write the current content to be stored into the preset new shard table and the historical form if the current content to be stored contained in the target storage data cannot be queried in the historical form and the current content to be stored is the newly added incremental data.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data storage method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data storage method according to any one of claims 1 to 7.

Images