Method, device and equipment for processing sub-library and sub-table data and medium
Through the library and table division method based on user level and number identification, the problem of data cannot be effectively dispersed in the prior art is solved, and more efficient storage space utilization and query support are achieved.
Patent Information
- Application Number
- CN202510421213.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-03
- Publication Date
- 2025-07-08
AI Technical Summary
The existing database and table division methods usually use the penultimate first and second digits of the username to divide the database, and the penultimate third and fourth digits to divide the database, resulting in the inability to effectively disperse the data, and there are empty database and large database and large tables, and the storage space is not effectively utilized.
By obtaining the user ID and order number identification, the user level is determined based on pre-set hierarchical rules, the data is divided into databases using different library rules, and the table is divided based on the order number identification. Character conversion and calculation rules are used to convert the user ID and order number identification into pure number format to determine the data repository and storage table.
It effectively reduces the situation of empty databases and large databases and large databases, improves the utilization rate of storage space, unifies the routing rules of data from different sources, expands the application fields, and supports better scope query and data discreteness.
Smart Images
Figure CN120277115A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of data processing and digital medicine. Specifically, it relates to a method, device, equipment and medium for processing sharded database and sharded table data. Background Art
[0002] With the development of the Internet and the gradual expansion of Internet medical services and fintech, a large amount of medical data will be generated in online medical treatment, such as in the processes of registration, outpatient service, examination, and drug collection; a large amount of transaction data will also be generated in the financial transaction process, such as in the processes of signing, payment, credit, and mortgage. The data traffic carried by the network system is getting larger and larger, and distributed servers and microservices have emerged accordingly. The pressure on the application layer is dispersed through machine expansion, and the pressure on the data storage layer can no longer be met only by a single database on a single machine. The data storage begins to expand horizontally and vertically, and sharding the database and sharding the table are one of the expansion means.
[0003] For sharding the database and sharding the table, a rule needs to be defined to store business data in different databases and different tables, and it can be queried and aggregated together through the business order number. However, currently, the sharding rules for the database and the table usually use the last two digits of the user name for database sharding and the third and fourth digits from the end of the user name for table sharding, resulting in ineffective dispersion of data, and situations such as empty databases and empty tables, large databases and large tables will occur, and the storage space is not effectively utilized. Therefore, it has become a difficult point to formulate a new method for sharding the database and sharding the table. Summary of the Invention
[0004] In view of the above situation, the embodiments of this application provide a method, device, equipment and medium for processing sharded database and sharded table data, aiming to solve the above problems or at least partially solve the above problems.
[0005] In a first aspect, this application provides a method for processing sharded database and sharded table data, including:
[0006] Obtain a user identifier and an order number identifier;
[0007] Based on a pre-set level rule, determine the user level corresponding to the user identifier;
[0008] When it is determined that the user level corresponding to the user identifier is the first level, shard the data based on the first database sharding rule according to the user identifier; when it is determined that the user level corresponding to the user identifier is the second level, shard the data based on the second database sharding rule according to the user identifier;
[0009] Shard the data based on a pre-set table sharding rule according to the order number identifier.
[0010] In a second aspect, a device for processing sharded database and sharded table data is provided, including:
[0011] An acquisition module, configured to acquire a user identifier and an order number identifier;
[0012] A level module, configured to determine the user level corresponding to the user identifier based on a preset level rule;
[0013] A database partitioning module, configured to, when determining that the user level corresponding to the user identifier is the first level, partition data according to the user identifier based on a first database partitioning rule; when determining that the user level corresponding to the user identifier is the second level, partition data according to the user identifier based on a second database partitioning rule;
[0014] A table partitioning module, configured to partition data according to the order number identifier based on a preset table partitioning rule.
[0015] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the database partitioning and table partitioning data processing method as described in the first aspect are implemented.
[0016] In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the database partitioning and table partitioning data processing method as described in the first aspect are implemented.
[0017] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:
[0018] The present application processes data through the user identifier, partitions the data using a preset table partitioning rule, and then partitions the table using the order number identifier and the table partitioning rule. The basis for database partitioning and table partitioning is no longer limited to the user identifier, which can effectively disperse the data, reduce the occurrence of empty databases and empty tables, and large databases and large tables, and the storage space is effectively utilized. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The accompanying drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The illustrative embodiments and descriptions thereof of the present application are used to explain the present application, and do not constitute an improper limitation of the present application. In the drawings:
[0020] Figure 1 is a schematic application environment diagram of the database partitioning and table partitioning data processing method in an embodiment of the present invention;
[0021] Figure 2 is a schematic flowchart of the database partitioning and table partitioning data processing method in an embodiment of the present invention;
[0022] Figure 3 is Figure 2Schematic flowchart of a specific embodiment of step S30 in
[0023] Figure 4 is Figure 2 Schematic flowchart of another specific embodiment of step S30 in
[0024] Figure 5 is Figure 2 Schematic flowchart of another specific embodiment of step S30 in
[0025] Figure 6 is Figure 2 Schematic flowchart of a specific embodiment of step S40 in
[0026] Figure 7 is Figure 2 Schematic flowchart of another specific embodiment of step S40 in
[0027] Figure 8 Schematic diagram of a structure of the sub-library and sub-table data processing device in an embodiment of the present invention;
[0028] Figure 9 Schematic diagram of a structure of a computer device in an embodiment of the present invention;
[0029] Figure 10 Schematic diagram of another structure of a computer device in an embodiment of the present invention. Specific embodiments
[0030] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.
[0031] It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such use can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the term "including" and its variants should be interpreted as an open term meaning "including but not limited to".
[0032] The technical solutions provided by the embodiments of the present application will be described in detail below in conjunction with the drawings.
[0033] As introduced above, currently, the rules for database and table sharding usually use the last two digits of the username for database sharding and the third and fourth last digits of the username for table sharding. This results in ineffective data dispersion, with situations such as empty databases and tables, as well as large databases and tables, and the storage space is not utilized effectively. To solve this technical problem, the embodiments of the present application provide a method for processing database and table sharding data.
[0034] The method for processing database and table sharding data provided by the embodiments of the present invention can be applied in an application environment such as Figure 1 where the device communicates with the server through a network. The server can obtain the user identifier and the order number identifier through the device; based on the pre-set level rules, determine the user level corresponding to the user identifier; when it is determined that the user level corresponding to the user identifier is the first level, based on the user identifier, perform database sharding on the data according to the first database sharding rule; when it is determined that the user level corresponding to the user identifier is the second level, based on the user identifier, perform database sharding on the data according to the second database sharding rule; based on the pre-set table sharding rule, perform table sharding on the data according to the order number identifier. The present application processes through the user identifier, performs database sharding on the data using the pre-set table sharding rule, and then performs table sharding using the table sharding rule through the order number identifier. The basis for database and table sharding is no longer limited to the user identifier, which can effectively disperse the data, reduce the occurrence of empty databases and tables, as well as large databases and tables, and the storage space is effectively utilized. Among them, the device can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The present invention will be described in detail through specific embodiments below.
[0035] Please refer to Figure 2 as shown in Figure 2 which is a schematic flowchart of a method for processing database and table sharding data provided by an embodiment of the present invention, including the following steps:
[0036] S10: Obtain the user identifier and the order number identifier.
[0037] Exemplarily, the user identifier is the username or merchant name. For different data sources, the form of the username or merchant name may vary. It may be a pure numerical value or may contain special characters such as letters and symbols. The single-order identifier is the actual business order number or business transaction number. For different data sources, the form of the username or merchant name may vary. It may be a pure numerical value or may contain special characters such as letters and symbols. The user identifier and the single-order identifier can be obtained simultaneously, or the user identifier can be obtained first and then the single-order identifier under this user identifier, or the user identifier can be obtained first and then the single-order identifier under this user identifier when needed. Taking online drug purchase as an example, the user identifier can be the user's login username, and the single-order identifier can be the order number for placing an order. Taking outpatient treatment as an example, the user identifier can be the registration number generated when the user registers, and the single-order identifier can be the numbers on various bill forms such as the user's subsequent examination reports, medical treatment reports, medical treatment forms, and prescription pickup forms. Taking a financial shopping platform as an example, the user identifier can be the user registration number, and the single-order identifier can be the user's commodity browsing record data, order number data for placing an order, payment data, and purchase record data.
[0038] S20: Determine the user level corresponding to the user identifier based on the pre-set level rules.
[0039] Exemplarily, the user's level is set in the pre-set level rules.
[0040] In one implementation, the user identifier has been pre-set. For example, the user identifiers of each department in a hospital and each doctor are all known. According to past experience, the user identifiers can be marked with levels, such as being divided into the first level, the second level, etc. Or a mapping relationship between the user identifier and the user level can be pre-set. The corresponding user level can be obtained based on the user identifier.
[0041] In another implementation, the user identifier can be marked with a level based on the historical data volume corresponding to the user identifier. For example, when the data volume corresponding to the user identifier is greater than the preset data volume, the user identifier is marked as the second level; when the data volume corresponding to the user identifier is less than or equal to the preset data volume, the user identifier is marked as the first level.
[0042] Exemplarily, the above-mentioned data volume refers to the business data volume generated daily. The business data volume generated daily by users at the second level is greater than that of users at the first level. The data can be medical data, such as personal health records, prescriptions, inspection reports, etc. The data can also be financial data, such as transaction data, payment data, business data, or purchase data.
[0043] S30: When it is determined that the user level corresponding to the user identifier is the first level, perform database partitioning on the data according to the user identifier based on the first database partitioning rule; when it is determined that the user level corresponding to the user identifier is the second level, perform database partitioning on the data according to the user identifier based on the second database partitioning rule.
[0044] Exemplarily, performing database partitioning on the service data corresponding to the user identifiers of the first level and the second level respectively improves the efficiency of database partitioning, enables the effective dispersion of data, reduces the occurrence of empty databases and empty tables, as well as large databases and large tables, and effectively utilizes the storage space.
[0045] Exemplarily, in one embodiment, as Figure 3 shown, when it is determined that the user level corresponding to the user identifier in step S30 is the first level, performing database partitioning on the data according to the user identifier based on the first database partitioning rule includes:
[0046] S311: Identify the special characters in the user identifier, and based on the pre-set first character processing rule, convert the special characters in the user identifier into numerical values to obtain the target user identifier.
[0047] Exemplarily, the first character processing rule is used to convert the special characters in the user identifier into numerical values, converting the user identifier into a pure numerical form. This setting can process data from multiple sources, broadening the application fields and scenarios.
[0048] Exemplarily, identify the special characters in the user identifier, where the special characters are characters other than the numerical format, such as letters. The first character processing rule pre-sets the mapping relationship between special characters and numbers. For example, A is mapped to the number 1, B is mapped to the number 2, C is mapped to the number 3, D is mapped to the number 4, and so on. For example, when user A visits the doctor, the user identifier 12A5C878B9 generated during registration is converted into the target user identifier 1215387829 according to the first character processing rule. For example, when user A shops financially, the user identifier 12A5C878B7 generated by the registration platform is converted into the target user identifier 1215387827 according to the first character processing rule.
[0049] S312: Obtain the storage repository of the data according to the last two numerical values of the target user identifier.
[0050] Exemplarily, for example, the last two digits of the target user identifier 1215387829 are 29, so the storage repository of the data is repository 29. For example, the last two digits of the target user identifier 1215387827 are 27, so the storage repository of the data is repository 27.
[0051] Exemplarily, in order to shorten the character length of the target user identifier, in another embodiment, as Figure 4As shown, when it is determined in step S30 that the user level corresponding to the user identifier is the first level, the data is partitioned based on the first partitioning rule according to the user identifier, including:
[0052] S311': Identify special characters in the user identifier, convert the special characters in the user identifier into numerical values based on a pre-set first character processing rule, and calculate the converted user identifier into a target user identifier based on a pre-set first calculation rule.
[0053] Exemplarily, identify special characters in the user identifier. The special characters are characters other than the numerical format, such as letters. A mapping relationship between special characters and numbers is pre-set in the first character processing rule. For example, A is mapped to the number 1, B is mapped to the number 2, C is mapped to the number 3, D is mapped to the number 4, and so on. For example, when user A visits a doctor, the user identifier 12A5C878B9 generated during registration is converted into the target user identifier 1215387829 through the first character processing rule. For example, when user A shops financially, the user identifier 12A5C878B7 generated by the registration platform is converted into the target user identifier 1215387827 through the first character processing rule.
[0054] Exemplarily, the first calculation rule is used to shorten the character length of the user identifier. In this embodiment, a ratio value is pre-set in the first calculation rule, and the product of the numbers of the user identifier and the ratio value is calculated to obtain the target user identifier. For example, the ratio value is 1 / 3, and the user identifier 1215387829 converted into a numerical value is calculated as 1215387829 / 3 to obtain the target user identifier 405129277. For example, the ratio value is 1 / 2, and the user identifier 1215387827 converted into a numerical value is calculated as 1215387827 / 2 to obtain the target user identifier 607693914. The target user identifier is rounded to an integer when taking values.
[0055] S312': Obtain the storage repository of the data according to the last two numerical values of the target user identifier.
[0056] Exemplarily, for example, the last two digits of the target user identifier 405129277 are 77, then the storage repository of the data is repository 77; the last two digits of the target user identifier 607693914 are 14, then the storage repository of the data is repository 14.
[0057] Exemplarily, in one implementation, as Figure 5 shown, when it is determined in step S30 that the user level corresponding to the user identifier is the second level, the data is partitioned based on the second partitioning rule according to the user identifier, including:
[0058] S321: Add a time character to the user identifier based on the time character addition rule to obtain a target user identifier.
[0059] Exemplarily, the time character addition rule is used to add a time character representing any one of year, quarter, and month after the last digit of the user identifier. The specific format of the time character can be set according to requirements. For example, the time character N2025 represents the time character of the year 2025, J01 represents the time character of the first quarter, and Y01 represents the time character of January. At least one of the year time character, quarter time character, and month time character is added during addition. For example, when user A visits a doctor, the user identifier 12A5C878B9 generated during registration. If the time character addition rule is used to add the year character N2025, the target user identifier after addition is 12A5C878B9N2025; if the time character addition rule is used to add the year time character N2025 and the quarter time character J01, the target user identifier after addition is 12A5C878B9N2025J01; if the time character addition rule is used to add the year time character N2025, the quarter time character J01, and the month time character Y01, the target user identifier after addition is 12A5C878B9N2025J01Y01. For example, when user A shops financially, the user identifier 12A5C878B7 generated by the registration platform. If the time character addition rule is used to add the year character N2026, the target user identifier after addition is 12A5C878B7N2026; if the time character addition rule is used to add the year time character N2026 and the quarter time character J03, the target user identifier after addition is 12A5C878B7N2026J03; if the time character addition rule is used to add the year time character N2026, the quarter time character J03, and the month time character Y01, the target user identifier after addition is 12A5C878B7N2026J03Y01.
[0060] S322: Obtain the data repository according to the last two digits of the target user identifier.
[0061] Exemplarily, identify that the data of the target user identifier belongs to the data within the time range of the added time character, and store the identified data in the data repository obtained from the last two digits of the target user identifier.
[0062] For example, the added time character is N2025, and the target user identifier is 12A5C878B9N2025. Store the data belonging to the year 2025 under this target user identifier in the 25 repository.
[0063] In step S30, a repository for business data is obtained. After the data is partitioned into different repositories, the method further includes: identifying the amount of data corresponding to the user identifier that has been stored in its corresponding repository, and based on a preset storage threshold, storing the remaining data corresponding to the user identifier in a specified repository.
[0064] Exemplarily, the amount of data stored in the repository is fixed. To avoid the storage difference of the business data under this user identifier in the repository from becoming increasingly large compared to other repositories, in this embodiment, the business data exceeding the storage threshold is stored in other repositories through a specified route.
[0065] Exemplarily, the storage threshold can be a specific data storage amount or a ratio value of the stored data amount to the total data amount of the repository. In this embodiment, the storage threshold is 10%. After the amount of data corresponding to the user identifier that has been stored in its corresponding repository exceeds 10% of the total data amount of the repository, the remaining data corresponding to the user identifier is stored in a specified repository. For example, when it is identified that the business data volume of a certain user identifier is large and the original repository for the business data is repository 25, when it is identified that the amount of data stored in repository 25 for this user identifier exceeds 10% of the total storage capacity of repository 25, the remaining business data of this user identifier is stored in the preset repository 99 through a specified route, avoiding the storage difference of repository 25 from becoming increasingly large compared to other repositories.
[0066] Further, when performing data query, when the required data cannot be found in the original repository, it can be queried from the preset specified repository according to the specified route.
[0067] S40: Based on the order number identifier, partition the data according to a preset data partitioning rule.
[0068] Exemplarily, as Figure 6 shown, step S40 partitions the data based on the order number identifier according to a preset data partitioning rule, including:
[0069] S41: Based on a preset second character processing rule, convert the special characters in the order number identifier into numerical values to obtain a target order number identifier.
[0070] Exemplarily, the second character processing rule is used to convert the special characters in the order number identifier into numerical values, converting the order number identifier into a pure numerical form. This setting can process data from multiple sources, broadening the application fields and scenarios.
[0071] Exemplarily, special characters in the single-order identifier are recognized. The special characters are characters other than the numeric format, such as letters. A mapping relationship between the special characters and numbers is preset in the second character processing rule. For example, A is mapped to the number 1, B is mapped to the number 2, C is mapped to the number 3, D is mapped to the number 4, and so on. For example, when user A visits a doctor, the single-order identifier 12A5C878B9D of the test report is converted into the target single-order identifier 12153878294 according to the second character processing rule. For example, when user A shops financially, the single order number 12A5C878B9D of the payment data is converted into 12153878294 according to the second character processing rule.
[0072] S42: Obtain the storage table of the data according to the last two numerical values of the target single-order identifier.
[0073] Exemplarily, for example, the last two digits of the target single-order identifier 12153878294 are 94, then the storage table of the data is Table 94.
[0074] Exemplarily, in order to shorten the character length of the target single-order identifier, in another embodiment, as Figure 7 shown, step S40 performs sub-table partitioning on the data according to the single-order identifier based on the preset sub-table partitioning rule, including:
[0075] S41': Recognize the special characters in the single-order identifier, convert the special characters in the single-order identifier into numerical values based on the preset second character processing rule, and calculate the converted single-order identifier with numerical values to obtain the target single-order identifier based on the preset second calculation rule.
[0076] Exemplarily, special characters in the single-order identifier are recognized. The special characters are characters other than the numeric format, such as letters. A mapping relationship between the special characters and numbers is preset in the second character processing rule. For example, A is mapped to the number 1, B is mapped to the number 2, C is mapped to the number 3, D is mapped to the number 4, and so on. For example, when user A visits a doctor, the single-order identifier 12A5C878B9D of the test report is converted into the target single-order identifier 12153878294. For example, when user A shops financially, the single order number 12A5C878B9D of the payment data is converted into 12153878294.
[0077] Exemplarily, the second calculation rule is used to shorten the character length of the single number identifier. In this embodiment, a proportional value is preset in the second calculation rule, and the number of the single number identifier is multiplied by the proportional value to obtain the target single number identifier. For example, when the proportional value is 1 / 3, when user A visits a doctor, the single number identifier 12153878294 converted into a numerical value in the test report is calculated as 12153878294 / 3 to obtain the target single number identifier 4051292765; when user A shops financially, the single number identifier 12153878294 converted into a numerical value in the payment data is calculated as 12153878294 / 3 to obtain the target single number identifier 4051292765. When taking the value of the target single number identifier, round it to an integer.
[0078] S42': Obtain the storage table of the data according to the last two numerical values of the target single number identifier.
[0079] Exemplarily, for example, the last two digits of the target single number identifier 4051292765 are 65, then the storage table of the data is Table 65.
[0080] It can be seen that in the above solution, the present application processes through the user identifier, uses the preset sub-table rule to divide the data into databases, and then uses the sub-table rule to divide the table through the single number identifier. The basis for database partitioning and table partitioning is no longer limited to the user identifier, which can effectively disperse the data and reduce the occurrence of empty databases and empty tables, as well as large databases and large tables, and the storage space is effectively utilized.
[0081] In this embodiment, by converting the user identifier and the single number identifier into a pure digital format and then performing simple calculations, the performance loss is negligible. The routing rules for data from different sources are unified, and the application field is expanded. The business data under the same user identifier after processing is in the same database, and range queries can be better supported. Users with a large amount of data can take timely measures, and can guide the data to other specified storage databases through specified routing, reducing the continuous accumulation of data. Second-level users can also be automatically generated annually, quarterly, or monthly, and combined with query degradation to achieve discrete data without manual intervention.
[0082] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
[0083] In one embodiment, a database and table partitioning data processing device is provided, and the database and table partitioning data processing device corresponds one-to-one to the database and table partitioning data processing method in the above embodiment. As Figure 8 shown, the database and table partitioning data processing device includes an acquisition module 101, a grading module 102, a database partitioning module 103, and a table partitioning module 104. The detailed descriptions of each functional module are as follows:
[0084] An acquisition module 101, configured to acquire a user identifier and an order number identifier;
[0085] A level module 102, configured to determine the user level corresponding to the user identifier based on a preset level rule;
[0086] A sub-library module 103, configured to, when the user level corresponding to the user identifier is a first level, perform sub-library of data according to the user identifier based on a first sub-library rule; when the user level corresponding to the user identifier is a second level, perform sub-library of data according to the user identifier based on a second sub-library rule;
[0087] A sub-table module 104, configured to perform sub-table of data according to the order number identifier based on a preset sub-table rule.
[0088] Specifically, the sub-library module 103 is further configured to
[0089] Identify special characters in the user identifier, and based on a preset first character processing rule, convert the special characters in the user identifier into numerical values to obtain a target user identifier;
[0090] Obtain the storage library of data according to the last two numerical values of the target user identifier.
[0091] Specifically, the sub-library module 103 is further configured to
[0092] Identify special characters in the user identifier, based on a preset first character processing rule, convert the special characters in the user identifier into numerical values, and based on a preset first calculation rule, calculate the user identifier converted into numerical values to obtain a target user identifier;
[0093] Obtain the storage library of data according to the last two numerical values of the target user identifier.
[0094] Specifically, the sub-library module 103 is further configured to
[0095] Based on a time character addition rule, add time characters to the user identifier to obtain a target user identifier;
[0096] Obtain the storage library of data according to the last two numerical values of the target user identifier.
[0097] Specifically, the sub-library module 103 is further configured to
[0098] Identify the amount of data that has been stored in the storage library corresponding to the user identifier, and based on a preset storage threshold, store the remaining data corresponding to the user identifier in a specified storage library.
[0099] Specifically, the sub - table module 104 is further configured to,
[0100] Based on a pre - set second character processing rule, convert the special characters in the single - order identifier into numerical values to obtain a target single - order identifier;
[0101] Obtain the storage table of the data according to the last two numerical values of the target single - order identifier.
[0102] Specifically, the sub - table module 104 is further configured to,
[0103] Identify the special characters in the single - order identifier, based on a pre - set second character processing rule, convert the special characters in the single - order identifier into numerical values, and based on a pre - set second calculation rule, calculate the single - order identifier converted into numerical values to obtain a target single - order identifier;
[0104] Obtain the storage table of the data according to the last two numerical values of the target single - order identifier.
[0105] The present invention provides a data processing device for database and table partitioning. Through deep integration of user historical behavior data, the application obtains user historical semantic features and interest preference features, which go beyond the scope of literal matching and can more accurately match user needs to achieve personalized recommendation. At the same time, by utilizing the parallel processing ability of the twin - tower model, the response speed of the recommendation system can be accelerated and the recommendation efficiency can be improved. In this embodiment, by converting the user identifier and the single - order identifier into a pure - digital format and then performing simple calculations, the performance loss is negligible. The routing rules for different - source data are unified, and the application field is expanded. The business data under the same user identifier after processing is in the same database, and range queries can be better supported. Users with large amounts of data can take timely measures. Data can be guided to other specified storage databases through specified routing, reducing the continuous accumulation of data. Secondary users can also be automatically generated annually, quarterly, or monthly, and combined with query degradation to achieve discrete data without manual intervention.
[0106] For the specific limitations of the data processing device for database and table partitioning, reference can be made to the limitations of the data processing method for database and table partitioning in the above text, which will not be elaborated here. Each module in the above - mentioned data processing device for database and table partitioning can be implemented in whole or in part through software, hardware, and their combinations. The above - mentioned modules can be embedded in the processor of the computer device in hardware form or be independent of it, or be stored in the memory of the computer device in software form so that the processor can call and execute the operations corresponding to the above - mentioned modules.
[0107] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 9As shown in the figure. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage media. The network interface of the computer device is used to communicate with external device terminals via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a data processing method for database sharding and table partitioning.
[0108] In one embodiment, a computer device is provided. The computer device can be a device terminal, and its internal structure diagram can be as Figure 10 As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage media. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the device terminal side of a data processing method for database sharding and table partitioning.
[0109] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:
[0110] Obtain a user identifier and an order number identifier;
[0111] Based on a pre-set level rule, determine the user level corresponding to the user identifier;
[0112] When it is determined that the user level corresponding to the user identifier is the first level, based on the user identifier, shard the data according to the first database sharding rule; when it is determined that the user level corresponding to the user identifier is the second level, based on the user identifier, shard the data according to the second database sharding rule;
[0113] According to the order number identifier, partition the data according to a pre-set table partitioning rule.
[0114] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, the following steps are implemented:
[0115] Obtain a user identifier and an order number identifier;
[0116] Determine the user level corresponding to the user identifier based on a pre-set level rule;
[0117] When it is determined that the user level corresponding to the user identifier is the first level, perform database partitioning on the data based on the user identifier and the first database partitioning rule; when it is determined that the user level corresponding to the user identifier is the second level, perform database partitioning on the data based on the user identifier and the second database partitioning rule;
[0118] Perform data table partitioning on the data based on the single order identifier and a pre-set table partitioning rule.
[0119] It should be noted that for the functions or steps that can be achieved by the above-mentioned computer-readable storage medium or computer device, reference can be made to the relevant descriptions on the server side and the device side in the foregoing method embodiments. To avoid repetition, they will not be described in detail here.
[0120] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.
[0121] Those skilled in the art can clearly understand that for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.
[0122] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A method for processing sharded databases and sharded tables, characterized in that: Obtain a user identifier and an order number identifier; Based on a pre-set level rule, determine the user level corresponding to the user identifier; When it is determined that the user level corresponding to the user identifier is the first level, based on the user identifier, shard the data according to the first database sharding rule; When it is determined that the user level corresponding to the user identifier is the second level, based on the user identifier, shard the data according to the second database sharding rule; Based on the order number identifier, shard the data according to a pre-set table sharding rule.
2. The method according to claim 1, characterized in that: When it is determined that the user level corresponding to the user identifier is the first level, based on the user identifier, sharding the data according to the first database sharding rule includes: Identify special characters in the user identifier, and based on a pre-set first character processing rule, convert the special characters in the user identifier into numerical values to obtain a target user identifier; Obtain the storage database of the data according to the last two numerical values of the target user identifier.
3. The method according to claim 1, characterized in that: When it is determined that the user level corresponding to the user identifier is the first level, based on the user identifier, sharding the data according to the first database sharding rule includes: Identify special characters in the user identifier, and based on a pre-set first character processing rule, convert the special characters in the user identifier into numerical values, and based on a pre-set first calculation rule, calculate the user identifier converted into numerical values to obtain a target user identifier; Obtain the storage database of the data according to the last two numerical values of the target user identifier.
4. The method according to claim 1, characterized in that: When it is determined that the user level corresponding to the user identifier is the second level, based on the user identifier, sharding the data according to the second database sharding rule includes: Based on a time character addition rule, add time characters to the user identifier to obtain a target user identifier; Obtain the storage database of the data according to the last two numerical values of the target user identifier.
5. The method according to claim 1, characterized in that: After sharding the data, the method further includes: Identify the amount of data corresponding to the user identifier that has been stored in its corresponding storage database, and based on a pre-set storage threshold, store the remaining data corresponding to the user identifier in a specified storage database.
6. The method according to claim 1, characterized in that: Based on the order number identifier, sharding the data according to a pre-set table sharding rule includes: Based on a pre-set second character processing rule, convert special characters in the order number identifier into numerical values to obtain a target order number identifier; Obtain the storage table of the data according to the last two numerical values of the target order number identifier.
7. The method according to claim 1, characterized in that: Based on the order number identifier, sharding the data according to a pre-set table sharding rule includes: Identify the special characters in the single order identifier, convert the special characters in the single order identifier into numerical values based on the pre-set second character processing rule, and calculate the converted single order identifier based on the pre-set second calculation rule to obtain the target single order identifier; Obtain the storage table of the data according to the last two numerical values of the target single order identifier.
8. A data processing device for database and table sharding, characterized in that Comprising: An acquisition module, configured to acquire a user identifier and a single order identifier; A level module, configured to determine the user level corresponding to the user identifier based on the pre-set level rule; A sub-library module, configured to perform sub-library of data according to the user identifier based on the first sub-library rule when the user level corresponding to the user identifier is the first level; When it is determined that the user level corresponding to the user identifier is the second level, perform sub-library of data according to the user identifier based on the second sub-library rule; A sub-table module, configured to perform sub-table of data according to the single order identifier based on the pre-set sub-table rule.
9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the data processing method for sub-library and sub-table as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the steps of the data processing method for sub-library and sub-table as described in any one of claims 1 to 7 are implemented.