A business data processing method, device, equipment and storage medium

Abstract

The present application discloses a business data processing method, device, equipment and storage medium, the method comprising: obtaining business data from an original server; storing data in the business data that meets business conditions in a new server according to a combination of business attribute hash and data component hash; obtaining the stored data from the new server by a multi-shard concurrent traversal method; and sending the stored data to a user account corresponding to the stored data when the acquired stored data meets the sending requirements. The business data can be hashed and stored by a hash combination routing method to avoid hash collisions of the data, and the stored data can be traversed by a multi-shard concurrent traversal method to quickly traverse data with the same business attributes, thereby improving traversal performance.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing, and in particular, to a business data processing method, apparatus, device and storage medium. Background Art

[0002] Data traversal refers to scanning all or part of the existing data and passing the obtained data to the next node for processing. Existing data traversal solutions generally include traditional MYSQL data traversal and memory-based data traversal.

[0003] In the process of implementing the present invention, it is found that the following technical problems exist in the prior art: In the scenario of massive data, the traditional MYSQL data traversal method has the defect of substandard performance, and even cannot realize the traversal of massive data. For example, when the amount of data in a single table reaches 5 million, the performance will drop sharply, resulting in the inability to realize the full traversal. The traversal method based on the memory carrier takes up about 8GB of memory space in the case of 30 million business data, and about 800GB of space is required for 3 billion business data, and backup or status records need to be made at the hardware level, which requires additional overhead, resulting in high costs. Summary of the invention

[0004] In order to solve at least one of the above technical problems, the embodiments of the present application provide the following solution.

[0005] In a first aspect, an embodiment of the present application further provides a method for processing business data, the method comprising:

[0006] Get business data from the original server;

[0007] The data that meets the business conditions in the business data is stored in the new server according to the combination of the business attribute hash and the data component hash;

[0008] Get the stored data from the new server by concurrently traversing multiple shards;

[0009] When the acquired stored data meets the sending requirement, the stored data is sent to the user account corresponding to the stored data.

[0010] In a second aspect, an embodiment of the present application further provides a service data processing device, the device comprising:

[0011] The acquisition module is used to obtain business data from the original server;

[0012] A storage module is used to store data that meets the business conditions in the business data to the new server in a manner of combining the business attribute hash and the data component hash;

[0013] The acquisition module is also used to obtain stored data from the new server by concurrently traversing multiple shards;

[0014] The sending module is used to send the stored data to the user account corresponding to the stored data when the acquired stored data meets the sending requirements.

[0015] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a business data processing method as provided in any embodiment of the present application is implemented.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the business data processing method provided in any embodiment of the present application is implemented.

[0017] The embodiment of the present application provides a business data processing method, device, equipment and storage medium, the method comprising: obtaining business data from the original server; storing the data in the business data that meets the business conditions to the new server according to the business attribute hash and data component hash combination method; obtaining the stored data from the new server by multi-shard concurrent traversal method; when the acquired stored data meets the sending requirements, sending the stored data to the user account corresponding to the stored data. The hash combination routing method can realize hash storage of business data to avoid hash collision of data, and traversing the stored data by multi-shard concurrent traversal method can realize fast traversal of data with the same business attributes, thereby improving traversal performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flowchart of a method for processing business data in an embodiment of the present application;

[0019] Figure 2 It is a flow chart of a method for storing data that meets business conditions in business data to a new server in an embodiment of the present application by combining business attribute hashes with data component hashes;

[0020] Figure 3 is a flow chart of a method for combining a business attribute hash with a data component hash in an embodiment of the present application;

[0021] Figure 4 It is a flow chart of a method for acquiring stored data from a new server by concurrently traversing multiple shards in an embodiment of the present application;

[0022] Figure 5 is a structural diagram of a business data processing device in an embodiment of the present application;

[0023] Figure 6 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0025] In addition, in the embodiments of the present application, words such as "optionally" or "exemplarily" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "optionally" or "exemplarily" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "optionally" or "exemplarily" is intended to present related concepts in a specific way.

[0026] In order to facilitate a clearer understanding of the solution provided in the embodiments of the present application, the relevant concepts involved in the embodiments of the present application are further explained here, as follows:

[0027] Elasticsearch: is a search server based on Lucene that provides a distributed, multi-user, full-text search engine that implements data transfer based on a RESTful web interface. Elasticsearch is used in cloud computing and can implement real-time search. It is stable, reliable, and easy to install and use. Clients are available in Java, .NET (C#), PHP, Python, Apache Groovy, Ruby, and many other languages.

[0028] Figure 1 A flowchart of a business data processing method provided in an embodiment of the present application, which can be applied to a database storing massive data, such as Figure 1 As shown, the method may include but is not limited to the following steps:

[0029] S101. Obtain business data from an original server.

[0030] Exemplarily, the original server in the embodiment of the present application can use elasticsearch, which can retain recent historical records to provide a traceback traversal function, and the hard disk space of the elasticsearch server is relatively large relative to the memory space, so there is no risk of data being unable to be stored due to business changes. In the embodiment of the present application, the number of original servers can be one or more, such as assuming that there are three elasticsearchs as original servers for storing business data. Optionally, business data can be obtained from the elasticsearch cluster by means of cursor traversal (scroll) in the prior art.

[0031] The business data in this step may include user-related data in various network platform scenarios, for example, user coupons, user orders, items in orders, and other data information on e-commerce platforms.

[0032] S102. Store the data that meets the business conditions in the business data to the new server according to the combination of the business attribute hash and the data component hash.

[0033] The business conditions in this step can be conditions for filtering business data that are adaptively set according to the usage requirements of the scenario. Taking the e-commerce platform as an example, if the business data is a user's coupon, then the remaining time of the user's coupon from the expiration date can be set as less than or equal to the warning period as a business condition for filtering data, where the warning period can be understood as the advance time for sending a reminder message to the user account. For example, assuming the warning period is 10 days, then based on the combination of business attribute hash and data component hash, relevant data with a remaining valid time of less than or equal to 10 days from the coupon expiration date can be filtered out from the massive user coupon data, and the filtered data can be stored in the new server.

[0034] It should be noted that the warning period here can be one or more, and different warning periods can be set for different types of coupons. For example, the warning period for coupons for toiletries is 7 days, the warning period for coupons for books is 10 days, the warning period for coupons for daily necessities is 20 days, etc. Regardless of the type of coupon, the data can be filtered according to the above business conditions.

[0035] Optionally, the new server may also use elasticsearch, that is, a new elasticsearch other than the original elasticsearch cluster used to store business data may be used as a new server to store data that meets business conditions. When storing data that meets business conditions in the new server, the msgpack technology may be used to compress the data to save server storage resources.

[0036] S103. Obtain the stored data from the new server by concurrently traversing multiple shards.

[0037] In this step, multiple slice concurrent traversal can be understood as traversing multiple slices in the server in parallel and traversing one slice in series. Through the method of multiple slice concurrent traversal, fast traversal can be achieved to obtain data stored on multiple slices in the new server, thereby improving traversal performance.

[0038] S104: When the acquired stored data meets the sending requirement, the stored data is sent to a user account corresponding to the stored data.

[0039] The sending requirement in this step can be a trigger condition that can be set according to the scenario requirements. Similarly, taking the coupons of the above-mentioned e-commerce platform as an example, the cumulative amount of coupons can be used as the sending requirement for sending data, that is, when the cumulative amount of coupons reaches a certain amount, data can be sent to the corresponding user account; or, the number of coupons can be used as the sending requirement for sending data, for example, when the number of coupons reaches a certain number, data can be sent to the corresponding user account; or, the cumulative amount can be combined with the screening conditions of the above-mentioned business data as a sending requirement, for example, if the remaining time from the expiration date of any coupon is equal to the warning period, and the cumulative amount of all coupons including any coupon reaches a certain amount, then the relevant data message can be sent to the corresponding user account at one time.

[0040] Of course, the above sending requirements are only exemplary descriptions, and those skilled in the art may adaptively set relevant conditions according to the requirements of the network platform or different scenarios, and the embodiments of the present application are not limited to this.

[0041] The embodiment of the present application provides a business data processing method, which may include: obtaining business data from the original server; storing the data that meets the business conditions in the business data to the new server according to the business attribute hash and data component hash combination method; obtaining the stored data from the new server by multi-shard concurrent traversal method; and sending the stored data to the user account corresponding to the stored data when the acquired stored data meets the sending requirements. The business data can be hashed and stored by hash combination routing to avoid data hash collisions, and the stored data can be traversed by multi-shard concurrent traversal method to quickly traverse the data with the same business attributes, thereby improving traversal performance.

[0042] like Figure 2 In one example, the implementation of step S102 may include but is not limited to the following steps:

[0043] S201. Scan the data that meets the business conditions by using a multi-shard concurrent traversal method according to the business attributes of the business data, and use the data that meets the business conditions as target data.

[0044] Exemplarily, the business attributes of the above business data may include a user's unique identification pin, that is, taking the user identification pin as the object and adopting a multi-shard concurrent traversal method to traverse the data, all data corresponding to the user identification pin and meeting the business conditions can be scanned out.

[0045] For example, taking the user coupons in the above e-commerce platform scenario as an example, assume that the business conditions set include: coupon expiration date <= current time + warning period. If a user has 10 coupons, 5 of which have expiration dates < current time + warning period, and 1 has expiration date = current time + warning period, that is, the user has 6 coupons that meet the set business conditions, then the data information of the user's 6 coupons can be quickly scanned through multi-shard traversal, and the data information of these 6 coupons is the target data corresponding to the business attributes in this step.

[0046] S202. Determine the shards of each target data according to a combination of a business attribute hash and a data component hash.

[0047] In an embodiment of the present application, the business attribute hash may include the hash value of the same business attribute in each target data stored in the new server. Exemplarily, the implementation method of this step may include: determining the hash value of the same business attribute in each target data, for each target data, combining the hash value of the data component of the current target data with the hash value of the same business attribute, and taking the modulus of the number of shards according to the combined hash value to obtain the shard identifier corresponding to the current target data. Then, the shard corresponding to the shard identifier is determined as the shard to which the current target data belongs.

[0048] The hash value combination in the above implementation process can be understood as the addition of hash values. The data component can be used to scan data, and one piece of data corresponds to a data component of the scanned data.

[0049] By combining the hash values ​​for routing, it is possible to ensure that data with the same hash value of the business attribute is hashed and stored, thereby avoiding uneven data distribution caused by hash collisions.

[0050] S203: Store each target data in the corresponding slice in the new server.

[0051] After the shards corresponding to each target data are determined through the above process, each target data can be stored in the corresponding shard in the new server, so as to achieve the purpose of hashing and storing each target data.

[0052] like Figure 3 As shown, in one example, the implementation of combining the business attribute hash with the data component hash in the above step S202 may include but is not limited to the following implementations:

[0053] S301. Modulo the hash value of the data component of the current target data according to the size of the routing hash value.

[0054] The size of the routing hash value in this step can be understood as the routing range, which is the grouping of shards, that is, taking a group of shards as the object, and taking the remainder of the hash value of the data component of the current target data. Since each data corresponds to its own data component, the data component factor is considered in the routing process, and the data can be hashed and stored.

[0055] S302: Combine the remainder result with the hash value of the same business attribute.

[0056] Suppose the routing hash value size is partition_rout_size, the hash value of the data component corresponding to the data is hash(docId), and the hash value of the same business attribute is hash(routing), then the combination of the above business attribute hash and data component hash can be expressed as hash(routing)+hash(docId)%partition_rout_size.

[0057] Furthermore, the expression method of determining the shard identifier corresponding to the current target data in step S202 may include: shardNo=(hash(routing)+hash(docId)%partition_rout_size)%

[0058] primary_shard_nums

[0059] Among them, shardNo indicates the shard ID, and primary_shard_nums indicates the number of shards.

[0060] like Figure 4 As shown, in an example, the implementation of the above step S103 may include:

[0061] S401. Traverse the business attributes of data in a single shard in a cursor traversal manner to determine the data stored in the single shard of the new server.

[0062] In the process of data cleaning, the combination of business attribute hash and data component hash is used to screen and store data, so as to ensure that all data with the same business attribute are on the same shard. In this step, the data can be scanned by cursor traversal according to the business attribute of the data, and the data corresponding to the same business attribute in the single shard can be determined in units of business attribute, so as to compactly traverse the data with the same attribute.

[0063] S402: traverse the multiple shards in a multi-shard concurrent traversal manner to obtain the stored data from the new server.

[0064] In the embodiment of the present application, by combining multi-shard concurrent traversal and cursor traversal, it is possible to achieve better traversal performance and quickly traverse the target data corresponding to the business attributes, which has a better traversal advantage in massive data scenarios. After traversing and obtaining the data, the data can be decompressed by msgpack technology and then related business processing can be performed, such as sending a reminder message of business data to the user account.

[0065] In an example, the implementation method of sending the stored data to the user account corresponding to the stored data in the above step S104 may include: sending the stored data to the user account corresponding to the stored data at a specified time.

[0066] The specified time in this implementation process can be a trigger time for sending data set according to the needs of the scenario, and the setting of the specified time is used to flexibly trigger data sending. That is, if the stored data meets the sending requirements, the stored data can be sent to the user account corresponding to the stored data at the specified time. For example, after traversing the stored data, if the sending requirements are met, such as the expiration date of 5 coupons < the time of the day + the warning period, and the expiration date of 1 coupon = the time of the day + the warning period, and the cumulative amount of these 6 coupons is greater than the set amount threshold, then the data can be sent immediately to the user account corresponding to the data, or the traversed data can be sent to the corresponding user account after a preset time interval. For example, data cleaning is performed at 3 am, business data that meets business conditions is obtained, and it is stored in a new server. If the relevant business data meets the sending requirements, the data traversed from the new server can be sent to the user account at 9 am.

[0067] Figure 5 A business data processing device provided in an embodiment of the present application, such as Figure 5 As shown, the device may include an acquisition module 501, a storage module 502, and a sending module 503;

[0068] The acquisition module is used to acquire business data from the original server;

[0069] A storage module is used to store data that meets the business conditions in the business data to the new server in a manner of combining the business attribute hash and the data component hash;

[0070] The acquisition module is also used to obtain stored data from the new server by concurrently traversing multiple shards;

[0071] The sending module is used to send the stored data to the user account corresponding to the stored data when the acquired stored data meets the sending requirements.

[0072] In one example, the storage module may include a scanning unit, a determining unit, and a storage unit;

[0073] The scanning unit is used to scan the data that meets the business conditions by using a multi-shard concurrent traversal method according to the business attributes of the business data, and take the data that meets the business conditions as the target data;

[0074] A determination unit, used to determine the shards of each target data according to a combination of a business attribute hash and a data component hash;

[0075] The storage unit is used to store each target data in the corresponding slice in the new server.

[0076] Exemplarily, the above-mentioned determination unit can be used to perform the following process to determine the shards of each target data, for example, determine the hash value of the same business attribute in each target data; for each target data, combine the hash value of the data component of the current target data with the hash value of the same business attribute, and take the modulus of the number of shards according to the combined hash value to obtain the shard identifier corresponding to the current target data; determine the shard corresponding to the shard identifier as the shard to which the current target data belongs.

[0077] Furthermore, the determination unit may also be configured to modulo the hash value of the data component of the current target data according to the size of the routing hash value, and combine the modulo result with the hash value of the same service attribute.

[0078] In one example, an acquisition module is used to traverse the business attributes of the data in a single shard by means of a cursor traversal to determine the data stored in the single shard of the new server; and to traverse multiple shards by means of multiple shard concurrent traversal to obtain the stored data from the new server.

[0079] Furthermore, the acquisition module can also be used to scan the data in a cursor traversal manner according to the business attributes of the data; and determine the data corresponding to the same business attributes in a single shard of the new server based on the business attributes.

[0080] In one example, the sending module is further used to send the stored data to a user account corresponding to the stored data at a specified time.

[0081] The above-mentioned business data processing device can execute Figure 1-Figure 4 The provided business data processing method has corresponding devices and beneficial effects in the method.

[0082] Figure 6 A schematic diagram of the structure of an electronic device provided in Embodiment 6 of the present invention is shown in FIG. Figure 6 As shown, the electronic device includes a controller 601, a memory 602, an input device 603, and an output device 606; the number of the controller 601 in the electronic device can be one or more. Figure 6 A controller 601 is taken as an example; the controller 601, the memory 602, the input device 603 and the output device 606 in the cloud management platform can be connected by a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0083] The memory 602 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and modules, such as Figure 1-Figure 4 Program instructions / modules corresponding to the business data processing method in the embodiment (for example, the acquisition module 501, the storage module 502, and the sending module 503 in the business data processing device). The controller 601 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 602, that is, implements the above-mentioned business data processing method.

[0084] The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 602 may further include a memory remotely arranged relative to the controller 601, and these remote memories may be connected to the terminal / server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0085] The input device 603 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 606 may include a display device such as a display screen.

[0086] The present application also provides a storage medium containing computer executable instructions, which are used to execute a business data processing method when executed by a computer controller. The method includes: Figure 1-Figure 4 Steps shown.

[0087] Through the above description of the implementation method, the technicians in the relevant field can clearly understand that the present application can be implemented with the help of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0088] It is worth noting that the modules included in the above-mentioned business data processing device are only divided according to functional logic, but are not limited to the above-mentioned division method. As long as the corresponding functions can be achieved, it is not used to limit the protection scope of this application.

[0089] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A business data processing method, characterized in that: include: Get business data from the original server; Scan the data that meets the business conditions by using a multi-shard concurrent traversal method according to the business attributes of the business data, and use the data that meets the business conditions as target data; Determine the shards of each target data according to the combination of the business attribute hash and the data component hash; For each of the target data, the method of combining the business attribute hash of the current target data with the data component hash includes: taking the remainder of the hash value of the data component of the current target data according to the size of the routing hash value; combining the remainder result with the hash value of the business attribute of the current target data; Storing each of the target data in a corresponding shard in the new server; Obtaining stored data from the new server by concurrently traversing multiple shards; When the acquired stored data meets the sending requirement, the stored data is sent to a user account corresponding to the stored data.

2. The method according to claim 1, characterized in that Determining the shards of each target data according to the combination of the business attribute hash and the data component hash includes: Determine the hash value of the same business attribute in each of the target data; For each of the target data, the hash value of the data component of the current target data is combined with the hash value of the same business attribute, and the number of shards is modulo the combined hash value to obtain a shard identifier corresponding to the current target data; The slice corresponding to the slice identifier is determined as the slice to which the current target data belongs.

3. The method according to claim 1, characterized in that Acquiring the stored data from the new server by concurrently traversing multiple shards includes: Traversing the business attributes of the data in a single shard in a cursor traversal manner to determine the data stored in the single shard of the new server; The multiple shards are traversed in a multi-shard concurrent traversal manner to obtain the stored data from the new server.

4. The method according to claim 3, characterized in that Traversing the business attributes of the data in a single shard in a cursor traversal manner to determine the data stored in the single shard of the new server includes: Scan data using cursor traversal based on the business attributes of the data; Taking the business attribute as a unit, determine the data corresponding to the same business attribute in a single shard of the new server.

5. The method according to claim 1, characterized in that Sending the stored data to a user account corresponding to the stored data includes: The stored data is sent to a user account corresponding to the stored data at a specified time.

6. A business data processing device, characterized in that: include: The acquisition module is used to obtain business data from the original server; A storage module is used to store the data that meets the business conditions in the business data in a new server in a manner of combining the business attribute hash and the data component hash; wherein the storage module includes a scanning unit, a determination unit and a storage unit; the scanning unit is used to scan the data that meets the business conditions in a manner of concurrent traversal of multiple shards according to the business attributes of the business data, and use the data that meets the business conditions as target data; the determination unit is used to determine the shard of each target data in a manner of combining the business attribute hash and the data component hash; the storage unit is used to store each target data in a corresponding shard in the new server; for each target data, the manner of combining the business attribute hash of the current target data with the data component hash includes: taking the remainder of the hash value of the data component of the current target data according to the size of the routing hash value; and combining the remainder result with the hash value of the business attribute of the current target data; The acquisition module is further used to acquire the stored data from the new server by concurrently traversing multiple shards; The sending module is used to send the stored data to the user account corresponding to the stored data when the acquired stored data meets the sending requirements.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the business data processing method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the business data processing method according to any one of claims 1 to 5 is implemented.

Images