Data processing method, device, electronic device and storage medium

Abstract

A data processing method, device, electronic device and storage medium based on big data processing, the method comprising: obtaining a server list and data to be processed; the server list comprises a plurality of servers; monitoring the status of the server in real time and updating the server list in time; calculating the position parameter corresponding to each server according to the server list; calculating the total length of the server list as the length parameter; slicing the data to be processed to obtain slicing data and calculating the HASH value corresponding to the slicing data; establishing a mapping between each HASH value and each position parameter; according to the mapping between the HASH value and the position parameter, each server processes the corresponding data in parallel. The data processing method of the present invention processes the data in slicing, ensures the unique correspondence between the data and the server, is conducive to improving the data processing speed, and ensures that the data will not be missed.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] With the rapid development and popularization of the Internet, the amount of data in many Internet companies has exploded. Data has penetrated into every industry and field, becoming a weather vane for corporate development. However, faced with huge and complex data, as well as users' demands for quick response to products, the ability to quickly process large amounts of data is particularly important for a company. Traditional systems face a large amount of data, and the system generally starts one and only one scheduled task to search and process the data, and then repeat the process. This will cause the system to have a bottleneck period in data processing within a period of time, which often fails to meet business requirements and limits the performance of the product to some extent. Summary of the invention

[0003] In view of this, it is necessary to provide a data processing method, device, electronic device and storage medium for improving the processing speed of big data.

[0004] A data processing method, comprising:

[0005] When an acquisition instruction is detected, a server list and data to be processed are acquired; the server list includes a plurality of servers;

[0006] When the first monitoring instruction is detected, the status of the server is monitored in real time and the server list is updated in time;

[0007] When a first calculation instruction is detected, calculating a location parameter corresponding to each server according to the server list;

[0008] When a second calculation instruction is detected, calculating the total length of the server list as a length parameter;

[0009] When a shard designation is detected, the data to be processed is sharded to obtain sharded data and a HASH value corresponding to the sharded data is calculated;

[0010] When a matching instruction is detected, a mapping between each of the HASH values ​​and each of the position parameters is established;

[0011] When a processing instruction is detected, each of the servers processes corresponding data in parallel according to the mapping between the HASH value and the location parameter.

[0012] Preferably, the step of real-time monitoring the status of the server and timely updating the server list includes:

[0013] Receive heartbeat signals from each server;

[0014] Determining whether the heartbeat signal is interrupted;

[0015] When the heartbeat signal is detected to be interrupted, the interruption time is accumulated;

[0016] Determining whether the accumulated interruption time is greater than a predetermined time;

[0017] When the accumulated interruption time is greater than the predetermined time, deleting the server from the server list;

[0018] When the heartbeat signal is not interrupted, return to the step of receiving the heartbeat signal sent by each server.

[0019] Preferably, the step of slicing the data to be processed to obtain slicing data and calculating the HASH value corresponding to the slicing data includes:

[0020] Get the IP address of each server;

[0021] Calculate the position parameter of the corresponding server in the server list according to the IP address;

[0022] The position parameters are stored in the Redis database.

[0023] Preferably, the step of establishing a mapping between each of the HASH values ​​and each of the position parameters comprises:

[0024] Slice the data to be processed to obtain slicing data and calculate the primary key value of the slicing data;

[0025] Calculate the HASH value of each of the shard data using a predetermined function according to the length parameter and the primary key value;

[0026] The HASH value is stored in the Redis database.

[0027] Preferably, the step of establishing a mapping between each of the HASH values ​​and each of the position parameters comprises:

[0028] Obtain a HASH value from the Redis database as a query HASH value;

[0029] Obtain one of the position parameters from the Redis database as a query position parameter;

[0030] Determine whether the query HASH value is consistent with the query location parameter;

[0031] When the query HASH value is consistent with the query location parameter, a mapping between the query HASH value and the query location parameter is established to match a row of data corresponding to the query HASH value to the server corresponding to the query location parameter;

[0032] When the query HASH value is inconsistent with the query position parameter, identifying that the query HASH value does not match the query position parameter, and returning to the step of obtaining one of the position parameters from the Redis database as the query position parameter;

[0033] Determine whether the HASH value is the last HASH value;

[0034] When the HASH value is the last HASH value, identifying that all the HASH values ​​have been matched with the position parameters;

[0035] When the HASH value is not the last HASH value, return to the step of obtaining a HASH value from the Redis database as a query HASH value.

[0036] Preferably, the data processing method further includes:

[0037] When the second monitoring instruction is detected, the performance of each server is monitored and the server list is dynamically adjusted.

[0038] Preferably, the step of monitoring the performance of each server and dynamically adjusting the server list comprises:

[0039] Calculate the total amount of data matched by each server at specified intervals;

[0040] Determine whether the total amount of data is greater than a specified threshold;

[0041] When the total amount of data is less than the specified threshold, returning to the step of calculating the total amount of data matched by each server at every specified time;

[0042] When the total amount of data is greater than or equal to the specified threshold, the server is removed from the server list.

[0043] In addition, in order to achieve the above object, the present invention also provides a data processing device, the data processing device comprising:

[0044] An acquisition module, used to acquire a server list and data to be processed when an acquisition instruction is detected; the server list includes multiple servers;

[0045] A first monitoring module, configured to monitor the status of the server in real time and update the server list in time when a first monitoring instruction is detected;

[0046] A first calculation module, configured to calculate a location parameter corresponding to each server according to the server list when a first calculation instruction is detected;

[0047] A second calculation module, configured to calculate the total length of the server list as a length parameter when a second calculation instruction is detected;

[0048] A sharding module, used for sharding the data to be processed to obtain sharded data and calculating the HASH value corresponding to the sharded data when a sharding instruction is detected;

[0049] A matching module, used for establishing a mapping between each of the HASH values ​​and each of the position parameters when a matching instruction is detected;

[0050] The processing module is used for, when a processing instruction is detected, to cause each server to process corresponding data in parallel according to the mapping between the HASH value and the position parameter.

[0051] In addition, in order to achieve the above-mentioned object, the present invention further provides an electronic device, wherein the electronic device includes a processor and a memory, and the processor is configured to perform the following steps when executing a computer program stored in the memory:

[0052] When an acquisition instruction is detected, a server list and data to be processed are acquired; the server list includes a plurality of servers;

[0053] When the first monitoring instruction is detected, the status of the server is monitored in real time and the server list is updated in time;

[0054] When a first calculation instruction is detected, calculating a location parameter corresponding to each server according to the server list;

[0055] When a second calculation instruction is detected, calculating the total length of the server list as a length parameter;

[0056] When a shard designation is detected, the data to be processed is sharded to obtain sharded data and a HASH value corresponding to the sharded data is calculated;

[0057] When a matching instruction is detected, a mapping between each of the HASH values ​​and each of the position parameters is established;

[0058] When a processing instruction is detected, each of the servers processes corresponding data in parallel according to the mapping between the HASH value and the location parameter.

[0059] A storage medium, the storage medium being a computer-readable storage medium, storing at least one instruction, wherein when the at least one instruction is executed by a processor, the following steps are implemented:

[0060] When an acquisition instruction is detected, a server list and data to be processed are acquired; the server list includes a plurality of servers;

[0061] When the first monitoring instruction is detected, the status of the server is monitored in real time and the server list is updated in time;

[0062] When a first calculation instruction is detected, calculating a location parameter corresponding to each server according to the server list;

[0063] When a second calculation instruction is detected, calculating the total length of the server list as a length parameter;

[0064] When a shard designation is detected, the data to be processed is sharded to obtain sharded data and a HASH value corresponding to the sharded data is calculated;

[0065] When a matching instruction is detected, a mapping between each of the HASH values ​​and each of the position parameters is established;

[0066] When a processing instruction is detected, each of the servers processes corresponding data in parallel according to the mapping between the HASH value and the location parameter.

[0067] The above-mentioned data processing method, device, electronic device and storage medium perform data sharding processing based on oracle calculation and ensure the unique correspondence between data and server, which is conducive to improving data processing speed. At the same time, all servers are monitored according to heartbeat monitoring, so that when a server has a problem, it is removed from the server list and the data is reallocated, ensuring that data is not missed. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0069] Figure 1 It is a schematic diagram of the data processing method of the present invention.

[0070] Figure 2 for Figure 1 Detailed flowchart of step S11 in FIG.

[0071] Figure 3for Figure 1 A detailed flowchart of step S12 is shown in FIG.

[0072] Figure 4 for Figure 1 Detailed flowchart of step S14 in FIG.

[0073] Figure 5 for Figure 1 Detailed flowchart of step S15 in FIG.

[0074] Figure 6 for Figure 1 Detailed flowchart of step S17 in FIG.

[0075] Figure 7 It is a functional module diagram of the data processing device of the present invention.

[0076] Figure 8 A schematic diagram of the device structure of the hardware operating environment of an embodiment of the present invention.

[0077] Main component symbols

[0078] Data processing device 1

[0079] Get Module 10

[0080] The first monitoring module 20

[0081] The first calculation module 30

[0082] The second calculation module 40

[0083] Sharding module 50

[0084] Matching module 60

[0085] Processing module 70

[0086] The second monitoring module 80

[0087] Memory 102

[0088] Communication bus 104

[0089] Processor 106

[0090] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0091] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0092] The terms "first", "second", "third", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any variation thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other steps or modules that are inherent to these processes, methods, products or devices.

[0093] The specific implementation of the data processing method of the present invention is described below with reference to the accompanying drawings.

[0094] In at least one embodiment of the present invention, the data processing method is applied to a data processing system consisting of at least one mobile terminal and multiple servers. The data processing system provides a visual interface. The visual interface is used to provide a human-computer interaction interface to the user, and the user can connect to the data processing system through an electronic device such as a mobile phone or a computer. Data is transmitted between the electronic device and the server according to a preset protocol. Preferably, the preset protocol includes, but is not limited to, any one of the following: HTTP protocol (Hyper Text Transfer Protocol), HTTPS protocol (Hyper Text Transfer Protocol over Secure Socket Layer, HTTP protocol with security as the goal), etc. In at least one embodiment of the present invention, the server can be a server group composed of several functional servers. The electronic device can be any terminal with a network connection function, for example, the electronic device can be a mobile device such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (Personal Digital Assistant, PDA), a game console, an interactive network television (Internet Protocol Television, IPTV), a smart wearable device, a navigation device, etc., or a fixed device such as a desktop computer, a digital TV, etc. The mobile terminal has a data storage device (see Figure 8). The data storage device can be used to store data codes. The data processing method is used to distribute the received data based on the oracle sharding processing method, and adjust the data distribution method in real time by monitoring the working status of the server.

[0095] See also Figure 1 , which is a schematic diagram of the data processing method of the present invention.

[0096] S10, when an acquisition instruction is detected, a server list and data to be processed are acquired.

[0097] In at least one embodiment of the present invention, the server list includes server names and corresponding IP addresses stored in a list form. The data processing system further includes an Oracle database and a Redis database. The Oracle database is a distributed database that stores a variety of functional functions. The Redis is an open source memory storage database that supports the storage of multiple data types.

[0098] S11, when the first monitoring instruction is detected, monitor the status of the server in real time and update the server list in time.

[0099] Please also read Figure 2 In at least one embodiment of the present invention, the step of real-time monitoring of the status of the server and timely updating the server list may further include:

[0100] S111, receiving a heartbeat signal from each server;

[0101] S112, determining whether the heartbeat signal is interrupted;

[0102] S113, when the heartbeat signal is detected to be interrupted, accumulating the interruption time;

[0103] S114, determining whether the accumulated interruption time is greater than a predetermined time;

[0104] S115: When the accumulated interruption time is greater than the predetermined time, deleting the server from the server list.

[0105] When the heartbeat signal is not interrupted, return to step S111.

[0106] When the accumulated interruption time is less than or equal to the predetermined time, the process returns to step S111.

[0107] In at least one embodiment of the present invention, the heartbeat signal is a pulse signal of a predetermined frequency. Each server may send a heartbeat signal of a different predetermined frequency to distinguish between different servers.

[0108] S12, when a first calculation instruction is detected, calculating a location parameter corresponding to each server according to the server list.

[0109] In at least one embodiment of the present invention, the position parameter is used to identify the position of the server in the server list.

[0110] Please also read Figure 3 In at least one embodiment of the present invention, the step of calculating the position parameter of each server in the server list according to the server list may further include:

[0111] S121, obtaining the IP address of each server;

[0112] S122, calculating the position parameter of the corresponding server in the server list according to the IP address;

[0113] S123, storing the position parameter in the Redis database.

[0114] In at least one embodiment of the present invention, the IP address of the server is stored in the Redis database in the form of a list. The position parameter is a position identifier of the server in the server list.

[0115] S13, when a second calculation instruction is detected, calculating the total length of the server list as a length parameter.

[0116] In at least one embodiment of the present invention, the length parameter may be the total number of the servers.

[0117] S14, when a sharding instruction is detected, the data to be processed is sharded to obtain sharded data and a HASH value corresponding to the sharded data is calculated.

[0118] In at least one embodiment of the present invention, each row of data in the to-be-processed data list has a unique corresponding primary key, and the primary key serves as an identification value for identifying the row of data.

[0119] Please also read Figure 4 In at least one embodiment of the present invention, the step of slicing the list of data to be processed to obtain slicing data and calculating the HASH value corresponding to the slicing data may further include:

[0120] S141, sharding the data to be processed to obtain sharded data and calculating the primary key value of the sharded data;

[0121] S142, calculating the HASH value of each of the shard data using a predetermined function according to the length parameter and the primary key value;

[0122] S143, storing the HASH value in the Redis database.

[0123] In at least one embodiment of the present invention, the predetermined function is ora_hash(id,ipCount-1). Wherein, the id is the primary key value corresponding to any row of data in the list of data to be processed, and the ipCount is the length parameter. The HASH value is a natural number less than the length parameter. In at least one embodiment of the present invention, the HASH value is stored in a stack. For example, when the length parameter is 11, the HASH value calculated according to the predetermined function is any natural number between 0 and 10.

[0124] S15, when a matching instruction is detected, establishing a mapping between each of the HASH values ​​and each of the position parameters.

[0125] Please also read Figure 5 In at least one embodiment of the present invention, the step of establishing a mapping between each of the HASH values ​​and each of the position parameters may further include:

[0126] S151, obtaining a HASH value from the Redis database as a query HASH value;

[0127] S152, obtaining one of the position parameters from the Redis database as a query position parameter;

[0128] S153, determining whether the query HASH value is consistent with the query location parameter;

[0129] S154, when the query HASH value is consistent with the query location parameter, establish a mapping between the query HASH value and the query location parameter to match a row of data corresponding to the query HASH value to the server corresponding to the query location parameter;

[0130] When the query HASH value is inconsistent with the query position parameter, it is identified that the query HASH value does not match the query position parameter, and the process returns to step S152;

[0131] S155, determining whether the HASH value is the last HASH value;

[0132] S156, when the HASH value is the last HASH value, identifying that all the HASH values ​​have been matched with the position parameters;

[0133] When the HASH value is not the last HASH value, return to step S151.

[0134] S16, when a processing instruction is detected, each of the servers processes corresponding data in parallel according to the mapping between the HASH value and the location parameter.

[0135] S17, when the second monitoring instruction is detected, monitor the performance of each server and dynamically adjust the server list.

[0136] Please also read Figure 6 In at least one embodiment of the present invention, the step of monitoring the performance of each server and dynamically adjusting the server list may further include:

[0137] S171, calculating the total amount of data matched by each server at specified intervals;

[0138] S172, determining whether the total amount of data is greater than a specified threshold;

[0139] S173: When the total amount of data is greater than or equal to the specified threshold, remove the server from the server list.

[0140] When the total amount of data is less than the specified threshold, return to step S171.

[0141] In at least one embodiment of the present invention, all of the above instructions may be data request instructions received by an electronic device. The electronic device may include a keyboard, a touch screen, etc., but the user input method in the example embodiment of the present disclosure is not limited thereto. It may be generated by a user through a specific operation on a visual interface. Specifically, the user's operation includes, but is not limited to: a sliding operation, a clicking operation (such as a single-click operation, a double-click operation, etc.). Specifically, the preset button may be a physical button on the electronic device, or a virtual button on the electronic device, etc. (for example, the virtual button may be a virtual icon on the display of the electronic device, etc.), and the present invention is not limited thereto.

[0142] The above data processing method processes the data in shards to ensure the unique correspondence between the data and the server, which is conducive to improving the data processing speed. At the same time, all servers are monitored according to the heartbeat monitoring so that when a server has a problem, it is removed from the server list and the data is reallocated to ensure that no data is missed.

[0143] Please refer to Figure 7The present invention provides a data processing device 1, which is applied to a data processing system composed of at least one electronic device and multiple servers. Data is transmitted between the electronic device and the server according to a preset protocol. Preferably, the preset protocol includes, but is not limited to, any one of the following: HTTP protocol (Hyper Text Transfer Protocol), HTTPS protocol (Hyper Text Transfer Protocol over Secure Socket Layer, HTTP protocol with security as the goal), etc. The electronic device can be any terminal with a network connection function, for example, the electronic device can be a mobile device such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (Personal Digital Assistant, PDA), a game console, an interactive network television (Internet Protocol Television, IPTV), a smart wearable device, a navigation device, etc., or a fixed device such as a desktop computer, a digital TV, etc.

[0144] In one embodiment of the present invention, the data processing device 1 comprises:

[0145] The acquisition module 10 is used to acquire the server list and the data to be processed when an acquisition instruction is detected.

[0146] In at least one embodiment of the present invention, the server list includes server names and corresponding IP addresses stored in a list form. The data processing system further includes an Oracle database and a Redis database. The Oracle database is a distributed database that stores a variety of functional functions. The Redis is an open source memory storage database that supports the storage of multiple data types.

[0147] The first monitoring module 20 is used to monitor the status of the server in real time and update the server list in time when the first monitoring instruction is detected.

[0148] The first monitoring module 20 further receives the heartbeat signal sent by each server and determines whether the heartbeat signal is interrupted. When the heartbeat signal is detected to be interrupted, the first monitoring module 20 further accumulates the interruption time and determines whether the accumulated interruption time is greater than the predetermined time. When the accumulated interruption time is greater than the predetermined time, the first monitoring module 20 further deletes the server from the server list.

[0149] When the heartbeat signal is not interrupted or the accumulated interruption time is less than or equal to the predetermined time, the first monitoring module 20 does not perform any operation.

[0150] In at least one embodiment of the present invention, the heartbeat signal is a pulse signal of a predetermined frequency. Each server may send a heartbeat signal of a different predetermined frequency to distinguish between different servers.

[0151] The first calculation module 30 is used to calculate the location parameter corresponding to each server according to the server list when a first calculation instruction is detected.

[0152] In at least one embodiment of the present invention, the position parameter is used to identify the position of the server in the server list.

[0153] The first calculation module 30 further obtains the IP address of each server, calculates the position parameter of the corresponding server in the server list according to the IP address, and stores the position parameter in the Redis database.

[0154] In at least one embodiment of the present invention, the IP address of the server is stored in the Redis database in the form of a list. The position parameter is a position identifier of the server in the server list.

[0155] The second calculation module 40 is used to calculate the total length of the server list as a length parameter when a second calculation instruction is detected.

[0156] In at least one embodiment of the present invention, the length parameter may be the total number of the servers.

[0157] The slicing module 50 is used to slice the data to be processed to obtain sliced ​​data and calculate the HASH value corresponding to the sliced ​​data when a slice instruction is detected.

[0158] In at least one embodiment of the present invention, each row of data in the to-be-processed data list has a unique corresponding primary key, and the primary key serves as an identification value for identifying the row of data.

[0159] The sharding module 50 further includes sharding the data to be processed to obtain sharded data and calculating the primary key value of the sharded data, calculating the HASH value of each sharded data using a predetermined function according to the length parameter and the primary key value, and storing the HASH value in the Redis database.

[0160] In at least one embodiment of the present invention, the predetermined function is ora_hash(id,ipCount-1). Wherein, the id is the primary key value corresponding to any row of data in the list of data to be processed, and the ipCount is the length parameter. The HASH value is a natural number less than the length parameter. In at least one embodiment of the present invention, the HASH value is stored in a stack. For example, when the length parameter is 11, the HASH value calculated according to the predetermined function is any natural number between 0 and 10.

[0161] The matching module 60 is used to establish a mapping between each of the HASH values ​​and each of the position parameters when a matching instruction is detected.

[0162] The matching module 60 further obtains one of the HASH values ​​from the Redis database as the query HASH value, obtains one of the position parameters from the Redis database as the query position parameter, and determines whether the query HASH value is consistent with the query position parameter. When the query HASH value is consistent with the query position parameter, the matching module 60 further establishes a mapping between the query HASH value and the query position parameter to match a row of data corresponding to the query HASH value to the server corresponding to the query position parameter. When the query HASH value is inconsistent with the query position parameter, it is identified that the query HASH value does not match the query position parameter, and the matching module 60 further obtains one of the position parameters from the Redis database as the query position parameter. The matching module 60 further determines whether the HASH value is the last HASH value. When the HASH value is the last HASH value, it is identified that all the HASH values ​​and the position parameters are matched. When the HASH value is not the last HASH value, the matching module 60 further obtains one of the HASH values ​​from the Redis database as the query HASH value.

[0163] The processing module 70 is used to, when a processing instruction is detected, cause each server to process corresponding data in parallel according to the mapping between the HASH value and the position parameter.

[0164] The second monitoring module 80 is used to monitor the performance of each server and dynamically adjust the server list when a second monitoring instruction is detected.

[0165] The second monitoring module 80 further calculates the total amount of data matched by each server at specified intervals, and determines whether the total amount of data is greater than a specified threshold. When the total amount of data is less than the specified threshold, the second monitoring module 80 further calculates the total amount of data matched by each server at specified intervals. When the total amount of data is greater than or equal to the specified threshold, the second monitoring module 80 further removes the server from the server list.

[0166] See also Figure 8 , which is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a processor 106 , a memory 102 and a communication bus 104 .

[0167] The memory 102 is used to store program codes. The memory 102 can be a circuit with storage function without physical form in an integrated circuit, such as a memory stick, a TF card (Trans-flash Card), a smart media card (smart media card), a secure digital card (secure digital card), a flash memory card (flash card) and other storage devices. The memory 102 can communicate data with the processor 106 via a communication bus 104. The memory 102 may include an operating system, a network communication module and a data processing program. The operating system is a program for managing and controlling the hardware and software resources of an electronic device, and supports the operation of a data processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the memory 102, and to communicate with other hardware and software in a data processing device.

[0168] The processor 106 may include one or more microprocessors or digital processors. The processor 106 may call the program code stored in the memory 102 to execute related functions. For example, Figure 7 Each module described in the above is a program code stored in the memory 102 and executed by the processor 106 to implement a data processing method. The processor 106 is also called a central processing unit (CPU), which is a large-scale integrated circuit and a computing core (Core) and a control core (Control Unit).

[0169] The processor 106 is used to execute multiple computer instructions stored in the memory 102 to implement the data processing method. The processor 106 can execute multiple instructions to implement the following steps:

[0170] S10, when an acquisition instruction is detected, a server list and data to be processed are acquired.

[0171] In at least one embodiment of the present invention, the server list includes server names and corresponding IP addresses stored in a list form. The data processing system further includes an Oracle database and a Redis database. The Oracle database is a distributed database that stores a variety of functional functions. The Redis is an open source memory storage database that supports the storage of multiple data types.

[0172] S11, when the first monitoring instruction is detected, monitor the status of the server in real time and update the server list in time.

[0173] In at least one embodiment of the present invention, the step of real-time monitoring of the status of the server and timely updating the server list may further include:

[0174] S111, receiving a heartbeat signal from each server;

[0175] S112, determining whether the heartbeat signal is interrupted;

[0176] S113, when the heartbeat signal is detected to be interrupted, accumulating the interruption time;

[0177] S114, determining whether the accumulated interruption time is greater than a predetermined time;

[0178] S115: When the accumulated interruption time is greater than the predetermined time, deleting the server from the server list.

[0179] When the heartbeat signal is not interrupted, return to step S111.

[0180] When the accumulated interruption time is less than or equal to the predetermined time, the process returns to step S111.

[0181] In at least one embodiment of the present invention, the heartbeat signal is a pulse signal of a predetermined frequency. Each server may send a heartbeat signal of a different predetermined frequency to distinguish between different servers.

[0182] S12, when a first calculation instruction is detected, calculating a location parameter corresponding to each server according to the server list.

[0183] In at least one embodiment of the present invention, the position parameter is used to identify the position of the server in the server list.

[0184] In at least one embodiment of the present invention, the step of calculating the position parameter of each server in the server list according to the server list may further include:

[0185] S121, obtaining the IP address of each server;

[0186] S122, calculating the position parameter of the corresponding server in the server list according to the IP address;

[0187] S123, storing the position parameter in the Redis database.

[0188] In at least one embodiment of the present invention, the IP address of the server is stored in the Redis database in the form of a list. The position parameter is a position identifier of the server in the server list.

[0189] S13, when a second calculation instruction is detected, calculating the total length of the server list as a length parameter.

[0190] In at least one embodiment of the present invention, the length parameter may be the total number of the servers.

[0191] S14, when a sharding instruction is detected, the data to be processed is sharded to obtain sharded data and a HASH value corresponding to the sharded data is calculated.

[0192] In at least one embodiment of the present invention, each row of data in the to-be-processed data list has a unique corresponding primary key, and the primary key serves as an identification value for identifying the row of data.

[0193] In at least one embodiment of the present invention, the step of slicing the list of data to be processed to obtain slicing data and calculating the HASH value corresponding to the slicing data may further include:

[0194] S141, sharding the data to be processed to obtain sharded data and calculating the primary key value of the sharded data;

[0195] S142, calculating the HASH value of each of the shard data using a predetermined function according to the length parameter and the primary key value;

[0196] S143, storing the HASH value in the Redis database.

[0197] In at least one embodiment of the present invention, the predetermined function is ora_hash(id,ipCount-1). Wherein, the id is the primary key value corresponding to any row of data in the list of data to be processed, and the ipCount is the length parameter. The HASH value is a natural number less than the length parameter. In at least one embodiment of the present invention, the HASH value is stored in a stack. For example, when the length parameter is 11, the HASH value calculated according to the predetermined function is any natural number between 0 and 10.

[0198] S15, when a matching instruction is detected, establishing a mapping between each of the HASH values ​​and each of the position parameters.

[0199] In at least one embodiment of the present invention, the step of establishing a mapping between each of the HASH values ​​and each of the position parameters may further include:

[0200] S151, obtaining a HASH value from the Redis database as a query HASH value;

[0201] S152, obtaining one of the position parameters from the Redis database as a query position parameter;

[0202] S153, determining whether the query HASH value is consistent with the query location parameter;

[0203] S154, when the query HASH value is consistent with the query location parameter, establish a mapping between the query HASH value and the query location parameter to match a row of data corresponding to the query HASH value to the server corresponding to the query location parameter;

[0204] When the query HASH value is inconsistent with the query position parameter, it is identified that the query HASH value does not match the query position parameter, and the process returns to step S152;

[0205] S155, determining whether the queried HASH value is the last HASH value;

[0206] S156, when the HASH value is the last HASH value, identifying that all the HASH values ​​have been matched with the position parameters;

[0207] When the HASH value is not the last HASH value, return to step S151.

[0208] S16, when a processing instruction is detected, each of the servers processes corresponding data in parallel according to the mapping between the HASH value and the location parameter.

[0209] S17, when the second monitoring instruction is detected, monitor the performance of each server and dynamically adjust the server list.

[0210] In at least one embodiment of the present invention, the step of monitoring the performance of each server and dynamically adjusting the server list may further include:

[0211] S171, calculating the total amount of data matched by each server at specified intervals;

[0212] S172, determining whether the total amount of data is greater than a specified threshold;

[0213] S173: When the total amount of data is greater than or equal to the specified threshold, remove the server from the server list.

[0214] When the total amount of data is less than the specified threshold, return to step S171.

[0215] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0216] The present invention also provides a storage medium. The storage medium is a computer-readable storage medium. The computer-readable storage medium stores computer instructions. The computer instructions can be stored in the memory 102, and when executed by one or more processors 106, the data processing method described in the above method embodiment is implemented, for example Figure 1 S10-S17 shown are not described in detail here.

[0217] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the modules, which is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or module can be electrical or other forms.

[0218] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0219] In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or each module may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0220] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0221] It should also be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0222] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that: The data processing method comprises: When an acquisition instruction is detected, a server list and data to be processed are acquired, wherein the server list includes a plurality of servers, the server list includes server names and corresponding IP addresses stored in a list form, and the IP addresses of the servers are stored in a Redis database in a list form; When the first monitoring instruction is detected, the status of the server is monitored in real time and the server list is updated in time; When the second monitoring instruction is detected, the performance of the server is monitored and the server list is dynamically adjusted, including: calculating the total amount of data matched by each server at specified intervals; determining whether the total amount of data is greater than a specified threshold; when the total amount of data is less than the specified threshold, calculating the total amount of data matched by each server at specified intervals; when the total amount of data is greater than or equal to the specified threshold, removing the server from the server list; When the first calculation instruction is detected, the IP address of each server is obtained, the position parameter of the corresponding server in the server list is calculated according to the IP address, and the position parameter is stored in the Redis database, where the position parameter is the position identifier of the server in the server list; When a second calculation instruction is detected, calculating the total length of the server list as a length parameter, wherein the length parameter is the total number of the servers; When a sharding instruction is detected, the data to be processed is sharded to obtain sharded data and the primary key value of the sharded data is calculated, the HASH value of each sharded data is calculated using the ora_hash function according to the length parameter and the primary key value, and the HASH value is stored in the Redis database, the primary key value is an identification value that identifies each row of data in the data to be processed, and the HASH value is a natural number less than the length parameter; When a matching instruction is detected, a mapping between each of the HASH values ​​and each of the position parameters is established; When a processing instruction is detected, each of the servers processes corresponding data in parallel according to the mapping between the HASH value and the location parameter.

2. The data processing method according to claim 1, characterized in that: The step of real-time monitoring the status of the server and timely updating the server list includes: Receive heartbeat signals from each server; Determining whether the heartbeat signal is interrupted; When the heartbeat signal is detected to be interrupted, the interruption time is accumulated; Determining whether the accumulated interruption time is greater than a predetermined time; When the accumulated interruption time is greater than the predetermined time, deleting the server from the server list; When the heartbeat signal is not interrupted, return to the step of receiving the heartbeat signal sent by each server.

3. The data processing method according to claim 1, characterized in that: The step of establishing a mapping between each of the HASH values ​​and each of the position parameters comprises: Obtain a HASH value from the Redis database as a query HASH value; Obtain one of the position parameters from the Redis database as a query position parameter; Determine whether the query HASH value is consistent with the query location parameter; When the query HASH value is consistent with the query location parameter, a mapping between the query HASH value and the query location parameter is established to match a row of data corresponding to the query HASH value to the server corresponding to the query location parameter; When the query HASH value is inconsistent with the query position parameter, identifying that the query HASH value does not match the query position parameter, and returning to the step of obtaining one of the position parameters from the Redis database as the query position parameter; Determine whether the HASH value is the last HASH value; When the HASH value is the last HASH value, identifying that all the HASH values ​​have been matched with the position parameters; When the HASH value is not the last HASH value, return to the step of obtaining a HASH value from the Redis database as a query HASH value.

4. A data processing device, characterized in that: The data processing device comprises: An acquisition module, used for acquiring a server list and data to be processed when an acquisition instruction is detected; the server list includes a plurality of servers, the server list includes server names and corresponding IP addresses stored in a list form, and the IP addresses of the servers are stored in a Redis database in a list form; A first monitoring module, configured to monitor the status of the server in real time and update the server list in time when a first monitoring instruction is detected; The second monitoring module is used to monitor the performance of the server and dynamically adjust the server list when a second monitoring instruction is detected, including: calculating the total amount of data matched by each server at specified intervals; judging whether the total amount of data is greater than a specified threshold; when the total amount of data is less than the specified threshold, calculating the total amount of data matched by each server at specified intervals; when the total amount of data is greater than or equal to the specified threshold, removing the server from the server list; A first calculation module, configured to obtain an IP address of each server when a first calculation instruction is detected, calculate a position parameter of the corresponding server in the server list according to the IP address, and store the position parameter in the Redis database, where the position parameter is a position identifier of the server in the server list; A second calculation module, configured to calculate the total length of the server list as a length parameter when a second calculation instruction is detected, wherein the length parameter is the total number of the servers; A sharding module is used for sharding the data to be processed to obtain sharded data and calculating the primary key value of the sharded data when a sharding instruction is detected, calculating the HASH value of each sharded data by using the ora_hash function according to the length parameter and the primary key value, and storing the HASH value in the Redis database, wherein the primary key value is an identification value for identifying each row of data in the data to be processed, and the HASH value is a natural number less than the length parameter; A matching module, used for establishing a mapping between each of the HASH values ​​and each of the position parameters when a matching instruction is detected; The processing module is used for, when a processing instruction is detected, to cause each server to process corresponding data in parallel according to the mapping between the HASH value and the position parameter.

5. An electronic device, characterized in that: The electronic device comprises a processor and a memory, and the processor is configured to implement the data processing method according to any one of claims 1 to 3 when executing a computer program stored in the memory.

6. A storage medium, characterized in that: The storage medium is a computer-readable storage medium storing at least one instruction, and when the at least one instruction is executed by a processor, the data processing method according to any one of claims 1 to 3 is implemented.

Images