Data processing method and device, computer equipment and computer readable storage medium

Abstract

The invention discloses a data processing method and device, computer equipment and a computer readable storage medium. The method comprises the following steps: firstly, determining a first data node and first data corresponding to the first data node according to an input data identifier; then grouping the first data to obtain a plurality of data groups; thirdly, establishing sub-directories corresponding to the plurality of data groups, and respectively writing the second data of each data group into the sub-directories corresponding to the data groups; and finally, generating a second data node corresponding to the second data based on the subdirectory. In this way, parallel migration of multiple groups of data can be achieved, and therefore the migration efficiency of the data in the data nodes is achieved. And moreover, the capacity expansion efficiency of the data service node is improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and specifically to a data processing method, apparatus, computer equipment, and computer-readable storage medium. Background Art

[0002] Data service nodes are server nodes used to store, process and provide data. In distributed systems, they can be used to manage and process large amounts of data and provide data access and services to users. When the amount of data is large, it is usually necessary to expand the data service nodes. By increasing the number of data service nodes, the performance and capacity of the system can be improved, the load can be shared and the data processing capabilities can be improved. In the process of expanding the capacity of data service nodes, factors such as load balancing and data consistency usually need to be considered to ensure that the expansion process does not cause system interruptions or data loss. Summary of the invention

[0003] The purpose of this application is to provide a data processing method, apparatus, computer device and computer-readable storage medium.

[0004] In a first aspect, an embodiment of the present application provides a data processing method, including:

[0005] Determine a first data node corresponding to the data identifier, and determine a plurality of first data corresponding to the first data node;

[0006] Grouping the plurality of first data to obtain a plurality of data groups;

[0007] Create subdirectories for each data group;

[0008] The second data in the data group is written into the subdirectory corresponding to the data group, and a second data node corresponding to the second data is generated based on the subdirectory, wherein the second data in each data group is a part of the first data.

[0009] In a second aspect, an embodiment of the present application provides a data processing device, including:

[0010] A determination module, used to determine a first data node corresponding to the data identifier, and determine a plurality of first data corresponding to the first data node;

[0011] A grouping module, used for grouping a plurality of first data to obtain a plurality of data groups;

[0012] A creation module is used to create a subdirectory for each data group;

[0013] A generation module is used to write the second data in the data group into a subdirectory corresponding to the data group, and generate a second data node corresponding to the second data based on the subdirectory, wherein the second data in each data group is a part of the first data.

[0014] In a third aspect, an embodiment of the present application further provides a computer device, comprising a memory storing a plurality of instructions; a processor loads instructions from the memory to execute the steps of any data processing method provided in the embodiment of the present application.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for a processor to load to execute the steps of any data processing method provided in the embodiment of the present application.

[0016] In a fifth aspect, an embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of any data processing method provided in the embodiment of the present application.

[0017] The present application first determines the first data node and the first data corresponding to the first data node according to the input data identifier. Then the first data is grouped to obtain multiple data groups, the data in each data group is the second data, and the second data is a part of the first data. Then, a subdirectory corresponding to the multiple data groups is established, and the second data of each data group is written into the subdirectory corresponding to each data group. Finally, a second data node corresponding to the second data is generated based on the subdirectory. In this way, by grouping the original data (first data) stored in the first data node and then writing it into multiple subdirectories in the form of data groups, parallel migration of multiple groups of data can be achieved, thereby achieving migration efficiency of data in the data node. In addition, a new data node (second data node) can be generated based on the subdirectory corresponding to the data group, thereby improving the expansion efficiency of the data service node. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 It is a schematic diagram of data storage in a data service node;

[0020] Figure 2 It is a flowchart of a method for extending a data service node;

[0021] Figure 3 It is a schematic diagram of an application environment of a data processing method provided in an embodiment of the present application;

[0022] Figure 4 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of an example of a configuration data group provided in an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of writing second data into a subdirectory provided in an embodiment of the present application;

[0025] Figure 7 It is a flowchart of a data processing method provided in a specific embodiment of the present application;

[0026] Figure 8 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0027] Fig. 9 It is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application. At the same time, in the description of the embodiments of the present application, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0029] In a distributed system, data is usually managed through multiple data service nodes, and data access and services are provided to users. Taking an application (APP) as an example, after a user logs in to the APP, a registration identification code (RID) is assigned and a customer identification code (CID) is uploaded at the same time, where RID is an identification code generated based on the user and APP dimensions, and CID is an identification code generated based on the user's device. Then, a database mapping table is established based on the mapping relationship between the application identification code (APPID), RID and CID. Assuming that the mapping table name is M, the current mapping table M contains three fields: APPID, RID and CID. When a message needs to be pushed to an APP, the RID can be used to query the mapping table M to obtain the corresponding CID, and other necessary information of the user when the message is pushed can be obtained through the CID, and finally the message is pushed to the user's device. Since the number of users is usually large, such as the order of hundreds of millions, it is easy to cause a large amount of data in the mapping table. In this way, if the mapping table M is queried during the peak period of user activity, the query time may be long.

[0030] In order to improve the efficiency of querying the mapping table M during peak hours, some methods will create a Message Transfer Service (MTS) data service node, embed a RocksDB database, initialize a RocksDB instance, use a scheduled task to synchronize the data of the M table, and use an interface to transfer the mapping data to the MTS service node. The MTS data service node uses RID as the key, CID as the value, and APPID as the column family name in RocksDB to store the full data of the APP in the column family corresponding to the RocksDB database embedded in MTS. In this way, using the feature of RocksDB that supports multi-dimensional data partitioning, APPID is used as the column family name to isolate data between different APPs. An MTS data service node can store the full mapping data of multiple APPs.

[0031] Figure 1 It is a schematic diagram of data service node storing data, such as Figure 1As shown, it is assumed that there are mapping table data for APP1 and APP2. Among them, the APP1 mapping table data is stored in the MTS1 data service node and the MTS2 data service node, and the APP2 mapping table data is stored in the MTS2 data service node and the MTS3 data service node. This method allows the full mapping data of each APP to be stored in multiple MTS data service nodes. In this way, when querying the mapping data, it is routed to any MTS data service node containing the corresponding APP mapping relationship through the APPID, which reduces the query pressure of RocksDB in the MTS data service node and expands the throughput, which is equivalent to load balancing for the MTS data service node. However, each time the full APP mapping data is synchronized using a scheduled task, due to the large amount of APP mapping data, the query speed from the database is slow, which easily leads to the horizontal expansion of the MTS data service node. It takes a long time to synchronize the mapping data to RocksDB, resulting in a relatively slow expansion speed of the data service node. It usually takes several days to complete the expansion of an MTS data service node.

[0032] exist Figure 1 Based on the storage method of the data service node shown in the figure, taking the extended data service node as the MTS4 data service node as an example, the following is combined with Figure 2 A method for extending a data service node is described. Figure 2 As shown, it is a flowchart of a method for expanding a data service node. The method for expanding a data service node includes steps 201-205.

[0033] Step 201: manually create a new MTS4 data service node and start the service.

[0034] Step 202: Initialize the RocksDB instance of the MTS4 data service node.

[0035] Step 203: Start a scheduled task loop to pull the data of the APP1 and APP2 mapping tables M in batches, use APPID as the column family to distinguish data isolation, and write the mapping data to the RocksDB instance of the MTS4 data service node.

[0036] Step 204: After the data is pulled, data verification is performed to confirm whether the MTS data service node has completed horizontal expansion. For example, the mapping table data RID and CID can be randomly extracted to query the RocksDB database. If the returned data is normal data, it means that the horizontal expansion of the MTS4 node is completed.

[0037] Step 205: After the MTS data service node is horizontally expanded, a mapping data service is provided, so that the mapping data of APP1 and APP2 can be queried through the MTS4 node.

[0038] The above data service node expansion method can use a scheduled task to pull the full mapping data of APP1 and APP2 to horizontally expand the MTS data service node. However, due to the large amount of mapping data, when expanding the MTS data service node, it takes a long time to synchronize the mapping data, and the expansion speed of the data service node is slow.

[0039] Based on this, the embodiment of the present application provides a more efficient way to expand the capacity of a data service node. Based on the data in the RocksDB of the existing data service node, the data is divided into multiple data groups in a custom grouping manner, and sent to the distributed stream processing platform in parallel according to the data group. Among them, the distributed stream processing platform may include Kafka message middleware. Then, the consumer RocksDB instance is dynamically created and managed according to the capacity of the data group using methods such as Docker and smart contracts, and the subdirectories of the data nodes to be newly created are consumed in parallel in the manner of data groups and written to at the same time, thereby creating a new data node, and the data in the subdirectories of the new data node are mounted to the main directory created in advance for storing the new data node data. In this way, the expansion of the data service node can be realized by simply entering a specified data identifier, such as APPID, and the subdirectories of the data service node are created based on the capacity of the data group, and the expansion process is dynamically scalable. In this way, the total time for expanding the data service node can be reduced to a few hours, greatly improving the expansion efficiency of the data service node.

[0040] In order to better understand a data processing method, apparatus, computer device, and computer-readable storage medium provided in an embodiment of the present application, an application environment applicable to an embodiment of the present application is first described below.

[0041] See also Figure 3 , Figure 3 A schematic diagram of an application environment of a data processing method provided by an embodiment of the present application is shown. As an example, the data processing method is run on Figure 3 As an example, the server 310 shown in the figure can be connected to the terminal device 320 through a network. Among them, the network is used to provide a medium for a communication link between the server 310 and the terminal device 320. The network can include various connection types, such as wired communication links, wireless communication links, etc., which are not limited in the embodiments of the present application. Optionally, in other embodiments, the data processing method can also be run in other types of electronic devices such as smart phones and laptops.

[0042] It should be understood that Figure 3The server 310, network and terminal device 320 are merely illustrative. Any number of servers, networks and terminal devices may be configured as required. For example, the server 310 may be a physical server or a server cluster composed of multiple servers, and the terminal device 320 may be a mobile phone, a tablet, a desktop computer, a laptop computer, or the like. It is understood that the embodiments of the present application may also allow multiple terminal devices 320 to access the server 310 at the same time.

[0043] The following is a detailed description in conjunction with the accompanying drawings. In this embodiment, the execution subject is taken as an example of the server 310. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.

[0044] Please refer to Figure 4 , Figure 4 4 is a flow chart of a data processing method provided in an embodiment of the present application. In an embodiment of the present application, the specific flow of the data processing method may include steps 401-404, which are described in detail below.

[0045] Step 401: Determine a first data node corresponding to a data identifier, and determine a plurality of first data corresponding to the first data node.

[0046] In the embodiment of the present application, the data identifier refers to an identifier used to isolate data from different data sources when storing data. Taking the RocksDB database as an example, the data of the APP is stored in the column family corresponding to the RocksDB database embedded in the MTS. By utilizing the feature that RocksDB supports multi-dimensional data partitioning, the data in APPs with different APPIDs can be stored. For example, the APPID is used as the column family name, that is, the data identifier, to isolate the mapping data between different APPs. The data corresponding to each data identifier is the first data, and the data service node storing the first data is the first data node. The embodiment of the present application can establish a mapping relationship between the data service node and the data identifier in advance. In this way, the user only needs to enter the data identifier to find the first data node storing the first data corresponding to the data identifier. As an example, Table 1 is a mapping table N established based on the mapping relationship between the data service node name and the APPID in the embodiment of the present application. As shown in Table 1, the data of APP1 is stored in MTS1 and MTS2, and the data of APP2 is stored in MTS2 and MTS3.

[0047] Table 1

[0048]

[0049]

[0050] Assume that the APPID of the APP that the user inputs that needs to expand the first data is X=APP1, APP2, that is, the first data that needs to be expanded is the data of APP1 and APP2. At this time, by looking up Table 1, it can be obtained that the node containing the data of APP1 and APP2 is the MTS2 node. Let the MTS2 node be the Y node, then the Y node is the first data node. In addition, if the APPID entered by the user does not have a corresponding first data node, the expansion process is terminated directly, and the user is informed that there is no data corresponding to the APPID.

[0051] In an embodiment of the present application, when the first data node corresponding to the data identifier is determined, the incremental data writing function of the first data node can be stopped. Among them, the incremental data writing function refers to the function of updating data based on an existing data set. There is no need to replace the entire data set through incremental data writing, and the existing data will not be affected. In an embodiment of the present application, the newly expanded data service node is the second data node. In order to ensure the consistency of data between the second data node and the first data node, before migrating the first data in the first data node, it is necessary to stop the incremental data writing function of the first data node. In this way, it can be ensured that no new data will be added to the first data node during the migration process. After stopping the incremental data writing function of the first data node, a main directory can be created to store the data of the second data node.

[0052] Step 402: Group the plurality of first data to obtain a plurality of data groups. In the embodiment of the present application, the data in each data group is the second data, and the second data is a part of the plurality of first data.

[0053] In an embodiment of the present application, in order to improve the migration efficiency of the first data, the first data can be customized and grouped to obtain multiple data groups. For example, the first data can be grouped according to the data size. By dividing multiple data capacities in advance and creating a corresponding number of instances based on each data capacity, the functions of parallel consumption and parallel writing to the main directory between multiple data groups can be realized, greatly reducing the migration time of the first data. In one example, the number of instances can be dynamically created by Docker, so as to configure the instance of each data group. In another example, assuming that the data identifier input by the user includes multiple, when the first data is grouped, it can be further grouped based on different data identifiers. For example, assuming that the data identifier input by the user includes APP1 and APP2, a custom grouping can be performed for APP1, and a custom grouping can be performed for APP2. The grouping method of each type of data identifier can be the same or different, and is not limited here. It should be noted that the grouping method of the first data in the embodiment of the present application can be customized according to user needs, so as to dynamically create and manage multiple data groups.

[0054] Step 403: Create a subdirectory for each data group.

[0055] In an embodiment of the present application, a subdirectory refers to a directory that stores data in each data group respectively. The number of subdirectories can be the same as the number of data groups, so that each subdirectory can be used to store the data of a data group. In an example, assuming that the data identifier input by the user includes APP1 and APP2, APP1 includes n data groups after grouping, and APP2 includes m data groups after grouping, then the final subdirectory can include n+m data groups, which is not limited here. By creating a subdirectory corresponding to the data group, the data of each data group can be written in parallel. In addition, the data capacity of each subdirectory can be calculated based on the grouping of the data group, so that each subdirectory can be dynamically set according to the data size of the corresponding data group, saving the storage space of the server. Compared with the related art in which the first data is sequentially written into the directory of the new data service node, the parallel writing of the embodiment of the present application is more efficient.

[0056] Step 404: write the second data in the data group into a subdirectory corresponding to the data group, and generate a second data node corresponding to the second data based on the subdirectory.

[0057] In an embodiment of the present application, the data in each data group is the second data, the data node storing the second data is the second data node, and the second data of each data group corresponds to a subdirectory. In this way, the second data in the data group can be written in parallel to the corresponding subdirectory. Then, the second data node is generated based on the subdirectory. In one example, a distributed stream processing platform can be created to write the second data of the data group in parallel to the partition of the distributed stream processing platform, and then the second data in the distributed stream processing platform partition is consumed simultaneously based on the instance of each data group, and then the second data is written in parallel to the subdirectory based on each instance. After completing the write operation of the second data, Docker can create a second data node, which includes the above-mentioned subdirectory. Then the subdirectory in the second data node is mounted to the pre-established main directory, so that the corresponding second data can be queried through the second data node, and the second data is consistent with the first data of the first data node. In this way, for the second data stored by the data group, a distributed decentralized data migration method can be implemented, which improves the expansion efficiency of the second data node.

[0058] In an embodiment of the present application, when a second data node is generated, the incremental data writing function of the first data node and the second data node can be enabled so that the newly added data can be written into the first data node and the second data node at the same time, thereby ensuring the consistency of the data stored in the first data node and the second data node.

[0059] The embodiment of the present application can realize parallel migration of multiple groups of data by grouping the original data (first data) stored in the first data node and then writing them into multiple subdirectories in the form of data groups, thereby realizing efficient migration of data in the data nodes. In addition, a new data node (second data node) can be generated based on the subdirectory corresponding to the data group, thereby improving the expansion efficiency of the data service node.

[0060] In step 402, a plurality of data groups may be determined according to a plurality of pre-set data capacity intervals, wherein the number of data capacities is the same as the number of data groups. Then, according to the data size of each first data, each first data is divided into a data group corresponding to the data capacity where the data size is located as the second data in the data group.

[0061] In the embodiment of the present application, the data capacity refers to the interval divided by the data size. For the first data in the first data node, the data size of each data can be calculated.

[0062] Figure 5 is a schematic diagram of an example of a configuration data group provided in an embodiment of the present application. Figure 5For example, for the data in the first data node, assuming that the largest data is 100GB, it can be divided into four data capacities: (0GB, 5GB], (5GB, 20GB], (20GB, 50GB], and (50GB, 100GB]. In this way, based on the four pre-set data capacities, the first data can be divided into four data groups, each data group corresponding to a data capacity. Then each first data is divided into a data group that matches the data size of the first data, so that each data group includes second data that matches the data capacity.

[0063] Taking the above four data capacities as an example, assume that the four data capacities are represented by A, B, C and D respectively, that is, data capacity A is (0GB, 5GB], data capacity B is (5GB, 20GB], data capacity C is (20GB, 50GB], and data capacity D is (50GB, 100GB]. Then, four data groups 1, 2, 3 and 4 are established respectively. Data group 1 corresponds to data capacity A, data group 2 corresponds to data capacity B, data group 3 corresponds to data capacity C, and data group 4 corresponds to data capacity D. Assume that the first The multiple first data in the data node are 4GB, 6GB, 8GB, 30GB, 40GB and 100GB respectively. Then, 4GB can be divided into data group 1, 6GB and 8GB can be divided into data group 2, 30GB and 40GB can be divided into data group 3, and 100GB can be divided into data group 4. Thus, four data groups can be obtained, and the second data in each data group is a part of the first data. The data size of the second data in each data group matches the data capacity corresponding to the data group.

[0064] In step 403, a number of instances corresponding to the number of second data included in the data group may be created first. Then, based on the number of instances corresponding to the data group, a subdirectory may be created for the data group. The data capacity of the subdirectory is determined based on the number of instances.

[0065] In the embodiment of the present application, the same number of subdirectories can be created based on the number of data groups. In this way, each subdirectory can be used to store data of one data group. At the same time, the data capacity of each subdirectory can be determined based on the number of instances of each data group. Figure 5 For example, assuming that the number of instances corresponding to the data group (0GB, 5GB] is 1, the data capacity of the subdirectory corresponding to the data group can be 5GB. For another example, if the number of instances corresponding to the data group (5GB, 20GB] is 2, the data capacity of the subdirectory corresponding to the data group can be 40GB. Similarly, the data capacity of each subdirectory can be determined based on the maximum value of the data capacity corresponding to the data group and the number of instances. In this way, the storage space of the server can be saved and the efficiency of writing the second data in parallel to the subdirectory can be improved.

[0066] In an embodiment of the present application, before writing the second data in the data group into the corresponding subdirectory, multiple partitions may be established on the distributed stream processing platform based on the identification information of the data group, wherein each partition corresponds to the identification information of a data group.

[0067] In an embodiment of the present application, a distributed stream processing platform is used to process real-time data streams. Among them, Kafka is one of the core components of the distributed stream processing platform, which has scalability and fault tolerance and can process large-scale data streams. The distributed stream processing platform can adopt a distributed architecture to store data in different partitions, thereby realizing parallel processing and load balancing of data. Therefore, an embodiment of the present application can establish multiple partitions on a distributed stream processing platform in advance based on the identification information of the data group, and each partition corresponds to a data group. For example, partition 1 corresponds to data group 1, which is used to store the second data in a data group with a data capacity of (0GB, 5GB], partition 2 corresponds to data group 2, which is used to store the second data in a data group with a data capacity of (5GB, 20GB], and so on, each partition can store the second data in the corresponding data group.

[0068] In step 404, the second data in each data group can be sent in parallel to multiple partitions on the distributed stream processing platform. The partition data stored in each partition corresponds to the second data of one of the data groups. In one example, a message topic K (Kafka topic K) can be created, and then multiple threads can be started to read the first data in the first data node simultaneously according to the divided data groups. Through the custom partitioner of the distributed stream processing platform, the second data of each data group can be sent to the specified partition for partition data isolation. Then start the instance created for the data group, and consume the partition data in multiple partitions through the distributed stream processing platform. For example, the partition data of Kafka topic K is consumed in parallel according to the instance created for each data group. In addition, the partition data of multiple partitions are written in parallel to the subdirectory.

[0069] Figure 6 is a schematic diagram of writing the second data into a subdirectory provided in an embodiment of the present application. Figure 6 As shown, it is assumed that subdirectory S1 ( / data / S / S1) is the subdirectory corresponding to the data group where [5GB, 20GB] is located, and subdirectory S2 ( / data / S / S2) is the subdirectory corresponding to the data group where [0GB, 5GB] is located. Among them, [0GB, 5GB] includes instance 3, and its data size is 4GB. [5GB, 20GB] includes instance 1 and instance 2, and the data size is 11GB. Then the subdirectory written to instance 1 and instance 2 is S1, and the subdirectory written to instance 3 is S2.

[0070] In step 404, a main directory S for storing data of the second data node may be created first. Thus, when all the second data in the data group are written into the corresponding subdirectory, the subdirectory may be mounted to the main directory S, and the second data node may be generated.

[0071] In an embodiment of the present application, after all the second data is written to the corresponding subdirectory, the RocksDB instance of Docker can be destroyed to release resources. Then, the smart contract is called to create the second data node Z, and the main directory S is mounted. Among them, a smart contract refers to a program that automatically executes the terms of the contract. For example, Ethereum, and can automatically execute code when preset conditions are met. Smart contracts can be used to build decentralized services. The use of smart contracts to achieve dynamic creation and management of second data nodes greatly improves the degree of decentralization, and can automatically expand and shrink resources according to actual load. The method for creating the second data node Z in an embodiment of the present application can refer to the method of dynamically creating an instance corresponding to the data group described above. In this way, the dynamic creation of the second data node can be achieved, and the flexibility of data service node creation is improved.

[0072] In an embodiment of the present application, a smart contract can be used to manage the life cycle of a data service node in RocksDB. Smart contracts can implement several functions such as creation, expansion, monitoring and management. Among them, creating and building a RocksDB Docker image can be achieved by building a dockerfile. Then develop and deploy smart contracts, and use the Truffle framework to deploy smart contracts to the Ethereum test network or main network. Then create a configuration to describe the RocksDB cluster. Next, create a custom controller X to monitor changes in RocksDB CRDs, and call smart contract methods as needed to create, update or destroy RocksDB data service nodes. Finally, deploy the custom controller X.

[0073] In this way, dynamic expansion is triggered, and the number of instances to be created is dynamically determined according to the size of each data. You only need to set the value of the number of instances to be expanded. Using the Ethereum platform and smart contracts to dynamically create and manage RocksDB data service nodes can greatly improve the degree of decentralization and automatically expand and shrink resources according to actual load.

[0074] Figure 7 is a flow chart of a data processing method provided in a specific embodiment of the present application. Figure 7As shown, taking the RocksDB data service node dynamic expansion method as an example, assuming that the data identifier is APPID, the first data node is Y node, the second data node is Z node, the main directory is S, and the distributed stream processing platform includes Kafka. In this method, steps 701-712 may be included.

[0075] Step 701: Get input APPID: X, where X is the APP corresponding to the mapping data that needs to be expanded.

[0076] Step 702: Determine the first data node as node Y according to the input X. Node Y is used to store the first data in X.

[0077] Step 703: Stop the incremental data writing function of the RocksDB embedded in the Y node.

[0078] Step 704: Create a main directory S. The main directory S is used to store data in the Z node.

[0079] Step 705: Calculate the data size of the first data in the Y node.

[0080] Step 706: divide the first data into multiple data groups according to multiple preset data capacities through Docker, and the data in each data group is the second data.

[0081] Step 707: Create an instance corresponding to each data group according to the number of second data, and create a subdirectory for the data group based on the number of instances.

[0082] Step 708: Create a Kafka topic K, start multiple threads to read the second data of the Y node at the same time, and send it to a specified partition of Kafka. Each partition corresponds to a data group identifier.

[0083] Step 709: Start an instance of each data group and consume the partitioned data in Kafka at the same time.

[0084] Step 710: Write each partition data into the corresponding subdirectory.

[0085] Step 711: After writing is completed, destroy the RocksDB instance through Docker, mount the subdirectory to the main directory, and create a Z node.

[0086] Step 712: Enable the incremental data writing function of the Y node and the Z node.

[0087] The embodiment of the present application sends the data in RocksDB to Kafka according to the set grouping, dynamically creates and manages the number of RocksDB consumer instances according to the size of the grouped data on the consumer side, realizes the parallel consumption of multiple partitioned data, and writes the grouped data in parallel to the corresponding subdirectory, and then mounts the subdirectory with the main directory, which can greatly improve the expansion time of the new data service node. In addition, with the help of Kafka and Docker, data migration can be quickly realized, and distributed decentralized migration can realize the rapid expansion of data service nodes.

[0088] It should be understood that, although each step in the flowcharts involved in the above-mentioned embodiments is displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0089] Figure 8 8 is a schematic diagram of the structure of a data processing device 800 provided in an embodiment of the present application. Figure 8 As shown, the data processing device 800 may include a determination module 801, a grouping module 802, an establishment module 803 and a generation module 804. The determination module 801 is used to determine the first data node corresponding to the data identifier, and determine the multiple first data corresponding to the first data node. The grouping module 802 is used to group the multiple first data to obtain multiple data groups. The establishment module 803 is used to establish a subdirectory for each data group. The generation module 804 is used to write the second data in the data group into the subdirectory corresponding to the data group, and generate the second data node corresponding to the second data based on the subdirectory, wherein the second data in each data group is a part of the first data.

[0090] Among them, the determination module 801, the grouping module 802, the establishment module 803 and the generation module 804 can be used to execute steps 401-404 in the embodiments corresponding to the above-mentioned data processing method respectively. For the specific implementation methods of these modules and more details, please refer to the corresponding method part, which will not be repeated here one by one.

[0091] Based on the same inventive concept, the embodiment of the present application also provides a computer device, which can be a server or a terminal device, and the computer device includes a memory and a processor, and a computer program is stored in the memory, and the processor implements the steps of the above data processing method when executing the computer program. Thereby, various functions are realized, for example: determining the first data node corresponding to the data identifier, and determining the multiple first data corresponding to the first data node; grouping the multiple first data to obtain multiple data groups; establishing a subdirectory for each data group; writing the second data in the data group into the subdirectory corresponding to the data group, and generating the second data node corresponding to the second data based on the subdirectory, wherein the second data in each data group is a part of the first data.

[0092] The embodiment of the present application can realize parallel migration of multiple groups of data by grouping the original data (first data) stored in the first data node and then writing them into multiple subdirectories in the form of data groups, thereby realizing efficient migration of data in the data nodes. In addition, a new data node (second data node) can be generated based on the subdirectory corresponding to the data group, thereby improving the expansion efficiency of the data service node.

[0093] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0094] In one embodiment, the computer device is a terminal device, for example, its internal structure diagram can be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data processing method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0095] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0096] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0097] Since the computer program stored in the computer-readable storage medium can execute any data processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any data processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0098] Based on the same inventive concept, the embodiment of the present application further provides a computer program product or a computer program, which includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations in the above embodiments.

[0099] It should be noted that the object data (including but not limited to user device information, user personal information, etc.) and conversation data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0100] Any reference to the memory, database or other media used in the embodiments provided in the present application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (Re RAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0101] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0102] In the above data processing device, computer-readable storage medium, computer equipment, and computer program product embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the above-described data processing device, computer-readable storage medium, computer program product, computer equipment, and its corresponding units can refer to the description of the data processing method in the above embodiment, and will not be repeated here.

[0103] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0104] The above is a detailed introduction to a data processing method, device, computer equipment, computer-readable storage medium and computer program product provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Determine a first data node corresponding to the data identifier, and determine a plurality of first data corresponding to the first data node; Grouping the plurality of the first data to obtain a plurality of data groups; Creating a subdirectory for each of the data groups; The second data in the data group is written into the subdirectory corresponding to the data group, and a second data node corresponding to the second data is generated based on the subdirectory, wherein the second data in each data group is a part of the first data.

2. The data processing method according to claim 1, characterized in that: The grouping of the plurality of the first data comprises: Determining a plurality of data groups according to a plurality of pre-set data capacities; According to the data size of each of the first data, each of the first data is divided into a data group corresponding to the data capacity of the data size; The number of the data capacities is the same as the number of the data groups.

3. The data processing method according to claim 2, characterized in that: The step of establishing a subdirectory for each data group includes: Creating a corresponding number of instances of the data group according to the number of second data included in the data group; Based on the number of instances corresponding to the data group, a subdirectory is created for the data group, and the data capacity of the subdirectory is determined based on the number of instances.

4. The data processing method according to claim 3, characterized in that: Writing the second data in the data group into the subdirectory corresponding to the data group includes: Sending the second data in each of the data groups in parallel to a plurality of partitions on the distributed stream processing platform, wherein the partition data stored in each partition corresponds to the second data of one of the data groups; The partition data in the plurality of partitions is consumed by the distributed stream processing platform, and the partition data of the plurality of partitions is written into the subdirectories in parallel.

5. The data processing method according to claim 4, characterized in that: Before writing the second data in the data group into the subdirectory corresponding to the data group, the method further includes: A plurality of partitions are established on the distributed stream processing platform based on the identification information of the data group, and each partition corresponds to one piece of identification information.

6. The data processing method according to claim 1, characterized in that: The generating a second data node corresponding to the second data based on the sub-directory includes: Creating a home directory for storing data of the second data node; When all the second data in the data group are written into the corresponding sub-directory, the sub-directory is mounted to the main directory, and the second data node is generated.

7. The data processing method according to claim 1, characterized in that: The method further comprises: When the first data node corresponding to the data identifier is determined, stopping the incremental data writing function of the first data node; When the second data node is generated, the incremental data writing function of the first data node and the second data node is enabled.

8. A data processing device, characterized in that: include: A determination module, used to determine a first data node corresponding to a data identifier, and determine a plurality of first data corresponding to the first data node; A grouping module, used for grouping the plurality of first data to obtain a plurality of data groups; A creation module, used for creating a subdirectory for each of the data groups; A generation module is used to write the second data in the data group into the subdirectory corresponding to the data group, and generate a second data node corresponding to the second data based on the subdirectory, wherein the second data in each data group is a part of the first data.

9. A computer device, characterized in that: It comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps of the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the data processing method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product comprises a computer program, and the computer program is used by a processor to execute the steps of the data processing method according to any one of claims 1 to 7.

Images