Data processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: receiving a data stream object; the data stream object comprises data to be written into the first database; splitting the data stream object into data fragments corresponding to a preset data type; according to the data volume of the data fragments, determining fragment categories of the data fragments; the fragment category comprises a first fragment category and a second fragment category, and the data volume of the first data fragment belonging to the first fragment category is smaller than the data volume of the second data fragment belonging to the second fragment category; storing all data in the first data fragment into the same storage partition, and splitting all data in the second data fragment into different storage partitions; and writing the data in the storage partition into the first database. The scheme provided by the invention can reduce the overhead of writing the data into the multiple tables and avoid resource waste.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the rapid development of the Internet and the explosive growth of data, enterprises have an increasing demand for data storage, query and analysis. Accurately writing data into a database can help enterprises achieve efficient data storage, fast query and accurate analysis. Therefore, as the volume of data continues to grow, how to write a large amount of data in real time and reliably will become a very important link in the field of data processing.

[0003] In the related art, it is usually necessary to split the data to be written into a database into data streams corresponding to each single table according to the data type, wherein the data types corresponding to the data written in each single table are the same.

[0004] However, there are many types of data to be written into the database, and each type of data requires an HTTP persistent connection for data writing. In this way, the number of HTTP persistent connections required will also be very large, resulting in excessive overhead for writing data into multiple tables. In addition, since the data flow of each type of data is different, when the data volume corresponding to some types of data is very small, this will cause the HTTP persistent connection for processing these types of data to be idle for a long time, resulting in a waste of resources and extremely low performance of writing data into multiple tables. Summary of the invention

[0005] The present disclosure provides a data processing method, device, electronic device and computer-readable storage medium, which can reduce the overhead of writing data into multiple tables and avoid resource waste. The specific scheme is as follows:

[0006] In a first aspect, an embodiment of the present disclosure provides a data processing method, the method comprising:

[0007] receiving a data stream object; the data stream object comprising data to be written into a first database;

[0008] Splitting the data stream object into data slices corresponding to preset data types;

[0009] Determine a shard category of the data shard according to the data volume of the data shard; the shard category includes a first shard category and a second shard category, and the data volume of the data shard belonging to the first shard category is less than the data volume of the data shard belonging to the second shard category;

[0010] All data in the first data shard are stored in the same storage partition, and all data in the second data shard are split and stored in different storage partitions; the first data shard is a data shard of the first shard category, and the second data shard is a data shard of the second shard category;

[0011] The data in the storage partition is written into the first database.

[0012] In a second aspect, an embodiment of the present disclosure provides a data processing device, the device comprising:

[0013] A receiving unit, configured to receive a data stream object; the data stream object includes data to be written into the first database;

[0014] A splitting unit, used to split the data stream object into data slices corresponding to preset data types;

[0015] a determining unit, configured to determine a shard category of the data shard according to the data volume of the data shard; the shard category includes a first shard category and a second shard category, and the data volume of the data shard belonging to the first shard category is less than the data volume of the data shard belonging to the second shard category;

[0016] A partitioning unit, configured to store all data in a first data shard into a same storage partition, and split all data in a second data shard into different storage partitions; the first data shard is a data shard of the first shard category, and the second data shard is a data shard of the second shard category;

[0017] A writing unit is used to write the data in the storage partition into the first database.

[0018] In a third aspect, the present disclosure further provides an electronic device, including:

[0019] Processor; and

[0020] The memory is used to store a data processing program. After the electronic device is powered on and the program is run by the processor, the method described in the first aspect is executed.

[0021] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium storing a data processing program, which is executed by a processor to perform the method described in the first aspect.

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] The data processing method provided by the present disclosure includes: receiving a data stream object; the data stream object includes data to be written into a first database; splitting the data stream object into data slices corresponding to a preset data type; determining the slice category of the data slice according to the data volume of the data slice; the slice category includes a first slice category and a second slice category, and the data volume of the data slice belonging to the first slice category is less than the data volume of the data slice belonging to the second slice category; storing all the data in the first data slice in the same storage partition, and splitting all the data in the second data slice into different storage partitions; the first data slice is a data slice of the first slice category, and the second data slice is a data slice of the second slice category; writing the data in the storage partition into the first database.

[0024] It can be seen that the data processing method provided by the present disclosure determines the data shards corresponding to each data type according to the data types contained in the received data stream object. Afterwards, the shard category is determined for the data shard according to the data volume corresponding to the data shard. Due to the data sharding, the shard category is determined for the data shard, that is, the data shard corresponding to the data stream object is determined as a data shard with a larger data volume and a data shard with a smaller data volume according to the data volume. The data in the second data shard of the second shard category with a larger data volume is split and stored in different storage partitions, and the data in the first data shard of the first shard category with a smaller data volume is stored in the same storage partition. In this way, compared with the prior art in which the data of each data type is divided into multiple storage partitions and the data of one data type is written into the database through multiple data requests, all the data in the first data shard with a smaller data volume is stored in one storage partition, and the data can be written into the database through one http long connection, which obviously reduces the number of http long connections corresponding to the first data shard with a smaller data volume, thereby reducing the overhead of data writing. In addition, the data processing method provided by the present invention avoids evenly distributing the data in the first data shard with a smaller data volume to each storage partition, so that the amount of data processed by the corresponding http long connection will not be too small, avoiding waste of resources due to a large number of http long connections being idle for too long, so that resources are reasonably utilized and the performance of writing data to multiple tables is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a system architecture diagram of a data processing method provided by an embodiment of the present disclosure;

[0026] Figure 2 It is a detailed diagram of the system architecture of the data processing method provided by the embodiment of the present disclosure;

[0027] Figure 3 It is a schematic diagram of writing data into multiple tables in the prior art;

[0028] Figure 4 is a flow chart of a data processing method provided by an embodiment of the present disclosure;

[0029] Figure 5 is a schematic diagram of dynamically detecting a first data segment and a second data segment in a data processing method provided by an embodiment of the present disclosure;

[0030] Figure 6 is a schematic diagram of thread pool asynchronous processing provided by an embodiment of the present disclosure;

[0031] Figure 7 is a structural block diagram of an example of a data processing device provided by an embodiment of the present disclosure;

[0032] Figure 8 It is a structural block diagram of an example of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Many specific details are described in the following description to facilitate a full understanding of the present disclosure. However, the present disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present disclosure, so the present disclosure is not limited by the specific implementation disclosed below.

[0034] It should be noted that the terms "first", "second", "third", etc. in the claims, specifications and drawings of the present disclosure are used to distinguish similar objects and are not used to describe a specific order or sequence. The data used in this way are interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having" and their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Before describing the embodiments of the present disclosure in detail, related concepts are first introduced and the prior art is further described.

[0036] 1. Introduction to related concepts

[0037] 1. Kafka (distributed messaging system): Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all action stream data of consumers on the website.

[0038] The Kafka system consists of the following three nodes: producer (message producer), broker (server), and consumer (message consumer). Kafka contains one or more servers, which are called brokers. Producers are used to generate messages and publish them to brokers. Brokers are used to store and maintain message queues of various topics on disk. Consumers can read messages from brokers.

[0039] Each message published to Kafka has a category, which is called a Topic. Each Topic contains one or more Partitions. When an event stream enters Kafka, it becomes a Topic. In this disclosure, taking virtual games as an example, all the data of each virtual game can belong to a Topic.

[0040] 2. Doris (analytic database): Doris is a high-performance, real-time analytical database that can return query results for massive data in sub-second response time, providing high-performance analysis and report query functions on large data sets at a low cost. Doris can not only support high-concurrency point query scenarios, but also support high-throughput complex analysis scenarios. Doris can better meet the use scenarios such as report analysis, instant query, and unified data warehouse construction. Users can build applications such as user behavior analysis, log retrieval analysis, and order analysis on it.

[0041] 3. Flink (stream data processing engine): Flink is a stream processing framework and a distributed stream data flow engine written in Java and Scala. Flink executes any stream data program in a data parallel and pipeline manner. Flink can consume Kafka data in real time and write it to Doris.

[0042] Among them, Flink can directly read full data and incremental change data from databases such as MySQL and PostgreSQL through a source component. The source component can be Flinkcdc, which is the flink-cdc-connectors component developed by the Flink community. It can monitor and capture changes in databases such as MySQL and PostgreSQL, and write them to the message middleware for other services to subscribe and consume.

[0043] 4. MySQL (relational database): MySQL is a relational database. MySQL can be used as a separate application in a client-server network environment, or it can be embedded in other software as a library.

[0044] 2. Further description of the prior art

[0045] As a high-performance, scalable and open source data warehouse, Doris (analytic database) has great advantages in meeting the needs of enterprises for big data analysis and real-time data processing. Doris can help enterprises achieve efficient data storage, quickly respond to query results with sub-second response time, and perform real-time analysis of the query results, thereby improving the competitiveness and innovation capabilities of enterprises. Therefore, how to write a large amount of data into Doris in real time and reliably becomes extremely important.

[0046] To address this problem, the existing technology usually uses the distributed message queue Kafka as the data source, consumes Kafka data in real time based on stream data processing engines such as Flink, and writes it to Doris. The high performance and high reliability of Flink can ensure the real-time and consistency of large amounts of data writing. All upstream data is written into a single table, which can quickly write data.

[0047] However, in actual applications, there are many types of upstream data, and the types of data change frequently. If all kinds of data are written into a single table, the data volume in the single table will be very large and the data types will be extremely complex. When developers query and analyze each type of data, the performance will be very low.

[0048] Therefore, in order to improve the performance of data query and analysis, it is necessary to split the data read by Flink from Kafka into data streams corresponding to each data type according to the data type to be written to the database, and write each data stream into the corresponding data table through an http long connection. However, since Flink executes a data writing task through multiple data partitions in parallel, each data partition will have data of various types. In the case of a large number of upstream data types, the number of data streams split out in each partition is also very large. For example, if there are 100 types of data to be written to the database, each data partition in Flink will have 100 types of data. In this way, each data partition needs 100 http long connections for data writing. Since each http long connection requires a thread for maintenance, the thread overhead is also very large, resulting in a burden on data maintenance.

[0049] In addition, because the data volumes corresponding to different types of data are different, some data streams correspond to larger data volumes, while others correspond to smaller data volumes, which results in some threads being idle while others being extremely busy, causing load imbalance between threads and serious data skew, resulting in extremely low performance when writing data to multiple tables. Moreover, when the data volumes corresponding to a large number of data types are very small, this will cause a large number of http long connections to be idle for too long, resulting in serious waste of resources when writing data to multiple tables.

[0050] Based on the above reasons, in order to reduce the overhead of writing data into multiple tables and avoid wasting resources, the first embodiment of the present disclosure provides a data processing method, which is applied to an electronic device, which may be a desktop computer, a laptop computer, a mobile phone, a tablet computer, a server, a terminal device, etc., or other electronic devices capable of data processing, which are not specifically limited in the embodiments of the present disclosure.

[0051] Before introducing the data processing method provided by the present disclosure, Figure 1 and Figure 2 The system architecture of the data processing method provided by the embodiment of the present disclosure and the specific details in the system architecture are introduced.

[0052] like Figure 1 1 is a system architecture diagram of a data processing method provided in an embodiment of the present disclosure, and the data processing system 100 includes a message system 101, a stream data processing engine 102, a first database 103, and a second database 104. It should be noted that in the embodiment of the present disclosure, the message system 101 is kafka, the stream data processing engine 102 is flink, the first database 103 is doris, and the second database 104 is mysql. The present disclosure is not limited to this.

[0053] Among them, the message system 101 (kafka) may include data under multiple categories (topics) such as the first category (topic), the second category (topic), ... the nth category (topic), etc. Each category can be understood as a project. For example, the data generated in the virtual game A is the first category, and the data generated in the virtual game B is the second category; the stream data processing engine 102 (flink) can read the data in the message system 101 (kafka) in real time, obtain the data stream to create a flink task, and a flink task is the data in a topic. The stream data processing engine 102 (flink) can read the mapping relationship between the data field value and the doris table name from the second database 104 (mysql) in real time or according to a certain period, and divide the data in a flink task by the doris table name according to the read data type mapping relationship. Flink finally writes each data into the first database 103 (doris).

[0054] Among them, the message system 101 (kafka) and the stream data processing engine 102 (flink), the second database 104 (mysql) and the stream data processing engine 102 (flink), and the stream data processing engine 102 (flink) and the first database 103 (doris) can be connected through a network, and the network can be a wired network or a wireless network, which is not specifically limited in the present disclosure.

[0055] The wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including hypertext mark-up language (HTML), extensible markup language (XML), etc. In addition, conventional encryption technologies such as secure socket layer (SSL), transport layer security (TLS), virtual private network (VPN), and Internet protocol security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0056] like Figure 2 FIG. 2 is a detailed diagram of the system architecture of the data processing method provided by the embodiment of the present disclosure, which includes the following steps S201 to S208:

[0057] Step S201: Flink obtains data stream objects from the message queue of Kafka in real time;

[0058] Step S202: In MySQL, dynamically update the mapping relationship between the data field value and the doris table name;

[0059] Step S203: Flink reads the mapping relationship from MySQL, converts it into a mapping relationship broadcast stream object, and broadcasts it to the concurrent threads in Flink, so that the data in each thread is parsed based on the mapping relationship;

[0060] Step S204: In Flink, the doris table name corresponding to each piece of data is parsed based on the mapping relationship, and the data stream object is split into data belonging to each doris table (doris table 1, doris table 2, doris table 3, doris table 4 and doris table 5);

[0061] Step S205: In Flink, dynamically detect the amount of data corresponding to each Doris table to determine whether the split Doris table belongs to a large table or a small table.

[0062] Step S206: In Flink, partition the data of the large and small tables so that the data in a large table is evenly split and stored in each storage partition, and the data in the same small table is stored in the same storage partition. Specifically, for the large table: the data is evenly sent to each storage partition, and for the small table: the doris table name is hashed and modulo, and the data in the same small table is stored in the same storage partition.

[0063] Step S207: In Flink, the data in each storage partition is batched and transmitted, and the data in each batch is divided into sub-batches according to the table name. Specifically, when the amount of data in each partition accumulates to a preset amount, the data in each partition is split into sub-batches according to the mapping relationship between the data field value and the doris table name.

[0064] Step S208: Flink writes the data of each split sub-batch into Doris through an http request.

[0065] Among them, Flink customizes BroadcastProcessFunction (broadcast processing function), which includes processElement (processing element) and processBroadcast (processing broadcast). ProcessElement (processing element) is used to parse the doris table name corresponding to each data based on the current mapping relationship. ProcessBroadcast (processing broadcast) is used to update the mapping relationship broadcast stream object when the data type mapping relationship read from MySQL is updated.

[0066] like Figure 3 As shown, it is a schematic diagram of writing data into multiple tables in the prior art. After receiving the data stream object, the data to be written into each doris table is split according to the data type, and the data to be written into each doris table is allocated to each partition in flink. One partition contains data of all data types, and the data in each partition is split into sub-batches and written into the doris database in batches.

[0067] The following will be Figure 2 The data processing method provided by the present disclosure is shown Figure 3 The prior art shown is compared to introduce the optimization performed by the data processing method provided by the present disclosure in the process of writing data into multiple tables:

[0068] 1. Reduce the number of http long connections and reduce network overhead.

[0069] Figure 2 In the table, partition 1 contains the data in doris table 1, doris table 4, and doris table 2, and partition 2 contains the data in doris table 1, doris table 4, doris table 3, and doris table 5. Figure 3 In the process of writing data to the same batch of data stream objects, partition 1 and partition 2 both contain data from doris table 1, doris table 2, doris table 3, doris table 4, and doris table 5. Figure 2 The processing method provided by the present disclosure is divided into 7 sub-batches in total. Figure 3 The prior art method shown divides a total of 10 sub-batches. Therefore, the data processing method provided by the present disclosure can effectively reduce the number of http long connections and reduce network overhead.

[0070] 2. Improved resource utilization of http long connections.

[0071] Take doris table 2 with a small amount of data as an example. Figure 2 In the doris table, all data in table 2 is stored in partition 1. Figure 3 In the example, part of the data in doris table 2 is stored in partition 1, and part of the data is stored in partition 2. Figure 2 In the processing method provided by the present disclosure shown, all data in doris table 2 can be written into the database through a http long connection, and Figure 3 In the prior art method shown, two HTTP long connections are required to write all the data in doris table 2 into the database. It can be seen that Figure 3 The amount of data processed by an HTTP long connection that processes the data in doris table 2 is only Figure 2 The amount of data processed by the HTTP long connection in processing the data in doris table 2 is half. In the case of a large number of doris tables with smaller data volumes, Figure 3 The existing technology shown will result in a small amount of data processed by many http long connections, resulting in serious waste of resources. Figure 2 In the processing method provided by the present disclosure shown, the data in each doris table with a smaller data volume is stored in a partition. In this way, only one http long connection is required to process the data in the doris table with a smaller data volume, so that the data volume in the http long connection will not be too small, thereby improving the resource utilization of the http long connection.

[0072] The following, combined Figure 4 , Figure 5 as well as Figure 6 The data processing method provided by the embodiment of the present disclosure is introduced.

[0073] As an embodiment of the present disclosure, the data processing method provided by the present disclosure can be applied to a stream data processing engine, in which a storage partition is deployed, such as Figure 4 As shown, the data processing method includes the following steps S401 to S405.

[0074] Step S401: Receive a data stream object.

[0075] The data stream object includes data to be written into the first database.

[0076] It should be noted that the data stream object is a data stream composed of data read from the message queue by the stream data processing engine and to be written into the first database. In the embodiment of the present disclosure, the stream data processing engine is flink, the message queue is the message queue in kafka, and the first database is Doris. In practical applications, the data processing method can also be applied to other devices with the same function as the stream data processing engine. The stream data processing engine can also be other engines with the same function as flink that can write data into the database in real time and efficiently. The message queue can also be other systems that can store data, and the database can also be other databases that can store data. The present disclosure does not specifically limit this.

[0077] In order to improve data throughput, Kafka uses a multi-partition method to cache streaming data. At the same time, Flink will schedule multiple upstream data partitions for data consumption when reading Kafka, which can effectively improve data throughput and data writing efficiency. In Flink, tasks can be processed in parallel. Assuming that the parallelism of tasks in Flink is 10, it means that there are 10 upstream data partitions in Flink that can execute the current task. In this way, the data in Kafka will be evenly divided into 10 parts and sent to each upstream data partition in Flink.

[0078] The data in the received data stream object is data generated by an application, which may be an application that needs to be downloaded and installed, or an application that can be used instantly, including but not limited to at least one of the following: an e-commerce platform, instant messaging software, and a virtual game. The data generated by the application is written into the first database storing data with the authorization of the user, so that the developer can perform query analysis based on the written data to better improve the application.

[0079] Step S402: Split the data stream object into data segments corresponding to preset data types.

[0080] This step is used to split the received data stream object into data slices according to data types, wherein the data types corresponding to the data in each data slice are the same.

[0081] In actual applications, data of one data type is written into a data table in the database for storing data. In this way, when querying data, the corresponding data table can be quickly found to query the corresponding data, or when analyzing data of one data type, only the data in the corresponding data table can be analyzed, and the data outside the data table can be shielded. Therefore, writing one type of data into one data table can improve the efficiency of data query and analysis.

[0082] For example, in a virtual game, with user authorization, the login log corresponding to the login behavior is written into the login data table, the registration log corresponding to the registration behavior is written into the registration data table, the chat log corresponding to the chat behavior is written into the chat data table, the recharge log corresponding to the recharge behavior is written into the recharge data table, and the game behavior log corresponding to the game behavior is written into the game behavior database. In this way, when it is necessary to analyze the registration situation of the virtual game, only the data in the registration database can be selected for analysis to obtain the specific analysis results of the registration behavior.

[0083] Based on this, in the embodiment of the present disclosure, the data stream object is first split into data fragments according to the data type. It should be noted that the data type can be preset, and each data in the data stream object can carry an identifier of the data type. In this way, when the data stream object is received, the data can be classified according to the preset data type to obtain the data fragment corresponding to each data type.

[0084] As shown in Table 1, it is a schematic table of splitting a batch of data stream objects of a virtual game into data slices in the data processing method provided by an embodiment of the present disclosure:

[0085] Table 1.

[0086]

[0087] Since the amount of data corresponding to each data type is different, the amount of data corresponding to each split data shard is also different. For example, the amount of game behavior data is much larger than the amount of registration data. If data is only written after the data stream object is split into data shards according to data type, the amount of data processed by some http requests will be extremely large, while the amount of data processed by other http requests will be very small, which will cause serious data skew.

[0088] Therefore, in order to avoid data skew when writing data to multiple tables, the split data shards are classified according to the data volume, and different types of data shards perform different operations to ensure that the data volume difference in each http request is small and can be relatively balanced. Therefore, in the present disclosure, after splitting the data shards, the following steps S403 and S404 are performed:

[0089] Step S403: Determine the shard category of the data shard according to the data volume of the data shard.

[0090] The shard categories include a first shard category and a second shard category, and the data volume of the data shards belonging to the first shard category is less than the data volume of the data shards belonging to the second shard category.

[0091] This step is used to classify the data shards, and divide the split data shards into data shards of the second shard category with larger data volume and data shards of the first shard category with smaller data volume.

[0092] In a possible implementation, data slices can be classified by setting a data volume reference value. Specifically, the size relationship between the data volume of the split data slices and the data volume reference value is determined. When the data volume of a certain data slice is less than the data volume reference value, the data slice is determined as a data slice belonging to the first slice category with a smaller data volume; when the data volume of a certain data slice is greater than or equal to the data volume reference value, the data slice is determined as a data slice belonging to the second slice category with a larger data volume.

[0093] In another possible implementation, considering that the division of shard categories is based on the data volume of each data shard in a batch of data stream objects, a large data volume or a small data volume is a relative concept. Based on this, data shards of the second shard category with a relatively large data volume and data shards of the first shard category with a relatively small data volume can be dynamically determined according to each split data shard. Specifically, after the data stream object is split into data shards, the data shards are sorted according to the data volume, and a preset number of data shards with a large data volume in the sorting are selected as data shards of the second shard category, and the rest are data shards of the first shard category.

[0094] In the embodiments of the present disclosure, other methods can be used to classify the data volume of data shards, for example, a data volume benchmark value can be set according to time periods. In time periods with high user activity, a higher data volume benchmark value can be set due to a large amount of data generated, and in time periods with low user activity, a lower data volume benchmark value can be set due to a small amount of data generated. The present disclosure does not specifically limit the classification method of data shards.

[0095] As shown in Table 2, it is a schematic table of the classification of the data fragments split in Table 1:

[0096] Table 2.

[0097]

[0098] Through this technical means, the data shards are classified to obtain data shards with a large amount of data belonging to the second shard category and data shards with a small amount of data belonging to the first shard category. In this way, different operations can be performed on data shards of different shard categories in subsequent steps.

[0099] Step S404: All data in the first data shard are stored in the same storage partition, and all data in the second data shard are split and stored in different storage partitions.

[0100] The first data shard is a data shard of the first shard category, and the second data shard is a data shard of the second shard category.

[0101] This step is used to perform different partitioning operations on data shards of different shard categories, so as to distribute and store the data in a second data shard in various storage partitions, and store the data in a first data shard in the same storage partition.

[0102] It should be noted that, since the data types in a batch of data stream objects are extremely complex, there may be hundreds of data types, among which the data volume corresponding to most data types is small. In this case, after this step is completed, a storage partition will include the data of each second data shard and all the data of at least one first data shard.

[0103] In a specific implementation, the data in the second data shard can be evenly distributed to each storage partition in flink, and considering that data of the same data type can be written through one http request, data of one data type is one data shard, therefore, the data in one data shard can be written through one http request. In this way, the data in the first data shard with a smaller data volume is distributed to one storage partition, and the data in the second data shard with a larger data volume is distributed to each storage partition, so that the difference in the data volume of each data type in one storage partition is significantly reduced, which is relatively balanced. In addition, the data in the first data shard with a smaller data volume can be written through one http request, so that the number of http requests in flink can be effectively reduced, reducing the loss incurred by data maintenance.

[0104] As shown in Table 3, it is an example of storing the data of each shard category in Table 2 into the storage partition:

[0105] Table 3.

[0106]

[0107] It should be noted that the data stored in different storage partitions of the same data shard is not the same, and one piece of data is only stored in one storage partition.

[0108] Step S405: writing the data in the storage partition into the first database.

[0109] This step is used to write the data in the storage partition into the database, specifically, to write the data in the storage partition into the corresponding data table according to the data type.

[0110] The first database is provided with various data tables, each of which is used to store data of a data type. In this way, the data in each storage partition can be successfully written into the corresponding data table.

[0111] The data processing method provided by the present disclosure includes: receiving a data stream object; the data stream object includes data to be written into a first database; splitting the data stream object into data slices corresponding to a preset data type; determining the slice category of the data slice according to the data volume of the data slice; the slice category includes a first slice category and a second slice category, and the data volume of the data slice belonging to the first slice category is less than the data volume of the data slice belonging to the second slice category; storing all the data in the first data slice in the same storage partition, and splitting all the data in the second data slice into different storage partitions; the first data slice is a data slice of the first slice category, and the second data slice is a data slice of the second slice category; writing the data in the storage partition into the first database.

[0112] It can be seen that the data processing method provided by the present disclosure determines the data shards corresponding to each data type according to the data types contained in the received data stream object. Afterwards, the shard category is determined for the data shard according to the data volume corresponding to the data shard. Due to the data sharding, the shard category is determined for the data shard, that is, the data shard corresponding to the data stream object is determined as a data shard with a larger data volume and a data shard with a smaller data volume according to the data volume. The data in the second data shard of the second shard category with a larger data volume is split and stored in different storage partitions, and the data in the first data shard of the first shard category with a smaller data volume is stored in the same storage partition. In this way, compared with the prior art in which the data of each data type is divided into multiple storage partitions and the data of one data type is written into the database through multiple data requests, all the data in the first data shard with a smaller data volume is stored in one storage partition, and the data can be written into the database through one http long connection, which obviously reduces the number of http long connections corresponding to the first data shard with a smaller data volume, thereby reducing the overhead of data writing. In addition, the data processing method provided by the present invention avoids evenly distributing the data in the first data shard with a smaller data volume to each storage partition, so that the amount of data processed by the corresponding http long connection will not be too small, avoiding waste of resources due to a large number of http long connections being idle for too long, so that resources are reasonably utilized and the performance of writing data to multiple tables is effectively improved.

[0113] In addition, since all data in a second data shard with a larger data volume is split and stored in different storage partitions, and thus written to the database through multiple http long connections, and all data in a first data shard with a smaller data volume is stored in the same storage partition, and thus written to the database through one http long connection, this will ensure that the amount of data processed by each http long connection will not be too much or too little, and can effectively reduce the difference in data volume between different http long connections, so that the amount of data corresponding to each http long connection is more balanced, avoiding data skew.

[0114] Optionally, step S402 may be implemented according to the following steps:

[0115] According to the preset data type mapping relationship, the data stream object is split into data slices corresponding to the preset data types.

[0116] In the disclosed embodiment, the preset data type mapping relationship may be a mapping relationship between a preset data field value and a data table storing data, wherein the data field value may represent the data type, and the field represents a variable associated with an object or a class. Generally, a "column" in a data table is called a "field", and each field contains information on a certain topic. For example, in an address book data table, name and contact number are attributes of the content filled in each column, and therefore, these columns are called "name" field and "contact number" field.

[0117] In actual applications, the preset data type mapping relationship is used to maintain which Doris table in Doris each log data in each topic of Kafka needs to be written to. In a specific implementation, the data table to which the data will be written can be determined based on the data field value corresponding to the data. In the database, the corresponding data table can be set according to the needs. After the data table is set, the mapping relationship between the data field value and the data table can be set, and the mapping relationship can be used to distinguish which data is written to which data table.

[0118] A data table may correspond to a table identifier, which can be understood as a unique identification identifier of the data table in the database in which it is located. Each time a data table is created in the database, a unique table identifier will be assigned to the data table. There is a one-to-one correspondence between the data table and the table identifier. According to the table identifier, a unique data table can be determined in the database storing the data table. The table identifier can be a string of characters, a vector, or a picture. In the embodiment of the present disclosure, the table name of the data table is used as the table identifier corresponding to the data table, and the present disclosure does not specifically limit this.

[0119] For example, for the login log, the data field value log_id is 4001, and the table name of the login data table doris written by the login log is ods_login_role. Then the mapping relationship between the data field value of the login log and the stored data table is 4001=>ods_login_role. In this way, when Flink writes data to the login log, it can parse the data field of the login log and obtain the data field value of 4001. Then, through the mapping relationship, it is determined that the table name of the data table to be written by the login log is ods_login_role.

[0120] As shown in Table 4, it is a schematic table of an example of the mapping relationship between data field values ​​and data types in a data table provided by an embodiment of the present disclosure:

[0121] Field Value doris table name 1001 ods_login_role 1002 ods_register_role 1003 ods_paid

[0122] In this way, the data stream object can be split by the preset data type mapping relationship to obtain various data fragments, and the data field values ​​corresponding to the data in each data fragment are the same, so the data in each data fragment will be written into the same data table. It should be noted that the data splitting here can be understood as identifying the data, and the data to be written into different data tables are distinguished by different identification information.

[0123] In addition, in actual applications, the mapping relationship may be updated frequently. For example, during a certain festival, the developer sets up a new activity for the festival in the virtual game. The log corresponding to the activity should be written into a separate data table. In this way, the developer can quickly query and analyze the data of the new activity, so as to make better improvements to the setting of the activity later.

[0124] In the prior art, the data type mapping relationship is usually saved in a configuration file corresponding to flink, and the configuration file is updated each time the data type mapping relationship is updated. In this way, flink needs to be restarted to write data each time the data type mapping relationship is updated, which adds a lot of extra loss. In addition, data writing is a continuous and uninterrupted process. In this way, data writing will be interrupted each time the data type mapping relationship is updated. After the data type mapping relationship is updated, the data will be written to the database, which will lead to a large amount of data hoarding, resulting in a heavy data load, making the data writing performance poor.

[0125] Therefore, in order to be able to quickly write data into the database based on the updated data type mapping relationship when the data type mapping relationship is updated, the data type mapping relationship is stored in a dedicated second database (relational database) in the present disclosure. The embodiment of the present disclosure takes MySQL as the second database as an example. In actual applications, the second database storing the data type mapping relationship can also be other databases that can achieve the same function, and the present disclosure does not specifically limit this.

[0126] Therefore, the data processing method provided by the embodiment of the present disclosure may further include the following steps:

[0127] In response to an update of the data type mapping relationship stored in the second database, acquiring an updated data type mapping relationship from the second database;

[0128] Accordingly, the step of "splitting the data stream object into data slices corresponding to the preset data types according to the preset data type mapping relationship" can be implemented by the following steps:

[0129] According to the updated data type mapping relationship, the data stream object is split into data slices corresponding to the preset data types.

[0130] In a specific implementation, MySQL that stores data type mapping relationships is connected to flink. When the data type is updated, the data type mapping relationship is updated in MySQL. For example, if a recharge activity is added during the New Year's Day period, a data table corresponding to the New Year's Day recharge activity is set in the first database doris, and a mapping relationship between the data field value of the New Year's Day recharge activity and the data table is added in the second database.

[0131] Specifically, a source operator can be defined in flink to read the data type mapping relationship from MySQL. It should be noted that flink can read the mapping relationship from MySQL in real time or according to a preset period, which is not limited in the present disclosure.

[0132] It should be noted that since Flink processes data stream objects in parallel through different threads, in order to enable each thread to quickly complete data splitting according to the data type mapping relationship after reading the mapping relationship, a broadcast operator can be set in Flink. The data type mapping relationship is converted into a broadcastable variable (mapping relationship broadcast stream object) through the Broadcast operator and broadcast to each concurrent thread. In this way, the data in each thread can be parsed based on the data type mapping relationship and split into various data fragments.

[0133] In a specific implementation, flink connects the data stream object with the broadcast stream object to obtain a stream object that contains both the data stream object and the data type mapping relationship. In order to quickly and correctly split the data stream object into corresponding data fragments based on the updated data type mapping relationship when the data type mapping relationship is updated, a BroadcastProcessFunction (broadcast processing function) operator can also be defined in flink in this disclosure, and the processElement (processing element) and processBroadcast (processing broadcast) methods are implemented in the BroadcastProcessFunction operator.

[0134] Specifically, the processElement method is used to implement specific data processing logic. Based on the currently broadcasted data type mapping relationship, each data in the data stream object is parsed, and the data in the data stream object is split into data slices. The processBroadcast method is used to update the mapping relationship broadcast stream object in real time according to the read data type mapping relationship to ensure that the data type mapping relationship indicated by the mapping relationship broadcast stream object is the latest data type mapping relationship.

[0135] In this way, the data stream objects received by Flink are divided according to the data type mapping relationship dynamically maintained in the second database, so that the data shards corresponding to each data table can be obtained accurately and efficiently, avoiding the data loss caused by manually updating the data type mapping relationship.

[0136] For example, the data stream object to be written currently contains three types of logs: login, registration, and recharge. Their data field values ​​are 1001, 1002, and 1003, respectively, and the corresponding doris table names are ods_login_role, ods_register_role, and ods_paid. If the existing technology is used, the data type mapping relationship in Table 4 can be fixed in the configuration file. At this time, after the flink task is started, the data field values ​​of each data in the data stream object can still be parsed, and then the corresponding doris table name can be found.

[0137] However, if a new chat-related data type is currently added, the data field value corresponding to this data type is 1004, and the corresponding doris table name is ods_chat. When splitting data shards, since there is no data type mapping relationship of 1004=>ods_chat in the flink configuration file, it is impossible to parse out which data table the data with a data field value of 1004 belongs to. At this time, it is necessary to manually modify the configuration file, update the data type mapping relationship, and restart the flink task. If the solution provided by the embodiment of the present disclosure is adopted, the configuration information is saved in MySQL, the flink task will obtain the latest data type mapping relationship from MySQL in real time, and split the data stream object into individual data shards based on the latest data type mapping relationship, without the need for frequent manual updates of the configuration file and restarting the flink task. It reduces the waste of manpower and material resources in the process of writing data into multiple tables, reduces the cost of manual operation and maintenance, and is easier to manage.

[0138] In an optional implementation, data shards may be dynamically classified into shard categories in the following manner:

[0139] Sort the data shards by data volume;

[0140] A preset number of data shards with larger data volumes are determined as second data shards belonging to the second shard category, and data shards other than the second data shards in the data shards are determined as first data shards belonging to the first shard category.

[0141] This step is used to dynamically classify the data slices. For example, the data slices can be sorted from large to small or from small to large according to the data volume. When sorting from large to small, the first 10 can be selected as the second data slices with large data volume, or the first 10% can be selected as the second data slices with large data volume, and the remaining data slices are the first data slices with small data volume.

[0142] In the embodiment of the present disclosure, in order to quickly determine whether each data shard is a first data shard with a smaller data volume or a second data shard with a larger data volume at the current moment, the data volume corresponding to each data shard can be counted in real time. Therefore, before step S403, the data processing method provided by the embodiment of the present disclosure may also include the following steps:

[0143] Get the shard ID corresponding to the data shard;

[0144] Add 1 to the value of the data volume corresponding to the shard identifier in the data volume statistics table to obtain the data volume corresponding to the data shard.

[0145] In this step, a data volume statistics table is maintained in Flink, which records the shard identifier and data volume corresponding to the data shard. Each time a data is received, the data shard corresponding to the data is determined, the shard identifier corresponding to the data shard is obtained, and the data volume corresponding to the shard identifier in the data volume statistics table is increased by 1. In this way, the data volume corresponding to each data shard can be recorded in real time.

[0146] In a specific implementation, a RichMapFunction operator may be defined in flink, and a HashMap object and a LinkedHashMap object may be maintained inside the operator.

[0147] Among them, the HashMap object is used to record the data shard to which the data arriving at the current moment belongs, and the current data volume corresponding to the data shard, and the LinkedHashMap object is used to record the data shard that currently belongs to the second shard category with a larger data volume and the data volume corresponding to each data shard belonging to the second shard category. It should be noted that in the HashMap object and the LinkedHashMap object, the data shard can be recorded by the shard identifier corresponding to the data shard.

[0148] It should be noted that, since the data in a data shard will be written into a data table, the shard identifier in this step may also be the table identifier of the corresponding data table.

[0149] Specifically, each time the RichMapFunction operator in Flink receives a piece of data, it updates the HashMap object information in real time according to the table identifier of the data table to which the data belongs, and increases the amount of data corresponding to the table identifier in the HashMap by one. In this way, the Hashmap can accurately record the amount of data corresponding to each current data shard (that is, the amount of data for each table identifier).

[0150] Afterwards, the data shards are sorted according to the data volume recorded in the Hashmap object, and the data shards with the largest data volume are selected as the second data shards. In this way, it can be determined whether the current data shard belongs to the first data shard with the smaller data volume or the second data shard with the larger data volume based on whether the identification information corresponding to the current data shard exists in the LinkedHashMap.

[0151] In this way, no matter how much data is in each data slice in the current data stream object, the second data slice with a relatively large data volume and the first data slice with a relatively small data volume can be determined therefrom.

[0152] Since data continuously flows in in the form of data streams, and the data volume of the same data type in different time periods may vary greatly, the data volume of different types of data in different time periods may change, which may cause the data of a certain data type in certain time periods to be the second data shard with a larger data volume, while the data of the data type in other time periods to be the first data shard with a smaller data volume.

[0153] For example, during the time period from 10:00am to 12:00am, more users play mode one in the virtual game, and fewer users play mode two, and during the time period from 17:00pm to 19:00pm, more users play mode two in the virtual game, and fewer users play mode one; then, during the time period from 10:00am to 12:00am, the amount of data corresponding to mode one is greater than the amount of data corresponding to mode two, and during the time period from 17:00pm to 19:00pm, the amount of data corresponding to mode two is greater than the amount of data corresponding to mode one.

[0154] Therefore, the data flow corresponding to the data shard in different time periods may change. In order to avoid accumulating the data volume corresponding to a data shard in the HashMap object, thereby causing inaccurate shard categories for the data shard, the data in the data volume statistics table in the embodiment of the present disclosure is updated according to a preset period. Before the step of "adding 1 to the value of the data volume corresponding to the shard identifier in the data volume statistics table", the data processing method provided by the present disclosure also includes the following steps:

[0155] Determine the statistical duration of the data volume statistics table; check whether the statistical duration has reached the preset period;

[0156] Therefore, the step of "adding 1 to the value of the data volume corresponding to the shard identifier in the data volume statistics table" can be implemented according to the following steps: when the statistical time length is less than the preset period, add 1 to the value of the data volume corresponding to the shard identifier in the data volume statistics table.

[0157] Correspondingly, the data processing method provided by the present disclosure further includes the following steps: when the statistical time length is greater than or equal to a preset period, clearing the data in the data volume statistical table.

[0158] Specifically, a preset period may be set for the HashMap object, and the data volume of each data shard in each preset period may be counted, so that each data shard may be classified into a shard category in each preset period.

[0159] After the data is counted in each cycle, the data counted in the HashMap object can be cleared. In practical applications, due to the large amount of data and the large number of data types, the amount of calculation required for real-time sorting and size table classification of the data in the HashMap object is large, which in turn affects the performance of data writing. In order to reduce the performance consumption caused by real-time sorting and size table classification, in the present disclosure, while the data amount is counted according to a preset cycle, the data amount of each data slice can also be sorted according to the preset cycle, so that the first data slice and the second data slice are classified according to the preset cycle.

[0160] For example, a timer scheduling thread is started in Flink to clean up the data in the HashMap object in a ten-minute cycle. Before cleaning, the data shards recorded in the HashMap object need to be sorted according to the data volume, and the top 10% of the data table is taken, marked as the second data shard with a larger data volume, and synchronized to the LinkedHashMap object.

[0161] like Figure 5 FIG. 1 is a schematic diagram of dynamically detecting a first data segment and a second data segment in a data processing method provided by an embodiment of the present disclosure, and includes the following steps:

[0162] When receiving the data, get the table name tb of the doris table to which the data belongs;

[0163] Check whether Hshmap contains a table named tb; if so, add 1 to the value of the table named tb in Hshmap; if not, set the value of the table named tb in Hshmap to 1;

[0164] Check whether the statistical cycle has reached the preset cycle; if so, update linkedHashmap and clear the statistical data in Hshmap; if not, continue to return to the step of "when receiving data, obtain the table name tb of the doris table to which the data belongs";

[0165] Check whether linkedHashmap contains table name tb; if so, mark table name tb as a large table; if not, mark table name tb as a small table.

[0166] In addition, in order to avoid the loss of the status recorded in the data volume statistics table due to the failure of the Flink task, the data volume statistics table is set with snapshot status and recovery status; the snapshot status is used to save the data volume statistics table to the snapshot space; the recovery status is used to restore the data volume statistics table according to the data volume statistics table stored in the snapshot space when the data volume statistics table is lost.

[0167] Specifically, you can manage HashMap objects through Flink's checkpoint mechanism. Connect RichMapFunction to the ListCheckpointed interface, and add the snapshotState and restoreState methods of RichMapFunction to the ListCheckpointed interface.

[0168] In this way, the snapshot state can save the data recorded in the HashMap object to the snapshot space. In the case of a failure in flink data writing that causes the current state of the HashMap object to be lost, the recovery state can quickly restore the data recorded in the HashMap object based on the data stored in the snapshot space. Among them, the factors that lead to data writing failure in flink may include but are not limited to at least one of the following: machine hardware failure, disk damage, memory overflow, and network fluctuation.

[0169] In practical applications, in order to avoid data skew and keep the data in each http request balanced, the "storing all the data in the first data shard into the same storage partition" in step S404 can be implemented according to the following steps:

[0170] Get the number of partitions of the storage partition;

[0171] Determining a hash value of the first data shard;

[0172] Take the remainder of the hash value to the number of partitions, and store the data in the first data shard into the storage partition corresponding to the remainder;

[0173] The step of "splitting all the data in the second data shard and storing them in different storage partitions" can be implemented according to the following steps: according to the number of partitions, the data in the second data shard is evenly distributed and stored in different storage partitions.

[0174] Specifically, the present disclosure may be provided with a data balancing partitioning strategy, and the determined data shards of different sharding categories are partitioned according to the data balancing partitioning strategy.

[0175] The following is a detailed introduction to the data balance partition strategy:

[0176] It should be noted that the data in the same data shard can be written into the same data table through one http request because the data types are the same, and data of different data types need to be written into different data tables through different http requests. In order to reduce the number of http requests, the present disclosure can allocate the data in a data shard belonging to the same table to the same storage partition, so that the data belonging to the same data table can be written into the data table through only one http request, thereby improving the performance of data writing.

[0177] However, considering that the amount of data in the second data shard is relatively large, if the data in the second data shard is written into the corresponding data table through an http request, the amount of data written in one http request will be abnormally large.

[0178] Therefore, in the present disclosure, in order to balance data and the number of http requests, the data in a second data shard with a larger data volume can be evenly distributed to each storage partition, and the data in a first data shard with a smaller data volume can be allocated to the same storage partition.

[0179] In practical applications, since the data of the first data shard and the second data shard are evenly distributed in the upstream data partition itself when Flink reads data from the Kafka message queue, when the present disclosure is implemented, when the number of upstream data partitions is the same as the number of storage partitions, the data of the second data shard in each upstream data partition can be directly transferred to the corresponding storage partition, and only the data of the same first data shard can be allocated to the same storage partition.

[0180] In an optional implementation, for the first data shard, a hash value can be calculated for the first data shard, and the remainder of the number of partitions is taken according to the hash value. Since the hash value corresponding to the data in the same first data shard is the same, this ensures that the data in the same first data shard can be distributed to the same storage partition, so as to be written to the same data table through one http request, thereby reducing the network overhead of data writing.

[0181] A hash value, also known as a hash function (or a hash algorithm, also known as a hash function), is a method of creating a small digital "fingerprint" from any data. It should be noted that the hash value of the first data shard may specifically be the hash value of the table name of the corresponding data table.

[0182] For example, the number of downstream partitions is n, and the hashcode method in Java is called for the table name corresponding to the current data to obtain a hash value x, and then x is modulo n to obtain a number i. The modulo result i represents that the current data is to be transmitted to the downstream storage partition numbered i. Hashcode is a coding method. In Java, each object has a hashcode, and Java can identify each object through this hashcode.

[0183] In practical applications, each storage partition may correspond to a storage identifier, and the value obtained by taking the remainder of the hash value of the first data shard over the number of partitions may correspond to the storage identifier, and one remainder corresponds to one storage identifier. In this way, a unique storage identifier can be determined based on the calculated remainder, and then a unique storage partition can be determined, so that the data in the first data shard can be stored in the corresponding storage partition.

[0184] Assume that the number of storage partitions is 3, there are 8 first data shards with smaller data volume, and the hash values ​​of the first data shard 1 to the first data shard 8 are 1, 2, 3, 4, 5, 6, 7, 8 respectively. Then the values ​​obtained by taking the hash values ​​of the first data shard 1 to the first data shard 8 modulo the number of partitions are 1, 2, 0, 1, 2, 0, 1, 2 respectively. Then the first data shard 1 to the first data shard 8 are stored in the storage partitions corresponding to the storage identifiers 1, 2, 0, 1, 2, 0, 1, 2 respectively.

[0185] As shown in Table 5, it is a schematic table of an example of a storage partition in which data in the first data shard provided by an embodiment of the present disclosure is stored:

[0186] Table 5.

[0187]

[0188] Since the data in the same first data shard corresponds to the same data shard and the calculated hash value is the same, the value obtained by taking the modulus of the number of partitions must be the same, which ensures that the data in the same first data shard can be transmitted to the same storage partition.

[0189] The data processing method provided by the present disclosure can solve the problem of unbalanced load of different http requests caused by the large difference in data volume between different data partitions, and can also solve the problem that there are too many types of data in the same storage partition, resulting in too many batches divided, resulting in too many http requests and too much network overhead. The data processing method provided by the present disclosure achieves a relatively optimized trade-off between alleviating data skew and reducing network overhead.

[0190] In addition, in order to improve data throughput and further reduce network overhead, the present disclosure may adopt a batch method to write data into the database. Therefore, step S405 may be implemented by the following steps:

[0191] Determine the amount of data in the storage partition;

[0192] When the data volume of the storage partition reaches a preset data volume, the data in the storage partition is written into the first database.

[0193] Specifically, the data volume in the storage partition is counted in real time, and when the data volume in a storage partition reaches a preset data volume, the data in the storage partition is written into the data table corresponding to the first database. In this way, the data throughput can be effectively improved.

[0194] Since a storage partition stores data of multiple data shards, the data stored in a storage partition will be written into multiple data tables. Therefore, step S405 can be implemented by the following steps:

[0195] The data in the storage partition is divided into write batches according to the data type mapping relationship; and the data in the write batches is written into the first database through a network request.

[0196] The data contained in each write batch corresponds to the same data type; the network requests corresponding to the data in different write batches are different.

[0197] Specifically, when dividing the data in a storage partition into sub-batches, the data to be written (the data in a storage partition that meets the preset data volume) can be dynamically divided into multiple sub-batches according to the mapping relationship between the data field value and the data table storing the data, and a separate http request is sent for each sub-batch to write the data.

[0198] Furthermore, since multiple http requests may cause data congestion when writing data synchronously, which may result in a long time for writing data, the present disclosure may also use a thread pool to perform asynchronous network requests, and multiple http requests may be handed over to multiple threads for asynchronous concurrent processing, thereby increasing the overall data writing speed and reducing the time required for data writing. The thread pool is a form of multi-threaded processing, in which tasks are added to a queue during processing, and then automatically started after the thread is created.

[0199] like Figure 6 As shown, it is a schematic diagram of thread pool asynchronous processing provided by an embodiment of the present disclosure. Flink includes multiple storage partitions such as storage partition 1, storage partition 2, storage partition 3, ... storage partition n, and each storage partition can be divided into multiple sub-batches such as sub-batch 1, sub-batch 2, sub-batch 3, ... sub-batch n. After the sub-batches are divided, each sub-batch is added to the thread pool, and the data of each sub-batch is asynchronously written into the first database doris through multiple threads.

[0200] Corresponding to the data processing method provided by the first embodiment of the present disclosure, the second embodiment of the present disclosure further provides a data processing device, such as Figure 7 As shown, the data processing device 700 is applied to a stream data processing engine, in which a storage partition is deployed, and the data processing device 700 includes:

[0201] The receiving unit 701 is used to receive a data stream object; the data stream object includes data to be written into the first database;

[0202] A splitting unit 702, configured to split the data stream object into data fragments corresponding to preset data types;

[0203] A determining unit 703 is configured to determine a shard category of the data shard according to the data volume of the data shard; the shard category includes a first shard category and a second shard category, and the data volume of the data shard belonging to the first shard category is less than the data volume of the data shard belonging to the second shard category;

[0204] A partitioning unit 704 is used to store all data in a first data shard into a same storage partition, and split all data in a second data shard into different storage partitions; the first data shard is a data shard of the first shard category, and the second data shard is a data shard of the second shard category;

[0205] The writing unit 705 is used to write the data in the storage partition into the first database.

[0206] Optionally, the splitting unit 702 is used to split the data stream object into data fragments corresponding to preset data types according to a preset data type mapping relationship.

[0207] Optionally, the data type mapping relationship is stored in the second database; the splitting unit 702 is further used to: in response to the data type mapping relationship stored in the second database being updated, obtain the updated data type mapping relationship from the second database;

[0208] The splitting unit 702 is specifically used to split the data stream object into data fragments corresponding to preset data types according to the updated data type mapping relationship.

[0209] Optionally, the determination unit 703 is used to: sort the data shards according to data volume; determine a preset number of data shards with larger data volume as second data shards belonging to a second shard category, and determine the data shards among the data shards except the second data shards as first data shards belonging to a first shard category.

[0210] Optionally, the write unit 705 is used to: split the data in the storage partition into write batches according to the data type mapping relationship; the data types corresponding to the data contained in each write batch are the same; write the data in the write batch into the first database through a network request; the network requests corresponding to the data in different write batches are different.

[0211] Optionally, the writing unit 705 is used to: determine the data volume of the storage partition; when the data volume of the storage partition reaches a preset data volume, write the data in the storage partition into the first database.

[0212] Optionally, the determination unit 703 is further used to: obtain a shard identifier corresponding to the data shard; and add 1 to a value of the data volume corresponding to the shard identifier in a data volume statistics table to obtain the data volume corresponding to the data shard.

[0213] Optionally, the determination unit 703 is also used to: determine the statistical duration of the data volume statistics table; query whether the statistical duration reaches the preset period; and when the statistical duration is less than the preset period, add 1 to the value of the data volume corresponding to the shard identifier in the data volume statistics table.

[0214] Optionally, the determining unit 703 is further configured to: clear the data in the data volume statistics table when the statistical duration is greater than or equal to the preset period.

[0215] Optionally, the partition unit 704 is used to: obtain the number of partitions of the storage partition; determine the hash value of the first data shard; take the remainder of the hash value with the number of partitions, and store the data in the first data shard in the storage partition corresponding to the remainder; according to the number of partitions, evenly distribute the data in the second data shard and store them in different storage partitions.

[0216] Corresponding to the data processing method provided by the first embodiment of the present disclosure, the third embodiment of the present disclosure further provides an electronic device for data processing. Figure 8 As shown, a stream data processing engine is deployed in the electronic device 800, and the electronic device 800 includes: a processor 801; and a memory 802, which is used to store a program of a data processing method. After the device is powered on and the program of the data processing method is run by the processor, the following steps are performed:

[0217] receiving a data stream object; the data stream object comprising data to be written into a first database;

[0218] Splitting the data stream object into data slices corresponding to preset data types;

[0219] Determine a shard category of the data shard according to the data volume of the data shard; the shard category includes a first shard category and a second shard category, and the data volume of the data shard belonging to the first shard category is less than the data volume of the data shard belonging to the second shard category;

[0220] All data in the first data shard are stored in the same storage partition, and all data in the second data shard are split and stored in different storage partitions; the first data shard is a data shard of the first shard category, and the second data shard is a data shard of the second shard category;

[0221] The data in the storage partition is written into the first database.

[0222] Corresponding to the data processing method provided in the first embodiment of the present disclosure, the fourth embodiment of the present disclosure provides a computer-readable storage medium storing a program of the data processing method, the program is executed by a processor to perform the following steps:

[0223] receiving a data stream object; the data stream object comprising data to be written into a first database;

[0224] Splitting the data stream object into data slices corresponding to preset data types;

[0225] Determine a shard category of the data shard according to the data volume of the data shard; the shard category includes a first shard category and a second shard category, and the data volume of the data shard belonging to the first shard category is less than the data volume of the data shard belonging to the second shard category;

[0226] All data in the first data shard are stored in the same storage partition, and all data in the second data shard are split and stored in different storage partitions; the first data shard is a data shard of the first shard category, and the second data shard is a data shard of the second shard category;

[0227] The data in the storage partition is written into the first database.

[0228] It should be noted that for the detailed description of the apparatus, electronic device and computer-readable storage medium provided in the second, third and fourth embodiments of the present disclosure, reference may be made to the relevant description of the first embodiment of the present disclosure, and no further details will be given here.

[0229] Although the present disclosure is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present disclosure. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure shall be based on the scope defined by the claims of the present disclosure.

[0230] In a typical configuration, a node device in a blockchain includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0231] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0232] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), random access memory (RAM) of other attributes, read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage media or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0233] 2. Those skilled in the art should understand that the embodiments of the present disclosure may be provided as methods, systems or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0234] Although the present disclosure is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present disclosure. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure shall be based on the scope defined by the claims of the present disclosure.

Claims

1. A data processing method, characterized in that: The method comprises: receiving a data stream object; the data stream object comprising data to be written into a first database; Splitting the data stream object into data slices corresponding to preset data types; Determine a shard category of the data shard according to the data volume of the data shard; the shard category includes a first shard category and a second shard category, and the data volume of the data shard belonging to the first shard category is less than the data volume of the data shard belonging to the second shard category; All data in the first data shard are stored in the same storage partition, and all data in the second data shard are split and stored in different storage partitions; the first data shard is a data shard of the first shard category, and the second data shard is a data shard of the second shard category; The data in the storage partition is written into the first database.

2. The method according to claim 1, characterized in that The step of splitting the data stream object into data segments corresponding to preset data types includes: According to the preset data type mapping relationship, the data stream object is split into data slices corresponding to the preset data types.

3. The method according to claim 2, characterized in that The data type mapping relationship is stored in the second database; the method further includes: In response to the data type mapping relationship stored in the second database being updated, acquiring an updated data type mapping relationship from the second database; The step of splitting the data stream object into data slices corresponding to preset data types according to the preset data type mapping relationship includes: According to the updated data type mapping relationship, the data stream object is split into data slices corresponding to the preset data types.

4. The method according to claim 1, characterized in that: The determining the shard category of the data shard according to the data amount of the data shard comprises: Sorting the data slices according to data volume; The data shards with a preset number of larger data volumes are determined as second data shards belonging to a second shard category, and the data shards other than the second data shards among the data shards are determined as first data shards belonging to a first shard category.

5. The method according to claim 2, characterized in that: Writing the data in the storage partition into the first database includes: Splitting the data in the storage partition into write batches according to the data type mapping relationship; the data types corresponding to the data contained in each write batch are the same; The data in the write batch is written into the first database through a network request; different write batches of data correspond to different network requests.

6. The method according to claim 1, characterized in that Writing the data in the storage partition into the first database includes: Determining the data volume of the storage partition; When the data amount of the storage partition reaches a preset data amount, the data in the storage partition is written into the first database.

7. The method according to claim 1, characterized in that Before determining the shard category of the data shard according to the data amount of the data shard, the method further includes: Obtaining a shard identifier corresponding to the data shard; The value of the data volume corresponding to the shard identifier in the data volume statistics table is increased by 1 to obtain the data volume corresponding to the data shard.

8. The method according to claim 7, characterized in that The data in the data volume statistics table is updated according to a preset period, and before adding 1 to the value of the data volume corresponding to the shard identifier in the data volume statistics table, the method further includes: Determine the statistical duration of the data volume statistical table; Check whether the statistical duration reaches the preset period; The step of adding 1 to the value of the data volume corresponding to the shard identifier in the data volume statistics table includes: When the statistical duration is less than the preset period, the value of the data volume corresponding to the fragment identifier in the data volume statistical table is increased by 1.

9. The method according to claim 7, characterized in that: The method further comprises: When the statistical time length is greater than or equal to the preset period, the data in the data volume statistical table is cleared.

10. The method according to claim 1, characterized in that The storing all the data in the first data shard into the same storage partition includes: Obtain the partition quantity of the storage partition; Determining a hash value of the first data shard; Taking the remainder of the hash value over the number of partitions, and storing the data in the first data shard into the storage partition corresponding to the remainder; The step of splitting all the data in the second data shard into different storage partitions includes: According to the number of partitions, the data in the second data shard is evenly distributed and stored in different storage partitions.

11. The method according to claim 7, characterized in that The data volume statistics table is provided with a snapshot state and a recovery state; the snapshot state is used to save the data volume statistics table to the snapshot space; The recovery state is used to restore the data volume statistics table according to the data volume statistics table stored in the snapshot space when the data volume statistics table is lost.

12. A data processing device, characterized in that: The device comprises: A receiving unit, configured to receive a data stream object; the data stream object includes data to be written into the first database; A splitting unit, used to split the data stream object into data slices corresponding to preset data types; a determining unit, configured to determine a shard category of the data shard according to the data volume of the data shard; the shard category includes a first shard category and a second shard category, and the data volume of the data shard belonging to the first shard category is less than the data volume of the data shard belonging to the second shard category; A partitioning unit, configured to store all data in a first data shard into a same storage partition, and split all data in a second data shard into different storage partitions; the first data shard is a data shard of the first shard category, and the second data shard is a data shard of the second shard category; A writing unit is used to write the data in the storage partition into the first database.

13. An electronic device, characterized in that: The electronic device comprises: Processor; and The memory is used to store a data processing program. After the electronic device is powered on and the program is run by the processor, the method according to any one of claims 1 to 11 is executed.

14. A computer-readable storage medium, characterized in that: A data processing program is stored, and the program is run by a processor to execute the method according to any one of claims 1 to 11.

Images