Data writing method, device, and storage medium

Abstract

Embodiments of the present application provide a data writing method, device and storage medium, the method is applied to a database server, and the method comprises the following steps: obtaining to-be-written data and data partition parameters in a data analysis system, wherein the data analysis system runs on the database server; determining a data writing mode according to the to-be-written data and the data partition parameters; and writing the to-be-written data into a corresponding data storage area according to the data writing mode. According to the present application, the data writing mode is determined according to the to-be-written data and the data partition parameters in the data analysis system, so that the data writing can be adapted to different situations, and the data writing efficiency can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a data writing method, device and storage medium. BACKGROUND

[0002] Hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation and loading, and is a mechanism for storing, querying and analyzing large-scale data stored in Hadoop.

[0003] When performing a data analysis task based on Hive, data needs to be written into a Hive table first, and then data analysis is performed based on the written data to obtain a data analysis result. Compared with the data analysis process, the process of writing data into the Hive table is a bottleneck of performing the data analysis task, and the rate of writing data into the Hive table seriously affects the performance of the data analysis task.

[0004] Therefore, how to optimize the rate of writing data into the Hive table is a problem to be solved. SUMMARY

[0005] Embodiments of the present application provide a data writing method, device and storage medium, aiming to provide a scheme for optimizing the data writing rate.

[0006] In a first aspect, the present application provides a data writing method, the method is applied to a database server, and the method comprises the following steps:

[0007] obtaining to-be-written data and a data partition parameter of the to-be-written data in a data analysis system, wherein the data analysis system runs on the database server;

[0008] determining a data writing mode according to the to-be-written data and the data partition parameter;

[0009] writing the to-be-written data into a corresponding data storage area according to the data writing mode.

[0010] Optionally, the data writing mode is determined according to the to-be-written data and the data partition parameter, and specifically comprises the following steps:

[0011] determining a total number of data storage areas corresponding to all to-be-written data according to the to-be-written data and the data partition parameter;

[0012] determining the data writing mode according to the total number of data storage areas.

[0013] In the above technical solution, the total number of data storage areas corresponding to all to-be-written data can reflect the complexity of the data writing process, and the data writing mode is determined according to the total number, which can adapt to different complexity of data writing requirements to improve the data writing rate.

[0014] Optionally, the data writing mode is determined according to the total number of data storage areas, and specifically includes:

[0015] If the total number of data storage areas is one, the data writing mode is a static partition writing mode.

[0016] If the total number of data storage areas is multiple, the data writing mode is determined according to the total data amount of all data to be written and the requested resource occupation amount.

[0017] In the above technical solution, when the total number is one, it means that the data writing process is relatively simple, and the static partition writing mode is adopted. If the total number is multiple, the data writing mode is further determined according to the total data amount and the requested resource occupation amount, so as to improve the data writing mode.

[0018] Optionally, the data writing mode is determined according to the total data amount of all data to be written and the requested resource occupation amount, and specifically includes:

[0019] If the total data amount is less than a preset data amount threshold, the data writing mode is a dynamic partition writing mode.

[0020] If the total data amount reaches the preset data amount threshold, the data writing mode is determined according to the requested resource occupation amount.

[0021] Optionally, the data writing mode is determined according to the requested resource occupation amount, and specifically includes:

[0022] It is judged whether the ratio of the requested resource occupation amount to the total amount of resources of the database server exceeds a preset proportion threshold.

[0023] If not, the data writing mode is a mode of writing into a database first and then writing into a data analysis system.

[0024] If yes, the data writing mode is a multi-thread writing mode.

[0025] Optionally, the data analysis system is Hive, and the mode of writing into a database first and then loading into the data analysis system is a mode of writing into a distributed file system first and then loading into Hive.

[0026] In the above technical solution, when the data amount is small, the dynamic partition writing mode is adopted, when the resource is sufficient and the data amount is large, the multi-thread writing mode is adopted, and when the resource is insufficient and the data amount is large, the mode of writing into a distributed file system first and then loading into Hive is adopted. Different data writing modes are determined according to different situations, so as to improve the data writing rate.

[0027] Optionally, the total number of data storage areas corresponding to all data to be written is determined according to the data to be written and data partition parameters, and specifically includes:

[0028] According to the data partition parameter, the type quantity of the partition data in all the to-be-written data is counted.

[0029] According to the type quantity of the partition data, the total quantity of the data storage areas is determined.

[0030] Optionally, the to-be-written data is written into the corresponding data storage area according to the data writing mode, and the method specifically comprises the following steps.

[0031] According to the data writing mode, the writing code template is determined from the code template set.

[0032] According to the to-be-written data and the writing code template, the data writing code is generated.

[0033] The data writing code is run, so as to write the to-be-written data into the corresponding data storage area.

[0034] Optionally, the code template set comprises the writing code template of the static partition writing mode, the writing code template of the dynamic partition writing mode, the writing code template of the multi-thread writing mode, and the writing code template of the database-first data analysis system writing mode.

[0035] In the above technical solution, the code templates under various writing modes are pre-written, the data writing under various data writing modes can be self-adapted, and the data writing efficiency is improved.

[0036] In a second aspect, the present application provides a data writing device, and the device comprises:

[0037] The obtaining module is used for obtaining to-be-written data and data partition parameters of the to-be-written data in a data analysis system, wherein the data analysis system runs on a database server.

[0038] The processing module is used for determining a data writing mode according to the to-be-written data and the data partition parameters.

[0039] The processing module is further used for writing the to-be-written data into a corresponding data storage area according to the data writing mode.

[0040] In a third aspect, the present application provides a database server, which comprises a memory and a processor.

[0041] The memory is used for storing the processor-executable instructions.

[0042] The processor is configured to implement the data writing method according to the first aspect and the optional solutions.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium, and the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the computer-executable instructions are used to implement the data writing method according to the first aspect and the optional solutions.

[0044] In a fifth aspect, the present application provides a computer program product comprising instructions which, when executed by a processor, implement the data writing method of the first aspect and the optional solutions.

[0045] The embodiments of the present application provide a data writing method and device and a storage medium. The data writing mode is determined according to the data to be written and data partition parameters in a data analysis system, and then the data is written according to the corresponding data writing mode. The data writing mode is different when the data partition parameters and the data to be written are different. The data writing mode is adapted to data writing in different situations, and the data writing efficiency can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 A structural schematic diagram of a data writing system provided by an embodiment of the present application is shown in FIG. 1.

[0047] Figure 2 A flowchart of a data writing method provided by another embodiment of the present application is shown in FIG. 2.

[0048] Figure 3 A flowchart of determining a data writing mode provided by another embodiment of the present application is shown in FIG. 3.

[0049] Figure 4 A principle schematic diagram of a data writing method provided by another embodiment of the present application is shown in FIG. 4.

[0050] Figure 5 A structural schematic diagram of a data writing device provided by another embodiment of the present application is shown in FIG. 5.

[0051] Figure 6 A structural schematic diagram of a database server provided by another embodiment of the present application is shown in FIG. 6. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0053] Writing data into a data analysis system memory area is the first step for the data analysis system to perform data analysis tasks. Embodiments of the present application provide a data writing method, device and storage medium, aiming to provide a scheme for optimizing data writing rate. The technical concept of the present application is to determine the corresponding data writing mode according to the to-be-written data and the set data partition parameter in the data analysis system. The to-be-written data and the set data partition parameter in the data analysis system can reflect the complexity of the data writing process. The data writing mode can be adjusted according to the data writing requirements of different complexity to improve the data writing rate and the rate of the data analysis process.

[0054] As shown in Figure 1 The present application provides a data writing system, which includes a client 101 and a database server cluster. The database server cluster includes a plurality of database servers 102, which are divided into master database servers and slave database servers. The master database servers and the slave database servers are in communication connection. The client 101 and the master database servers are in communication connection. The client 101 is used to send to-be-written data to the master database servers 102. The master database servers 102 are used to execute the following data writing method to write the to-be-written data into the corresponding storage area.

[0055] As shown in Figure 2 The present application provides a data writing method, which is used in a database server. The data writing method specifically includes the following steps:

[0056] S201, obtaining to-be-written data and a data partition parameter of the to-be-written data in a data analysis system.

[0057] The database server receives a write data request input by a client and obtains the to-be-written data by analyzing the write data request.

[0058] The database server runs a data analysis system and a database system. When writing data into the database system, the data partition parameter in the data analysis system is set according to the user's partition requirement for data storage. After setting the data partition parameter, the to-be-written data is written into the storage area used by the data analysis system, and then the to-be-written data is written into the storage area used by the database system by the data analysis system, so as to realize data landing.

[0059] Taking Hive as an example, when the client sends a write data request to the database server, the partition requirement is also set in the write request, for example: writing to-be-written data every day, that is, writing to-be-written data of the same day into the same storage area.

[0060] S202, determine a data writing mode according to the to-be-written data and the data partition parameter.

[0061] The data writing mode is determined according to the number of the storage areas of the data analysis system corresponding to the to-be-written data, which is determined by analyzing all the to-be-written data according to the data partition parameter. If the number of the storage areas is relatively small, the data can be directly written into the storage areas of the data analysis system. If the total number of the storage areas is relatively large, multi-thread storage, dynamic partition storage or other storage modes are needed to be used to speed up the writing rate.

[0062] S203, write the to-be-written data into the corresponding data storage area according to the data writing mode.

[0063] The data writing mode is determined according to the number of the storage areas of the data analysis system corresponding to the to-be-written data, which is determined by analyzing all the to-be-written data according to the data partition parameter. If the number of the storage areas is relatively small, the data can be directly written into the storage areas of the data analysis system. If the total number of the storage areas is relatively large, multi-thread storage, dynamic partition storage or other storage modes are needed to be used to speed up the writing rate.

[0064] In the data writing method provided by the embodiment of the present application, the data writing mode is determined according to the to-be-written data and the data partition parameter in the data analysis system, and then the data is written according to the corresponding data writing mode. The data partition parameter and the to-be-written data are different, and the data writing mode is also different, which can improve the data writing efficiency.

[0065] The embodiment of the present application provides a data writing method. The data writing method is used in a database server, and specifically includes the following steps:

[0066] S301, obtain to-be-written data and a data partition parameter of the to-be-written data in a data analysis system.

[0067] The step has been described in detail in the above embodiment, and will not be described here.

[0068] S302, determine a data writing mode according to the to-be-written data and the data partition parameter.

[0069] The total number of the data storage areas corresponding to all the to-be-written data is determined by analyzing all the to-be-written data and the data partition parameter according to the to-be-written data and the data partition parameter. Then, the data writing mode is determined according to the total number of the data storage areas. If the number of the storage areas is relatively small, the data can be directly written into the storage areas of the data analysis system. If the total number of the storage areas is relatively large, multi-thread storage, dynamic partition storage or other storage modes are needed to be used to speed up the writing rate.

[0070] In the determination of the total number of data storage areas corresponding to all the data to be written, the type number of partition data in all the data to be written is counted according to the data partition parameter, and the total number of data storage areas is determined according to the type number of partition data.

[0071] For example, the data to be written is the consumption data of January, and the data partition parameter is to store by day. It can be determined that the date of the timestamp in the consumption data of January is the partition data, the date type number in all the consumption data is counted, that is, the type number of partition data in the consumption data is counted, and the total number of date types is obtained. The date types include January 1, January 2, …, January 31. If the consumption data of January includes 15 days of consumption data, the total number of dates is 15, that is, the type number of partition data is 15, and it can be determined that the consumption data of January corresponds to 15 data storage areas.

[0072] Preferably, as shown in Figure 3 After determining the total number of data storage areas corresponding to all the data to be written, the data writing mode is determined according to steps S1001 to S1003.

[0073] S1001, determine whether the total number of data storage areas corresponding to all the data to be written is one, if yes, go to S1002, otherwise, go to S1003.

[0074] S1002, if the total number of data storage areas is one, the data writing mode is a static partition writing mode.

[0075] If the total number of data storage areas is one, that is, the data to be written is written into the same data storage area, the storage process is relatively simple, and the static partition writing mode can be used to write the data to be written into the corresponding storage area.

[0076] S1003, if the total number of data storage areas is multiple, determine the data writing mode according to the total data amount and the request resource amount of all the data to be written.

[0077] If the total number of data storage areas is multiple, that is, the data to be written is written into multiple data storage areas, the storage process is relatively complex, and the total data amount and the request resource amount of the data to be written are required to determine the data writing mode. The request resource amount can be obtained from the data writing request sent by the client.

[0078] Preferably, continuing to refer to Figure 3 When it is determined that the total number of data storage areas corresponding to all the data to be written is multiple, the data writing mode is determined according to steps S2001 to S2004:

[0079] S2001, whether the total data amount of all data to be written is less than a preset data amount threshold, if yes, go to S2002, otherwise, go to S2003.

[0080] S2002, if the total data amount is less than the preset data amount threshold, the data writing mode is a dynamic partition writing mode.

[0081] Among them, when the data to be written needs to be written to multiple storage areas, and the total data amount of the data to be written is relatively small, the dynamic partition writing mode is selected. That is, the operating system automatically partitions and writes.

[0082] S2003, if the total data amount reaches the preset data amount threshold, continue to judge whether the ratio of the request resource amount and the total amount of the remaining resources of the database server exceeds the preset proportion threshold, if yes, go to S2004, otherwise go to S2005.

[0083] Among them, the preset proportion threshold can be 1, that is, whether the request resource amount exceeds the total amount of the remaining resources of the database server is judged.

[0084] S2004, the data writing mode is a mode of writing to the database first and then writing to the data analysis system.

[0085] When the data to be written needs to be written to multiple storage areas, and the total data amount of the data to be written is relatively large, and the remaining resources of the database server are insufficient, the mode of writing to the database first and then writing to the data analysis system is adopted, which can improve the data writing efficiency.

[0086] S2005, the data writing mode is a multi-thread writing mode.

[0087] Among them, when the data to be written needs to be written to multiple storage areas, and the total data amount of the data to be written is relatively large, and the remaining resources of the database server are sufficient, more resources of the database server can be occupied to improve the efficiency of storing large data amount.

[0088] S303, write the data to be written to the corresponding data storage area according to the data writing mode.

[0089] Among them, the data to be written can be written to the data storage area in an adaptive manner of different data writing modes. Specifically, steps S3001 to S3003 are adopted.

[0090] S3001, determine a writing code template from a code template set according to the data writing mode.

[0091] The write code template corresponding to each data write mode is pre-written, and the code template set includes a plurality of write code templates. After determining the data write mode according to the to-be-written data and the data partition parameter, the write code template is determined from the code template set according to the data write mode.

[0092] S3002, generating data write code according to the to-be-written data and the write code template.

[0093] After determining the write code template, the to-be-written data is filled into the write code template to generate the data write code.

[0094] S3003, running the data write code to write the to-be-written data into the corresponding data storage area.

[0095] After obtaining the data write code, the to-be-written data can be written into the storage area used by the data analysis system.

[0096] In the data write method provided in the embodiments of the present application, the storage area corresponding to the to-be-written data is determined according to the to-be-written data and the data partition parameter in the data analysis system, and the data write mode is determined according to the type number of the storage area. If the type number is multiple, the write mode is determined according to the total data amount of the to-be-written data and the applied resource occupation. Different data write modes are used for different situations, which can improve the efficiency of data write.

[0097] The data write method provided in the embodiments of the present application will be described below in combination with Hive. First, the data write modes supported by Hive are described, and then the specific steps of the data write method provided in the embodiments of the present application are described.

[0098] The data write modes of Hive include static partition data write mode, dynamic partition data write mode, multi-thread data write mode, and the mode of first writing to HDFS and then loading to Hive.

[0099] The static partition data write mode is applied to the case where the storage area corresponding to the data is single. Specifically, the partition write data is specified, such as writing the consumption data of a certain day, writing the order data of a certain day, etc.

[0100] The dynamic partition data writing mode is suitable for the case that the data corresponds to multiple storage areas, but has the problems of slow data writing rate and easy generation of a large number of small files. If the number of the storage areas corresponding to the data is large, such as the case of re-flashing one-year data with a day as the partition parameter, each partition needs to perform a scan, delete and write file operation, and the data writing rate is slow. If the number of the parallel working instances when writing data is M and the number of partitions is N, MxN small files are generated, and too many small files will seriously affect the performance of the NameNode.

[0101] The multi-thread data writing mode is suitable for the case that the number of partitions is large but the data volume of each partition is small. Each thread can be used to execute a small number of working instances to avoid generating too many small files while ensuring concurrency. The disadvantage of the multi-thread data writing mode is that the computing engine needs to support multi-thread programming, and some computing engines, such as MapReduce, do not support controllable multi-thread programming, and the number of working instances of each thread cannot be accurately controlled. Moreover, the multi-thread data writing mode consumes a large amount of resources, and if the resources are insufficient, the task execution speed will be affected.

[0102] The first-write-then-load-to-Hive mode has very fast speed, but this mode does not support the Orc storage format, which is a widely used Hive storage format. Moreover, this mode needs manual partitioning, and the interface for writing to HDFS only supports specifying a single HDFS directory. If the data needs to be partitioned, the partitioning code needs to be manually maintained.

[0103] The following will be described in combination with Figure 4 The data writing method provided by another embodiment of the present application specifically includes the following steps:

[0104] S401, obtaining to-be-written data and a data partition parameter of the to-be-written data in a data analysis system.

[0105] The client extracts the to-be-written data from the received data analysis task and analyzes the to-be-written data and the data partition parameter. The analysis is mainly performed in four aspects:

[0106] First, the total data volume of the to-be-written data is analyzed, the total number M of the to-be-written data is counted, N pieces of data are sampled from the to-be-written data, and the total data volume T of the N pieces of data is calculated. The total data volume of all the to-be-written data can be estimated according to the following formula.

[0107]

[0108] Second, the requested resource occupation amount is obtained. According to the database server cluster interface called in the data analysis task, the requested resource occupation amount, such as the memory size and the number of threads, can be obtained.

[0109] Thirdly, obtain the data partition parameter, and obtain the file storage format and the partition field through the script in the Hive client by using the command to obtain the table structure to be written into the Hive.

[0110] Fourthly, after obtaining the partition field of the Hive table, analyze how many enumeration combinations of all the partition fields in the data to be written are, so as to obtain the type quantity of the storage area corresponding to all the data to be written.

[0111] For example, if the Hive table has three partition fields, namely “year”, “month” and “day”, and if the timestamps of all the data to be written are January 1, 2021, then the partition field has only one enumeration combination, that is, all the data to be written is written into the same storage area. If the timestamps of all the data to be written are January 1, 2021 and January 2, 2021, then the partition field has two enumeration combinations, that is, the number of storage areas where the data to be written is written is two.

[0112] S402, determining the data writing mode according to the data to be written and the data partition parameter.

[0113] In S402, after the analysis of the data to be written and the data partition parameter is completed, the data writing mode is determined according to the analysis result.

[0114] If the partition field has only one enumeration combination, that is, all the data to be written is written into the same storage area, the static partition writing Hive table mode is adopted.

[0115] If the partition field has multiple enumeration combinations, and the total data quantity is less than the preset data quantity threshold, the dynamic partition writing Hive table mode is adopted.

[0116] If the partition field has multiple enumeration combinations, and the total data quantity reaches the preset data quantity threshold, and the resource quantity to be occupied is less than the remaining resource quantity of the database server, that is, the partitions are more, the data quantity is larger, and the resource is more sufficient, the multi-thread writing Hive table mode is adopted.

[0117] If the partition field has multiple enumeration combinations, and the total data quantity reaches the preset data quantity threshold, and the resource quantity to be occupied exceeds the remaining resource quantity of the database server, that is, the partitions are more, the data quantity is larger, and the resource is insufficient, the mode of writing into the HDFS first and then loading into the Hive table is adopted.

[0118] S403, writing the data to be written into the corresponding data storage area according to the data writing mode.

[0119] The four data writing mode writing code templates can be pre-written to form a code template set. The code template set includes a static partition writing mode writing code template, a dynamic partition writing mode writing code template, a multi-thread writing mode writing code template, and a writing code template of a mode of writing data into a database first and then writing data into a data analysis system.

[0120] A lib library function is written, and the input parameters are to-be-written data and an identifier of a data writing mode. The lib library function code template set is used as a basis. The data writing mode determined in S402 and the to-be-written data are input into the lib library function to generate data writing code. The data writing code is run to write the to-be-written data into a corresponding data storage area.

[0121] In the data writing method provided in the embodiments of the present application, the parameters related to the determined Hive writing mode are analyzed according to the to-be-written data and the Hive table structure, the Hive writing mode is determined according to the related parameters, the corresponding code is generated by matching the analyzed Hive writing mode according to the pre-written code template, and the data is written in the determined Hive table mode by running the code, thereby improving the efficiency of data writing and the efficiency of data analysis tasks.

[0122] As shown in Figure 5 Another embodiment of the present application provides a data writing apparatus 500, which specifically includes:

[0123] The acquisition module 501 is configured to acquire to-be-written data and a data partition parameter of the to-be-written data in a data analysis system, where the data analysis system runs on a database server.

[0124] The processing module 502 is configured to determine a data writing mode according to the to-be-written data and the data partition parameter.

[0125] The processing module 502 is further configured to write the to-be-written data into a corresponding data storage area according to the data writing mode.

[0126] Optionally, the processing module 502 is specifically configured to:

[0127] determine a total number of data storage areas corresponding to all to-be-written data according to the to-be-written data and the data partition parameter;

[0128] determine the data writing mode according to the total number of data storage areas.

[0129] Optionally, the processing module 502 is specifically configured to:

[0130] if the total number of data storage areas is one, the data writing mode is a static partition writing mode;

[0131] If the total number of data storage areas is multiple, the data writing mode is determined according to the total data amount of all data to be written and the requested resource amount.

[0132] Optionally, the processing module 502 is specifically configured to:

[0133] If the total data amount is less than the preset data amount threshold, the data writing mode is a dynamic partition writing mode.

[0134] If the total data amount reaches the preset data amount threshold, the data writing mode is determined according to the requested resource amount.

[0135] Optionally, the processing module 502 is specifically configured to:

[0136] determine whether the ratio of the requested resource amount to the total amount of resources of the database server exceeds a preset proportion threshold;

[0137] If not, the data writing mode is a mode of writing data into the database first and then writing data into the data analysis system.

[0138] If yes, the data writing mode is a multi-thread writing mode.

[0139] Optionally, the data analysis system is Hive, and the mode of writing data into the database first and then loading into the data analysis system is a mode of writing data into a distributed file system first and then loading into Hive.

[0140] Optionally, the processing module 502 is specifically configured to:

[0141] count the type number of partition data in all data to be written according to the data partition parameter;

[0142] determine the total number of data storage areas according to the type number of partition data.

[0143] Optionally, the processing module 502 is specifically configured to:

[0144] determine a writing code template from a code template set according to the data writing mode;

[0145] generate data writing code according to the data to be written and the writing code template;

[0146] run the data writing code to write the data to be written into a corresponding data storage area.

[0147] Optionally, the code template set includes a writing code template of a static partition writing mode, a writing code template of a dynamic partition writing mode, a writing code template of a multi-thread writing mode, and a writing code template of a mode of writing data into a database first and then writing data into a data analysis system.

[0148] As Figure 6As shown, another embodiment of the present application provides a test server 600, which comprises a transmitter 601, a receiver 602, a memory 603, and a processor 604.

[0149] The transmitter 601 is configured to transmit instructions and data, the receiver 602 is configured to receive instructions and data, the memory 603 is configured to store computer-executable instructions, and the processor 604 is configured to execute the computer-executable instructions stored in the memory to implement each step of the data writing method performed by the above-mentioned embodiments. For details, please refer to the related description in the foregoing data writing method embodiments.

[0150] Optionally, the memory 603 can be independent or integrated with the processor 604. When the memory 603 is independently arranged, the processing device further comprises a bus for connecting the memory 603 and the processor 604.

[0151] The embodiments of the present application further provide a computer-readable storage medium, which stores computer-executable instructions. When the processor executes the computer-executable instructions, the data writing method performed by the above-mentioned processing device is implemented.

[0152] The embodiments of the present application further provide a computer program product, which comprises instructions. When the processor executes the instructions, the data writing method performed by the above-mentioned processing device is implemented.

[0153] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data write method, characterized by, The method is applied to a database server, and the method comprises: acquiring to-be-written data and a data partition parameter of the to-be-written data in a data analysis system, wherein the data analysis system runs on the database server; the data partition parameter is set according to a partition requirement of a user for data storage; determining a data writing mode according to the to-be-written data and the data partition parameter; counting a type quantity of partition data in all to-be-written data according to the data partition parameter; determining a total quantity of data storage areas according to the type quantity of the partition data; and determining the data writing mode according to the total quantity of the data storage areas; the determination of the data writing mode according to the total quantity of the data storage areas specifically comprises: if the total quantity of the data storage areas is a plurality, determining the data writing mode according to a total data quantity of all to-be-written data and a requested resource occupation quantity; if the total data quantity reaches a preset data quantity threshold, determining whether a ratio of the requested resource occupation quantity to a total quantity of idle resources of the database server exceeds a preset proportion threshold; if yes, the data writing mode is a mode of writing into a database first and then into the data analysis system; if no, the data writing mode is a multi-thread writing mode.

2. The method of claim 1, wherein, the determination of the data writing mode according to the total quantity of the data storage areas specifically comprises: if the total quantity of the data storage areas is one, the data writing mode is a static partition writing mode.

3. The method of claim 2, wherein, the determination of the data writing mode according to the total data quantity of all to-be-written data and the requested resource occupation quantity specifically comprises: if the total data quantity is less than the preset data quantity threshold, the data writing mode is a dynamic partition writing mode.

4. The method according to claim 1 or 2, characterized in that, the data analysis system is Hive, and the mode of writing into the database first and then into the data analysis system is a mode of writing into a distributed file system first and then into Hive.

5. The method according to any one of claims 1 to 3, characterized in that, the writing of the to-be-written data into corresponding data storage areas according to the data writing mode specifically comprises: determining a writing code template from a code template set according to the data writing mode; generating data writing code according to the to-be-written data and the writing code template; running the data writing code to write the to-be-written data into corresponding data storage areas.

6. The method of claim 5, wherein, the code template set comprises a writing code template of a static partition writing mode, a writing code template of a dynamic partition writing mode, a writing code template of a multi-thread writing mode, and a writing code template of a mode of writing into a database first and then into a data analysis system.

7. A data writing apparatus, characterized by comprising: the apparatus comprises: an acquisition module, configured to acquire to-be-written data and a data partition parameter in a data analysis system, wherein the data analysis system runs on a database server; the data partition parameter is set according to a partition requirement of a user for data storage; a processing module, configured to determine a data writing mode according to the to-be-written data and the data partition parameter; the processing module is further configured to write the to-be-written data into corresponding data storage areas according to the data writing mode. The processing module is specifically configured to: count a type quantity of partition data in all to-be-written data according to the data partition parameter; determine a total quantity of data storage areas according to the type quantity of the partition data; determine the data writing mode according to the total quantity of the data storage areas; if the total quantity of the data storage areas is a plurality, determine the data writing mode according to a total data quantity of all to-be-written data and a requested resource occupation quantity; if the total data quantity reaches a preset data quantity threshold, determine whether a ratio of the requested resource occupation quantity to a total quantity of idle resources of the database server exceeds a preset proportion threshold; if yes, the data writing mode is a mode of writing into a database first and then writing into a data analysis system; and if no, the data writing mode is a multi-thread writing mode.

8. A database server, characterized by Comprise: a memory, a processor; a memory; a memory for storing the processor-executable instructions; The processor is configured to implement the data writing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by the processor to implement the data writing method according to any one of claims 1 to 6.

10. A computer program product comprising instructions, characterized in that, The instructions are executed by the processor to implement the data writing method according to any one of claims 1 to 6.

Images