Data processing method and device, storage medium and program product

Abstract

The invention discloses a data processing method and device, a storage medium and a program product, and relates to the technical field of data processing.The data processing method comprises the steps that multiple fragment files of a to-be-downloaded file corresponding to all actuators are downloaded concurrently through at least two actuators; when one workflow engine in the at least two workflow engines monitors that downloading of any fragment file is completed, the fragment file is converted into a conversion file in a preset format through the workflow engine, and the conversion file is loaded to a target database; and combining all the conversion files in the target database to synchronize the to-be-downloaded file to the target database. The data processing efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, storage medium and program product. Background Art

[0002] ETL (Extract, Transform, Load) is an important part of building a data warehouse. For example, when synchronizing data to a database such as Hive, the data from the data source is downloaded, extracted and converted through SFTP (Secure File Transfer Protocol), and finally the converted data is loaded into a database such as Hive.

[0003] In the related technology, when synchronizing data to a database such as Hive, Jsch is used to establish a link and download files to HDFS (Hadoop Distributed File System), and then the files are converted into Orc (Optimized Row Columnar) files and loaded into Hive. However, this solution performs serial blocking processing on each node, resulting in low data processing efficiency. Summary of the invention

[0004] The main purpose of this application is to provide a data processing method, device, storage medium and program product, aiming to solve the technical problem of low data processing efficiency.

[0005] To achieve the above objectives, the present application proposes a data processing method, comprising:

[0006] Concurrently downloading multiple fragment files of the file to be downloaded corresponding to each executor through at least two executors;

[0007] When one of the at least two workflow engines detects that the download of any segment file is completed, the workflow engine converts any segment file into a conversion file of a preset format, and loads the conversion file into a target database;

[0008] Merge all converted files in the target database to synchronize the files to be downloaded to the target database.

[0009] In one embodiment, the step of concurrently downloading multiple segment files of a file to be downloaded corresponding to each executor through at least two executors includes:

[0010] For each file to be downloaded, obtain the number of first file records in the file to be downloaded and the preset number of fragments of the fragmented file;

[0011] Determine the number of second file records in each shard file based on the number of first file records and the preset number of shards;

[0012] Based on the second file record number, at least two executors are controlled to concurrently download the number of file records in the to-be-downloaded file corresponding to each executor to obtain a plurality of segmented files.

[0013] In one embodiment, based on the first file record number and the preset number of shards, the step of determining the second file record number in each shard file includes:

[0014] If the first file record number is divisible by the preset number of fragments, the result value after the first file record number is divided by the preset number of fragments is used as the second file record number;

[0015] If the number of first file records cannot be divided by the preset number of shards, the number of second file records is determined using formula 1;

[0016] Among them, formula 1 is:

[0017]

[0018] splitCount represents the number of second file records, fileCount represents the number of first file records, N represents the preset number of fragments, and splitSeq represents the sequence number of the fragment file.

[0019] In one embodiment, when one of the at least two workflow engines detects that the download of any segment file is complete, the step of converting any segment file into a conversion file of a preset format by the workflow engine includes:

[0020] Obtain the first target file to be downloaded, the allocated memory of the executor, the spark memory ratio, the reserved memory of the executor, the number of cores of a single executor, and the user memory; the first target file to be downloaded is the file to be downloaded with the largest file size among the multiple files to be downloaded;

[0021] Use the allocated memory, spark memory ratio, reserved memory, the number of cores of a single executor, and formula 2 to determine the available memory per core;

[0022] Among them, formula 2 includes:

[0023]

[0024] usableMemory indicates the available memory of a single core, executorMemory indicates the allocated memory, memoryFraction indicates the spark memory ratio, and numCores indicates the number of cores of a single executor;

[0025] The ratio of the first target file to be downloaded to the preset number of fragments is used as the size of a single fragment file;

[0026] The product of the size of a single shard file and the number of concurrent connections is used as the maximum file size that can be processed in a single batch.

[0027] The ratio of the maximum file size to the available memory of a single core is used as the optimal number of partitions;

[0028] Determine a first product of the total number of executors and the number of cores of a single executor, and a second product of the optimal number of partitions after expanding by a preset multiple;

[0029] The larger value of the first product and the second product is used as the actual number of partitions;

[0030] When one of the at least two workflow engines detects that the download of any shard file is complete, the shard file is segmented using the actual number of partitions to obtain a plurality of shard sub-files; wherein the number of the plurality of shard sub-files is the same as the actual number of partitions;

[0031] All the sliced ​​sub-files are converted into conversion files in a preset format through at least one workflow engine.

[0032] In one embodiment, the workflow engine includes a first workflow engine and a second workflow engine, the slice sub-files have file numbers, and the step of converting all the slice sub-files into conversion files of a preset format by at least one workflow engine includes:

[0033] Using the first workflow engine, converting the odd-numbered slice sub-files in all the slice sub-files into conversion files in a preset format;

[0034] The second workflow engine is used to convert the slice sub-files with even-numbered file numbers in all the slice sub-files into conversion files in a preset format.

[0035] In one embodiment, the step of merging all converted files in the target database to synchronize the files to be downloaded to the target database includes:

[0036] For any converted file converted by the workflow engine, determining whether the accumulated number of converted files is equal to a preset threshold;

[0037] If the cumulative number of converted files is equal to a preset threshold, the converted files of the preset threshold number are merged to obtain at least one merged file, and after the merged file is obtained, the cumulative number is cleared to zero;

[0038] For any workflow engine, if it is detected that all the fragmented sub-files are converted into transformed files, all the merged files and all the transformed files are merged to synchronize the files to be downloaded to the target database.

[0039] In one embodiment, after the multiple segment files of the file to be downloaded corresponding to each executor are downloaded concurrently by at least two executors, the method further includes:

[0040] Obtain a configuration file; the configuration file includes the file name of each file to be downloaded, the file size of each file to be downloaded, and the last modification time of each file to be downloaded;

[0041] Determine whether there is a target download log; wherein the target download log is a download log generated after all the files to be downloaded are downloaded, and the target download log includes the file name of the file to be downloaded, the file size of each file to be downloaded, and the last modification time of each file to be downloaded;

[0042] If the target download log exists, the target download log is verified using the configuration file to obtain a verification result; wherein the verification result is a verification pass or a verification fail;

[0043] If the verification result is that the verification is passed, it is determined that all files to be downloaded have been downloaded;

[0044] If the verification result is that the verification fails, the inconsistent second target file to be downloaded is determined based on the download log and the configuration file, and the second target file to be downloaded is downloaded.

[0045] In addition, to achieve the above objectives, the present application also proposes a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method described above.

[0046] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0047] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0048] One or more technical solutions proposed in this application have at least the following technical effects:

[0049] Compared with the related art, after downloading the file to be downloaded to HDFS, the file to be downloaded is converted into an Orc file, and the Orc file is loaded into Hive. In this application, multiple shard files of the file to be downloaded corresponding to each executor are downloaded concurrently by at least two executors, that is, the download task of downloading the file to be downloaded is distributed to each executor, rather than the download task is concentrated on one executor, and multiple shard files of the file to be downloaded can be downloaded at the same time, thereby improving the download efficiency and concurrent processing capability. At the same time, when any workflow engine among multiple workflow engines detects that the download of any shard file is completed, the workflow engine converts any shard file into a conversion file of a preset format, that is, multiple workflow engines perform conversion operations on multiple downloaded shard files and load the conversion files into the target database. The loading efficiency of the shard files can be improved by concurrently loading the downloaded file shards by multiple workflow engines. That is, this application can avoid serial blocking processing in the download stage or the loading stage by concurrent downloading in the download stage and concurrent loading in the loading stage, thereby improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 A schematic diagram of a process for synchronizing data to a HIVE database in the related technology of this application;

[0053] Figure 2 A flowchart of the first embodiment of the data processing method of the present application is provided;

[0054] Figure 3 A schematic diagram of the download verification process after downloading the segmented files of the file to be downloaded for this application;

[0055] Figure 4 This is a schematic diagram of the in-heap memory model of the executor of this application;

[0056] Figure 5 This is a schematic diagram of the relationship between various memories in the heap memory model of the executor of this application;

[0057] Figure 6This is a schematic diagram of the core operation timeline of the executor of this application;

[0058] Figure 7 A schematic diagram of a loading timeline when a single workflow engine of this application converts a sliced ​​sub-file into a conversion file of a preset format;

[0059] Figure 8 A schematic diagram of a loading timeline when the present application uses two workflow engines to convert a segmented sub-file into a conversion file of a preset format;

[0060] Fig. 9 This is a schematic diagram of the process of merging multiple shard sub-files for this application;

[0061] Fig.10 A flow chart of loading verification for this application;

[0062] Fig.11 This is the process diagram of SparkSQL concurrent ETL for this application;

[0063] Fig.12 A schematic diagram of the process of concurrently downloading the fragmented files of the file to be downloaded for this application;

[0064] Fig.13 A flowchart showing the concurrent recording of shard sub-files for this application;

[0065] Fig.14 Schematic diagram of the device structure of the hardware operating environment involved in the data processing method in the embodiment of the present application.

[0066] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0067] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0068] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0069] The main solution of the embodiment of the present application is: use at least two executors to concurrently download multiple segment files of the file to be downloaded corresponding to each executor; when a workflow engine among at least two workflow engines detects that the download of any segment file is completed, use a workflow engine to convert any segment file into a conversion file of a preset format, and load the conversion file into a target database; merge all conversion files in the target database to synchronize the file to be downloaded to the target database.

[0070] In this embodiment, for the convenience of description, the following description is made with the data processing device as the execution subject.

[0071] ETL (Extract, Transform, Load) is an important part of building a data warehouse. For example, when synchronizing data to a database such as Hive, the data from the data source is downloaded, extracted and converted through SFTP (Secure File Transfer Protocol), and finally the converted data is loaded into a database such as Hive.

[0072] In the related technology, when synchronizing data to a database such as Hive, Jsch is used to establish a link and download the file to HDFS (Hadoop Distributed File System), and then the file is converted into an Orc (Optimized Row Columnar) file and loaded into Hive. Figure 1 shown.

[0073] However, there are the following defects in loading files using the above solution:

[0074] 1. Single-thread download takes a long time: The download adopts a single-thread mode, which is time-consuming and unstable. This problem is more prominent when downloading large files. For example, it took up to 4 hours to download a certain type of large-volume business data (about 600 million data), which seriously affected the output of downstream systems.

[0075] 2. High cost of job retry: Downloading large files is highly dependent on cluster stability. If the job fails due to network jitter, IO exceptions, etc. during the download process, the downloaded files will be cleared. Rerunning the job can only download them again, which increases the time cost of file downloads. Multiple failures will cause data delays in downstream systems.

[0076] 3. Too many small files in HDFS: Affected by the parallelism of SparkSQL and the block size of HDFS, the number of small files after large files are converted can reach thousands. Massive small files will occupy a large amount of NameNode mapping space, increase disk seek time, and make downstream jobs take longer to read data;

[0077] 4. Overall management of SFTP split files: The exchange platform provides a file splitting function, but how to implement concurrent downloads of split files, perform integrity checks on downloaded files, and ensure idempotence of results throughout the entire process have become issues that need to be urgently addressed in overall file management.

[0078] Therefore, the above solution performs serial blocking processing at each node, resulting in low data processing efficiency.

[0079] To this end, the present application provides a solution. Compared with the related art, after downloading the file to be downloaded to HDFS, the file to be downloaded is converted into an Orc file, and the Orc file is loaded into Hive. The present application uses at least two executors to concurrently download multiple shard files of the file to be downloaded corresponding to each executor, that is, the download task of downloading the file to be downloaded is distributed to each executor, rather than concentrating the download task on one executor, and multiple shard files of the file to be downloaded can be downloaded at the same time, thereby improving the download efficiency and concurrent processing capability. At the same time, when any workflow engine among multiple workflow engines detects that the download of any shard file is completed, the workflow engine converts any shard file into a conversion file of a preset format, that is, multiple workflow engines perform conversion operations on multiple downloaded shard files and load the conversion files into the target database. The loading efficiency of the shard files can be improved by concurrently loading the downloaded file shards by multiple workflow engines. That is, the present application can avoid serial blocking processing in the download stage or the loading stage by concurrent downloading in the download stage and concurrent loading in the loading stage, thereby improving the data processing efficiency.

[0080] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a data processing device, etc. The following takes a data processing device as an example to illustrate this embodiment and the following embodiments.

[0081] Based on this, the embodiment of the present application provides a data processing method, referring to 2, Figure 2 This is a flowchart of the first embodiment of the data processing method of the present application.

[0082] In this embodiment, the data processing method includes steps S10 to S40:

[0083] Step S10: concurrently downloading multiple segment files of the file to be downloaded corresponding to each executor through at least two executors.

[0084] In this embodiment, the SFTP3.0 server stores a data file, and the SFTP3.0 server can segment the data file to obtain multiple data sub-files, which are the files to be downloaded. The segment files can be the sub-files to be downloaded after the files to be downloaded are segmented.

[0085] Specifically, the data processing device can create and persist a lightweight DataSet (dataset) with a partition number equal to the concurrency number, and use broadcast variables to evenly distribute the download tasks of multiple files to be downloaded to each executor. Each executor can independently perform SFTP login and file segmentation operations for downloading files to be downloaded.

[0086] It is understandable that when downloading the fragmented files of the file to be downloaded, in order to ensure that the fragmented files are as uniform as possible and balance the download resources, further as an optional implementation, step S10 specifically includes:

[0087] Step S101: for each file to be downloaded, the number of first file records in the file to be downloaded and the preset number of fragments of the fragmented file are obtained.

[0088] Step S102: determining the number of second file records in each fragment file based on the number of first file records and the preset number of fragments.

[0089] It is understandable that in order to avoid processing delay or resource waste caused by too many second file records in the shard file, it is necessary to ensure that the number of second file records in each shard file is as uniform as possible. Therefore, in order to ensure that the number of second file records in the shard file is as uniform as possible and balance the download resources, further, as an optional implementation, step S102 specifically includes:

[0090] Step S1021: If the first file record number is divisible by the preset number of fragments, the result value after the first file record number is divided by the preset number of fragments is used as the second file record number.

[0091] Step S1022: If the first file record number cannot be divided by the preset number of shards, the second file record number is determined using formula 1.

[0092] Among them, formula 1 is:

[0093]

[0094] splitCount represents the number of second file records, fileCount represents the number of first file records, N represents the preset number of fragments, and splitSeq represents the sequence number of the fragment file.

[0095] In this embodiment, the first file record number may be the total number of independent data entries in the file to be downloaded.

[0096] Specifically, after obtaining the first file record number in each file to be downloaded, determine whether the first file record number is divisible by the preset number of fragments. When the first file record number is divisible by the preset number of fragments, the result value after the first file record number is divided by the preset number of fragments is used as the second file record number, that is, the second file record number in all fragment files is the same, and its value is: N>0, splitCount represents the number of second file records, fileCount represents the number of first file records, and N represents the preset number of fragments. When the number of first file records cannot be divided by the preset number of fragments, in order to ensure that the number of second file records in the fragment files is as even as possible and to balance the download resources, the remainder padding method can be used to keep the number of second file records in the first N-1 fragment files consistent, and the number of second file records in the Nth fragment file is the remaining number of file records, that is, the number of second file records in the first N-1 fragment files and the number of second file records in the Nth fragment file are calculated using Formula 1.

[0097] In this embodiment, when the first file record number is divisible by the preset number of shards, the result value after the first file record number is divided by the preset number of shards is used as the second file record number. When the first file record number is not divisible by the preset number of shards, the remainder is filled in to make the second file record numbers in the first N-1 shard files consistent, and the second file record number in the Nth shard file is the remaining file record number, thereby ensuring that the second file record numbers in the shard files are as uniform as possible and achieving the purpose of balancing download resources.

[0098] Step S103: Based on the second file record number, control at least two executors to concurrently download the file records in the to-be-downloaded file corresponding to each executor to obtain a plurality of segmented files.

[0099] Specifically, after the SFTP3.0 server splits the data file into multiple files to be downloaded, it can generate a configuration file corresponding to the multiple files to be downloaded. The configuration file may include the file name of each file to be downloaded, the number of first file records in each file to be downloaded, the file size of each file to be downloaded, the last modification time of each file to be downloaded, and the total number of files to be downloaded. The data processing device can establish a connection with the SFTP3.0 server through Jsch and download the configuration file from the SFTP3.0 server. After obtaining the configuration file, the number of first file records in each file to be downloaded can be determined from the configuration file, and the number of second file records in each fragment file can be calculated based on the number of first file records and the preset number of fragments of the fragment file. After the number of second file records in each fragment file is calculated, when each executor independently performs the SFTP login and file fragmentation operation of downloading the file to be downloaded, the executor can extract the records of the specified range from the file to be downloaded based on the second file record number, and save it as a separate file, that is, as a fragment file.

[0100] In this embodiment, when the first file record number is divisible by the preset number of shards, the result value after the first file record number is divided by the preset number of shards is used as the second file record number. When the first file record number is not divisible by the preset number of shards, the remainder is filled in to make the second file record numbers in the first N-1 shard files consistent, and the second file record number in the Nth shard file is the remaining file record number, thereby ensuring that the second file record numbers in the shard files are as uniform as possible and achieving the purpose of balancing download resources.

[0101] It is understandable that, in the process of downloading the files to be downloaded, when the number of files to be downloaded is too large or the data volume of the files to be downloaded is too large, it will take a long time to download. In order to reduce the error reporting cost of the operation, further, as an optional implementation, after step S10, it also includes:

[0102] Step S11, obtaining a configuration file; the configuration file includes the file name of each file to be downloaded, the file size of each file to be downloaded, and the last modification time of each file to be downloaded.

[0103] Step S12, determining whether there is a target download log.

[0104] The target download log is a download log generated after all files to be downloaded are downloaded. The target download log includes the file names of the files to be downloaded, the file sizes of the files to be downloaded, and the last modification time of the files to be downloaded.

[0105] Step S13: If the target download log exists, the target download log is verified using the configuration file to obtain a verification result.

[0106] The verification result is verification passed or verification failed.

[0107] Step S14: If the verification result is that the verification is passed, it is determined that all files to be downloaded have been downloaded.

[0108] Step S15: If the verification result is that the verification fails, the inconsistent second target file to be downloaded is determined based on the download log and the configuration file, and the second target file to be downloaded is downloaded.

[0109] Specifically, while the data processing device downloads the segment files of the file to be downloaded through at least two executors, it can also establish a connection with the SFTP3.0 server through Jsch and download the configuration file from the SFTP3.0 server.

[0110] It can be understood that the files to be downloaded are data sub-files obtained after the SFTP3.0 server splits the data files. For all the files to be downloaded, after all the files to be downloaded are downloaded, a target download log corresponding to the downloaded data files will be generated. Figure 3 As shown, Figure 3 The download check flow chart after downloading the fragmented files of the files to be downloaded. After completing the download of all the files to be downloaded, determine whether there is a target download log. When there is a target download log, check whether the file name in the configuration file, the file size of each file to be downloaded, and the last modification time are consistent with the records in the target download log. If all items are consistent, there is no need to re-download the files to be downloaded, determine that all the files to be downloaded have been downloaded, and enter the file loading stage. When any of the file name in the verification configuration file, the file size of each file to be downloaded, and the last modification time is inconsistent with the record in the target download log, jump to finishTagDtl to verify the download log, and re-download the inconsistent files, that is, determine the inconsistent second target to be downloaded files based on the download log and the configuration file, and download the second target to be downloaded files. When downloading the second target to be downloaded file, first check whether the various records of the files downloaded in the finishTagDtl directory are consistent with the SFTP3.0 server, that is, determine whether the downloaded files to be downloaded have changed. If they are consistent, only the files to be downloaded that have not been downloaded are downloaded. If they are inconsistent, re-download the inconsistent files to be downloaded and the files to be downloaded that have not been downloaded.

[0111] In this embodiment, when the file to be downloaded is successfully downloaded to HDFS, the download log of the file will be recorded in the finishTagDtl directory. The download log includes the file name, data volume, file size, file encoding and the last file modification time on the SFTP3.0 server. When all files are downloaded, the total download log will be recorded in the finishTag directory. The download log will record the total data volume, total file size and the sum of file modification time of all files. By using the configuration file to verify the target download log, the cost of job error reporting can be reduced, and the downloaded files are retained when the job reports an error, and the file data is verified when the job is restarted. If the verification passes, the downloaded files can be used directly.

[0112] Step S20, when one of the at least two workflow engines detects that the download of any segment file is complete, the workflow engine converts any segment file into a conversion file of a preset format, and loads the conversion file into the target database.

[0113] In this embodiment, the conversion file in the preset format may be an orc file. The target database may be a HIVE database.

[0114] It should be noted that for spark batch jobs, it is very important to set an appropriate degree of parallelism to make full use of cluster resources. If the degree of parallelism is too high and the number of tasks is too small, a large amount of task scheduling overhead will be generated; if the degree of parallelism is too low and the number of tasks is too small, the parallel computing capability of the cluster cannot be brought into play, resulting in slow task execution and even the risk of memory overflow. Since the concurrent download scheme is based on the Spark heap memory model, on the basis of controlling the number of files loaded by a single batch stream, it is necessary to calculate the number of partitions of a single batch of shard files, that is, the actual number of partitions when performing a split operation on any shard file. Therefore, further, as an optional implementation, step S20 specifically includes:

[0115] Step S201, obtain the first target file to be downloaded, the allocated memory of the executor, the spark memory ratio, the reserved memory of the executor, the number of cores of a single executor, and the user memory.

[0116] The first target file to be downloaded is the file to be downloaded with the largest file size among the multiple files to be downloaded.

[0117] Step S202, using the allocated memory, spark memory ratio, reserved memory, the number of cores of a single executor and formula 2, determine the available memory of a single core.

[0118] Among them, formula 2 includes:

[0119]

[0120] usableMemory indicates the available memory of a single core, executorMemory indicates the allocated memory, memoryFraction indicates the spark memory ratio, and numCores indicates the number of cores of a single executor.

[0121] Step S203: taking the ratio of the first target file to be downloaded to the preset number of fragments as the size of a single fragment file.

[0122] Step S204: The product of the size of a single shard file and the number of concurrent connections is used as the maximum file size that can be processed in a single batch.

[0123] Step S204: taking the ratio of the maximum file size to the available memory of a single core as the optimal number of partitions.

[0124] Step S205, determining a first product of the total number of executors and the number of cores of a single executor, and a second product of the optimal number of partitions after being enlarged by a preset multiple.

[0125] Step S206: taking the larger value of the first product and the second product as the actual number of partitions.

[0126] Step S207: When one of the at least two workflow engines detects that the download of any segment file is complete, the actual number of partitions is used to split the segment file into multiple segment sub-files.

[0127] The number of multiple shard sub-files is the same as the actual number of partitions.

[0128] Step S208: convert all the sliced ​​sub-files into conversion files of a preset format through at least one workflow engine.

[0129] In this embodiment, refer to Figure 4 As shown, Figure 4 This is a schematic diagram of the executor's on-heap memory model. The executor's on-heap memory area can be divided into Execution memory, Storage memory, User Memory, and Reserved Memory (Reserved Memory = 300MB). After Spark 1.5, Execution memory and Storage memory can be shared, and the total size is controlled by the value of spark.memory.fraction (spark memory ratio), where the default value of spark.memory.fraction is 0.6. Figure 5 As shown, Figure 5This is a schematic diagram of the relationship between the various memories in the executor's in-heap memory model. Execution memory: mainly used to store temporary data in the calculation process such as Shuffle, Join, Sort, Aggregation, etc. Storage memory: mainly used to store spark's cache data, such as RDD (Resilient Distributed Dataset) cache and unroll data. User memory (User Memory): mainly used to store data required for RDD conversion operations, such as RDD dependency information, and for sparkSQL, there is also the memory usage of UDF (User-Defined Function). Note: This part of the data cannot be spilled to disk, and if it is not enough, OOM will occur. Reserved memory (Reserved Memory): system reserved memory, which will be used to store Spark internal objects.

[0130] The concurrency number can be the number of download tasks that are performed simultaneously, which represents how many download operations can be processed at the same time. For example, when the concurrency number is 10, 10 download tasks will be started at the same time.

[0131] Specifically, the allocated memory of the executor, the spark memory ratio, the number of cores of a single executor, and the reserved memory can be stored locally in the data processing device. When executing step S201, the allocated memory of the executor, the spark memory ratio, the number of cores of a single executor, and the reserved memory can be called locally from the data processing device. After obtaining the allocated memory of the executor, the spark memory ratio, the number of cores of a single executor, and the reserved memory, the single-core available memory can be obtained using Formula 2, that is, usableMemory indicates the available memory of a single core, executorMemory indicates the allocated memory, memoryFraction indicates the spark memory ratio, and numCores indicates the number of cores of a single executor. Since the reserved memory value is 300MB, Formula 2 can be adaptively changed to:

[0132]

[0133] After obtaining the available memory of a single core, the ratio of the number of records of the first target file to be downloaded to the number of records of the second file can be used to determine the size of a single shard file. That is, the ratio of the first target file to be downloaded to the preset number of shards is used as the size of a single shard file. The specific value is: Where splitSize represents the size of a single shard file, FileSize represents the size of the file to be downloaded, and split represents the preset number of shards. After obtaining the size of a single shard file, the product of the size of a single shard file and the number of concurrent connections can be used as the maximum file size that can be processed in a single batch, that is, maxFileSize = splitSize × numfiles, maxFileSize represents the maximum file size that can be processed in a single batch, and numfiles is the maximum number of files that can be processed in a single batch stream, which generally defaults to the number of concurrent connections. After determining the maximum file size, the ratio of the maximum file size to the available memory on a single core can be used as the optimal number of partitions, that is, optimalaNumPartitions represents the optimal number of partitions. Finally, the first product of the total number of executors and the number of cores of a single executor is determined, as well as the second product of the optimal number of partitions after being enlarged by a preset multiple. The larger value of the first product and the second product is used as the actual number of partitions, that is, actualNumPartitions = max(nunCores×numExecutors, optimalNumPartitions×1.5), where actualNumPartitions represents the actual number of partitions, nunCores represents the number of cores of a single executor, and numExecutors represents the number of executors. It should be noted that the recommended configuration of the optimal number of partitions is 2 to 3 times the actual number of cores. Since this embodiment uses at least two workflow engines, the multiplication factor is configured to 1.5. The actual operating effect is as follows. Figure 6 As shown, Figure 6 This is a schematic diagram of the executor core running timeline. The green part is the core running calculation process. By applying this algorithm, we can make full use of the memory while avoiding OOM (Out of Memory), run the entire core concurrently, and ensure that the running time of the loading task is basically the same.

[0134] After obtaining the actual number of partitions, when one of the at least two workflow engines detects that any shard file has been downloaded, the actual number of partitions can be used to segment any shard file that has been downloaded to obtain multiple shard sub-files. After obtaining multiple shard sub-files, all shard sub-files can be converted into a conversion file of a preset format by at least one workflow engine.

[0135] It should be noted that when a workflow engine is used to convert all the sliced ​​sub-files into a conversion file of a preset format, there will be a core idle problem, that is, when the workflow engine is processing the sliced ​​sub-files of the current batch, the next batch of sliced ​​sub-files cannot be started for processing, especially when most cores have completed the loading task of the current stream, they will be in an idle state due to the influence of a few inefficient cores, resulting in a barrel effect, as shown in the reference Figure 7 As shown, Figure 7 The schematic diagram of the loading timeline when a single workflow engine converts a sliced ​​sub-file into a conversion file of a preset format. Therefore, in order to avoid the core being idle, further, as an optional real-time method, step S208 specifically includes:

[0136] Step S2081: using the first workflow engine, converting the odd-numbered slice sub-files in all the slice sub-files into conversion files in a preset format.

[0137] Step S2082: using the second workflow engine, converting the slice sub-files with even-numbered file numbers in all the slice sub-files into conversion files in a preset format.

[0138] The workflow engine includes a first workflow engine and a second workflow engine, and the slice sub-file has a file number.

[0139] Specifically, when a shard file is split into multiple shard sub-files, the shard sub-files can be sorted to obtain a shard sub-file sequence, and the shard sub-files can be numbered in sequence according to the order of each shard sub-file in the shard sub-file sequence as the file number of the shard sub-file, to obtain multiple shard sub-files with file numbers.

[0140] After obtaining multiple slice sub-files with file numbers, the first workflow engine can be used to convert all the slice sub-files with odd file numbers into conversion files in a preset format, and the second workflow engine can be used to convert all the slice sub-files with even file numbers into conversion files in a preset format.

[0141] In this embodiment, two workflow engines are used to convert all the slice sub-files into conversion files of a preset format, that is, the first workflow engine is used to convert the odd-numbered slice sub-files into conversion files of a preset format, and the second workflow engine is used to convert the even-numbered slice sub-files into conversion files of a preset format. Since the first workflow engine and the second workflow engine are completely decoupled when processing the slice sub-files, even if any one workflow engine is blocked, it will not affect the core processing of the other workflow engine, thereby realizing the maximum utilization of concurrent resources, such as Figure 8 As shown, Figure 8A schematic diagram of a loading timeline when two workflow engines are used to convert segmented sub-files into conversion files of a preset format.

[0142] Step S30, merging all converted files in the target database to synchronize the files to be downloaded to the target database.

[0143] It should be noted that when converting a shard file into a conversion file of a preset format, the shard file is further divided into multiple shard sub-files based on the actual number of partitions, and finally all the shard sub-files are converted into conversion files of the preset format, which makes the number of conversion files huge. In order to reduce the pressure when merging conversion files and control the merging time within a shorter time, further, as an optional implementation, step S30 specifically includes:

[0144] Step S301: for any converted file converted by a workflow engine, determine whether the accumulated number of converted files is equal to a preset threshold.

[0145] Step S302: if the accumulated number of converted files is equal to a preset threshold, the converted files of the preset threshold number are merged to obtain at least one merged file, and after the merged file is obtained, the accumulated number is cleared.

[0146] Step S303, for any workflow engine, if it is detected that all the segmented sub-files are converted into converted files, all the merged files and all the converted files are merged to synchronize the files to be downloaded to the target database.

[0147] Specifically, after the workflow engine converts the sliced ​​sub-files into the conversion files of the preset format, for each workflow engine, when the workflow engine starts to convert the sliced ​​sub-files into the conversion files of the preset format, the cumulative number of conversion files can be calculated from zero. When the cumulative number of conversion files is equal to the preset threshold, the conversion files of the preset threshold number are merged to obtain a merged file. After obtaining the merged file, the cumulative number is cleared to zero, and the above operation is repeated until the workflow engine converts all the sliced ​​sub-files into the conversion files of the preset format. After the sliced ​​sub-files corresponding to any workflow engine are converted into the conversion files of the preset format, all merged files and all conversion files corresponding to the workflow engine can be merged to synchronize the files to be downloaded to the target database.

[0148] To facilitate a better understanding of this embodiment, a specific example is provided below.

[0149] Reference Fig. 9 As shown, Fig. 9It is a schematic diagram of the merging process of multiple sharded sub-files, taking two workflow engines as an example and the target database being an HDFS database as an example, the first workflow engine (stream loading stream A) and the second workflow engine (stream loading stream B). After the first workflow engine and the second workflow engine convert the sharded sub-files into conversion files of a preset format, the conversion files will be written into the directory structure of HDFS, and the listener will respectively listen to the cumulative number of conversion files converted by the first workflow engine and the second workflow engine. When the cumulative number of conversion files converted by any of the first workflow engine or the second workflow engine is equal to a preset threshold, the conversion files of the preset threshold number under the workflow engine are merged to obtain a merged file, that is, the sharded sub-files under each workflow engine are merged by asynchronous merging, and the merged file obtained after the merger is synchronized to the HDFS database, and the merged file is written into the directory structure of the HDFS data volume. Finally, when the listener detects that the sub-files corresponding to any workflow engine in the first workflow engine or the second workflow engine are converted into conversion files, the listener can send a merge file instruction to the driver side, so that the driver side merges all the merged files and all the conversion files corresponding to the workflow engine, and after all the merged files and all the conversion files corresponding to the workflow engine, writes them into the directory structure of the HDFS database.

[0150] In this embodiment, for the conversion file obtained by the workflow engine conversion, small files are merged in two stages, which can not only improve the utilization of resources, but also reduce the pressure of the workflow engine to merge the conversion files after converting all the shard sub-files into conversion files. Even if a large number of small files are generated in the whole process, all small files can be merged in a relatively short time, thereby avoiding the situation that a large number of small files will occupy a large amount of NameNode mapping space, increase disk seek time, and cause downstream jobs to take longer to read data.

[0151] It should be noted that after converting the shard sub-files into the conversion files of the preset format, in order to prevent the merged files from not being accurately recognized by the stream native checkpoint, as an optional real-time method, a check mechanism can be implemented by recording the status when the loading is completed, as shown in Fig.10 As shown, Fig.10The figure is a flow chart of loading verification. When each workflow engine is loaded, a finish flag (completion flag) will be generated in the HDFS directory. If an error occurs during the subsequent synchronization of the to-be-downloaded file to the target database, such as the failure of synchronization of the to-be-downloaded file due to insufficient permissions or the non-existence of the hive table, the to-be-downloaded file with the error needs to be repaired, and then the loading process of the to-be-downloaded file is restarted. Before restarting the loading process, the system will check the loading status of the last to-be-downloaded file. If the check finds that the last download status is correct and the completion flag already exists, it means that the last loading of the to-be-downloaded file has been completed, then the system can directly load the data into the target database without having to re-load the to-be-downloaded file. If an error occurs during the verification phase of the Hive table, that is, the to-be-downloaded file will be converted to a null value during the processing process, and the table data volume will be compared with the loaded data volume after completion. If they are consistent, the data volume verification is passed. Finally, the field is checked for null values. If there is no null value in the table, the verification passes. Otherwise, it is considered that there is a field abnormality during the data loading process in Hive, and the verification fails. If the check fails, it will be considered that a problem has occurred in the loading process of the file to be downloaded. At this time, the loading process of the file to be downloaded will be restarted to ensure that the file to be downloaded is loaded correctly.

[0152] In this embodiment, compared with the related art, after downloading the file to be downloaded to HDFS, the file to be downloaded is converted into an Orc file, and the Orc file is loaded into Hive. In this embodiment, multiple shard files of the file to be downloaded corresponding to each executor are downloaded concurrently by at least two executors, that is, the download task of downloading the file to be downloaded is distributed to each executor, rather than the download task is concentrated on one executor, and multiple shard files of the file to be downloaded can be downloaded simultaneously, thereby improving the download efficiency and concurrent processing capability. At the same time, when any workflow engine among the multiple workflow engines detects that the download of any shard file is completed, the workflow engine converts any shard file into a conversion file of a preset format, that is, multiple workflow engines perform conversion operations on multiple downloaded shard files and load the conversion files into the target database, and the loading efficiency of the shard files can be improved by the way that multiple workflow engines perform concurrent loading on the downloaded file shards, that is, the embodiment can avoid serial blocking processing in the download stage or the loading stage by concurrent downloading in the download stage and concurrent loading in the loading stage, thereby improving the data processing efficiency.

[0153] To facilitate a better understanding of this embodiment, a specific example is provided below.

[0154] See also Figure 11-13 As shown, Fig.11 This is a flowchart of SparkSQL concurrent ETL. Fig.12Figure 13 is a flow chart of concurrently downloading the fragmented files of the file to be downloaded. The SFTP3.0 server splits the data file into six files to be downloaded. When SparkSQL concurrently downloads the files to be downloaded through at least two executors, it can download the fragmented files of the files to be downloaded and download the configuration file of the data file at the same time. After obtaining the configuration file, the configuration file can be verified, the path can be parsed, and the Schema can be constructed. After completing the download of the fragmented files of the file to be downloaded, the configuration file can be used to verify the file to be downloaded to ensure the integrity of the fragmented files of the file to be downloaded. When it is determined that the fragmented files of the file to be downloaded are not completely downloaded, the fragmented files that have not been downloaded will continue to be downloaded, or if the fragmented files of the file to be downloaded are not downloaded, the fragmented files that have not been downloaded will be downloaded. In the process of downloading the segmented files of the file to be downloaded, after any workflow engine in the workflow engine (Structured Streaming) detects that any segmented file has been downloaded, that is, after the first workflow engine (Streaming stream B) or the second workflow engine (Streaming stream A) detects that any segmented file has been downloaded, any segmented file is segmented according to the actual number of partitions to obtain multiple segmented sub-files with file numbers, and the first workflow engine is used to convert the odd-numbered segmented sub-files into a conversion file in a preset format, and the second workflow engine is used to convert the even-numbered segmented sub-files into a conversion file in a preset format. In the process of conversion, the listener can calculate the total amount of data that the first workflow engine and the second workflow engine need to process based on the first file record number, the preset number of segments, and the actual number of partitions in all the files to be downloaded, that is, the sum of the file record numbers of the odd-numbered segmented sub-files that the first workflow engine needs to process, and the sum of the file record numbers of the even-numbered segmented sub-files that the second workflow engine needs to process. When the first workflow engine or the second workflow engine is processing shard sub-files, when the listener detects that the cumulative number of converted files obtained by any workflow engine is equal to a preset threshold, multiple shard sub-files of the preset threshold number are merged to obtain a merged file, and when it is monitored that the amount of data processed by any workflow engine is equal to the total amount of data, all shard sub-files and all merged files are merged to obtain a new merged file corresponding to the workflow engine. That is, for any workflow engine, when the listener monitors that the amount of data processed by the workflow engine is equal to the total amount of data, it can use the task thread corresponding to the workflow engine to merge all shard sub-files and all merged files under the workflow engine to obtain a new merged file, and load the new merged file into the HIVE library.For example, task thread A is used to merge all merged files and all fragmented sub-files under the second workflow engine, and task thread A is used to merge all merged files and all fragmented sub-files under the first workflow engine.

[0155] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0156] The present application provides a data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method in the above-mentioned embodiment 1.

[0157] Reference below Fig.14 , which shows a schematic diagram of the structure of a data processing device suitable for implementing an embodiment of the present application. The data processing device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.14 The data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0158] like Fig.14As shown, the data processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0159] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0160] The data processing device provided by the present application adopts the data processing method in the above embodiment to solve the technical problem of low data processing efficiency. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0161] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0162] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0163] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data processing method in the above-mentioned embodiment.

[0164] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0165] The computer-readable storage medium may be included in the device; or may exist independently without being assembled into the data processing device.

[0166] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a data processing device, the data processing device: concurrently downloads multiple fragment files of the file to be downloaded corresponding to each executor through at least two executors; when a workflow engine among at least two workflow engines detects that the download of any fragment file is completed, converts any fragment file into a conversion file of a preset format through a workflow engine, and loads the conversion file into a target database; merges all conversion files in the target database to synchronize the file to be downloaded to the target database.

[0167] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0168] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0169] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0170] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data processing method, and can solve the technical problem of low data processing efficiency. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the data processing method provided in the above-mentioned embodiment, and will not be repeated here.

[0171] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data processing method when executed by a processor.

[0172] The computer program product provided by this application can solve the technical problem of low data processing efficiency. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the data processing method provided by the above embodiment, which will not be repeated here.

[0173] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The data processing method comprises: Concurrently downloading, by at least two executors, multiple fragment files of the file to be downloaded corresponding to each of the executors; When one of the at least two workflow engines detects that the download of any segment file is completed, the workflow engine converts the any segment file into a conversion file of a preset format, and loads the conversion file into a target database; All the converted files in the target database are merged to synchronize the files to be downloaded to the target database.

2. The data processing method according to claim 1, characterized in that: The step of concurrently downloading multiple fragment files of the file to be downloaded corresponding to each of the executors through at least two executors comprises: For each of the files to be downloaded, obtaining the number of first file records in the file to be downloaded and the preset number of fragments of the fragmented file; Determine the number of second file records in each of the fragment files based on the number of the first file records and the preset number of fragments; Based on the second file record number, at least two of the executors are controlled to concurrently download the number of file records in the to-be-downloaded file corresponding to each of the executors to obtain a plurality of the segmented files.

3. The data processing method according to claim 2, characterized in that: The step of determining the number of second file records in each of the fragment files based on the number of first file records and the preset number of fragments includes: If the first file record number is divisible by the preset number of fragments, the result value after the first file record number is divided by the preset number of fragments is used as the second file record number; If the first file record number cannot be divided by the preset number of shards, the second file record number is determined using formula 1; Among them, formula 1 is: splitCount represents the number of records in the second file, fileCount represents the number of records in the first file, N represents the preset number of fragments, and splitSeq represents the sequence number of the fragment file.

4. The data processing method according to claim 1, characterized in that: When one of the at least two workflow engines detects that the download of any segment file is complete, the step of converting the any segment file into a conversion file of a preset format by the one workflow engine comprises: Obtaining a first target file to be downloaded, the allocated memory of the executor, the spark memory ratio, the reserved memory of the executor, the number of cores of a single executor, and the user memory; the first target file to be downloaded is the file to be downloaded with the largest file size among the multiple files to be downloaded; Determine the available memory of a single core by using the allocated memory, the spark memory ratio, the reserved memory, the number of cores of a single executor and formula 2; Wherein, the formula 2 includes: usableMemory represents the available memory of a single core, executorMemory represents the allocated memory, memoryFraction represents the spark memory ratio, and numCores represents the number of cores of a single executor; The ratio of the first target file to be downloaded to the preset number of fragments is used as the size of a single fragment file; The product of the single shard file size and the number of concurrent connections is used as the maximum file size that can be processed in a single batch; The ratio of the maximum file size to the available memory of the single core is used as the optimal number of partitions; Determine a first product of the total number of the executors and the number of cores of a single executor, and a second product of the optimal number of partitions after being enlarged by a preset multiple; The larger value of the first product and the second product is used as the actual partition number; When one of the at least two workflow engines detects that the download of any segment file is complete, the actual number of partitions is used to divide the any segment file into multiple segment sub-files; wherein the number of the multiple segment sub-files is the same as the actual number of partitions; All the slice sub-files are converted into conversion files in a preset format through at least one of the workflow engines.

5. The data processing method according to claim 4, characterized in that: The workflow engine includes a first workflow engine and a second workflow engine, the slice sub-files have file numbers, and the step of converting all the slice sub-files into conversion files of a preset format by at least one of the workflow engines includes: Using the first workflow engine, converting the slice sub-files with odd file numbers in all the slice sub-files into conversion files in a preset format; The second workflow engine is used to convert the slice sub-files whose file numbers are even-numbered in all the slice sub-files into conversion files in a preset format.

6. The data processing method according to claim 4, characterized in that: The step of merging all the converted files in the target database to synchronize the files to be downloaded to the target database includes: For the conversion files obtained by any workflow engine conversion, determining whether the accumulated number of the conversion files is equal to a preset threshold; If the cumulative number of the conversion files is equal to a preset threshold, merging the conversion files of the preset threshold number to obtain at least one merged file, and after obtaining the merged file, clearing the cumulative number; For any workflow engine, if it is detected that all the segmented sub-files are converted into the converted files, all the merged files and all the converted files are merged to synchronize the files to be downloaded to the target database.

7. The data processing method according to claim 1, characterized in that: After the at least two executors concurrently download the multiple fragment files of the file to be downloaded corresponding to each of the executors, the method further includes: Obtain a configuration file; the configuration file includes the file name of each file to be downloaded, the file size of each file to be downloaded, and the last modification time of each file to be downloaded; Determine whether there is a target download log; wherein the target download log is a download log generated after all the files to be downloaded are downloaded, and the target download log includes the file name of the file to be downloaded, the file size of each file to be downloaded, and the last modification time of each file to be downloaded; If the target download log exists, the target download log is verified using the configuration file to obtain a verification result; wherein the verification result is a verification pass or a verification fail; If the verification result is that the verification is passed, it is determined that all the files to be downloaded have been downloaded; If the verification result is that the verification fails, an inconsistent second target file to be downloaded is determined based on the download log and the configuration file, and the second target file to be downloaded is downloaded.

8. A data processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images