Resource acquisition method, resource acquisition device, storage medium and electronic equipment

Abstract

The present disclosure relates to the field of computer technology, and specifically to a resource acquisition method, a resource acquisition device, a computer-readable storage medium, and an electronic device. The method comprises: scanning a resource server in real time to obtain a list of files to be processed, determining a first list of files to be downloaded based on the list of files to be processed; dividing multiple files in the first list of files to be downloaded into at least one file group based on file specification information; downloading multiple files in at least one file group using a streaming framework, and merging the multiple files in the file group into file blocks; and storing the file blocks in a storage area of ​​a data platform to complete the resource acquisition process. The technical solutions of the embodiments of the present disclosure can solve the problem of high latency during resource acquisition.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a resource acquisition method, a resource acquisition device, a computer-readable storage medium, and an electronic device. Background Art

[0002] With the rapid development of software and hardware, data is expanding, and various big data platforms are being developed. Big data platforms can process, store, calculate, and analyze massive amounts of data, for example, to predict related businesses or provide precise customization.

[0003] Before using a big data platform to process data, it is necessary to first collect data from relevant business parties. In existing solutions, data can be collected using ETL (data warehouse technology), but this solution generally has a collection time limit, such as collecting data at dawn every day.

[0004] However, the existing solution, which uses a timed collection method, cannot meet real-time requirements. When transmitting data, the different file sizes have a significant impact on the storage efficiency of the data platform. In addition, when collecting data centrally, the large amount of data puts greater pressure on network transmission, making it difficult to control the transmission rate.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0006] The purpose of the present disclosure is to provide a resource acquisition method and resource acquisition device, a computer-readable storage medium and an electronic device, which can solve the problem of high delay in resource acquisition.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0008] According to a first aspect of the present disclosure, a resource acquisition method is provided, comprising: scanning a resource server in real time to obtain a list of files to be processed, and determining a first list of files to be downloaded based on the list of files to be processed; wherein the first list of files to be downloaded includes specification information of multiple files in the first list of files to be downloaded; according to the specification information of the files, the multiple files in the first list of files to be downloaded are divided into at least one file group; wherein the specification information of the file group meets a preset file block specification; downloading the multiple files in the at least one file group using a streaming framework, and merging the multiple files in the file group into file blocks; and storing the file blocks in a storage area of ​​a data platform to complete the resource acquisition process.

[0009] Optionally, the real-time scanning resource server obtains a list of files to be processed, including: the real-time scanning resource server obtains a total file list, obtains multiple files in the total file list and compliance information corresponding to the files; and determines the list of files to be processed based on the compliance information corresponding to the files.

[0010] Optionally, the real-time scanning resource server obtains a list of files to be processed, including: the real-time scanning resource server obtains a total file list, obtains multiple files in the total file list and first specification information corresponding to the files; compares the first specification information with multiple files in the total file list obtained by the previous real-time scan and second specification information corresponding to the files to obtain a comparison result; determines the list of files to be processed based on the comparison result.

[0011] Optionally, determining the first list of files to be downloaded based on the list of files to be processed includes: obtaining the download status of multiple files in the list of files to be processed; wherein the download status includes downloading, downloading completed, and waiting to be downloaded; and determining the first list of files to be downloaded based on the download status of the multiple files.

[0012] Optionally, dividing the multiple files in the first list of files to be downloaded into at least one file group according to the specification information of the files includes: obtaining association information corresponding to the multiple files in the first list of files to be downloaded, and dividing the multiple files into at least one type file list according to the association information corresponding to the files; dividing the multiple files in the type file list into at least one file group according to the specification information of the files.

[0013] Optionally, after merging the multiple files in the file group into a file block, the method further includes: generating verification data for the file block; wherein the verification data includes verification specification information of the multiple files in the file group and verification specification information of the file block.

[0014] Optionally, storing the file block in the storage area of ​​the data platform includes: storing the file block in the storage area of ​​the data platform, and updating the file information corresponding to the file block to the cache directory corresponding to the data platform; verifying the file block in the storage area according to the verification data of the file block; after the file block in the storage area is successfully verified, updating the file information corresponding to the file block to the official directory corresponding to the data platform.

[0015] Optionally, verifying the file blocks in the storage area based on the verification data of the file blocks includes: obtaining specification information corresponding to the file blocks in the storage area; and verifying the file blocks in the storage area based on the verification specification information of the multiple files and the specification information corresponding to the file blocks in the storage area.

[0016] Optionally, after the file block verification in the storage area fails, the verification status corresponding to the file information of the file block in the cache directory corresponding to the data platform is updated to failure; a second list of files to be downloaded is generated based on the verification status, and the download and verification process is executed according to the second list of files to be downloaded until the file block corresponding to the second list of files to be downloaded is successfully verified.

[0017] Optionally, when the number of file block check failures in the storage area exceeds a preset threshold, an alarm is issued.

[0018] According to a second aspect of the present disclosure, a resource acquisition device is provided, comprising: a to-be-downloaded file determination module, configured to scan a resource server in real time to obtain a to-be-processed file list, and determine a first to-be-downloaded file list based on the to-be-processed file list; wherein the first to-be-downloaded file list includes specification information of multiple files in the first to-be-downloaded file list; a file group division module, configured to divide multiple files in the first to-be-downloaded file list into at least one file group based on the specification information of the files; wherein the specification information of the file group conforms to a preset file block specification; a streaming download module, configured to download multiple files in the at least one file group using a streaming framework, and merge the multiple files in the file group into a file block; and a resource storage module, configured to store the file block in a storage area of ​​a data platform to complete the resource acquisition process.

[0019] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the resource acquisition method as described in the first aspect is implemented.

[0020] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0021] one or more processors; and

[0022] The memory is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the resource acquisition method as described in the first aspect of the above embodiment.

[0023] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0024] In the resource collection method provided in one embodiment of the present disclosure, a download list can be obtained in real time, and file blocks can be downloaded through a streaming framework to complete the resource collection process. On the one hand, real-time file collection and downloading through a streaming framework can meet real-time requirements and avoid the high latency problem in data collection in the prior art. On the other hand, the steps of resource downloading can be distributed to different time points, avoiding the problem of high network pressure when centralized resource collection in the prior art, and can better control the network transmission rate. On the other hand, downloading files according to preset file block specifications improves the storage efficiency of the data platform, thereby improving the data throughput capacity of the data platform.

[0025] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0027] Figure 1 A flowchart schematically illustrates a resource acquisition method in an exemplary embodiment of the present disclosure;

[0028] Figure 2 A flowchart schematically illustrates determining a list of files to be processed based on compliance information corresponding to the files in an exemplary embodiment of the present disclosure;

[0029] Figure 3 A flowchart schematically illustrates determining a list of files to be processed according to comparison results in an exemplary embodiment of the present disclosure;

[0030] Figure 4 Schematically illustrates a flow chart of determining a first list of files to be downloaded according to download status of multiple files in an exemplary embodiment of the present disclosure;

[0031] Figure 5 A flowchart schematically illustrates dividing a plurality of files in a file type list into at least one file group according to specification information of the files in an exemplary embodiment of the present disclosure;

[0032] Figure 6 Schematically illustrates a flow chart of updating file information corresponding to a file block into a formal directory corresponding to a data platform in an exemplary embodiment of the present disclosure;

[0033] Figure 7A flowchart schematically illustrates verifying file blocks in a storage area according to verification specification information of multiple files and specification information corresponding to the file blocks in the storage area in an exemplary embodiment of the present disclosure;

[0034] Figure 8 A flowchart schematically illustrates a process of downloading and verifying files according to a second list of files to be downloaded until verification of file blocks corresponding to the second list of files to be downloaded succeeds in an exemplary embodiment of the present disclosure;

[0035] Figure 9 A schematic diagram schematically illustrates a resource collection method in an exemplary embodiment of the present disclosure;

[0036] Figure 10 The following schematically shows the composition of a resource collection device in an exemplary embodiment of the present disclosure;

[0037] Figure 11 The structure diagram of a computer system suitable for implementing the electronic device of the exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0038] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete and the concepts of the example embodiments will be fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In addition, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known structures, methods, devices, implementations, materials, or operations are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0039] The blocks shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. Specifically, these functional entities may be implemented in software, or in one or more software-hardened modules, or in different networks and / or processor devices and / or microcontroller devices.

[0040] With the rapid development of software and hardware, data is expanding, and various big data platforms are being developed. Big data platforms can process, store, calculate, and analyze massive amounts of data, for example, to predict related businesses or provide precise customization.

[0041] Before using a big data platform to process data, it is necessary to first collect data from relevant business parties. In existing solutions, data can be collected using ETL (data warehouse technology), but this solution generally has a collection time limit, such as collecting data at dawn every day.

[0042] However, the existing solution, which uses a timed collection method, cannot meet real-time requirements. When transmitting data, the different file sizes have a significant impact on the storage efficiency of the data platform. In addition, when collecting data centrally, the large amount of data puts greater pressure on network transmission, making it difficult to control the transmission rate.

[0043] According to the resource collection method provided in this exemplary embodiment, the resource server can be scanned in real time to obtain a list of files to be processed, a first list of files to be downloaded can be determined based on the list of files to be processed, multiple files in the first list of files to be downloaded can be divided into at least one file group based on the file specification information, multiple files in the at least one file group can be downloaded using a streaming framework, and multiple files in the file group can be merged into file blocks, and the file blocks can be stored in the storage area of ​​the data platform to complete the resource collection process. Figure 1 As shown, the resource acquisition method may include the following steps:

[0044] Step S110: Scan the resource server in real time to obtain a list of files to be processed, and determine a first list of files to be downloaded based on the list of files to be processed; wherein the first list of files to be downloaded includes specification information of multiple files in the first list of files to be downloaded;

[0045] Step S120: dividing the plurality of files in the first list of files to be downloaded into at least one file group according to the file specification information; wherein the file group specification information conforms to a preset file block specification;

[0046] Step S130, using a streaming framework to download multiple files in at least one file group, and merging the multiple files in the file group into file blocks;

[0047] Step S140: storing the file blocks in the storage area of ​​the data platform to complete the resource collection process.

[0048] In a resource collection method provided by an embodiment of the present disclosure, a download list can be obtained in real time, and file blocks can be downloaded through a streaming framework to complete the resource collection process. On the one hand, real-time file collection and downloading through a streaming framework can meet real-time requirements and avoid the high latency problem in data collection in the prior art. On the other hand, the steps of resource downloading can be distributed to different time points, avoiding the problem of high network pressure when centralized resource collection in the prior art, and can better control the network transmission rate. On the other hand, downloading files according to preset file block specifications improves the storage efficiency of the data platform, thereby improving the data throughput capacity of the data platform.

[0049] Below, steps S110 to S140 of the resource collection method in this exemplary embodiment will be described in more detail with reference to the accompanying drawings and embodiments.

[0050] Step S110: Scan the resource server in real time to obtain a list of files to be processed, and determine a first list of files to be downloaded according to the list of files to be processed;

[0051] In an exemplary embodiment of the present disclosure, a resource server can be scanned in real time to obtain a list of files to be processed, and a first list of files to be downloaded can be determined based on the list of files to be processed. The resource server may include a server that produces business data, and the list of files to be processed can be obtained by scanning the resource server. For example, an FTP (File Transfer Protocol) server or an SFTP (Secure File Transfer Protocol) server. Specifically, the resource server can be used to store business data, and files in the resource server can be obtained by scanning the resource server, and a list of files to be processed can be obtained based on these files. Specifically, the list of files to be processed can include all files in the resource server.

[0052] After obtaining the list of pending files, a first list of pending files can be determined based on the list of pending files. Specifically, some files in the list of pending files may be abnormal, have already been downloaded, or are currently being downloaded. In this case, these files need to be removed, and the list of pending files after removal is used as the first list of pending files. It should be noted that the present disclosure does not impose any particular limitation on the method for determining the first list of pending files based on the list of pending files.

[0053] The first list of files to be downloaded includes specification information for multiple files in the first list of files to be downloaded. Specifically, the first list of files to be downloaded may include multiple files, and each of the multiple files has corresponding specification information. The file specification information may be used to indicate numerical information about the memory occupied by the file. For example, it may include the file size, number of lines, number of items, etc. It should be noted that the present disclosure does not specifically limit the specific type of file specification information.

[0054] In an exemplary embodiment of the present disclosure, a resource server may be scanned in real time to obtain a total file list, obtain multiple files in the total file list and compliance information corresponding to the files, and determine a list of files to be processed based on the compliance information corresponding to the files. Figure 2 As shown, determining a list of files to be processed according to the compliance information corresponding to the files may include the following steps S210 to S220:

[0055] Step S210: Scan the resource server in real time to obtain a total file list, and obtain multiple files in the total file list and compliance information corresponding to the files;

[0056] In an exemplary embodiment of the present disclosure, a resource server can be scanned in real time to obtain a total file list, and multiple files in the total file list and the compliance information corresponding to the files can be obtained. Specifically, the total file list refers to a file list that includes all files in the resource server, and the files in the total file list have corresponding compliance information. Specifically, the compliance information of the file may include the file format, processing permissions, file name, etc. It should be noted that the specific type of compliance information corresponding to the files in the total file list of the present disclosure is not specifically limited.

[0057] Step S220: Determine a list of files to be processed based on the compliance information corresponding to the files.

[0058] In an exemplary embodiment of the present disclosure, after obtaining multiple files and corresponding compliance information in the total file list through the above steps, a list of files to be processed can be determined based on the compliance information corresponding to the files. Specifically, files whose compliance information does not meet the preset compliance standard can be removed, and the total file list after removal is determined as the list of files to be processed.

[0059] For example, if the multiple files in the total file list and the compliance information corresponding to the files are file names, a regular expression can be used to determine whether the file names meet the preset compliance standards. If the file names do not meet the preset compliance standards, the file corresponding to the file name can be removed from the total file list, and the total file list without the file name can be used as the file list to be processed. For another example, if the multiple files in the total file list and the compliance information corresponding to the files are formats, and the preset compliance standard is tmp format, all files that are not in tmp format need to be removed from the total file list, and the total file list after removal can be used as the file list to be processed.

[0060] It should be noted that the present disclosure does not impose any special limitation on the specific method of determining the list of files to be processed based on the compliance information corresponding to the files.

[0061] Through the above steps S210 to S220, the resource server can be scanned in real time to obtain a total file list, obtain multiple files in the total file list and the compliance information corresponding to the files, and determine a list of files to be processed based on the compliance information corresponding to the files.

[0062] In an exemplary embodiment of the present disclosure, a resource server can be scanned in real time to obtain a total file list, multiple files in the total file list and first specification information corresponding to the files are obtained, the first specification information is compared with multiple files in the total file list obtained in the previous real-time scan and second specification information corresponding to the files to obtain a comparison result, and a list of files to be processed is determined based on the comparison result. Figure 3 As shown, determining the list of files to be processed according to the comparison results may include the following steps S310 to S330:

[0063] Step S310: Scan the resource server in real time to obtain a total file list, and obtain multiple files in the total file list and first specification information corresponding to the files;

[0064] In an exemplary embodiment of the present disclosure, a resource server can be scanned in real time to obtain a total file list, and multiple files in the total file list and the first specification information corresponding to the files can be obtained. Specifically, the total file list refers to a file list that includes all files in the resource server, and the files in the total file list have corresponding first specification information. The first specification information of a file can be used to indicate numerical information of the memory occupied by the file. For example, it can include the size of the file, the number of lines, the number of items, etc. It should be noted that the present disclosure does not specifically limit the specific type of the first specification information of the file.

[0065] Step S320: Compare the first specification information with multiple files in the total file list obtained by the previous real-time scan and the second specification information corresponding to the files to obtain a comparison result;

[0066] Step S330: Determine a list of files to be processed based on the comparison result.

[0067] In an exemplary embodiment of the present disclosure, after obtaining the plurality of files in the total file list and the first specification information corresponding to the files through the above steps, the plurality of files in the total file list obtained by the previous real-time scan and the second specification information corresponding to the files can be obtained. Specifically, when performing a real-time scan, there is a certain time difference, for example, 1 second, that is, the result of the previous real-time scan is the result of the real-time scan performed 1 second ago, which includes the plurality of files and the second specification information corresponding to the files.

[0068] At this point, the plurality of files in the total file list and the first specification information corresponding to the files can be compared with the plurality of files in the total file list and the second specification information corresponding to the files to obtain a comparison result. Specifically, the comparison result can be used to indicate changes in the file specification information between the two scans. The list of files to be processed can be determined based on the comparison result.

[0069] For example, the file specification information is the file size. The total file list obtained by the current scan includes file A, file B, and file C, and their corresponding file sizes are file A-50MB, file B-70MB, and file C-80MB. The total file list obtained by the previous scan includes file A, file B, and file D, and their corresponding file sizes are file A-45MB, file B-50MB, and file D-90MB. The first specification information of the currently scanned file can be compared with the second specification information obtained by the previous scan to obtain that the results of the two scans of file A and file B are different. This can indicate that file A and file B are being uploaded. At this time, file A and file B can be ignored. After file A and file B are uploaded, file A and file B can be downloaded. In other words, file A and file B can be removed from the total file list at this time, and the total file list after removal is used as the list of files to be processed. It should be noted that the present disclosure does not specifically limit the method of determining the list of files to be processed based on the comparison results.

[0070] Through the above steps S310 to S330, the resource server can be scanned in real time to obtain a total file list, multiple files in the total file list and the first specification information corresponding to the files can be obtained, the first specification information can be compared with the multiple files in the total file list obtained by the last real-time scan and the second specification information corresponding to the files to obtain a comparison result, and the list of files to be processed can be determined based on the comparison result.

[0071] In an exemplary embodiment of the present disclosure, the download status of multiple files in the to-be-processed file list may be obtained, and the first to-be-downloaded file list may be determined according to the download status of the multiple files. Figure 4As shown, determining a first list of files to be downloaded according to the download status of multiple files may include the following steps S410 to S420:

[0072] Step S410, obtaining the download status of multiple files in the list of files to be processed;

[0073] In an example embodiment of the present disclosure, the download status of multiple files in the pending file list can be obtained. The download status of multiple files in the pending file list may include downloading, download completed, and pending download. Specifically, the download status of the files in the pending file list can be obtained in the download part of the current streaming framework. Specifically, if there is a file in the download part of the streaming framework that has been downloaded and is identical to the file in the pending file list, the download status of the file in the pending file list is download completed; if there is a file in the download part of the streaming framework that is already being downloaded and is identical to the file in the pending file list, the download status of the file in the pending file list is downloading; if there is no file in the download part of the streaming framework that is identical to the file in the pending file list, the download status of the file in the pending file list is pending download. It should be noted that the present disclosure does not specifically limit the specific method for obtaining the download status of multiple files in the pending file list.

[0074] Step S420: Determine a first list of files to be downloaded according to the download status of the multiple files.

[0075] In an exemplary embodiment of the present disclosure, after obtaining the download status of multiple files in the pending file list through the above steps, a first pending file list can be determined based on the download status of the multiple files. Specifically, files with a download status of "downloading" or "download completed" can be removed from the pending file list, and the removed pending file list can be used as the first pending file list. It should be noted that the present disclosure does not specifically limit the specific method for determining the first pending file list based on the download status of multiple files.

[0076] Through the above steps S410 to S420, the download status of multiple files in the list of files to be processed can be obtained, and the first list of files to be downloaded can be determined according to the download status of the multiple files.

[0077] Step S120, dividing the plurality of files in the first list of files to be downloaded into at least one file group according to the file specification information;

[0078] In an exemplary embodiment of the present disclosure, after obtaining the first list of files to be downloaded through the above steps, the multiple files in the first list of files to be downloaded can be divided into at least one file group according to the specification information of the multiple files in the first list of files to be downloaded. Specifically, when performing the division, the division can be performed according to the preset file block specifications. The preset file block specifications refer to the basic specifications when the data platform stores files. When the files are stored in the data platform according to the preset file block specifications, the storage efficiency of the files can be improved, and the data throughput capacity can be improved, and the data processing efficiency can be improved. For example, the data platform is a Hadoop platform. In this case, the preset file block specifications can be set to comply with the Hadoop cluster block size, such as 1GB. It should be noted that the present disclosure does not specifically limit the specific type and specific value of the preset file block specifications.

[0079] Furthermore, the file group specification information meeting the preset file block specification may include the total size of the multiple files before division being equal to the preset file block specification. For example, if the first to-be-downloaded file list includes file A, file B, file C, file D, and file E, and their corresponding file sizes are: file A - 300 MB, file B - 724 MB, file C - 225 MB, file D - 455 MB, and file E - 344 MB, and the preset file block specification is set to 1 GB, then files A and file B can be merged into one file group, and files C, file D, and file E can be merged into one file group.

[0080] Alternatively, the preset file block specification may be a specification interval. When the total specification of the multiple files before division is within the specification interval corresponding to the preset file block, the specification information of the file group also conforms to the preset file block specification. For example, the first list of files to be downloaded includes file A, file B, file C, file D, and file E, and their corresponding file sizes are: file A - 280MB, file B - 700MB, file C - 200MB, file D - 435MB, and file E - 350MB. The preset file block specification corresponds to a specification interval of 950MB to 1050MB. The total size of files A and file B is 980MB, which is within the aforementioned specification interval. Therefore, files A and file B can be divided into one file group. The total size of files C, file D, and file E is 985MB, which is within the aforementioned specification interval. Therefore, files C, file D, and file E can be divided into one file group.

[0081] In an exemplary embodiment of the present disclosure, association information corresponding to multiple files in a first list of files to be downloaded may be obtained, multiple files may be divided into at least one type file list according to the association information corresponding to the files, and multiple files in the type file list may be divided into at least one file group according to the specification information of the files. Figure 5As shown, dividing multiple files in the file type list into at least one file group according to the file specification information may include the following steps S510 to S520:

[0082] Step S510: Obtain association information corresponding to a plurality of files in a first list of files to be downloaded, and classify the plurality of files into at least one type file list according to the association information corresponding to the files;

[0083] In an exemplary embodiment of the present disclosure, association information corresponding to multiple files in the first list of files to be downloaded can be obtained. Specifically, the association information corresponding to the multiple files in the first list of files to be downloaded can be used to indicate the actual characteristics of the files. For example, the association information corresponding to the multiple files in the first list of files to be downloaded may include the business type to which the files belong, the region to which the files belong, the province to which the files belong, the file date, etc. It should be noted that the present disclosure does not specifically limit the specific type of association information corresponding to the multiple files in the first list of files to be downloaded.

[0084] In an exemplary embodiment of the present disclosure, after obtaining the association information corresponding to multiple files in the first file list to be downloaded through the above steps, the multiple files can be divided into at least one type file list according to the association information corresponding to the files. Specifically, multiple files can be divided into different categories according to the association information, and each category corresponds to a type file list. For example, the first file list to be downloaded includes file A, file B, and file C, and the association information is the file date. The multiple files and the association information corresponding to the multiple files are: file A-2021.10.20, file B-2021.10.20, file C-2021.10.15, that is, at this time, the multiple files can be divided into two type file lists, one type file list (2021.10.20) includes file A and file B, and the other type file list (2021.10.15) includes file C. It should be noted that the present disclosure does not specifically limit the specific method of dividing multiple files into at least one type file list according to the association information corresponding to the files.

[0085] Step S520: Divide the multiple files in the file type list into at least one file group according to the file specification information.

[0086] In an exemplary embodiment of the present disclosure, after obtaining the file specification information and the type file list through the above steps, the multiple files in the type file list can be divided into at least one file group according to the file specification information. For example, the type file list (2021.10.20) obtained through the above steps includes file A, file B, file C, and file D, and their corresponding file specification information (file size) is: file A-500MB, file B-524MB, file C-400MB, file D-624MB, and the preset file block specification is 1GB. At this time, file A and file B in the type file list (2021.10.20) can be divided into file group a, and file C and file D in the type file list (2021.10.20) can be divided into file group b. It should be noted that the present disclosure does not specifically limit the specific method of dividing multiple files in the type file list into at least one file group according to the file specification information.

[0087] Through the above steps S510 to S520, the association information corresponding to multiple files in the first file list to be downloaded can be obtained, and the multiple files can be divided into at least one type file list according to the association information corresponding to the files, and the multiple files in the type file list can be divided into at least one file group according to the specification information of the files.

[0088] Step S130, using a streaming framework to download multiple files in at least one file group, and merging the multiple files in the file group into file blocks;

[0089] In an exemplary embodiment of the present disclosure, after obtaining at least one file group through the above steps, a streaming framework can be used to download multiple files in the file group. Specifically, a streaming framework can continuously process data sets and significantly reduce latency. It has advantages such as high performance, high fault tolerance, state management, and functional primitives. Streaming frameworks such as Storm, Spark Streaming, Samza, and Flink can be used. This disclosure does not specifically limit the specific type of streaming framework used.

[0090] In an exemplary embodiment of the present disclosure, when downloading multiple files in a file group using a streaming framework, the multiple files in multiple file groups can be downloaded simultaneously using different download nodes. Alternatively, the multiple files in a file group can be downloaded using different download nodes. After the multiple files in a file group are downloaded, the multiple files in the next file group are downloaded. It should be noted that the present disclosure does not specifically limit the specific method for downloading multiple files in a file group using a streaming framework.

[0091] In an exemplary embodiment of the present disclosure, after downloading multiple files in at least one file group through the above steps, the multiple files in the file group can be merged into file blocks. Specifically, because the file groups are divided according to preset file block specifications, files belonging to the same file group can be directly merged into file blocks. In this case, the specifications of the file blocks meet the preset file block specifications. It should be noted that the present disclosure does not specifically limit the specific method for merging multiple files in a file group into file blocks.

[0092] Step S140: storing the file blocks in the storage area of ​​the data platform to complete the resource collection process.

[0093] In an exemplary embodiment of the present disclosure, after obtaining the file blocks through the above steps, the file blocks can be stored in the data platform's storage area, completing the resource acquisition process. Furthermore, after resource acquisition is complete, a notification of the completion of resource acquisition can be sent to downstream business parties, who can then use these resources for business computing and analysis.

[0094] It should be noted that the present disclosure does not impose any particular limitation on the method of storing the file blocks in the storage area of ​​the data receipt.

[0095] In an exemplary embodiment of the present disclosure, after the multiple files in the file group are merged into file blocks through the above steps, verification data for the file blocks can be generated. The verification data includes verification specification information for the multiple files in the file group and verification specification information for the file blocks. After the download is complete, the downloaded file blocks need to be stored in a storage area of ​​the data platform. The verification specification information of the files and the verification specification information of the file blocks can be used to verify whether errors occurred during the storage process of the downloaded file blocks. It should be noted that the present disclosure does not specifically limit the method of verification based on the verification data of the file blocks.

[0096] In an exemplary embodiment of the present disclosure, the file blocks can be stored in the storage area of ​​the data platform, and the file information corresponding to the file blocks can be updated to the cache directory corresponding to the data platform. The file blocks in the storage area are verified according to the verification data of the file blocks. After the verification of the file blocks in the storage area is successful, the file information corresponding to the file blocks is updated to the official directory corresponding to the data platform. Figure 6 As shown, updating the file information corresponding to the file block to the official directory corresponding to the data platform may include the following steps S610 to S630:

[0097] Step S610: Store the file block in the storage area of ​​the data platform, and update the file information corresponding to the file block to the cache directory corresponding to the data platform;

[0098] In an exemplary embodiment of the present disclosure, after downloading the file block through the above steps, the file block can be stored in the storage area of ​​the data platform, and the file information corresponding to the file block can be updated to the cache directory corresponding to the data platform. Specifically, the file information corresponding to the file block is used to represent the indication information of the multiple files included in the file block. For example, the indication information of the file may include the file name, file ID, etc. It should be noted that the present disclosure does not specifically limit the specific form of the file information corresponding to the file block. After storing the file block in the storage area of ​​the data platform, the above file information can be updated in the cache directory corresponding to the data platform, that is, the current cache file can be viewed in the cache directory of the data platform.

[0099] Step S620, verifying the file blocks in the storage area according to the verification data of the file blocks;

[0100] Step S630: After the file block in the storage area is successfully verified, the file information corresponding to the file block is updated to the official directory corresponding to the data platform.

[0101] In an exemplary embodiment of the present disclosure, after obtaining the verification data of the file block through the above steps, the file block stored in the storage area of ​​the data platform can be verified based on the verification data of the file block. For example, the specification information of the file stored in the storage area of ​​the data platform can be compared with the verification data of the file block (the verification specification information of the file block). When the file stored in the storage area of ​​the data platform is the same as the verification data of the file block (the verification specification information of the file block), it means that there is no error in the storage process of the file. At this time, it is determined that the file block in the storage area has passed the verification, and the file information corresponding to the file block can be updated to the official directory corresponding to the data platform, indicating that the file block has been successfully stored in the storage area of ​​the data platform.

[0102] It should be noted that the present disclosure does not impose any special limitation on the specific manner of verifying the file blocks in the storage area according to the verification data of the file blocks.

[0103] Through the above steps S610 to S630, the file blocks can be stored in the storage area of ​​the data platform, and the file information corresponding to the file blocks can be updated to the cache directory corresponding to the data platform. The file blocks in the storage area are verified according to the verification data of the file blocks. After the file blocks in the storage area are successfully verified, the file information corresponding to the file blocks is updated to the official directory corresponding to the data platform.

[0104] In an exemplary embodiment of the present disclosure, the specification information corresponding to the file blocks in the storage area can be obtained, and the file blocks in the storage area can be verified according to the verification specification information of multiple files and the specification information corresponding to the file blocks in the storage area. Figure 7 As shown, verifying the file blocks in the storage area according to the verification specification information of multiple files and the specification information corresponding to the file blocks in the storage area may include the following steps S710 to S720:

[0105] Step S710, obtaining specification information corresponding to the file blocks in the storage area;

[0106] In an exemplary embodiment of the present disclosure, specification information corresponding to a file block in a storage area can be obtained. Specifically, the specification information corresponding to a file block can be used to indicate numerical information about the memory occupied by the file block. For example, this information can include the size of the file block, the number of rows, the number of entries, etc. It should be noted that the present disclosure does not specifically limit the specific type of the file block specification information.

[0107] Step S720 , verifying the file blocks in the storage area according to the verification specification information of the multiple files and the specification information corresponding to the file blocks in the storage area.

[0108] In an exemplary embodiment of the present disclosure, after obtaining the specification information corresponding to the file block through the above steps, the file block in the storage area can be verified based on the verification specification information of multiple files and the specification information corresponding to the file block in the storage area. Specifically, the verification data of the file block includes the verification specification information of the multiple files corresponding to the file block. The verification specification information of the multiple files corresponding to the file block may include the size, number of rows, number of entries, etc. of the files. It should be noted that the present disclosure does not specifically limit the specific type of file verification specification information.

[0109] For example, if the verification specification information of a file and the specification information corresponding to the file block are file sizes, the verification specification information (file sizes) of multiple files can be added together to obtain total verification specification information, which is then compared with the specification information (file sizes) corresponding to the file blocks in the storage area. If the total verification specification information (file size) is the same as the specification information (file size) corresponding to the file blocks in the storage area, then the verification of the file blocks in the storage area is successful. It should be noted that the present disclosure does not specifically limit the method for verifying the file blocks in the storage area based on the verification specification information of multiple files and the specification information corresponding to the file blocks in the storage area.

[0110] Through the above steps S710 to S720, the specification information corresponding to the file blocks in the storage area can be obtained, and the file blocks in the storage area can be verified according to the verification specification information of multiple files and the specification information corresponding to the file blocks in the storage area.

[0111] In an exemplary embodiment of the present disclosure, after the file block verification in the storage area fails, the verification status corresponding to the file information of the file block in the cache directory corresponding to the data platform is updated to failure, and a second list of files to be downloaded is generated according to the verification status. The download and verification process is performed according to the second list of files to be downloaded until the file block corresponding to the second list of files to be downloaded succeeds in verification. Figure 8 As shown, the download and verification process is executed according to the second list of files to be downloaded until the file blocks corresponding to the second list of files to be downloaded are successfully verified, which may include the following steps S810 to S820:

[0112] Step S810: After the file block verification in the storage area fails, the verification status corresponding to the file information of the file block in the cache directory corresponding to the data platform is updated to failure;

[0113] In an example embodiment of the present disclosure, when verifying a file block in a storage area, there may be a situation where the verification fails. For example, the verification specification information (file size) of the file block is different from the specification information (file size) corresponding to the file block in the storage area, and the file block verification in the storage area fails. The verification status corresponding to the file information of the file block in the cache directory corresponding to the data platform can be updated to failure. Specifically, in the cache target of the data platform, the verification status of the file block can be marked, and the verification status can include verifying, success, and failure. When the verification status corresponding to the file information of the file block in the cache directory corresponding to the data platform is updated to failure, it means that the verification of the file block and the multiple files corresponding to the file block has failed, and the multiple files in the file block need to be downloaded again.

[0114] Step S820: Generate a second list of files to be downloaded according to the verification status, and execute the download and verification process according to the second list of files to be downloaded until the file blocks corresponding to the second list of files to be downloaded are successfully verified.

[0115] In an example embodiment of the present disclosure, after the verification status of the file information corresponding to multiple files of the file block that failed verification is updated to verification failure through the above steps, the cache directory corresponding to the data platform can be scanned, and the files with verification status failed can be summarized to obtain a second list of files to be downloaded, and the files obtained in the second list of files to be downloaded can be grouped according to the scheme of the present disclosure, and a streaming framework is used to download multiple files, and the multiple files are merged into file blocks, and the file blocks are stored in the storage area of ​​the data platform, and the file information corresponding to the file blocks is updated to the cache directory corresponding to the data platform, and the above verification process is executed. When the verification is successful, the file information corresponding to the file blocks is updated to the official directory corresponding to the data platform.

[0116] Furthermore, when the number of times that the file block check fails in the storage area exceeds a preset threshold, an alarm is issued.

[0117] Through the above steps S810 to S820, after the file block verification in the storage area fails, the verification status corresponding to the file information of the file block in the cache directory corresponding to the data platform is updated to failure, and a second list of files to be downloaded is generated based on the verification status. The download and verification process is executed according to the second list of files to be downloaded until the file block corresponding to the second list of files to be downloaded is successfully verified.

[0118] In a resource collection method provided by an embodiment of the present disclosure, a download list can be obtained in real time, and file blocks can be downloaded through a streaming framework to complete the resource collection process. On the one hand, real-time file collection and downloading through a streaming framework can meet real-time requirements and avoid the high latency problem in data collection in the prior art. On the other hand, the steps of resource downloading can be distributed to different time points, avoiding the problem of high network pressure when centralized resource collection in the prior art, and can better control the network transmission rate. On the other hand, downloading files according to preset file block specifications improves the storage efficiency of the data platform, thereby improving the data throughput capacity of the data platform.

[0119] In an exemplary embodiment of the present disclosure, Figure 9 As shown, the to-be-processed file acquisition module can scan the FTP / SFTP server in real time to obtain a to-be-processed file list. The download status determination module determines a first to-be-downloaded file list based on the download status of multiple files in the to-be-processed file list, and sends the obtained first to-be-downloaded file list to the file grouping module. In the file grouping module, the multiple files in the first to-be-downloaded file list are divided into at least one file group based on the file specification information, and the at least one file group is written into a message queue at group granularity. The streaming download module obtains the at least one file group from the message queue and uses a streaming framework to download multiple files in the at least one file group. The file merging module merges the multiple files in the at least one file group into file blocks, stores the merged file blocks in a storage area of ​​the data platform, writes verification messages corresponding to verification data of the file blocks into the message queue, and updates file information corresponding to the file blocks to a cache directory corresponding to the data platform. The verification message is obtained from the message queue, and the single file block is verified in the single file verification module. After the verification is successful, the multiple file blocks obtained in this scan are verified in the batch file verification module. After the verification is successful, the file information corresponding to the multiple file blocks obtained in this scan is updated to the official directory corresponding to the data platform.

[0120] It should be noted that the above figures are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0121] In addition, in an exemplary embodiment of the present disclosure, a resource collection device is also provided. Figure 10 As shown, a resource collection device 1000 includes: a file to be downloaded determination module 1010 , a file group division module 1020 , a streaming download module 1030 , and a resource storage module 1040 .

[0122] Among them, the module for determining files to be downloaded is used to scan the resource server in real time to obtain a list of files to be processed, and determine a first list of files to be downloaded based on the list of files to be processed; wherein, the first list of files to be downloaded includes specification information of multiple files in the first list of files to be downloaded; the file group division module is used to divide multiple files in the first list of files to be downloaded into at least one file group according to the specification information of the files; wherein, the specification information of the file group meets the preset file block specification; the streaming download module is used to download multiple files in at least one file group using a streaming framework, and merge multiple files in the file group into file blocks; the resource storage module is used to store the file blocks in the storage area of ​​the data platform to complete the resource collection process.

[0123] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, a resource server is scanned in real time to obtain a list of files to be processed, and the device also includes: a compliance information acquisition unit, which is used to scan the resource server in real time to obtain a total file list, obtain multiple files in the total file list and compliance information corresponding to the files; a first file list to be processed determination unit, which is used to determine the file list to be processed based on the compliance information corresponding to the files.

[0124] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, a resource server is scanned in real time to obtain a list of files to be processed, and the device also includes: a first specification information acquisition unit, which is used to scan the resource server in real time to obtain a total file list, and obtain multiple files in the total file list and the first specification information corresponding to the files; a comparison result determination unit, which is used to compare the first specification information with the multiple files in the total file list obtained by the previous real-time scan and the second specification information corresponding to the files to obtain a comparison result; and a second file list to be processed determination unit, which is used to determine the file list to be processed based on the comparison result.

[0125] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, a first list of files to be downloaded is determined according to the list of files to be processed, and the device further includes: a download status acquisition unit, used to obtain the download status of multiple files in the list of files to be processed; wherein the download status includes downloading, downloading completed, and waiting to be downloaded; a first list of files to be downloaded determination unit, used to determine the first list of files to be downloaded according to the download status of multiple files.

[0126] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, multiple files in the first list of files to be downloaded are divided into at least one file group according to the specification information of the files, and the device also includes: a type file list determination unit, used to obtain the associated information corresponding to the multiple files in the first list of files to be downloaded, and divide the multiple files into at least one type file list according to the associated information corresponding to the files; a file group division unit, used to divide the multiple files in the type file list into at least one file group according to the specification information of the files.

[0127] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, after the multiple files in the file group are merged into file blocks, the device also includes: a verification data generation unit, which is used to generate verification data for the file blocks; wherein the verification data includes verification specification information of the multiple files in the file group and verification specification information of the file blocks.

[0128] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, the file blocks are stored in the storage area of ​​the data platform, and the device also includes: a cache directory update unit, which is used to store the file blocks in the storage area of ​​the data platform, and update the file information corresponding to the file blocks to the cache directory corresponding to the data platform; a file block verification unit, which is used to verify the file blocks in the storage area according to the verification data of the file blocks; and a formal directory update unit, which is used to update the file information corresponding to the file blocks to the formal directory corresponding to the data platform after the file blocks in the storage area are successfully verified.

[0129] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, the file blocks in the storage area are verified according to the verification data of the file blocks, and the device further includes: a file block specification acquisition unit, used to obtain specification information corresponding to the file blocks in the storage area; and a file block verification unit, used to verify the file blocks in the storage area according to the verification specification information of multiple files and the specification information corresponding to the file blocks in the storage area.

[0130] In an exemplary embodiment of the present disclosure, based on the aforementioned scheme, the device also includes: a verification status updating unit, which is used to update the verification status corresponding to the file information of the file block in the cache directory corresponding to the data platform to failure after the file block verification in the storage area fails; a re-verification unit, which is used to generate a second list of files to be downloaded according to the verification status, and execute the download and verification process according to the second list of files to be downloaded until the file block corresponding to the second list of files to be downloaded is successfully verified.

[0131] In an exemplary embodiment of the present disclosure, based on the above solution, the apparatus further includes: an alarm unit, configured to issue an alarm when the number of file block check failures in the storage area exceeds a preset threshold.

[0132] Since the functional modules of the resource acquisition device of the exemplary embodiment of the present disclosure correspond to the steps of the exemplary embodiment of the resource acquisition method described above, for details not disclosed in the embodiment of the device of the present disclosure, please refer to the embodiment of the resource acquisition method described above.

[0133] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0134] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above-mentioned resource acquisition method is also provided.

[0135] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0136] Refer to the following Figure 11 1100 according to this embodiment of the present disclosure is described. Figure 11 The electronic device 1100 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0137] like Figure 11As shown, electronic device 1100 is implemented as a general-purpose computing device. Components of electronic device 1100 may include, but are not limited to, the aforementioned at least one processing unit 1110, the aforementioned at least one storage unit 1120, a bus 1130 connecting various system components (including storage unit 1120 and processing unit 1110), and a display unit 1140.

[0138] The storage unit stores program codes, which can be executed by the processing unit 1110, so that the processing unit 1110 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 1110 can perform the following steps: Figure 1 In step S110 shown in the figure, a resource server is scanned in real time to obtain a list of files to be processed, and a first list of files to be downloaded is determined based on the list of files to be processed; wherein, the first list of files to be downloaded includes specification information of multiple files in the first list of files to be downloaded; in step S120, multiple files in the first list of files to be downloaded are divided into at least one file group based on the specification information of the files; wherein, the specification information of the file group meets the preset file block specification; in step S130, multiple files in the at least one file group are downloaded using a streaming framework, and the multiple files in the file group are merged into file blocks; in step S140, the file blocks are stored in the storage area of ​​the data platform to complete the resource acquisition process.

[0139] For example, electronic devices can achieve Figure 1 The steps shown.

[0140] The storage unit 1120 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 1121 and / or a cache memory unit 1122 , and may further include a read-only memory unit (ROM) 1123 .

[0141] The storage unit 1120 may also include a program / utility 1124 having a set (at least one) of program modules 1125, such program modules 1125 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0142] The bus 1130 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0143] Electronic device 1100 can also communicate with one or more external devices 1170 (e.g., a keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1100, and / or any device that enables electronic device 1100 to communicate with one or more other computing devices (e.g., a router, modem, etc.). Such communication can occur via input / output (I / O) interface 1150. Furthermore, electronic device 1100 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via network adapter 1160. As shown, network adapter 1160 communicates with other modules of electronic device 1100 via bus 1130. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with electronic device 1100, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0144] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a device, etc.) to execute the method according to the embodiments of the present disclosure.

[0145] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which is stored a program product capable of implementing the aforementioned methods of this specification. In some possible embodiments, various aspects of the present disclosure may also be implemented in the form of a program product, which includes program code. When the program product is executed on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the "Exemplary Methods" section of this specification.

[0146] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0147] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0148] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0149] Furthermore, the above-mentioned figures are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the above-mentioned figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0150] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

Claims

1. A resource acquisition method, characterized in that: The method comprises: Scan the resource server in real time to obtain a list of files to be processed, and determine a first list of files to be downloaded based on the list of files to be processed; wherein the first list of files to be downloaded includes specification information of multiple files in the first list of files to be downloaded; Dividing the plurality of files in the first list of files to be downloaded into at least one file group according to the specification information of the files; wherein the specification information of the file group complies with a preset file block specification; Downloading multiple files in the at least one file group using a streaming framework, and merging the multiple files in the file group into file blocks; Storing the file blocks in a storage area of ​​the data platform to complete the resource collection process; Among them, the list of files to be processed is obtained in the following manner: real-time scanning of the resource server to obtain the total file list, obtaining multiple files in the total file list and the first specification information corresponding to the files; comparing the first specification information with the multiple files in the total file list obtained by the previous real-time scan and the second specification information corresponding to the files to obtain a comparison result; determining the list of files to be processed based on the comparison result; wherein, the comparison result is used to indicate the change in the specification information of the file between two scans, and the first specification information is used to indicate the numerical information of the memory occupied by the file.

2. The method according to claim 1, characterized in that The determining a first list of files to be downloaded according to the list of files to be processed includes: Obtaining the download status of multiple files in the pending file list; wherein the download status includes downloading, download completed, and pending download; A first list of files to be downloaded is determined according to the download status of the multiple files.

3. The method according to claim 1, characterized in that After merging the multiple files in the file group into a file block, the method further includes: Generate verification data for the file block; wherein the verification data includes verification specification information of multiple files in the file group and verification specification information of the file block.

4. The method according to claim 3, characterized in that Storing the file block in a storage area of ​​the data platform includes: The file block is stored in the storage area of ​​the data platform, and the file information corresponding to the file block is updated in the cache directory corresponding to the data platform; verifying the file blocks in the storage area according to the verification data of the file blocks; After the file block in the storage area is successfully verified, the file information corresponding to the file block is updated to the official directory corresponding to the data platform.

5. The method according to claim 4, characterized in that The verifying the file blocks in the storage area according to the verification data of the file blocks includes: Obtaining specification information corresponding to the file blocks in the storage area; The file blocks in the storage area are verified according to the verification specification information of the multiple files and the specification information corresponding to the file blocks in the storage area.

6. The method according to claim 4, characterized in that The method further comprises: After the file block verification in the storage area fails, updating the verification status corresponding to the file information of the file block in the cache directory corresponding to the data platform to failure; A second list of files to be downloaded is generated according to the verification status, and a download and verification process is performed according to the second list of files to be downloaded until the file blocks corresponding to the second list of files to be downloaded are successfully verified.

7. A resource collection device, characterized in that: The device comprises: a file-to-be-downloaded determining module configured to scan a resource server in real time to obtain a list of files to be processed, and determine a first list of files to be downloaded based on the list of files to be processed; wherein the first list of files to be downloaded includes specification information of multiple files in the first list of files to be downloaded; A file group division module, configured to divide the plurality of files in the first list of files to be downloaded into at least one file group according to the specification information of the files; wherein the specification information of the file group complies with a preset file block specification; a streaming download module, configured to download multiple files in the at least one file group using a streaming framework, and merge the multiple files in the file group into file blocks; A resource storage module is used to store the file blocks in the storage area of ​​the data platform to complete the resource collection process; Among them, the list of files to be processed is obtained in the following manner: real-time scanning of the resource server to obtain the total file list, obtaining multiple files in the total file list and the first specification information corresponding to the files; comparing the first specification information with the multiple files in the total file list obtained by the previous real-time scan and the second specification information corresponding to the files to obtain a comparison result; determining the list of files to be processed based on the comparison result; wherein, the comparison result is used to indicate the change in the specification information of the file between two scans, and the first specification information is used to indicate the numerical information of the memory occupied by the file.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: one or more processors; as well as The memory is configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 6.

Images