Data acquisition method and device, computer readable storage medium, processor

Abstract

The application discloses a data acquisition method and device, a computer readable storage medium and a processor. The method comprises the following steps: dividing predetermined data stored in a distributed file system HDFS into blocks to obtain a plurality of predetermined data blocks corresponding to the predetermined data; determining a to-be-sent data block corresponding to file start-stop position information and a to-be-sent file identifier of to-be-sent data when it is determined that a predetermined file identifier is inconsistent with an initial file identifier; sending the to-be-sent data block and the to-be-sent file identifier to a client; and sending part of data and the to-be-sent data block to a predetermined terminal as target data when it is determined that a current file identifier is consistent with the to-be-sent file identifier. The application solves the technical problem of the prior art that the HDFS for storing data needs to download the entire file to the local when downloading the file, and the downloading fails if network jitter occurs during the downloading process, and the resource is wasted.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a data acquisition method and apparatus, a computer-readable storage medium, and a processor. Background Technology

[0002] The sheer volume of data generated daily presents a significant challenge for storage, a problem addressed by HDFS. HDFS is a distributed file system designed to run on commodity hardware. It boasts high fault tolerance, is designed for deployment on low-cost hardware, provides high-throughput access to application data, and is well-suited for applications with large datasets. However, HDFS downloads files by downloading the entire file locally. This can lead to long download times, and network instability can cause download failures, requiring a re-download and wasting resources. When users need this data and attempt to download it, HDFS's API downloads the entire file locally. If a download fails due to network instability, the user is forced to re-download the entire file, resulting in a poor user experience.

[0003] Regarding the issue that HDFS, used for data storage in the aforementioned technologies, requires downloading the entire file to the local machine when downloading a file, and if network jitter occurs during the download process, the download will fail and have to be re-downloaded, wasting resources, there is currently no effective solution. Summary of the Invention

[0004] This invention provides a data acquisition method and apparatus, a computer-readable storage medium, and a processor to at least solve the technical problem in the related art where HDFS, used for storing data, requires downloading the entire file to the local machine when downloading a file. If network jitter occurs during the download process, the download will fail and have to be re-downloaded, wasting resources.

[0005] According to one aspect of the present invention, a data acquisition method is provided, comprising: receiving data request information sent by a client, wherein the data request information is used to request the download of target data; when the data request information is file start and end position information and an initial file identifier, dividing predetermined data stored in the HDFS distributed file system based on the file start and end file information to obtain a plurality of predetermined data blocks corresponding to the predetermined data, wherein the predetermined data is data in the target data excluding data already existing locally, the file start and end position information includes: the end position of the partial data, the end position of the target data, and the initial file identifier is the value of the partial data stored locally. The system obtains a file identifier for each of the multiple partial data blocks obtained by row segmentation; determines a predetermined file identifier for each of the multiple predetermined data blocks; when the predetermined file identifier is inconsistent with the initial file identifier, determines the data block to be sent corresponding to the file start and end position information and the file identifier to be sent for the data to be sent; and sends the data block to be sent and the file identifier to be sent to the client. Upon receiving the data block to be sent and the file identifier to be sent, the client calculates the current file identifier of the data block to be sent, and when the current file identifier is consistent with the file identifier to be sent, sends the partial data and the data block to be sent together as the target data to the predetermined terminal.

[0006] Optionally, when the data request information is file start and end position information and an initial file identifier, the predetermined data stored in the HDFS distributed file system is divided into blocks based on the file start and end file information to obtain multiple predetermined data blocks corresponding to the predetermined data. This includes: parsing the data request information to obtain the file start and end position information and the initial file identifier; searching for the predetermined data in the HDFS; and dividing the predetermined data into blocks on the HDFS based on the file start and end position information to obtain the multiple predetermined data blocks.

[0007] Optionally, when it is determined that the predetermined file identifier is inconsistent with the initial file identifier, determining the data block to be sent corresponding to the file start and end position information and the file identifier to be sent for the data block to be sent includes: comparing the predetermined file identifier with the initial file identifier to obtain a comparison result; when the comparison result indicates that there is an inconsistent file identifier between the predetermined file identifier and the initial file identifier, determining the inconsistent file identifier between the predetermined file identifier and the initial file identifier as the target file identifier; and determining the data block to be sent based on the target file identifier and the start and end position information.

[0008] Optionally, sending the data block to be sent and the file identifier to be sent to the client includes: fragmenting the data block to be sent on the HDFS to obtain multiple data fragments corresponding to the data block to be sent; sending the multiple data fragments and the file identifier to be sent to the client, so that when the client determines that the file identifiers of the multiple data fragments are consistent with the file identifier to be sent, it sends the partial data together with the data block to be sent as the target data to the predetermined terminal.

[0009] Optionally, the data acquisition method further includes: when the data request information is the file path of the target data, acquiring the target data from a predetermined storage medium according to the file path; dividing the target data into blocks on the HDFS to obtain multiple target data blocks corresponding to the target data; and sending the multiple target data blocks to the client so that the client can send the multiple target data blocks as the target data to the predetermined terminal.

[0010] Optionally, sending the plurality of target data blocks to the client includes: recording the start and end position information of each of the plurality of target data blocks; performing slicing processing on each of the plurality of target data blocks to obtain target data slices of each of the plurality of target data blocks; recording the start and end position information of the target data slices of each of the plurality of target data blocks; and sending the start and end position information of each of the plurality of target data blocks, the target data slices of each of the plurality of target data blocks, and the start and end position information of the target data slices of each of the plurality of target data blocks to the client.

[0011] Optionally, the data acquisition method further includes: while sending the start and end position information of each of the plurality of target data blocks, the target data slice of each of the plurality of target data blocks, and the start and end position information of the target data slice of each of the plurality of target data blocks to the client, determining the file identifier of the target data, the length of the target data, the byte array information of the target data slice of each of the plurality of target data blocks, and the total number of data slices; sending the file identifier of the target data, the length of the target data, the byte array information of the target data slice of each of the plurality of target data blocks, and the total number of data slices together to the client, wherein, after receiving the first target data slice, the client generates an empty file with the same length as the target data slice, skips the start position of the first target data slice, writes the byte array of the first target data slice to the empty file, and increments by 1 for each target data slice received.

[0012] According to another embodiment of the present invention, a data acquisition apparatus is also provided, comprising: a receiving unit, configured to receive data request information sent by a client, wherein the data request information is used to request the download of target data; and a block-splitting unit, configured to, when the data request information is file start and end position information and an initial file identifier, divide the predetermined data stored in the HDFS distributed file system into blocks based on the file start and end file information, thereby obtaining a plurality of predetermined data blocks corresponding to the predetermined data, wherein the predetermined data is data in the target data excluding the portion of data already existing locally, the file start and end position information includes: the end position of the portion of data, the end position of the target data, and the initial file identifier is obtained by dividing the portion of data stored locally into blocks. The system comprises: a file identifier for each of a plurality of predetermined data blocks; a first determining unit for determining a predetermined file identifier for each of the plurality of predetermined data blocks; a second determining unit for determining, when the predetermined file identifier is inconsistent with the initial file identifier, the data block to be sent corresponding to the file start and end position information and the file identifier to be sent for the data to be sent; and a sending unit for sending the data block to be sent and the file identifier to be sent to the client, wherein, after receiving the data block to be sent and the file identifier to be sent, the client calculates the current file identifier of the data block to be sent, and when the current file identifier is consistent with the file identifier to be sent, sends the partial data together with the data block to be sent as the target data to a predetermined terminal.

[0013] Optionally, the block division unit includes: a parsing module, used to parse the data request information to obtain the file start and end position information and the initial file identifier; a search module, used to search for the predetermined data in the HDFS; and a block division module, used to divide the predetermined data into blocks on the HDFS based on the file start and end position information to obtain the plurality of predetermined data blocks.

[0014] Optionally, the second determining unit includes: a comparison module, configured to compare the predetermined file identifier with the initial file identifier to obtain a comparison result; a first determining module, configured to determine the file identifier that is inconsistent between the predetermined file identifier and the initial file identifier as the target file identifier when the comparison result indicates that there is an inconsistent file identifier between the predetermined file identifier and the initial file identifier; and to determine the data block to be sent based on the target file identifier and the start and end position information.

[0015] Optionally, the sending unit includes: a fragmentation module, configured to fragment the data block to be sent on the HDFS to obtain multiple data fragments corresponding to the data block to be sent; and a first sending module, configured to send the multiple data fragments and the file identifier to be sent to the client, so that when the client determines that the file identifiers of the multiple data fragments are consistent with the file identifier to be sent, it will send the partial data together with the data block to be sent as the target data to the predetermined terminal.

[0016] Optionally, the data acquisition device further includes: an acquisition unit, configured to acquire the target data from a predetermined storage medium according to the file path when the data request information is the file path of the target data; a block division unit, further configured to divide the target data into blocks on the HDFS to obtain multiple target data blocks corresponding to the target data; and a sending unit, further configured to send the multiple target data blocks to the client, so as to use the client to send the multiple target data blocks as the target data to the predetermined terminal.

[0017] Optionally, the sending unit includes: a first recording module for recording the start and end position information of each of the plurality of target data blocks; a processing module for performing slicing processing on each of the plurality of target data blocks to obtain target data slices of each of the plurality of target data blocks; a second recording module for recording the start and end position information of the target data slices of each of the plurality of target data blocks; and a second sending module for sending the start and end position information of each of the plurality of target data blocks, the target data slices of each of the plurality of target data blocks, and the start and end position information of the target data slices of each of the plurality of target data blocks to the client.

[0018] Optionally, the data acquisition device further includes: a second determining unit, configured to determine the file identifier of the target data, the length of the target data, the byte array information of the target data slice of each of the plurality of target data blocks, and the total number of data slices while sending the start and end position information of each of the plurality of target data blocks, the target data slice of each of the plurality of target data blocks, and the start and end position information of the target data slice of each of the plurality of target data blocks to the client; the sending unit is further configured to send the file identifier of the target data, the length of the target data, the byte array information of the target data slice of each of the plurality of target data blocks, and the total number of data slices together to the client, wherein, after receiving the first target data slice, the client generates an empty file with the same length as the target data slice, skips the start position of the first target data slice, writes the byte array of the first target data slice to the empty file, and increments by 1 for each target data slice received.

[0019] According to another embodiment of the present invention, a computer-readable storage medium is also provided, characterized in that the computer-readable storage medium includes a stored program, wherein the program executes the data acquisition method described in any one of the above embodiments.

[0020] According to another embodiment of the present invention, a processor is also provided, the processor being used to run a program, wherein the program, when running, executes the data acquisition method described in any one of the above embodiments.

[0021] In this embodiment of the invention, a data request message sent by a client is received, wherein the data request message is used to request the download of target data; when the data request message is file start and end position information and an initial file identifier, the predetermined data stored in the HDFS distributed file system is divided into blocks based on the file start and end file information to obtain multiple predetermined data blocks corresponding to the predetermined data, wherein the predetermined data is the data in the target data excluding the data that already exists locally, the file start and end position information includes: the end position of the partial data, the end position of the target data, and the initial file identifier is the file identifier of each of the multiple partial data blocks obtained by dividing the partial data stored locally; the predetermined file identifier of each of the multiple predetermined data blocks is determined; when it is determined that the predetermined file identifier is inconsistent with the initial file identifier, the data block to be sent corresponding to the file start and end position information and the file identifier to be sent for the data to be sent are determined; the data block to be sent and the file identifier to be sent are sent to the client, wherein after receiving the data block to be sent and the file identifier to be sent, the client calculates the current file identifier of the data block to be sent, and when it is determined that the current file identifier is consistent with the file identifier to be sent, the partial data and the data block to be sent are sent together as target data to the predetermined terminal. The technical solution provided by this invention achieves the goal of finding the data that needs to be transmitted on HDFS after an interruption during the transmission of data requested by the client, and then transmitting this data in blocks. This method of transmitting data in blocks via HDFS breaks down the entire file into smaller parts, reduces the file transmission capacity, and improves the file transmission efficiency. This solves the technical problem in related technologies where HDFS, used for storing data, needs to download the entire file to the local machine when downloading a file. If network jitter occurs during the download process, the download will fail and have to be re-downloaded, wasting resources. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0023] Figure 1 This is a hardware structure block diagram of a mobile terminal for a data acquisition method according to an embodiment of the present invention.

[0024] Figure 2 This is a flowchart of a data acquisition method according to an embodiment of the present invention;

[0025] Figure 3 This is a flowchart of an optional data acquisition method according to an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of a data acquisition device according to an embodiment of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] As described in the background section, HDFS, used for storing data in related technologies, requires downloading the entire file to the local machine. If network fluctuations occur during the download process, the download will fail and must be restarted, wasting resources. Specifically, HDFS downloads a file only when the entire file is downloaded to the local machine; this can lead to excessive download time, and network fluctuations can cause download failures, requiring restarts and wasting resources. This invention addresses this by employing block-based transmission combined with MD5 checksums to achieve breakpoint resumption, reducing network resource pressure and waste. This invention provides a data acquisition method and apparatus, a computer-readable storage medium, and a processor.

[0030] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0031] The methods and embodiments provided in this invention can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a data acquisition method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0032] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data acquisition method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] According to an embodiment of the present invention, a method embodiment for data acquisition is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] Figure 2 This is a flowchart of a data acquisition method according to an embodiment of the present invention, such as... Figure 2 As shown, the data acquisition method includes the following steps:

[0035] Step S202: Receive data request information sent by the client, wherein the data request information is used to request the download of target data.

[0036] In this embodiment, the server can receive data request information sent by the client, wherein the data request information may be information sent by the user through a terminal device to request the download of target data.

[0037] Step S204: When the data request information is file start and end position information and initial file identifier, the predetermined data stored in HDFS is divided into blocks in the distributed file system HDFS based on the file start and end file information to obtain multiple predetermined data blocks corresponding to the predetermined data. The predetermined data is the data in the target data excluding the data that already exists locally. The file start and end position information includes: the end position of the partial data and the end position of the target data. The initial file identifier is the file identifier of each of the multiple partial data blocks obtained by dividing the partial data stored locally.

[0038] In this embodiment, the server can parse the data request information to determine what content is included in the data request information.

[0039] When the data request information contains the start and end positions of the target file and the initial file identifier, it indicates that the target data was interrupted during transmission due to network jitter or other factors. In this case, the interruption resume process can be used.

[0040] During the breakpoint resume process, the server can search for the breakpoint resume data from its own stored data based on the breakpoint location information, that is, the starting position of the data to be continued (the ending position of the data that has been successfully transmitted) and the ending position of the target data. Then, it can divide this part of the data into blocks on HDFS to obtain multiple data blocks, that is, multiple predetermined data blocks.

[0041] Step S206: Determine the predefined file identifier for each of the plurality of predefined data blocks.

[0042] After obtaining multiple predetermined data blocks in the above manner, a file identifier for each predetermined data block can be generated, that is, a predetermined file identifier.

[0043] Step S208: When it is determined that the predetermined file identifier is inconsistent with the initial file identifier, the data block to be sent corresponding to the file start and end position information and the file identifier to be sent for the data to be sent are determined.

[0044] In this embodiment, the server can compare the initial file identifier sent by the local client with the predetermined file identifier to determine whether they are consistent; and if they are inconsistent, the server can determine the data block to be sent corresponding to the file start and end position information and the file identifier to be sent for the data to be sent.

[0045] Step S210: Send the data block to be sent and the file identifier to be sent to the client. After receiving the data block to be sent and the file identifier to be sent, the client calculates the current file identifier of the data block to be sent. When it is determined that the current file identifier is consistent with the file identifier to be sent, the client sends part of the data together with the data block to be sent as target data to the predetermined terminal.

[0046] As described above, in the embodiments of the present invention, a data request message sent by a client can be received, wherein the data request message is used to request the download of target data; when the data request message is file start and end position information and an initial file identifier, the predetermined data stored in the HDFS distributed file system is divided into blocks based on the file start and end file information to obtain multiple predetermined data blocks corresponding to the predetermined data, wherein the predetermined data is the data in the target data excluding the data that already exists locally, the file start and end position information includes: the end position of the partial data, the end position of the target data, and the initial file identifier is the file identifier of each of the multiple partial data blocks obtained by dividing the partially stored data locally; the predetermined file identifier of each of the multiple predetermined data blocks is determined; when determining the predetermined file identifier... When the file identifier is inconsistent with the initial file identifier, the system determines the data block to be sent corresponding to the file start and end position information, as well as the file identifier to be sent for the data to be sent. The data block to be sent and the file identifier to be sent are then sent to the client. After receiving the data block to be sent and the file identifier to be sent, the client calculates the current file identifier of the data block to be sent. If the current file identifier matches the file identifier to be sent, the client sends a portion of the data along with the data block to be sent as target data to the predetermined terminal. This achieves the goal of finding the data that needs to be transmitted after interruption during the transmission of data requested by the client in HDFS, and then transmitting this portion of data in blocks. This method of transmitting data in blocks through HDFS breaks down the entire file into smaller parts, reduces the file transmission capacity, and improves the file transmission efficiency.

[0047] Therefore, the technical solution provided by the embodiments of the present invention solves the technical problem in the related art that HDFS, used for storing data, needs to download the entire file to the local machine when downloading a file. If network jitter occurs during the download process, the download will fail and can only be re-downloaded, wasting resources.

[0048] According to the above embodiments of the present invention, when the data request information is file start and end position information and initial file identifier, the predetermined data stored in HDFS is divided into blocks in the distributed file system HDFS based on the file start and end file information to obtain multiple predetermined data blocks corresponding to the predetermined data. This may include: parsing the data request information to obtain file start and end position information and initial file identifier; searching for predetermined data in HDFS; and dividing the predetermined data into blocks in HDFS based on the file start and end position information to obtain multiple predetermined data blocks.

[0049] In this embodiment, the data request information is parsed to obtain the file start and end position information and the initial file identifier; at the same time, the predetermined data, that is, the data that needs to be resumed from the breakpoint, is searched in HDFS; and the predetermined data is divided into blocks on HDFS based on the file start and end position information to obtain multiple predetermined data blocks.

[0050] According to the above embodiments of the present invention, when it is determined that the predetermined file identifier and the initial file identifier are inconsistent, determining the data block to be sent corresponding to the file start and end position information and the file identifier to be sent for the data to be sent includes: comparing the predetermined file identifier with the initial file identifier to obtain a comparison result; when the comparison result indicates that there is an inconsistent file identifier between the predetermined file identifier and the initial file identifier, determining the inconsistent file identifier between the predetermined file identifier and the initial file identifier as the target file identifier; and determining the data block to be sent based on the target file identifier and the start and end position information.

[0051] In this embodiment, the predetermined file identifier can be compared with the initial file identifier. If the comparison result indicates that they are inconsistent, the target file identifier that is inconsistent with the initial file identifier is determined. Then, the data block to be sent is determined based on the target file identifier and the start and end position information.

[0052] According to the above embodiments of the present invention, sending the data block to be sent and the file identifier to be sent to the client may include: fragmenting the data block to be sent on HDFS to obtain multiple data fragments corresponding to the data block to be sent; sending the multiple data fragments and the file identifier to be sent to the client, so that when the client determines that the file identifiers of the multiple data fragments are consistent with the file identifier to be sent, it will send a portion of the data together with the data block to be sent as target data to a predetermined terminal.

[0053] In this embodiment, the data to be sent can be fragmented on HDFS to obtain the poison attack data fragments corresponding to the data block to be sent. Multiple data fragments and the file identifier to be sent are sent to the client so that when the client determines that the file identifiers of multiple data fragments are consistent with the file identifier to be sent, it can send part of the data together with the data block to be sent as target data to the predetermined terminal.

[0054] According to the above embodiments of the present invention, the data acquisition method may further include: when the data request information is the file path of the target data, acquiring the target data from a predetermined storage medium according to the file path; dividing the target data into blocks on HDFS to obtain multiple target data blocks corresponding to the target data; and sending the multiple target data blocks to the client so that the client can send the multiple target data blocks as target data to a predetermined terminal.

[0055] In this embodiment, if it is determined that the client is sending the file path of the target data, the target data can be obtained from the predetermined storage medium according to the file path, and the target data can be divided into blocks on HDFS to obtain multiple target data blocks corresponding to the target data. The multiple target data blocks are then sent to the client so that the client can send the multiple target data blocks to the predetermined terminal.

[0056] According to the above embodiments of the present invention, sending multiple target data blocks to a client includes: recording the start and end position information of each of the multiple target data blocks; performing slicing processing on each of the multiple target data blocks to obtain target data slices of each of the multiple target data blocks; recording the start and end position information of the target data slices of each of the multiple target data blocks; and sending the start and end position information of each of the multiple target data blocks, the target data slices of each of the multiple target data blocks, and the start and end position information of the target data slices of each of the multiple target data blocks to the client.

[0057] In this embodiment, when the server sends multiple target data blocks to the client, it can record the start and end position information of each of the multiple target data blocks, perform slicing processing on each of the multiple target data blocks to obtain target data slices of each of the multiple target data blocks, and record the start and end position information of the multiple target data slices, the target data slices of each of the multiple target data blocks, and the start and end position information of the target data slices of each of the multiple target data blocks and send them to the client.

[0058] According to the above embodiments of the present invention, the data acquisition method further includes: while sending the start and end position information of each of the plurality of target data blocks, the target data slice of each of the plurality of target data blocks, and the start and end position information of the target data slice of each of the plurality of target data blocks to the client, determining the file identifier of the target data, the length of the target data, the byte array information of the target data slice of each of the plurality of target data blocks, and the total number of data slices; sending the file identifier of the target data, the length of the target data, the byte array information of the target data slice of each of the plurality of target data blocks, and the total number of data slices together to the client, wherein, after receiving the first target data slice, the client generates an empty file with the same length as the target data slice, skips the start position of the first target data slice, writes the byte array of the first target data slice to the empty file, and increments by 1 for each target data slice received.

[0059] In this embodiment, after the client receives the first slice, it can first generate an empty file of the same length, then skip the starting position and write the byte array to the file, incrementing by 1 for each slice received.

[0060] Figure 3 This is a flowchart of an optional data acquisition method according to an embodiment of the present invention, such as... Figure 3As shown, we can first write the method name `resumeDownload(full path to local file, full path to remote file)`; then, we check if the file exists locally based on the local file path; if the file exists locally, we divide the local file into blocks, record the start and end positions, and verify each block to generate an MD5 hash, sending this information to the server; the server divides the file into blocks on HDFS based on the start and end position information sent by the client, and calculates the MD5 hash of each block; we compare the MD5 values ​​of each block between the client and the server, and if the MD5 values ​​are inconsistent, we send that block separately, slice it, and send the MD5 hash of the complete file; after the client receives the file blocks, it calculates the MD5 hash of the complete file, and if the MD5 hash matches the one sent by the server, we resume the download. Successful reception; if the file is not found locally, the chunked transfer process is initiated directly. The client sends the full path of the remote file to the server. The server divides the file into chunks based on the file path, records the start and end positions of each chunk, slices the file within each chunk, records the start and end positions of each slice, and sends the start and end position information, byte array information, file length, complete MD5 value, and total number of chunks to the client. After receiving the first slice, the client first generates an empty file of the same length, then skips the start position and writes the byte array to the file, incrementing it by 1 for each received slice. If the number of slices received by the client equals the number of file slices sent, the client locally verifies the complete MD5 value of the file. If the MD5 verification is successful, the download is successful. If the MD5 verification fails, the next call to this method will directly follow the logic of resuming interrupted downloads.

[0061] The technical solution provided by this invention can encapsulate the functions of chunked transmission and resumable interrupted transmission into a method by encapsulating HDFS. Only one method name, resumeDownload(full path to local file, full path to remote file), is exposed externally. Internally, chunked transmission and resumable interrupted transmission are implemented, increasing transmission efficiency, saving time, improving user experience, and simplifying user development. This effectively overcomes the limitations of the existing HDFS file download method copyToLocalFile(false, src, dst, true), which can only download the entire file to the local machine. If network fluctuations occur, the entire file needs to be downloaded again, wasting time and bandwidth, resulting in a poor user experience.

[0062] By combining HDFS with message middleware, file fragmentation and block transmission are achieved. The MD5 value of each block is calculated, and the file is merged according to the merging rules. This allows for control of file transmission speed and completion of file integrity verification. If network anomalies occur during transmission, the file can be downloaded by using the fragment deduplication function and breakpoint recording information after network recovery, and by calculating MD5. It has the following beneficial effects: (1) HDFS block transmission breaks the entire file into smaller parts and transmits them in blocks. The size of each block is only 1 / N of the total file transmission capacity. Combined with multi-threading, the transmission rate is improved. (2) The fragment deduplication implementation logic reads the content of the HDFS file at a specified position, finds the fragment that has not been sent repeatedly, and combines MD5 verification to ensure successful file landing. (3) The HDFS block transmission and breakpoint resumption methods are encapsulated. Users do not need to perform secondary development and can directly call to achieve similar functions. (4) Transmission efficiency is improved and bandwidth is saved. Users do not need to worry about having to re-download the entire file due to network jitter, saving time.

[0063] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0064] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0065] According to embodiments of the present invention, a data acquisition apparatus for implementing the above-described data acquisition method is also provided. Figure 4 This is a schematic diagram of a data acquisition device according to an embodiment of the present invention, such as... Figure 4 As shown, the device includes: a receiving unit 41, a segmentation unit 43, a first determining unit 45, a second determining unit 47, and a sending unit 49. The data acquisition device will now be described.

[0066] The receiving unit 41 is used to receive data request information sent by the client, wherein the data request information is used to request the download of target data.

[0067] Segmentation unit 43 is used to segment the predetermined data stored in HDFS in the distributed file system HDFS based on the file start and end file information and the initial file identifier when the data request information is file start and end position information and the initial file identifier, to obtain multiple predetermined data blocks corresponding to the predetermined data. The predetermined data is the data in the target data excluding the data that already exists locally. The file start and end position information includes: the end position of the partial data and the end position of the target data. The initial file identifier is the file identifier of each of the multiple partial data blocks obtained by segmenting the partial data stored locally.

[0068] The first determining unit 45 is used to determine the predetermined file identifier for each of the plurality of predetermined data blocks.

[0069] The second determining unit 47 is used to determine the data block to be sent corresponding to the file start and end position information and the file identifier to be sent of the data to be sent when the predetermined file identifier is inconsistent with the initial file identifier.

[0070] The sending unit 49 is used to send the data block to be sent and the file identifier to be sent to the client. After receiving the data block to be sent and the file identifier to be sent, the client calculates the current file identifier of the data block to be sent. When it is determined that the current file identifier is consistent with the file identifier to be sent, the client sends part of the data together with the data block to be sent as target data to the predetermined terminal.

[0071] It should be noted that the receiving unit 41, the block division unit 43, the first determining unit 45, the second determining unit 47, and the sending unit 49 mentioned above correspond to steps S202 to S210 in the above embodiments. The five units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments.

[0072] As can be seen from the above, in the scheme described in the above embodiments of the present invention, a receiving unit can be used to receive data request information sent by a client, wherein the data request information is used to request the download of target data; when the data request information is file start and end position information and an initial file identifier, a block-splitting unit can divide the predetermined data stored in the HDFS distributed file system based on the file start and end file information to obtain multiple predetermined data blocks corresponding to the predetermined data, wherein the predetermined data is the data in the target data excluding the data that already exists locally, the file start and end position information includes: the end position of the partial data, the end position of the target data, and the initial file identifier is the file identifier of each of the multiple partial data blocks obtained by dividing the partially stored data locally; a first determining unit is used to determine the predetermined file identifier of each of the multiple predetermined data blocks; a second determining unit is used to determine the predetermined file identifier of each of the multiple predetermined data blocks. When the predetermined file identifier is inconsistent with the initial file identifier, the system determines the data block to be sent corresponding to the file start and end position information, as well as the file identifier to be sent for the data to be sent. The system then uses a sending unit to send the data block to be sent and the file identifier to be sent to the client. Upon receiving the data block to be sent and the file identifier to be sent, the client calculates the current file identifier of the data block to be sent. If the current file identifier matches the file identifier to be sent, the client sends a portion of the data along with the data block to be sent as target data to the predetermined terminal. This achieves the goal of finding the data that needs to be transmitted on HDFS after an interruption in the transmission of the data requested by the client, and then transmitting this portion of data in blocks. This method of transmitting data in blocks via HDFS breaks down the entire file into smaller parts, reducing file transmission capacity and improving file transmission efficiency.

[0073] Therefore, the technical solution provided by the embodiments of the present invention solves the technical problem in the related art that HDFS, used for storing data, needs to download the entire file to the local machine when downloading a file. If network jitter occurs during the download process, the download will fail and can only be re-downloaded, wasting resources.

[0074] Optionally, the block division unit includes: a parsing module for parsing the data request information to obtain file start and end position information and an initial file identifier; a search module for searching for the predetermined data in HDFS; and a block division module for dividing the predetermined data into blocks on HDFS based on the file start and end position information to obtain multiple predetermined data blocks.

[0075] Optionally, the second determining unit includes: a comparison module, used to compare the predetermined file identifier with the initial file identifier to obtain a comparison result; a first determining module, used to determine the file identifier that is inconsistent between the predetermined file identifier and the initial file identifier as the target file identifier when the comparison result indicates that there is an inconsistent file identifier between the predetermined file identifier and the initial file identifier; and to determine the data block to be sent based on the target file identifier and the start and end position information.

[0076] Optionally, the sending unit includes: a fragmentation module for fragmenting the data block to be sent on HDFS to obtain multiple data fragments corresponding to the data block to be sent; and a first sending module for sending the multiple data fragments and the file identifier to be sent to the client, so that when the client determines that the file identifiers of the multiple data fragments are consistent with the file identifier to be sent, it will send part of the data together with the data block to be sent as target data to a predetermined terminal.

[0077] Optionally, the data acquisition device further includes: an acquisition unit, configured to acquire target data from a predetermined storage medium according to the file path when the data request information is the file path of the target data; a block division unit, configured to divide the target data into blocks on HDFS to obtain multiple target data blocks corresponding to the target data; and a sending unit, configured to send the multiple target data blocks to the client so that the client can send the multiple target data blocks as target data to a predetermined terminal.

[0078] Optionally, the sending unit includes: a first recording module for recording the start and end position information of each of the multiple target data blocks; a processing module for performing slicing processing on each of the multiple target data blocks to obtain target data slices of each of the multiple target data blocks; a second recording module for recording the start and end position information of the target data slices of each of the multiple target data blocks; and a second sending module for sending the start and end position information of each of the multiple target data blocks, the target data slices of each of the multiple target data blocks, and the start and end position information of the target data slices of each of the multiple target data blocks to the client.

[0079] Optionally, the data acquisition device further includes: a second determining unit, configured to determine the file identifier of the target data, the length of the target data, the byte array information of the target data slice of each of the multiple target data blocks, and the total number of data slices while sending the start and end position information of each of the multiple target data blocks, the target data slice of each of the multiple target data blocks, and the start and end position information of the target data slice of each of the multiple target data blocks to the client; and a sending unit, configured to send the file identifier of the target data, the length of the target data, the byte array information of the target data slice of each of the multiple target data blocks, and the total number of data slices to the client together, wherein, after receiving the first target data slice, the client generates an empty file with the same length as the target data slice, skips the start position of the first target data slice, writes the byte array of the first target data slice to the empty file, and increments by 1 for each target data slice received.

[0080] According to another embodiment of the present invention, a computer-readable storage medium is also provided, characterized in that the computer-readable storage medium includes a stored program, wherein the program executes any of the above-described data acquisition methods.

[0081] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any communication device in a group of communication devices.

[0082] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: receiving data request information sent by a client, wherein the data request information is used to request the download of target data; when the data request information is file start and end position information and an initial file identifier, dividing the predetermined data stored in the HDFS distributed file system into blocks based on the file start and end file information to obtain multiple predetermined data blocks corresponding to the predetermined data, wherein the predetermined data is the data in the target data excluding the data that already exists locally, and the file start and end position information includes: the end position of the partial data, the end position of the target data, and the initial file identifier is the local... The system divides the stored partial data into blocks to obtain file identifiers for each of the multiple partial data blocks; determines the predetermined file identifier for each of the multiple predetermined data blocks; when the predetermined file identifier is inconsistent with the initial file identifier, it determines the data block to be sent corresponding to the file start and end position information and the file identifier to be sent for the data to be sent; and sends the data block to be sent and the file identifier to be sent to the client. After receiving the data block to be sent and the file identifier to be sent, the client calculates the current file identifier of the data block to be sent, and when it is determined that the current file identifier is consistent with the file identifier to be sent, it sends the partial data together with the data block to be sent as target data to the predetermined terminal.

[0083] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: parsing the data request information to obtain file start and end position information and an initial file identifier; searching for predetermined data in HDFS; and dividing the predetermined data into blocks on HDFS based on the file start and end position information to obtain multiple predetermined data blocks.

[0084] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: comparing a predetermined file identifier with an initial file identifier to obtain a comparison result; when the comparison result indicates that there is a file identifier that is inconsistent between the predetermined file identifier and the initial file identifier, determining the file identifier that is inconsistent between the predetermined file identifier and the initial file identifier as the target file identifier; and determining the data block to be sent based on the target file identifier and the start and end position information.

[0085] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: fragmenting the data block to be sent on HDFS to obtain multiple data fragments corresponding to the data block to be sent; sending the multiple data fragments and the file identifier to be sent to the client, so that when the client determines that the file identifiers of the multiple data fragments are consistent with the file identifier to be sent, it will send a portion of the data together with the data block to be sent as target data to a predetermined terminal.

[0086] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: when the data request information is the file path of the target data, the target data is obtained from the predetermined storage medium according to the file path; the target data is divided into blocks on HDFS to obtain multiple target data blocks corresponding to the target data; the multiple target data blocks are sent to the client so that the client can send the multiple target data blocks as target data to the predetermined terminal.

[0087] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: recording the start and end position information of each of the plurality of target data blocks; performing slicing processing on each of the plurality of target data blocks to obtain target data slices of each of the plurality of target data blocks; recording the start and end position information of the target data slices of each of the plurality of target data blocks; and sending the start and end position information of each of the plurality of target data blocks, the target data slices of each of the plurality of target data blocks, and the start and end position information of the target data slices of each of the plurality of target data blocks to the client.

[0088] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: while sending the start and end position information of each of the plurality of target data blocks, the target data slice of each of the plurality of target data blocks, and the start and end position information of the target data slice of each of the plurality of target data blocks to the client, determining the file identifier of the target data, the length of the target data, the byte array information of the target data slice of each of the plurality of target data blocks, and the total number of data slices; sending the file identifier of the target data, the length of the target data, the byte array information of the target data slice of each of the plurality of target data blocks, and the total number of data slices together to the client, wherein, after receiving the first target data slice, the client generates an empty file with the same length as the target data slice, skips the start position of the first target data slice, writes the byte array of the first target data slice to the empty file, and increments by 1 for each target data slice received.

[0089] According to another embodiment of the present invention, a processor is also provided, the processor being used to run a program, wherein the program, when running, executes the data acquisition method described in any one of the above embodiments.

[0090] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0091] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0092] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0093] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0095] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0096] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data acquisition method, characterized in that, include: Receive data request information sent by the client, wherein the data request information is used to request the download of target data; When the data request information is file start and end position information and initial file identifier, the predetermined data stored in the HDFS distributed file system is divided into blocks based on the file start and end file information to obtain multiple predetermined data blocks corresponding to the predetermined data. The predetermined data is the data in the target data excluding the data that already exists locally. The file start and end position information includes: the end position of the partial data and the end position of the target data. The initial file identifier is the file identifier of each of the multiple partial data blocks obtained by dividing the partial data stored locally. Determine the predefined file identifier for each of the plurality of predefined data blocks; When it is determined that the predetermined file identifier is inconsistent with the initial file identifier, the data block to be sent corresponding to the file start and end position information and the file identifier to be sent for the data to be sent are determined. The data block to be sent and the file identifier to be sent are sent to the client. After receiving the data block to be sent and the file identifier to be sent, the client calculates the current file identifier of the data block to be sent. When it is determined that the current file identifier is consistent with the file identifier to be sent, the client sends the partial data together with the data block to be sent as the target data to the predetermined terminal.

2. The data acquisition method according to claim 1, characterized in that, When the data request information is file start and end position information and an initial file identifier, the predetermined data stored in the HDFS distributed file system is divided into blocks based on the file start and end information to obtain multiple predetermined data blocks corresponding to the predetermined data, including: The data request information is parsed to obtain the file start and end position information and the initial file identifier; The predetermined data is obtained by searching the HDFS. The predetermined data is divided into blocks on the HDFS based on the start and end position information of the file, resulting in the plurality of predetermined data blocks.

3. The data acquisition method according to claim 1, characterized in that, When it is determined that the predetermined file identifier is inconsistent with the initial file identifier, the data block to be sent corresponding to the file start and end position information and the file identifier to be sent for the data to be sent are determined, including: The predetermined file identifier is compared with the initial file identifier to obtain the comparison result; When the comparison result indicates that there is a file identifier that is inconsistent between the predetermined file identifier and the initial file identifier, the file identifier that is inconsistent between the predetermined file identifier and the initial file identifier is determined to be the target file identifier; The data block to be sent is determined based on the target file identifier and the start and end position information.

4. The data acquisition method according to any one of claims 1 to 3, characterized in that, Sending the data block to be sent and the file identifier to be sent to the client includes: The data block to be sent is fragmented on the HDFS to obtain multiple data fragments corresponding to the data block to be sent. The plurality of data slices and the file identifier to be sent are sent to the client, so that when the client determines that the file identifiers of the plurality of data slices are consistent with the file identifier to be sent, the client will send the partial data together with the data block to be sent as the target data to the predetermined terminal.

5. The data acquisition method according to claim 1, characterized in that, Also includes: When the data request information is the file path of the target data, the target data is obtained from the predetermined storage medium according to the file path; The target data is divided into blocks on the HDFS to obtain multiple target data blocks corresponding to the target data; The plurality of target data blocks are sent to the client so that the client can use the plurality of target data blocks as target data to send to the predetermined terminal.

6. The data acquisition method according to claim 5, characterized in that, Sending the plurality of target data blocks to the client includes: Record the start and end position information of each of the plurality of target data blocks; Each of the plurality of target data blocks is sliced ​​to obtain a target data slice for each of the plurality of target data blocks; Record the start and end position information of the target data slice for each of the plurality of target data blocks; The start and end position information of each of the plurality of target data blocks, the target data slice of each of the plurality of target data blocks, and the start and end position information of the target data slice of each of the plurality of target data blocks are sent to the client.

7. The data acquisition method according to claim 5, characterized in that, Also includes: While sending the start and end position information of each of the plurality of target data blocks, the target data slice of each of the plurality of target data blocks, and the start and end position information of the target data slice of each of the plurality of target data blocks to the client, the file identifier of the target data, the length of the target data, the byte array information of the target data slice of each of the plurality of target data blocks, and the total number of data slices are determined. The file identifier of the target data, the length of the target data, the byte array information of each target data segment in the plurality of target data blocks, and the total number of data segments are sent to the client. After receiving the first target data segment, the client generates an empty file with the same length as the target data segment, skips the starting position of the first target data segment, and writes the byte array of the first target data segment into the empty file. The number is incremented by 1 for each target data segment received.

8. A data acquisition device, characterized in that, include: A receiving unit is configured to receive data request information sent by a client, wherein the data request information is used to request the download of target data; A block division unit is used to divide the predetermined data stored in the HDFS distributed file system based on the file start and end file information when the data request information is file start and end position information and an initial file identifier, to obtain multiple predetermined data blocks corresponding to the predetermined data. The predetermined data is the data in the target data excluding the data that already exists locally. The file start and end position information includes: the end position of the partial data and the end position of the target data. The initial file identifier is the file identifier of each of the multiple partial data blocks obtained by dividing the partial data stored locally. The first determining unit is used to determine the predetermined file identifier for each of the plurality of predetermined data blocks; The second determining unit is used to determine the data block to be sent corresponding to the file start and end position information and the file identifier to be sent of the data to be sent when it is determined that the predetermined file identifier is inconsistent with the initial file identifier. A sending unit is configured to send the data block to be sent and the file identifier to be sent to the client, wherein, after receiving the data block to be sent and the file identifier to be sent, the client calculates the current file identifier of the data block to be sent, and when it is determined that the current file identifier is consistent with the file identifier to be sent, it sends the partial data together with the data block to be sent as the target data to a predetermined terminal.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program performs the data acquisition method according to any one of claims 1 to 7.

10. A processor, characterized in that, The processor is used to run a program, wherein the program executes the data acquisition method according to any one of claims 1 to 7 when it runs.

Images