Data processing method, device, electronic device and storage medium

Abstract

The present disclosure provides a data processing method, device, electronic device and storage medium, which relate to the field of computer technology, and in particular to technical fields such as cloud computing, search, and information flow. The specific implementation scheme is: according to the business process of the target data download end, send a first acquisition request for the first file block of the data file to the data provider end of the point-to-point network; obtain the first file block of the data file provided by the data provider end according to the first acquisition request; store the first file block of the data file in the memory space corresponding to the business process, and the memory space corresponding to the business process is used to store data directly retrieved when the business process is running. The embodiment of the present disclosure can increase the data download speed.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to technical fields such as cloud computing, search, and information flow. Background Art

[0002] In the context of widespread application of cloud-native architecture technology, the business service startup time of each data download end increases linearly with the deployment time brought about by the complexity of the business service.

[0003] With the development of the Internet, higher requirements are placed on the data download speed of the data download end. There is an urgent need to improve the overall speed of downloading data files at each data download end of the data source under the cloud native architecture. Summary of the invention

[0004] The present disclosure provides a data processing method, apparatus, device and storage medium.

[0005] According to one aspect of the present disclosure, there is provided a data processing method, comprising:

[0006] Sending a first acquisition request for a first file block of a data file to a data provider of a peer-to-peer network according to a service process of a target data download terminal;

[0007] Acquire a first file block of a data file provided by a data provider according to a first acquisition request;

[0008] The first file block of the data file is stored in the memory space corresponding to the business process, and the memory space corresponding to the business process is used to store data directly retrieved when the business process is running.

[0009] According to another aspect of the present disclosure, there is provided a data processing device, comprising:

[0010] A first acquisition request sending module, used for sending a first acquisition request for a first file block of a data file to a data providing end of a peer-to-peer network according to a service process of a target data download end;

[0011] A first file block acquisition module, used to acquire a first file block of a data file provided by a data provider according to a first acquisition request;

[0012] The first file block storage module is used to store the first file block of the data file into the memory space corresponding to the business process. The memory space corresponding to the business process is used to store data directly called when the business process is running.

[0013] According to another aspect of the present disclosure, a data processing system is provided, comprising: a data source end and multiple data download ends connected to the data source end; at least one of the multiple data download ends comprises the data processing device provided by any one embodiment of the present disclosure.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method in any embodiment of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to enable a computer to execute the method in any embodiment of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method in any embodiment of the present disclosure is implemented.

[0020] According to the technical solution disclosed in the present invention, the first file block of a data file can be obtained through a data provider in a peer-to-peer network. When obtaining data, a data provider that is close to the requested data can be selected for data acquisition; at the same time, the obtained first file block is stored in the memory space corresponding to the business process without passing through the disk, and is thus not limited by the disk write speed, thereby improving the data download speed and the data write speed, thereby improving the overall data acquisition speed.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flowchart of a data processing method according to an embodiment of the present disclosure;

[0024] Figure 2 is a schematic diagram of a data source end and a data download end according to an example of the present disclosure;

[0025] Figure 3 is a flowchart of a data processing method according to another embodiment of the present disclosure;

[0026] Figure 4 is a flowchart of a data processing method according to another embodiment of the present disclosure;

[0027] Figure 5 is a schematic diagram of a data processing device according to an embodiment of the present disclosure;

[0028] Figure 6 is a schematic diagram of a data processing device according to another embodiment of the present disclosure;

[0029] Figure 7 is a schematic diagram of a data processing device according to another embodiment of the present disclosure;

[0030] Figure 8 is a schematic diagram of a data processing device according to another embodiment of the present disclosure;

[0031] Fig. 9 is a schematic diagram of a data processing device according to another embodiment of the present disclosure;

[0032] Fig.10 is a schematic diagram of a data processing device according to another embodiment of the present disclosure;

[0033] Fig.11 is a schematic diagram of a data processing system according to an embodiment of the present disclosure;

[0034] Fig.12 is an application network diagram of a data processing system according to an example of the present disclosure;

[0035] Fig.13 is a schematic diagram of module settings of a data download terminal according to an example of the present disclosure;

[0036] Fig.14 It is a block diagram of an electronic device used to implement the data processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0038] The present disclosure first provides a data processing method, such as Figure 1 As shown, including:

[0039] Step S11: sending a first acquisition request for a first file block of a data file to a data provider of a peer-to-peer network according to a service process of the target data download terminal;

[0040] Step S12: Acquire a first file block of a data file provided by the data provider according to the first acquisition request;

[0041] Step S13: storing the first file block of the data file into the memory space corresponding to the business process. The memory space corresponding to the business process is used to store data directly retrieved when the business process is running.

[0042] In this embodiment, the target data download end may be one of a plurality of data download ends that request to download the file blocks of the data file.

[0043] In a specific implementation, the target data download end may correspond to the data source end, such as Figure 2 As shown, in a data download network, there may be one data source terminal 21 and multiple data download terminals 22, and the data source terminal and the data download terminals 22 may be distributed in different regions, such as region 1, region 2, region 3, region 4, region 5, etc. The data download terminal 22 may download data from the data source terminal 21 to the data download terminal 22, so that multiple data download terminals 22 may provide data to the terminals 23 within the region of the data download terminal 22. The data source terminal may be a main server that provides data files, and the data download terminal may be a sub-server that obtains data files from the data source terminal and is distributed in multiple different regions. The data download terminal 22 is connected to both the data source terminal 21 and other data download terminals.

[0044] In a specific implementation, the target data download terminal can run at least one business process, and the business process can use the data file. If the data file is large, it can be divided into multiple file blocks, and the first file block can be one of the at least one file block of the data file.

[0045] When the business process runs on the target data download end, it needs to use data files, which can be obtained from the data source end or from other nearby data download ends. When the target data download end obtains the data file, it can obtain the file blocks of the data file in sequence.

[0046] In a specific implementation, based on the business process of the target data download end, a first acquisition request for the first file block of the data file is sent to the data provider end of the peer-to-peer network. It can be that based on the demand for data files by the business process of the target data download end, a first acquisition request for the first file block of the data file is sent to the data provider end of the peer-to-peer network.

[0047] In a specific implementation, a first acquisition request for the first file block of a data file is sent to a data provider of a peer-to-peer network based on a business process of a target data download end. Alternatively, a first acquisition request for the first file block of a data file is sent to a data provider of a peer-to-peer network based on operation information of the business process of the target data download end (such as request information for reading data).

[0048] In a possible implementation, the target data download end and the data provider end may both belong to the same peer-to-peer (P2P) network.

[0049] In a possible implementation, after receiving the first acquisition request, the data provider may provide the first file block or the seed file of the first file block to the target data downloader, so that the target data downloader can directly download the first file block or download the first file block according to the seed file.

[0050] In one possible implementation, storing the first file block of the data file in the memory space corresponding to the business process can be storing the first file block of the data file in the memory space where the program data and dependent data of the business process are located, that is, the memory space where the running data of the business process is located. Thus, when the business process needs to use the data file, the data file can be directly retrieved from the memory space corresponding to the business program without the need for transfer on the disk or other storage media at the target data download end.

[0051] In a specific implementation, the business process can be a business process of the underlying service module of the target data download end. According to the deployment time statistics of each underlying service module, the pre-startup preparation time caused by complex strategies and data is too long, among which data download is usually a stage that consumes more time. In the data download of a business process, if the dependent data files that need to be downloaded are too large and too many, more time is required. For example, it takes more than 16 minutes to download a 7GB data file from the network data source. The underlying reason is that the cloud-native containers (such as containers for search processes, containers for map processes, and other different containers) on the same physical machine are mixed and deployed, and the read and write rate of the disk HDD (Hard Disk Drive) is used together. The online machine service used to provide services to users is to ensure that multiple container services of the entire machine provide stable services, and the write speed of the single disk at the data download end is controlled at a rate of 20-40M / s.

[0052] Due to the fact that downloading large data files takes a long time, and with the trend of the overall network bandwidth of the physical computer room constantly increasing, the problem of how to improve the speed of downloading large data files to disk has gradually emerged. Usually, the target physical hardware for downloading data files of business processes is the disk under the default primary partition. For most service storage, it is placed in the primary partition by default. When the single disk write speed of the primary partition is tens of MB / s, the network data will arrive at the physical machine and wait in line to write to the disk. In this case, the time between data download and disk and module service waiting does not match (for example, remote acquisition speed is fast, local writing speed is slow), which brings about the problem of too much waiting time, that is, the extreme asymmetry between the fast network download data and the slow disk writing data speed.

[0053] The data processing method provided by the embodiment of the present disclosure can obtain the first file block of a data file through a data provider in a point-to-point network. When obtaining data, a data provider that is close to the requested data can be selected for data acquisition; at the same time, the obtained first file block is stored in the memory space corresponding to the business process without passing through the disk, so it is not limited by the disk write speed, thereby improving the data download speed and data write speed, and thus improving the overall data acquisition speed.

[0054] In one embodiment, according to the service process of the target data download end, sending a first acquisition request for a first file block of a data file to a data provider of a peer-to-peer network includes:

[0055] When it is determined through the virtual file system of the target data download end that the target data download end does not have the first file block requested by the first acquisition request, obtaining a data provider of the peer-to-peer network from a distribution server shared by multiple data download ends;

[0056] A first acquisition request is sent to a data provider.

[0057] In a possible implementation, when the business process of the target data download terminal needs to obtain a data file, a query request can be initiated to the data service of the target data download terminal through the virtual system, and the data service can be used to determine whether the target data download terminal has stored the data file that the business process needs to obtain. When the target data download terminal does not have the data file that the business process needs to obtain, a query is made to the common distribution server in the peer-to-peer network to determine which data download terminals in the peer-to-peer network have the file blocks of the data file that the business process needs to obtain, and the file blocks of the data file can be obtained preferentially from other data download terminals in the peer-to-peer network that are close to the target data download terminal, or the data download terminal that can obtain the file blocks of the data file the fastest can be downloaded preferentially.

[0058] In the case that other data downloading ends in the peer-to-peer network do not have the file blocks of the data file, the data source end may be determined as the data providing end.

[0059] In a possible implementation, the data file required by the business process may be divided into a plurality of file blocks, and the plurality of file blocks may be obtained from at least one data provider.

[0060] In one implementation, obtaining a data provider of a peer-to-peer network from a distribution server shared by a plurality of data download terminals may be obtaining one or more data providers from the peer-to-peer network.

[0061] In a specific implementation method, the virtual file system of the target data download end can interact with the data service of the target data download end, and the data service of the target data download end can interact with the target data download end, and further the download service of the target data download end can interact with the data provider in the peer-to-peer network to obtain the first file block of the data file.

[0062] In this embodiment, the data provider is determined by the distribution server in the peer-to-peer network, so that the data provider with the highest download efficiency can be determined according to the distribution of each data download terminal in the peer-to-peer network, thereby improving the overall data download speed.

[0063] In one embodiment, the point-to-point network also includes a data source end; the data provider end is other data download ends among multiple data download ends except the target data download end, and the other data download ends store the first file block of the data file directly or indirectly derived from the data source end; or, the data provider end is the data source end.

[0064] In one implementation, the data download end that can provide the file block fastest can be used as the data provider. For example, if the first file block is stored on both data download ends, the first file block can be requested preferentially from the data download end that is closer or has a faster transmission speed, and the data download end that is closer or has a faster transmission speed can be used as the data provider.

[0065] In another implementation, if each data downloading end in the peer-to-peer network has not downloaded the first file block of the data file, the data source end may be used as the data providing end.

[0066] In this embodiment, the data source end or other data download end in the point-to-point network can be used as the data providing end, so that the device for obtaining the first file block can be flexibly determined, which is conducive to improving the acquisition speed of data files as a whole.

[0067] In one embodiment, before obtaining the data provider of the peer-to-peer network from the distribution server shared by multiple data download terminals, such as Figure 3 ,include:

[0068] Step S31: using the business process of the target data download terminal to initiate a read request to the data service of the target data download terminal;

[0069] Step S32: Generate a read interface using the data service of the target data download end, and send a read request to the download service of the target data download end through the read interface;

[0070] Step S33: using the download service of the target data download terminal, starting a point-to-point download client at the target data download terminal;

[0071] Step S34: Initiate a data provider acquisition request for the first file block to the distribution server through the peer-to-peer download client.

[0072] In one possible implementation, a business process of the target data download end is used to initiate a read request to a data service of the target data download end. Specifically, this may include using a business process to generate a read request, and sending the read request to the data service of the target data download end through a virtual file system of the target data download end.

[0073] In a specific implementation, the peer-to-peer download client may be a client running on a target data download end and used for downloading data files.

[0074] In a specific implementation, the distribution server may query the data download end that has downloaded the first file block of the data file, and use the data download end that has downloaded the first file block and has a higher transmission speed as the data provider.

[0075] In a specific implementation, the target data download end may be added to the peer-to-peer network when the target data download end needs to download a file block of a data file.

[0076] In this embodiment, a read interface is generated through the data service, and the read interface interacts with the download service, so that the business process can directly interact with the download service for data without going through the disk of the target data download end, thereby realizing diskless downloading and improving the data disk landing speed (writing speed).

[0077] In one implementation, storing the first file block of the data file into the memory space corresponding to the business process includes:

[0078] storing the first file block in a buffer storage area corresponding to the data file according to the order of the first file block in the data file;

[0079] When the buffer storage area corresponding to the data file is full, the first file block in the buffer storage area is moved to the memory space corresponding to the business process; or when the business process calls the first file block in the buffer storage area, the first file block called by the business process is moved to the memory space corresponding to the business process.

[0080] In a possible implementation, the data file may be divided into a plurality of file blocks, and the plurality of file blocks are distributed in order in the data file. For example, if the size of the data file is 500M, the data of the 1M-100Mth, the data of the 101M-200Mth, the data of the 201M-300Mth, the data of the 301M-400Mth, and the data of the 401M-500Mth are divided respectively to form five file blocks, and the first file block may be any one of the five file blocks.

[0081] In one implementation, according to the order of the first file block in the data file, the first file block is stored in the buffer storage area corresponding to the data file. Specifically, according to the order of the first file block in the data file and the file blocks of the data file already in the target data download end, it can be determined that there are still file blocks in the order before the first file block that have not been downloaded to the target data download end, and then the download of the file blocks in the order before the first file block can be waited first; or when the space size of the buffer storage area allows the first file block and the file block before the first file block (the file block of the first order of the data file) to be placed, the file blocks before the first file block that have not been downloaded are stored in the buffer storage area together with the first file block. When the buffer storage area is full, or the file blocks in the order before the first file block are downloaded, when the service process starts to call the first file block, the first file block in the buffer storage area is transferred to the memory corresponding to the service process.

[0082] In one possible implementation, the first file block is stored in a buffer storage area corresponding to the data file according to the order of the first file block in the data file. When the first file block is a file block that is ranked first in the data file, the first file block is stored in a buffer storage area corresponding to the data file requested by the business process.

[0083] In a specific implementation, a buffer storage area may be set for each data file that needs to be obtained by a business process.

[0084] In this embodiment, a buffer storage area is set so that the file blocks obtained by the target download data terminal can be called sequentially by the business process, so that the business process can call the data file normally.

[0085] In one embodiment, Figure 4 As shown, the data file processing method also includes:

[0086] Step S41: receiving a second acquisition request for a file block of a data file sent by another data download terminal in the peer-to-peer network;

[0087] Step S42: according to the second acquisition request, the second file block of the data file stored on the target data download end is sent to other data download ends.

[0088] In a specific implementation, after obtaining the first file block or other file blocks of the data file, the target data download end may become a data provider to provide file blocks of the data file to other data download ends in the peer-to-peer network.

[0089] In a specific implementation, the second file block may be the first file block, or may be other file blocks divided by the data file.

[0090] In this embodiment, after obtaining the file blocks of the data file, the target data download end can become a data provider, thereby improving the data download efficiency in the entire point-to-point network.

[0091] In one embodiment, sending a first acquisition request for a first file block of a data file to a data provider of a peer-to-peer network includes:

[0092] Obtaining a seed file of a first file block for downloading a data file through a download service of a target data download terminal;

[0093] According to the seed file, a first acquisition request for a first file block of a data file is sent to a data provider of a peer-to-peer network.

[0094] In a specific implementation, the seed file of the first file block may be obtained from a distribution server shared by multiple data download terminals in a peer-to-peer network, or may be obtained from other storage locations.

[0095] In a specific implementation, the seed file may be generated according to the file blocks after the distribution server shared by multiple data providers in the peer-to-peer network divides the data file.

[0096] In this embodiment, a first acquisition request for the first file block is sent to the data provider according to the seed file, so that the first file block can be downloaded through the seed file.

[0097] In one implementation, the seed file is generated by the distribution server according to the distribution strategy of the distribution management controller shared by multiple data download terminals and the data file.

[0098] In this embodiment, the seed file is generated by the distribution server, so after the seed file is generated, the distribution server can distribute the seed file according to the set distribution strategy, so that the data download end can obtain the file blocks of the data file according to the seed file.

[0099] The present disclosure also provides a data processing device, such as Figure 5 As shown, including:

[0100] A first acquisition request sending module 51 is used to send a first acquisition request for a first file block of a data file to a data provider of a peer-to-peer network according to a service process of a target data download terminal;

[0101] A first file block acquisition module 52, configured to acquire a first file block of a data file provided by a data provider according to a first acquisition request;

[0102] The first file block storing module 53 is used to store the first file block of the data file into the memory space corresponding to the business process. The memory space corresponding to the business process is used to store data directly retrieved when the business process is running.

[0103] In one embodiment, Figure 6 As shown, the first acquisition request sending module includes:

[0104] The data provider acquisition unit 61 is used to acquire a data provider of a peer-to-peer network from a distribution server shared by multiple data download terminals when it is determined through the virtual file system of the target data download terminal that the target data download terminal does not have the first file block requested by the first acquisition request;

[0105] The sending unit 62 is used to send a first acquisition request to the data provider.

[0106] In one embodiment, the point-to-point network also includes a data source end; the data provider end is other data download ends among multiple data download ends except the target data download end, and the other data download ends store the first file block of the data file directly or indirectly derived from the data source end; or, the data provider end is the data source end.

[0107] In one embodiment, Figure 7 As shown, the data processing device also includes:

[0108] A read request initiating module 71 is used to initiate a read request to a data service of a target data download terminal using a business process of the target data download terminal;

[0109] A read interface module 72, used to generate a read interface using the data service of the target data download end, and send a read request to the download service of the target data download end through the read interface;

[0110] A starting module 73 is used to start a point-to-point download client at the target data download end using the download service of the target data download end;

[0111] The provider acquisition request module 74 is used to initiate a data provider acquisition request for the first file block to the distribution server through the point-to-point download client.

[0112] In one embodiment, Figure 8 As shown, the first file block storage module includes:

[0113] The cache unit 81 is used to store the first file block into a buffer storage area corresponding to the data file according to the order of the first file block in the data file;

[0114] A first transfer unit 82 is used to move the first file block in the buffer storage area to the memory space corresponding to the business process when the buffer storage area corresponding to the data file is full;

[0115] And / or, the second transfer unit 83 is used to move the first file block called by the business process to the memory space corresponding to the business process when the business process calls the first file block in the buffer storage area.

[0116] In one embodiment, Fig. 9 As shown, the data processing device also includes:

[0117] A second acquisition request receiving module 91, configured to receive a second acquisition request for a file block of a data file sent by another data download terminal in the peer-to-peer network;

[0118] The second file block sending module 92 is used to send the second file block of the data file stored on the target data download end to other data download ends according to the second acquisition request.

[0119] In one embodiment, Fig.10 As shown, the first acquisition request sending module includes:

[0120] The seed file unit 101 is used to obtain a seed file of a first file block for downloading a data file through a download service of a target data download terminal;

[0121] The seed file processing unit 102 is used to send a first acquisition request for a first file block of a data file to a data provider of a peer-to-peer network according to the seed file.

[0122] In one implementation, the seed file is generated by the distribution server according to the distribution strategy of the distribution management controller shared by multiple data download terminals and the data file.

[0123] The present disclosure also provides a data processing system. Fig.11As shown, it includes: a data source terminal 111 and multiple data download terminals 112 connected to the data source terminal; at least one of the multiple data download terminals includes a data processing device 113 provided by any embodiment of the present disclosure.

[0124] In one implementation, at least one of the multiple data download terminals in the data processing system includes the data service and download service of any one of the embodiments of the present disclosure.

[0125] In an example of the present disclosure, a data processing system implements data network acceleration based on diskless data downloading of a P2P network. After the data file of the data source segment is updated, the auxiliary system notifies each data download end of the event of the data file update. After the device of the data download end downloads the file block of the data file through the network, it receives the data packet transmitted in the network, directly stores it in the memory of the business process, and notifies the business process to perform the loading operation, thereby realizing a diskless remote data request and loading solution.

[0126] In the example of this disclosure, Fig.12 As shown, the data acceleration network can mainly include an online data source 121 and a data download terminal, and the data download terminal can further include a data distribution management 122, a distribution server 123, a download service 124, and a data service 125. Among them, the data service 125 can correspond to the business APP, and the download service 124 corresponds to the data service. Multiple data download terminals can share one distribution server 123 and one data distribution management 122, and one data service 125 can correspond to multiple download services 124.

[0127] Reference Fig.12 The data acceleration network can be managed by the PAAS (Platform As Service) controller for container management, and each container corresponds to a major business process. At the same time, the MIS (Management Information System) version management module can also be used to manage the distribution strategy generated by the data distribution management 122 and notify the PAAS controller.

[0128] In a specific implementation, the data service is used to connect to the business process to provide a data reading service interface and synchronize the state of interaction with the business process. The business process can be a business program (APP, Application) process, and the data of the business process at the data download end is obtained through download or re-download operations.

[0129] The data service is also used to interact with the download service, notify the data file to start downloading and start, and obtain the download status of the remote data file. At the same time, the data service is also used to write data files to the storage medium asynchronously at a low speed, perform persistence operations (such as writing to disk), to implement a backup plan for data download, and write part of the downloaded data to the disk to prepare for recovery when the download data is abnormal. By writing data files asynchronously at a low speed, it does not affect the online acquisition of data files and directly storing them in the memory space corresponding to the business process.

[0130] In addition, the data download end provides a read agent function for the data service on the local storage medium, obtains the file name in the virtual file system, and determines the distribution of each file block of the data file corresponding to the read file name, providing fast reading capability for abnormal recovery.

[0131] In a specific example, the download service can also be set at the data download end, which can be used to connect to the data service, receive the data file download instruction sent from the data service to the download service, and finally put the data file downloaded from the cloud P2P into the memory and deliver it to the data service at the same time. The above cloud can include the data source end and other data download ends that have downloaded data.

[0132] The download service can also be used to interact with the distribution service to obtain the specific information of the data file to be downloaded and the distribution information of the file blocks of the data file from the distribution server. At the same time, the download service is also used to register heartbeat information, which indicates that the data download end has a continuous download function, obtains the download file P2P seed file, and synchronizes the download status to the distribution server.

[0133] The download service can also be used to interact with nodes at the data source end and the peer data download end to exchange and download data blocks through the P2P network.

[0134] A distribution server (Tracker-Server) can be set up for all data download terminals in a peer-to-peer network, that is, all data download terminals in the peer-to-peer network can share the distribution server.

[0135] The distribution server is used to receive and execute data distribution management instructions, provide data file meta-information (including basic information such as the name of the data file) query function to the data download end, and feedback summary information to the management controller.

[0136] The distribution server can also be used to manage the download service status, collect the download progress of the download service, complete the instance of data file download, and set the data file instance to be in the download completed state.

[0137] At the same time, the distribution server is also used to split the data file into file blocks, make seed files for the split file blocks, and distribute the seed files to each data download terminal.

[0138] The function of data distribution management (Data-Controller) tends to be background management and is used to implement core decision-making functions.

[0139] Data distribution management can be called the management and controller of data distribution, which controls data file operations and manages multi-regional data files.

[0140] Data distribution management can be used for the registration and management of data files, to detect changes in data files provided by the data source, and to generate and manage data versions based on the data files provided by the data source.

[0141] Data distribution management can also be used for data monitoring and version report management, and can be used for query by the data download end in the system.

[0142] In another example of the present application, by de-diskifying the remote data acceleration network, the disk write speed limit that affects the online service using the dictionary is removed, reducing the overall time consumption of network data transmission to the online service reading into the memory, wherein the dictionary can be a data file, and further can be part of the dependent data of the business process. The main implementation mechanism, such as Fig.13 shown.

[0143] Reference Fig.13 , a virtual file system 131 and an online business service program 132 (Main App Process) are started on a physical device at a data download end, which is equivalent to Fig.12 During the operation of the business APP, at least one business process is started.

[0144] Still refer to Fig.13 The online business program uses the virtual file system 131 (Virtual File System) to mount the data file address in the data service 133 through a soft link (Soft Link), and the directory name of the data file is placed under the mount point (Mount Point) directory.

[0145] Among them, the user's cost can be reduced through the virtual file system 131, and the storage address of the file to be read is changed from the disk to the virtual file system, so that the change of reading the file is minimized.

[0146] Still refer to Fig.13, meta-info management, seed file management and data persistence (Persist) are designed and implemented in the data service 133 at the data download end. When the business process initiates a read stat operation of the remote dictionary of the mount point (MountPoint), the read action will be sent to the data service 133. When the data service 133 determines that there is no data to be read locally at the data download end, it starts the read interface and initiates a data request to the download service 134.

[0147] Still refer to Fig.13 After receiving the file reading instruction, the download service 134 starts the P2P network download client (P2P download) to download the remote data (file blocks of the data file) through the seed file. The downloaded data is transmitted back to the download service client through the network. The download service sends the data in batches to the data service. The data service merges and processes the data and returns it to the business service program, thereby completing a data acceleration network download process.

[0148] In the disclosed example, the remote data is directly downloaded and entered into the memory space corresponding to the business process through data services and download services, thus getting rid of the performance bottleneck limitation caused by writing to the disk, thereby significantly improving the speed of remote data downloading and memory loading.

[0149] In a P2P network, for fast file transfer and download, a data file in the network will be split into multiple small data packets (i.e., data blocks in other embodiments), and a hash value is calculated for each packet. According to the strategy generated by the data distribution management, multiple peer nodes are requested as concurrently as possible in the network as the receiving end of the data packet of the data file. For the data acceleration network to be implemented, it is required to directly provide data memory into the business program memory, so it is necessary to ensure the basic order of the file blocks.

[0150] In this disclosure example, still refer to Fig.13 , a buffer storage area 135 for data files is added to the download service 134. The buffer storage area 135 can download data in a P2P manner, or download data through a backup cluster download method to provide data recovery in case of abnormalities. When the download service 134 provides data to the data service 133, data transmission can be performed according to the set speed limit (Speed ​​Limit).

[0151] In the computer program, a buffer storage area is established for each dictionary (data file), and each program corresponds to multiple dictionaries. The buffer storage area can be used to store the data packets with the highest sorting in the data file. When downloading the data packets or data blocks of the data file, each data download end will give priority to obtaining the data in the buffer storage area, and use the data packets or data blocks in order. According to the data in the P2P network, according to the size of the buffer storage area and the set file block window size, the window is sequentially obtained by sliding, and finally the download service has the function of sequentially obtaining and returning data files. In the data service, by reading and controlling the download speed, the consumption range of machine resources can be limited, and the resource usage of the computer CPU and memory at different data download speeds can be controlled.

[0152] The disclosed embodiment improves the download rate of the P2P network in a controlled manner and adds a file block sliding window technical solution to the download service to achieve orderly and fast downloading and acquisition of network data within the window, thereby achieving a balance between speed and resource consumption.

[0153] The embodiments of the present disclosure can be applied to fields such as cloud computing, search, and information flow.

[0154] When applied to the field of information flow, users can be modeled in an offline system, and vectorized representations of information flow recommended content articles and videos can be generated, and finally index data files can be output as data sources for online computer programs. The online computer program uses the present invention to perform diskless network accelerated downloading of the required index data files, and places the required data files into the program memory. Finally, the computer program responds to the user's access request, so that the user can obtain the recommended content of the information flow.

[0155] At the same time, the technical solution provided by the embodiment of the present disclosure effectively speeds up the download speed of the remote dictionary (data file), establishes a set of diskless data download and loading solutions, so that the online service loading data files is free from the disk write speed limit problem, thereby greatly improving the deployment speed of online services. Currently, it is applied in real online search engine systems and information flow recall systems, shortening the original big data file download time to 50%-12%, shortening the search recommendation dictionary download time from more than 20 minutes to within 10 minutes, and shortening the information flow vector index download speed from 42 minutes to 5 minutes, achieving good business results.

[0156] The various embodiments of the present disclosure can be applied to the field of computer technology, and in particular, can be applied to technical fields such as cloud computing, big data, and information flow.

[0157] Furthermore, the embodiments of the present disclosure may be applied to the acquisition of data files of servers corresponding to products such as search engine online service systems, information flow recommendation systems, related search products, demand guidance products, and input box prompt word products.

[0158] The functions of each unit, module or sub-module in each device of the embodiments of the present disclosure can be found in the corresponding description in the above method embodiments, and will not be repeated here.

[0159] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0160] Fig.14 A schematic block diagram of an example electronic device 140 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0161] like Fig.14 As shown, the electronic device 140 includes a computing unit 141, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 142 or a computer program loaded from a storage unit 148 into a random access memory (RAM) 143. In the RAM 143, various programs and data required for the operation of the electronic device 140 can also be stored. The computing unit 141, the ROM 142, and the RAM 143 are connected to each other via a bus 144. An input / output (I / O) interface 145 is also connected to the bus 144.

[0162] A number of components in the electronic device 140 are connected to the I / O interface 145, including: an input unit 146, such as a keyboard, a mouse, etc.; an output unit 147, such as various types of displays, speakers, etc.; a storage unit 148, such as a disk, an optical disk, etc.; and a communication unit 149, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 149 allows the electronic device 140 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0163] The computing unit 141 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 141 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 141 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 148. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 140 via ROM 142 and / or communication unit 149. When the computer program is loaded into RAM 143 and executed by the computing unit 141, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 141 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).

[0164] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0165] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0166] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0167] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0168] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0169] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0170] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0171] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data processing method, comprising: Sending a first acquisition request for a first file block of a data file to a data provider of a peer-to-peer network according to a service process of a target data download terminal, wherein the target data download terminal runs at least one service process; Acquire a first file block of the data file provided by the data provider according to the first acquisition request; storing the first file block of the data file into a memory space corresponding to the business process, where the memory space corresponding to the business process is used to store data directly retrieved when the business process is running; The sending of a first acquisition request for a first file block of a data file to a data provider of a peer-to-peer network according to a service process of a target data download terminal comprises: When it is determined through the virtual file system of the target data download end that the target data download end does not have the first file block requested by the first acquisition request, acquiring a data provider of a peer-to-peer network from a distribution server shared by multiple data download ends; Sending the first acquisition request to the data provider; The point-to-point network further comprises a data source end; the data provider end is other data download ends among the multiple data download ends except the target data download end, and the other data download ends store the first file block of the data file directly or indirectly derived from the data source end; or, the data provider end is the data source end; Wherein, before obtaining the data provider of the peer-to-peer network from the distribution server shared by multiple data download terminals, the method includes: Using the business process of the target data download terminal, initiating a read request to the data service of the target data download terminal; Using the data service of the target data download end to generate a read interface, and sending the read request to the download service of the target data download end through the read interface; Using the download service of the target data download terminal, starting a point-to-point download client at the target data download terminal; A data provider acquisition request for the first file block is initiated to the distribution server through the peer-to-peer download client.

2. The method according to claim 1, wherein: The storing the first file block of the data file into the memory space corresponding to the business process includes: storing the first file block in a buffer storage area corresponding to the data file according to the order of the first file block in the data file; When the buffer storage area corresponding to the data file is full, the first file block in the buffer storage area is moved to the memory space corresponding to the business process; or when the business process calls the first file block in the buffer storage area, the first file block called by the business process is moved to the memory space corresponding to the business process.

3. The method according to any one of claims 1 to 2, wherein: Also includes: Receiving a second acquisition request for a file block of a data file sent by another data download terminal in the peer-to-peer network; According to the second acquisition request, the second file block of the data file stored on the target data download end is sent to the other data download ends.

4. The method according to any one of claims 1 to 2, wherein: The step of sending a first acquisition request for a first file block of a data file to a data provider of a peer-to-peer network comprises: Obtaining, through the download service of the target data download terminal, a seed file for downloading the first file block of the data file; According to the seed file, a first acquisition request for a first file block of a data file is sent to a data provider of a peer-to-peer network.

5. The method according to claim 4, wherein: The seed file is generated by the distribution server according to the distribution strategy of the distribution management controller shared by the multiple data download terminals and the data file.

6. A data processing device, comprising: A first acquisition request sending module, used for sending a first acquisition request for a first file block of a data file to a data provider of a peer-to-peer network according to a business process of a target data download terminal, wherein the target data download terminal runs at least one business process; A first file block acquisition module, configured to acquire a first file block of the data file provided by the data provider according to the first acquisition request; A first file block storage module, used to store the first file block of the data file into the memory space corresponding to the business process, where the memory space corresponding to the business process is used to store data directly retrieved when the business process is running; Wherein, the first acquisition request sending module includes: a data provider acquiring unit, configured to acquire a data provider of a peer-to-peer network from a distribution server shared by multiple data download terminals, when it is determined through a virtual file system of a target data download terminal that the target data download terminal does not have the first file block requested by the first acquisition request; A sending unit, configured to send the first acquisition request to the data provider; The point-to-point network further comprises a data source end; the data provider end is other data download ends among the multiple data download ends except the target data download end, and the other data download ends store the first file block of the data file directly or indirectly derived from the data source end; or, the data provider end is the data source end; Wherein, the device further comprises: A read request initiating module, used to initiate a read request to the data service of the target data downloading terminal using the business process of the target data downloading terminal; A read interface module, used to generate a read interface using the data service of the target data download end, and send the read request to the download service of the target data download end through the read interface; A starting module, used to use the download service of the target data download terminal to start the point-to-point download client at the target data download terminal; The provider acquisition request module is used to initiate a data provider acquisition request for the first file block to the distribution server through the point-to-point download client.

7. The device according to claim 6, wherein: The first file block storage module includes: a cache unit, configured to store the first file blocks in a buffer storage area corresponding to the data file according to the order of the first file blocks in the data file; A first transfer unit is used to move the first file block in the buffer storage area to the memory space corresponding to the business process when the buffer storage area corresponding to the data file is full; And / or, a second transfer unit is used to move the first file block called by the business process to the memory space corresponding to the business process when the business process calls the first file block in the buffer storage area.

8. The device according to any one of claims 6 to 7, wherein: The device also includes: A second acquisition request receiving module, used for receiving a second acquisition request for a file block of a data file sent by another data download terminal in the peer-to-peer network; The second file block sending module is used to send the second file block of the data file stored on the target data download end to the other data download end according to the second acquisition request.

9. The device according to any one of claims 6 to 7, wherein: The first acquisition request sending module includes: A seed file unit, used to obtain a seed file for downloading a first file block of the data file through a download service of the target data download terminal; The seed file processing unit is used to send a first acquisition request for a first file block of a data file to a data provider of a peer-to-peer network according to the seed file.

10. The device according to claim 9, wherein: The seed file is generated by the distribution server according to the distribution strategy of the distribution management controller shared by the multiple data download terminals and the data file.

11. A data processing system, wherein: include: A data source end and a plurality of data download ends connected to the data source end; At least one of the multiple data download terminals includes a data processing device as described in any one of claims 6-10.

12. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.

14. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images