Data backsource scheduling method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a data backsource scheduling method and device, electronic equipment and storage medium, relates to the field of artificial intelligence, in particular to cloud computing and cloud storage technology, and can be applied to the intelligent cloud scene. The specific implementation scheme is as follows: in the case that the local device does not store target data, it is determined to obtain the target data from a source device; target segmented data is received, wherein the target segmented data is data obtained by segmenting the target data; after receiving the target segmented data, the reception of other segmented data in the target data is suspended; and after completing a predetermined operation on the target segmented data, the reception of the other segmented data is continued.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, in particular to cloud computing and cloud storage technology, which can be applied to intelligent cloud scenarios. BACKGROUND

[0002] In related technologies, in the process of tracing back data from the source end, the target data required is directly returned from the source end to the client. When the bandwidth of the source end returning data is low or the bandwidth of a certain client receiving data is low, data congestion and backlog will occur, affecting normal data reading and writing. SUMMARY

[0003] The present disclosure provides a data source scheduling method and device for avoiding data source congestion, an electronic device and a storage medium.

[0004] According to an aspect of the present disclosure, a data source scheduling method is provided, comprising: determining to obtain target data from a source device in the case that the target data is not stored in a local device; receiving target segmented data, wherein the target segmented data is data obtained by segmenting the target data; pausing to receive other segmented data in the target data after receiving the target segmented data, wherein the other segmented data is segmented data other than the target segmented data in the target data; and continuing to receive the other segmented data after completing a predetermined operation on the target segmented data.

[0005] Optionally, before receiving the target segmented data, the method further comprises: receiving a read request of a client, wherein the read request is used to obtain the target data; establishing a transmission link corresponding to the read request, wherein the transmission link corresponds to an identifier of the client one by one, and the transmission link is used to transmit the target data segmented by a mirror source thread from the source device to other devices; and registering the transmission link in the mirror source thread to point the transmission link to the source device.

[0006] Optionally, receiving the target segmented data comprises: determining that the segmented data currently transmitted by the transmission link is the target segmented data; temporarily closing a read thread corresponding to the read request, and receiving the target segmented data based on the transmission link.

[0007] Optionally, after completing the predetermined operation on the target segmented data, continuing to receive the other segmented data comprises: storing the target segmented data in a cache of the local device through the mirror source thread; storing the target segmented data stored in the cache in an object storage system of the client and the local device according to the read request, and continuing to receive the other segmented data.

[0008] Optionally, the segmented data stored in the cache is stored into the object storage system of the client and the local device, including: waking up a read thread corresponding to the read request; and storing the target segmented data stored in the cache into the object storage system of the client and the local device based on the read thread.

[0009] Optionally, the other segmented data in the target data is paused, including: adjusting the transmission link to a locked state, wherein in the locked state, the other segmented data in the target data is paused.

[0010] Optionally, the other segmented data is continued to be received, including: closing the read thread, determining that the mirror source retrieval thread acquires the other segmented data from the source device; releasing the locked state of the transmission link, and after the locked state is released, receiving the other segmented data from the source device through the transmission link.

[0011] Optionally, the target segmented data is determined by: determining a target amount of data, wherein the data amount is used to indicate the data size of the target data, and the data amount is segmented according to a preset data amount to obtain the target segmented data; or detecting a network bandwidth of the transmission link, determining a transmissible data amount of the transmission link according to the network bandwidth, and segmenting the target data according to the transmissible data amount to obtain the target segmented data, wherein the greater the network bandwidth, the greater the data amount of the target segmented data.

[0012] According to another aspect of the present disclosure, a data source retrieval scheduling apparatus is provided, including: a determination module configured to determine to acquire target data from a source device in a case where the target data is not stored in a local device; a first receiving module configured to receive target segmented data, wherein the target segmented data is data obtained by segmenting the target data; a control module configured to pause receiving other segmented data in the target data after receiving the target segmented data, wherein the other segmented data is segmented data other than the target segmented data in the target data; and a second receiving module configured to continue to receive the other segmented data after a predetermined operation on the target segmented data is completed.

[0013] According to another aspect of the present disclosure, an electronic device is also provided, including: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the above data source retrieval scheduling methods.

[0014] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable a computer to perform any of the above data source retrieval scheduling methods.

[0015] According to another aspect of the present disclosure, there is also provided a computer program product comprising a computer program which, when executed by a processor, implements the data backsource scheduling method of any of the above.

[0016] It should be understood that the contents described in this part are not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0018] Figure 1 is a flowchart of an optional data backsource scheduling method according to the present disclosure;

[0019] Figure 2 is a task scheduling diagram of data backsource in an exemplary embodiment of the present disclosure;

[0020] Figure 3 is a schematic diagram of an application scenario of an optional data backsource scheduling method of the present disclosure;

[0021] Figure 4 is a structural schematic diagram of an optional data backsource scheduling device according to the present disclosure;

[0022] Figure 5 is a structural schematic diagram of another optional data backsource scheduling device according to the present disclosure;

[0023] Figure 6 shows a structural schematic diagram of another optional electronic device that can be used to implement the embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] The exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to help understanding, and should be considered only as exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, the description below omits the description of well-known functions and structures.

[0025] In order to facilitate those skilled in the art to better understand the embodiments related to the present disclosure, the technical terms or part of nouns that the embodiments of the present disclosure can involve are explained as follows:

[0026] Cache, in its original sense, refers to a kind of high-speed memory with faster access speed than general random access memory (English full name: Random Access Memory, abbreviation: RAM). It is usually not as system main memory that uses DRAM technology, but uses expensive but faster SRAM technology. The setting of cache is one of the important factors for all modern computer systems to exert high performance.

[0027] CURL (full name Command Line URL Viewer) is an open source command line tool and a cross-platform library for transferring data between servers and distributing to almost all new operating systems. CURL programming is used almost anywhere that needs to send or receive data through Internet protocols. CURL supports almost all Internet protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, etc.).

[0028] WGET is a free tool for automatically downloading files from the network, supporting downloading through the three most common TCP / IP protocols: HTTP, HTTPS, and FTP, and can use HTTP proxy. The name "WGET" comes from the combination of "World Wide Web" and "get". So-called automatic download means that WGET can continue to execute in the background after the user exits the system until the download task is completed.

[0029] Object storage data back source: The mirror back source function of object storage refers to that when the resource requested by the user of the website does not exist, the correct data can be obtained from the set source address through the back source rule, and then returned to the user of the website.

[0030] Bandwidth: Network bandwidth refers to the maximum bit data that can be passed in a fixed time (for example, 1 second). The application field of bandwidth is very wide, which can be used to identify the data transmission capacity of signal transmission, identify the data amount passing through the link in unit time, and identify the display capacity of the display. In analog signal systems, it is also called frequency width, which refers to the amount of data that can be transmitted in a fixed time, that is, the ability to transmit data in the transmission pipeline. It is usually represented by cycles per second or hertz (Hz). In digital devices, bandwidth refers to the amount of data passing through the link in unit time. It is usually represented by bps, that is, the number of bits that can be transmitted per second.

[0031] Digest algorithm is also called hash algorithm and hash algorithm. It converts data of arbitrary length into a fixed-length data string (usually represented by a 16-bit string) through a function.

[0032] Figure 1is a flowchart of a data backsource scheduling method according to the present disclosure, as shown in the figure, the method comprises: Figure 1

[0033] S102, in the case where the local device does not store the target data, determining to obtain the target data from the source device;

[0034] S104, receiving target segment data, wherein the target segment data is data obtained by segmenting the target data;

[0035] S106, after receiving the target segment data, pausing to receive other segment data in the target data; and

[0036] S108, after completing the predetermined operation on the target segment data, continuing to receive the other segment data.

[0037] Through the above-mentioned data backsource scheduling method, the target data can be read from the source device in the case where the local device does not store the target data, and the target data is not read at one time, but is received in the form of segments, and after receiving the target segment data, the reception of other segment data of the target data is paused, and then, after completing the predetermined operation on the target segment data, the reception of other segment data is processed. Through the above-mentioned method, the target data with large data volume can be segmented for transmission in the form of segment data, saving the processing resources in the data backsource process and reducing the data processing pressure of each processing thread. It can be understood that since the reception of other segment data is performed after the completion of the predetermined operation on the target segment data, the balance between the bandwidth of receiving segment data and the bandwidth of performing the predetermined operation is indirectly ensured.

[0038] In an exemplary embodiment, the read request of the client can be received before receiving the target segment data, wherein the read request is used to obtain the target data, and then the transmission link corresponding to the read request can be established. It should be noted that the transmission link corresponds to the identity of the client one by one, and the transmission link is used to transmit the target data segment obtained from the source device by the mirror backsource thread to other devices; the transmission link is registered in the mirror backsource thread to point to the source device. It can be understood that the transmission link corresponds to the identity of the client one by one, so for different clients, different transmission links are used for data transmission, ensuring that the data reading requests of each client are relatively independent, so even if the transmission link of a certain client fails or data congestion occurs, it will not affect the reading of data from the source device by other client devices.

[0039] ​It is easily noticed that, after receiving the read request of the client, if the target data is stored in the local device, the target data is directly read from the local device. It is also necessary to note that the other segment data is the segment data other than the target segment data in the target data.

[0040] It is necessary to note that the transmission link includes but is not limited to the link corresponding to WGET and the link corresponding to CURL. The transmission link can support various Internet protocols such as DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, and LDAP.

[0041] In some embodiments of the present disclosure, receiving the target segment data can be achieved by the following manner. Specifically, it is determined that the segment data currently transmitted by the transmission link is the target segment data. The read thread corresponding to the read request is temporarily closed, and the target segment data is received based on the transmission link.

[0042] It is easily noticed that, closing the read thread of the read request can pause the client device from reading data from the local device, saving the bandwidth resources, and achieving the balance between the bandwidth of transmitting data from the source device and the bandwidth of receiving data by the client.

[0043] In order to ensure that the data with higher priority is transmitted preferentially, in some other optional embodiments, receiving the target segment data can also be achieved by the following manner. The local device sends data identification information to the source device, wherein the data identification information can be used to indicate the acquisition priority of each segment data in the target data. If the priority is higher, it means that the local device needs to acquire the data preferentially. Accordingly, the source device can mark the segment data according to the data identification information, and send the data that needs to be acquired preferentially by the local device to the local device. For example, each segment data can be sorted according to the acquisition priority indicated by the data identification information. The higher the acquisition priority, the higher the sorting, and the data is preferentially sent to the local device.

[0044] In some embodiments of the present disclosure, after completing the predetermined operation on the target segment data, continuing to receive other segment data can be achieved by the following steps. The target segment data is stored in the cache of the local device through the mirror source thread, the target segment data stored in the cache is stored in the object storage system of the client and the local device according to the read request, and the other segment data is continuously received.

[0045] The mirror source thread is responsible for processing the source data alone, which can avoid the interference of other read-write requests on the source data processing process, and also avoid affecting other normal read-write requests of the client. Meanwhile, it is easily noticed that the target segmented data currently received is processed, and then other data is received, which avoids data blocking, saves the bandwidth of the mirror source thread, and reduces the processing pressure of the mirror source thread.

[0046] Specifically, storing the segmented data stored in the cache into the object storage system of the client and the local device includes: waking up the read thread corresponding to the read request; and storing the target segmented data stored in the cache into the object storage system of the client and the local device based on the read thread.

[0047] It can be understood that, since the read thread is temporarily closed in order to avoid occupying the bandwidth required by the mirror source thread, the read thread can be re-woken up here to ensure that the data storage works normally, and it should be noted that the read thread can store the target segmented data into the object storage system of the client and the local device at the same time to avoid occupying the resources of the cache for a long time. In addition, it should be noted that storing the obtained data into the object storage system of the local device can ensure that the data is obtained directly from the object storage system when the same data request is obtained next time, without obtaining the data from the source device again.

[0048] In order to reduce the data storage pressure of the cache and save storage space, the target segmented data in the cache can be deleted after the target segmented data stored in the cache is stored into the object storage system of the client and the local device, so as to avoid too much data in the cache, which causes the cache to run slowly and affects the read-write of other data.

[0049] Optionally, in order to avoid that the target segmented data in the storage system of the local device is mistakenly deleted, and when other client devices need to obtain the target segmented data, the data is obtained from the server again, the target segmented data can be set with a valid cache period. Within the valid cache period, even if the target segmented data in the storage system of the local device is mistakenly deleted, the data can still be obtained from the cache. For example, the valid cache period is set to 1h, so that within 1h, even if the target segmented data in the storage system of the local device is mistakenly deleted, the data can still be obtained from the cache. After 1h, the target segmented data can be cleared from the cache to save the storage space of the cache and ensure the normal read-write of other data.

[0050] It should be noted that, since the target data is segmented and transmitted in some examples of the present disclosure, in order to avoid the data stored in the object storage system or the client being incomplete, the target segmented data in the cache can not be deleted, and other segmented data received subsequently can also be cached, and the complete target data formed is cached in the cache, and then the target data stored in the object storage system or the client is checked, and the target data in the cache is deleted in the case of determining that the data is complete.

[0051] It can be understood that the above data checking can be realized in the following manner: a predetermined checking algorithm is used to calculate the first checking value of the target data in the cache, and the predetermined algorithm is used to calculate the second checking value of the target data stored in the storage system or the client, if the first checking value is the same as the second checking value, it is determined that the target data stored in the client and the object storage system is complete, and then the target data in the cache is deleted. Obviously, if the first checking value is different from the second checking value, it means that the target data in the client and the object storage system is missing, at this time, the missing segmented data needs to be determined, and the missing segmented data is stored from the cache to the client and the object storage system.

[0052] It should be noted that the above predetermined checking algorithm includes but is not limited to parity check, crc cyclic redundancy check, and modulo 2 division check algorithm.

[0053] In other embodiments, the above process of checking data integrity can also be realized in the following manner: a digest algorithm can be used to calculate the first digest value of the target data in the cache, and the first digest value is signed by using the private key of the client to obtain a signature value, and then the signature value is decrypted by using the public key of the client to obtain a second digest value, in the case that the first digest value is consistent with the second digest value, it is determined that the target data stored in the client and the object storage system is complete. Similarly, if the first digest value is different from the second digest value, it means that the target data in the client and the object storage system is missing, at this time, the missing segmented data needs to be determined, and the missing segmented data is stored from the cache to the client and the object storage system.

[0054] It should be noted that the above digest algorithm includes but is not limited to MD5, SHA1, SHA512 algorithm.

[0055] It should be noted that the other segment data in the target data is suspended, and the transmission link can be controlled to be in a locked state. After the transmission link is adjusted to the locked state, the other segment data in the target data can be suspended. In an exemplary embodiment, the state of the transmission link can be controlled by triggering a digital signal. For example, when the digital signal is 1, the transmission link is locked, and when the digital signal is 0, the transmission link is unlocked.

[0056] In some optional embodiments of the present disclosure, the other segment data is continuously received by the following steps. Specifically, the reading thread can be closed, the mirror source thread is determined to obtain the other segment data from the source device, and then the transmission link is unlocked. After the transmission link is unlocked, the other segment data of the source device obtained by the mirror source thread is received through the transmission link. It can be understood that the reading thread is closed to avoid mutual compression of the bandwidth required by the reading thread and the mirror source thread in the process of transmitting data, which affects the processing efficiency of each other. Especially in the case of low available bandwidth, closing the reading thread can effectively ensure that the mirror source thread obtains data from the source device normally.

[0057] It should be noted that the target segment data can be determined by the following method:

[0058] First, the amount of data of the target quantity is determined, wherein the amount of data is used to indicate the data size of the target data, the amount of data is divided according to the preset amount of data, and the target segment data is obtained. The second is to detect the network bandwidth of the transmission link, determine the transmittable data amount of the transmission link according to the network bandwidth, and divide the target data according to the transmittable data amount of the transmission link to obtain the target segment data. It should be noted that the greater the network bandwidth, the greater the data amount of the target segment data.

[0059] It can be understood that the above-mentioned segmentation method of dividing the target data according to the size of the data amount is relatively simple, and can quickly realize the segmentation of the target data. The method of dividing the target data according to the network bandwidth of the transmission link has higher real-time performance, and can dynamically divide the data amount of each segment data according to the network bandwidth, and has higher flexibility.

[0060] Figure 2 is a task scheduling diagram of data source in an exemplary embodiment of the present disclosure, as Figure 2 shown, the task scheduling process can be implemented by the following steps:

[0061] 1. Register CURL (transmission link) to the mirror source thread, and read data from the source device (referred to as source device) through the thread responsible for the mirror source (i.e. mirror source thread).

[0062] 2. Temporarily close the read thread corresponding to the client read request (i.e. the thread processing the client request, which can be referred to as client_thread), and wait to be awakened.

[0063] 3. Determine that the mirror backsource thread reads data from the source device, and then, after reading part of the data, caches the read part of the data in a cache (e.g. cache), and suspends receiving data on the CURL, and then awakens the client_thread.

[0064] 4. After the client_thread is awakened, it begins processing the data in the cache, and stores the data to the client and to the object storage system of the local device (i.e. writes the data to disk). After processing is complete, the suspend flag on the CURL is removed, i.e. the CURL is unlocked, so that it can continue to receive other data from the source device via the CURL.

[0065] 5. The above processes 2-4 are repeatedly executed until all of the data transmitted on the CURL is received, and all of the target data is sent to the client device and stored in the object storage system.

[0066] Figure 3 is an application scenario diagram of a data backsource scheduling method of the present disclosure, as shown in Figure 3 the client can first initiate a read request for target data to the local device. After receiving the read request, the local device determines whether the local device has stored the target data. If not, the local device obtains the target data from the source device, writes the obtained target data to the local device, and then sends the target data to the client device.

[0067] Figure 4 is a structural diagram of a data backsource scheduling apparatus according to the present disclosure, as shown in Figure 4 the data backsource scheduling apparatus comprises:

[0068] a determination module 40, configured to determine to obtain target data from a source device if the local device does not store the target data;

[0069] a first receiving module 42, configured to receive target segmented data, wherein the target segmented data is data obtained by segmenting the target data;

[0070] a control module 44, configured to suspend receiving other segmented data in the target data after receiving the target segmented data;

[0071] a second receiving module 46, configured to continue receiving the other segmented data after completing a predetermined operation on the target segmented data.

[0072] In the scheduling device, a determination module 40 is used to determine whether to obtain target data from a source device when the local device does not store the target data; a first receiving module 42 is used to receive target segmented data, wherein the target segmented data is data obtained by segmenting the target data; a control module 44 is used to suspend receiving other segmented data in the target data after receiving the target segmented data, wherein the above-mentioned other segmented data is segmented data in the target data other than the target segmented data; a second receiving module 46 is used to continue receiving other segmented data after completing a predetermined operation on the target segmented data, thereby realizing that the target data can be read from the source device when the local device does not store the target data, and Instead of reading the required target data all at once, the target data is received in the form of segments. After receiving the target segmented data, the reception of other segmented data of the target data is suspended. Then, after completing the predetermined operation on the target segmented data, the reception of other segmented data is resumed. Through the above method, target data with a large amount of data can be divided and transmitted in the form of segmented data, which saves processing resources in the process of returning data to the source and reduces the data processing pressure of each processing thread. It can be understood that since other segmented data are received after the predetermined operation is completed on the target segmented data, the balance between the bandwidth for receiving segmented data and the bandwidth for executing predetermined operations is indirectly guaranteed.

[0073] Figure 5 is a structural diagram of another data back-source scheduling device according to the disclosed embodiment, such as Figure 5 As shown, the apparatus further includes: a third receiving module 48, for receiving a read request from the client before receiving the target segmented data, wherein the read request is used to obtain the target data; an establishing module 50, for establishing a transmission link corresponding to the read request, wherein the transmission link corresponds one-to-one with the identifier of the client, and the transmission link is used to transmit the target data segment obtained by the mirror back to source thread from the source device to other devices; a registration module 52, for registering the transmission link in the mirror back to source thread to point the transmission link to the source device.

[0074] In some optional embodiments of the present disclosure, the first receiving module 42 includes: a determination unit, used to determine that the segmented data currently transmitted by the transmission link is the target segmented data; a receiving unit, used to temporarily close the read thread corresponding to the read request and receive the target segmented data based on the transmission link.

[0075] In some embodiments of the present disclosure, the second receiving module 46 includes: a first storage unit, used to store the target segmented data in the cache of the local device through the mirror back to the source thread; a second storage unit, used to store the target segmented data stored in the cache in the object storage system of the client and the local device according to the read request, and continue to receive other segmented data.

[0076] Optionally, the second storage unit comprises: a wake-up unit configured to wake up a read thread corresponding to the read request; and a third storage unit configured to store the target segment data stored in the cache into an object storage system of the client and the local device based on the read thread.

[0077] In some optional embodiments of the present disclosure, the control module 44 comprises a control unit configured to adjust the transmission link to a locked state in which the reception of other segment data in the target data is suspended.

[0078] Optionally, the second receiving module 46 comprises: a closing unit configured to close the read thread and determine that the mirror back-to-source thread acquires other segment data from the source device; and a releasing unit configured to release the locked state of the transmission link and receive the other segment data from the source device through the transmission link after the locked state is released.

[0079] According to embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0080] According to embodiments of the present disclosure, an electronic device is provided, which comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data back-to-source scheduling method of any one of the above.

[0081] Optionally, the electronic device can further comprise a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0082] In an example embodiment of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable a computer to perform the data back-to-source scheduling method of any one of the above.

[0083] Optionally, in the present embodiment, the non-transitory computer readable storage medium can be configured to store a computer program for performing the following steps:

[0084] S1, in the case where the local device does not store the target data, determining to acquire the target data from the source device;

[0085] S2, receiving target segment data, wherein the target segment data is data obtained by segmenting the target data;

[0086] S3, after receiving the target segment data, suspending the reception of other segment data in the target data, wherein the other segment data is segment data other than the target segment data in the target data.

[0087] S4, and after completing the predetermined operation on the target segment data, continue to receive the other segment data.

[0088] Optionally, in the embodiment, the non-transitory computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of a readable storage medium would include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0089] In an example embodiment of the present disclosure, a computer program product is also provided, comprising a computer program which, when executed by a processor, implements the scheduling method of data back source of any of the above.

[0090] Optionally, the computer program, when executed by the processor, can implement the following steps:

[0091] S1, determining to obtain target data from a source device in a case that the target data is not stored in a local device;

[0092] S2, receiving target segment data, wherein the target segment data is data obtained by segmenting the target data;

[0093] S3, pausing to receive other segment data in the target data after receiving the target segment data, wherein the other segment data is segment data other than the target segment data in the target data;

[0094] S4, and after completing the predetermined operation on the target segment data, continue to receive the other segment data.

[0095] In the related embodiments of the present disclosure, the target data can be read from the source device in the case that the target data is not stored in the local device, and instead of reading the required target data at one time, the target data is received in segments, and after receiving the target segment data, the receiving of other segment data of the target data is suspended, and then after completing the predetermined operation on the target segment data, the receiving of other segment data is processed again. Through the above method, the target data with large data volume can be segmented and transmitted in the form of segment data, saving the processing resources in the data back-to-source process and reducing the data processing pressure of each processing thread. It can be understood that since the receiving of other segment data is performed after the completion of the predetermined operation on the target segment data, the balance between the bandwidth of receiving segment data and the bandwidth of performing the predetermined operation is indirectly ensured.

[0096] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0097] As Figure 6 shown, the device 600 includes a computing unit 601 that can perform various suitable actions and processes in accordance with computer programs stored in a read-only memory (ROM) 602 or loaded into a random access memory (RAM) 603 from a storage unit 608. Various programs and data required by the device 600 to operate can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0098] Various components in the device 600 are connected to the I / O interface 605, including an input unit 606, such as a keyboard, a mouse, etc., an output unit 607, such as various types of displays, speakers, etc., a storage unit 608, such as a magnetic disk, an optical disk, etc., and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0099] The computing unit 601 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as the scheduling method of data backhauling. For example, in some embodiments, the scheduling method of data backhauling can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded onto the RAM 603 and executed by the computing unit 601, one or more steps of the methods described above, such as the scheduling method of data backhauling, can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the scheduling method of data backhauling by any other suitable means, such as by means of firmware.

[0100] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0101] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine or entirely on a remote machine or server.

[0102] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0103] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0104] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0105] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0106] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure can be achieved, which is not limited herein.

[0107] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for scheduling data back source, comprising: determining to acquire target data from a source device in a case that the target data is not stored in a local device; receiving target segment data, wherein the target segment data is data obtained by segmenting the target data; suspending reception of other segment data in the target data after receiving the target segment data, wherein the other segment data is segment data in the target data other than the target segment data; and continuing to receive the other segment data after completing a predetermined operation on the target segment data; receiving the target segment data comprises: determining that segment data currently transmitted by a transmission link corresponding to a read request of a client is the target segment data, wherein the read request is used to acquire the target data, the transmission link corresponds to an identity of the client in a one-to-one manner, and the transmission link is used to upload the target data acquired by a mirror back source thread from the source device to other devices after segmenting the target data; temporarily closing a read thread corresponding to the read request, and receiving the target segment data based on the transmission link.

2. The method of claim 1, wherein, Before the receiving the target segment data, the method further comprises: receiving the read request of the client; establishing the transmission link corresponding to the read request; registering the transmission link in the mirror back source thread to point the transmission link to the source device.

3. The method of claim 2, wherein, The continuing to receive the other segment data after completing the predetermined operation on the target segment data comprises: storing the target segment data in a buffer of the local device through the mirror back source thread; storing the target segment data stored in the buffer in an object storage system of the client and the local device according to the read request, and continuing to receive the other segment data.

4. The method of claim 3, wherein, The storing the segment data stored in the buffer in the object storage system of the client and the local device comprises: waking up the read thread corresponding to the read request; storing the target segment data stored in the buffer in the object storage system of the client and the local device based on the read thread.

5. The method of claim 4, wherein, The suspending reception of the other segment data in the target data comprises: adjusting the transmission link to a locked state, wherein in the locked state, the reception of the other segment data in the target data is suspended. 6.The method of claim 5, the continuing to receive the other segment data comprises: closing the read thread, and determining that the mirror back source thread acquires the other segment data from the source device; removing the locked state of the transmission link, and receiving the other segment data from the source device through the transmission link after removing the locked state. 7.The method of any one of claims 1-6, the target segment data is determined by: the data amount is used to indicate a data size of the target data, the data amount is segmented according to a preset data amount to obtain the target segment data; or determining a data amount of the target data, wherein ​ ​ Detecting a network bandwidth of the transmission link, determining a transmissible data amount of the transmission link according to the network bandwidth, segmenting the target data according to the transmissible data amount, and obtaining the target segmented data, wherein the greater the network bandwidth is, the greater the data amount of the target segmented data is.

8. A data back source scheduling apparatus, comprising: a determining module configured to determine to acquire target data from a source device in a case that a local device does not store the target data; a first receiving module configured to receive target segmented data, wherein the target segmented data is data obtained by segmenting the target data; the receiving of the target segmented data comprises: determining that segmented data currently transmitted by a transmission link corresponding to a read request of a client is the target segmented data, wherein the read request is used to acquire the target data, the transmission link corresponds to an identity of the client in a one-to-one manner, and the transmission link is used to upload the target data acquired by a mirror back source thread from the source device to other devices after segmenting the target data; temporarily closing a read thread corresponding to the read request, and receiving the target segmented data based on the transmission link; a control module configured to pause receiving other segmented data in the target data after the target segmented data is received, wherein the other segmented data is segmented data other than the target segmented data in the target data; and a second receiving module configured to continue receiving the other segmented data after a predetermined operation on the target segmented data is completed.

9. An electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-7.

11. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-7.

11. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-7.

Images