A data moving method, device, apparatus and storage medium

Abstract

The application discloses a data moving method and device, equipment and storage medium, comprising: calculating the hash value of each piece of data in the local cache, wherein the hash value is used to represent the node identification of receiving the data; creating an array with the same capacity as the local cache, and saving the hash values of the data in the array in sequence; establishing a mapping relationship between the array and the local cache, and determining the aggregated data from the local cache by referring to the mapping relationship; obtaining the target hash value corresponding to the aggregated data, and copying the aggregated data to the target node cache matched with the target hash value to move the data. By creating a hash array for marking and binding the data, the aggregated data is determined from the local cache by referring to the array, and the same aggregated data of adjacent receiving nodes is copied to the target node cache at a time, so that the number of copying times is reduced, and the I / O load pressure is relieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data moving method, device, equipment and storage medium. BACKGROUND

[0002] In a distributed database system, data moving is a key mechanism to realize parallel computing, and through reorganizing and distributing data of each node, a distributed execution plan can efficiently complete complex operations. Figure 1 As shown in the figure, the general process of data moving is generally as follows: data is batched into a local cache, each piece of data in the cache is subjected to hash calculation, data is copied to the cache of a target node according to the hash result, and when the cache of the target node reaches a sending condition, the data is sent to the target node.

[0003] Since each piece of data is subjected to hash calculation in the existing data moving process and then moved from the cache of a source node to the cache of a target node, the number of data copying and the amount of data are consistent, so there is a serious I / O performance bottleneck in the current data moving. SUMMARY

[0004] The present application provides a data moving method to reduce the number of data copying in the data moving process.

[0005] According to a first aspect of the present application, a data moving method is provided, comprising: calculating a hash value of each piece of data in a local cache, wherein the hash value is used to represent a node identifier receiving the data;

[0006] creating an array with the same capacity as the local cache, and saving the hash values of the data in sequence into the array;

[0007] establishing a mapping relationship between the array and the local cache, and determining aggregated data from the local cache by referring to the mapping relationship, wherein the aggregated data contains multiple pieces of data with the same hash value;

[0008] obtaining a target hash value corresponding to the aggregated data, and copying the aggregated data to a target node cache matching the target hash value to perform data moving.

[0009] According to another aspect of the present application, a data moving device is provided, comprising: a hash value calculation module configured to calculate a hash value of each piece of data in a local cache, wherein the hash value is used to represent a node identifier receiving the data;

[0010] an array creation module configured to create an array with the same capacity as the local cache, and save the hash values of the data in sequence into the array;

[0011] a data aggregation determining module, configured to establish a mapping relationship between the array and the local cache, and determine aggregation data from the local cache according to the mapping relationship, wherein the aggregation data contains a plurality of data with the same hash value;

[0012] a data replication module, configured to obtain a target hash value corresponding to the aggregation data, and replicate the aggregation data to a target node cache matched with the target hash value, so as to move data.

[0013] According to another aspect of the present application, a mobile device is provided, comprising:

[0014] at least one processor; and

[0015] a memory connected with the at least one processor; wherein,

[0016] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the method according to any one of the embodiments of the present application.

[0017] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to execute the method according to any one of the embodiments of the present application.

[0018] According to another aspect of the present application, a computer program product is provided, which comprises a computer program for enabling a processor to execute the method according to any one of the embodiments of the present application.

[0019] The present application has the beneficial technical effect that by creating a hash array for marking and binding data, and determining aggregation data from the local cache according to the array, and replicating the same aggregation data of adjacent receiving nodes to the target node cache once, the number of replication is reduced, and the I / O load pressure is relieved.

[0020] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present application, nor to limit the scope of the present application. Other features of the present application will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to make the technical solution in the embodiments of the present application clearer, the drawings needed in the embodiments description will be briefly introduced as follows. Obviously, the drawings in the embodiments description are only some of the embodiments of the present application, and all the other embodiments obtained by those skilled in the art without any creative effort based on the embodiments in the present application shall fall into the protection scope of the present application.

[0022] Figure 1 is a scene diagram of data movement in the prior art;

[0023] Figure 2 is a flow chart of a data movement method according to the first embodiment of the present application;

[0024] Figure 3 is a scene diagram of data movement according to the first embodiment of the present application;

[0025] Figure 4 is a flow chart of a data movement method according to the second embodiment of the present application;

[0026] Figure 5 is a structural diagram of a data movement device according to the third embodiment of the present application;

[0027] Figure 6 is a structural diagram of a mobile device according to the fourth embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the technical solution in the embodiments of the present application clearer, the drawings needed in the embodiments description will be briefly introduced as follows. Obviously, the drawings in the embodiments description are only some of the embodiments of the present application, and all the other embodiments obtained by those skilled in the art without any creative effort based on the embodiments in the present application shall fall into the protection scope of the present application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and in the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to such a process, method, product or device. In addition, the collected information is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards of relevant countries and regions, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal

[0030] Embodiment one

[0031] Figure 2 A flowchart of a data moving method is provided for the first embodiment of the present application. The present embodiment can be applied to the case of moving data. The method can be executed by a data moving device, which can be realized in the form of hardware and / or software. As shown in Figure 2 , the method comprises:

[0032] Step S101, calculating the hash value of each piece of data in the local cache.

[0033] Optionally, before calculating the hash value of each piece of data in the local cache, the method further comprises: reading data from a specified hard disk according to a distributed execution plan, wherein the type of the specified hard disk includes a mechanical hard disk, a solid state hard disk and a hybrid hard disk; and writing the read data into the local cache in batches.

[0034] Specifically, when data movement is required in a distributed system, each node in the system receives a distributed execution plan from the server. This plan includes which specific data from the hardware needs to be moved to other nodes. Therefore, the nodes read data from designated hard drives according to the plan. These designated hard drives can include mechanical hard drives, solid-state drives, and hybrid hard drives, etc. This embodiment does not limit the specific type of hard drive; as long as data storage is achieved, it falls within the scope of this application. After reading data from the hard drive, the read data is written to the node's local cache. To avoid excessive processing resource consumption during data writing, a batch writing method can be used. For example, each time a specified amount of data is read, the read data is written to the local cache without waiting for all data to be read before performing the write operation.

[0035] It should be noted that the hash value in this embodiment can be used to represent the node identifier of the receiving data. For example, when the hash value of the data is determined to be N2, it is determined that the data needs to be sent to the node with node identifier N2. Moreover, the hash value of each data is uniquely determined, so a data can only be sent to one node. Of course, this embodiment is only an example and does not limit the specific numerical content of the hash value.

[0036] Step S102: Create an array with the same capacity as the local cache, and save the hash values ​​of each data item into the array in order.

[0037] Among them, such as Figure 3 The diagram shown is a scenario illustration of this implementation. In this implementation, an array with the same capacity as the local cache is created, and the hash values ​​of each data are saved into the array in order. Therefore, each element in the array specifically stores the node identifier that receives the data. Since it is necessary to establish the correspondence between the local cache array and the local cache array later, it is necessary to ensure that the size of the local cache is the same as the capacity of the array, so as to ensure a one-to-one correspondence between the array and the local cache position later.

[0038] For example, if the local cache capacity is determined to be 7, then a corresponding array with a capacity of 7 needs to be created. When the local cache is full of data, the hash values ​​calculated from the 7 data points are sequentially saved into the array according to the order in which the data was stored. Of course, this embodiment is only an example and does not limit the size of the local cache and the array. As long as the two correspond, they are all within the protection scope of this application.

[0039] Step S103: Establish a mapping relationship between the array and the local cache, and determine the aggregated data from the local cache with reference to the mapping relationship.

[0040] Optionally, establish a mapping relationship between the array and the local cache, including: determining the sequence number position of each data in the local cache and the array position of each hash value in the array; establishing a correspondence between the sequence number position and the array position to bind the data with the hash value of the data.

[0041] Optionally, aggregated data can be determined from the local cache by referring to the mapping relationship, including: determining the target array position with the same hash value from the array, and determining the target sequence number position corresponding to the target data position according to the mapping relationship; and taking the data located at the target sequence number position as aggregated data.

[0042] Specifically, this implementation establishes a mapping relationship between the array and the local cache, ensuring a one-to-one match between each piece of data in the local cache and its corresponding hash value in the array. This is achieved by binding data to its hash value through a correspondence between the sequence number and the array position. The sequence number refers to the position of each array element in the local cache, while the array position refers to the position of the hash value within the array. Since the hash values ​​calculated for each piece of data are stored in the data in the order they match the data, establishing this correspondence between the sequence number and the array position effectively binds the data to its hash value. For example, when the sequence number of data x in the local cache is 1, the hash value N2 corresponding to data x is also located at position 1 in the array. Figure 3 As shown, by establishing the correspondence between the two types of positions, the binding of each piece of data to its hash value is achieved. Subsequently, when copying data, the corresponding data can be found based on the hash value in the array and the above mapping relationship.

[0043] In this embodiment, since the hash value in the data can identify which node the corresponding data was sent to, to avoid repeated processing of multiple data items sent to the same node, this embodiment identifies the multiple data items sent to the same node and treats them as aggregated data. Therefore, the aggregated data in this embodiment specifically contains multiple data items with the same hash value, and these multiple data items in the aggregated data are required to be adjacent. Furthermore, the aggregated data in this embodiment does not actually perform a data aggregation operation, because data aggregation also consumes certain computing resources. Therefore, the aggregated data in this embodiment essentially identifies adjacent data items sent to the same node without performing an aggregation operation. Also, in this embodiment, data items sent to the same node but not adjacent are not included in the scope of the aggregated data.

[0044] For example, such as Figure 3As shown, the hash values of the three data with the sequence number positions of 4, 6 and 7 in the local cache are all N3, but only the two data with the sequence number positions of 6 and 7 are adjacent, so the two data with the sequence number positions of 6 and 7 are taken as the aggregated data; the hash values of the two data with the sequence number positions of 1 and 2 in the local cache are all N2 and are adjacent, so the two data with the sequence number positions of 1 and 2 are taken as the aggregated data, so for the target hash value N2, there are mainly two sets of aggregated data, of course, the number of data contained in each aggregated data is not limited in the embodiment. Figure 3

[0045] In step S104, the target hash value corresponding to the aggregated data is obtained, and the aggregated data is copied into the target node cache matching the target hash value, so as to move the data.

[0046] Optionally, obtaining the target hash value corresponding to the aggregated data and copying the aggregated data into the target node cache matching the target hash value comprises: obtaining the target hash value bound to the aggregated data, and querying the node storage list according to the target hash value to obtain the target node, wherein the node storage list saves the correspondence between the hash value and the node; obtaining the target node cache pre-allocated for the target node, and copying the aggregated data into the target node cache once.

[0047] Specifically, in the embodiment, after the aggregated data is obtained, the target hash value bound to the aggregated data is obtained, for example, the aggregated data obtained above includes two sets, the first aggregated data contains two data with the sequence number positions of 1 and 2, and the second aggregated data contains two data with the sequence number positions of 6 and 7, the target hash value bound to the first aggregated data is N2, and the target hash value bound to the second aggregated data is N3, after the target hash value is obtained, the target node can be obtained according to the target hash value by querying the node storage list, and the correspondence between the hash value and the node is saved in the node storage list, for example, the correspondence between N2 and node 2 and the correspondence between N3 and node 3 are saved in the node storage list, and the specific content stored in the node storage list is not limited in the embodiment.

[0048] ​It should be noted that in the embodiment, the node cache is pre-allocated for different nodes, so after the target node cache corresponding to the aggregated data is determined according to the target hash value, the aggregated data can be copied into the target node cache once, thereby avoiding the problem of large I / O load pressure caused by performing multiple copy operations. For example, for the first aggregated data containing two data with serial number positions of 1 and 2, the two data contained in the aggregated data can be copied into the target node N2 cache at the same time; for the second aggregated data containing two data with serial number positions of 6 and 7, the two data contained in the aggregated data can be copied into the target node N3 cache at the same time. Therefore, for the local cache containing 7 data, only 5 copy operations are required to complete the movement of data from the local cache to the target node cache due to the aggregation. Of course, in the embodiment, only an example is given, and the specific number of data copy operations is not limited, as long as it is less than the number of single data, which is within the protection scope of the application. Moreover, the aggregated data copied at the same time is included in each target node cache, and the single data copied independently is also included. In the embodiment, the specific copy method of the data contained in the target node cache is not limited, as long as the data in the target node cache is sent to the target node, which is within the protection scope of the application. Figure 3

[0049] In the embodiment, the hash array for marking and binding the data is created, the aggregated data is determined from the local cache by referring to the array, and the aggregated data with the same adjacent receiving node is copied into the target node cache once, thereby reducing the number of copies and realizing the relief of I / O load pressure.

[0050] Embodiment Two

[0051] Figure 4 A data movement method flowchart is provided for the second embodiment of the application. Based on the above-mentioned embodiment, after the aggregated data is copied into the target node cache matched with the target hash value, the embodiment further includes: when it is determined that the target node cache satisfies the sending condition, performing compression processing on the aggregated data in the target node cache to obtain compressed data, performing serialization processing on the compressed data to obtain serialized data, and sending the serialized data to the target node for data movement.

[0052] In step S201, the hash value of each data in the local cache is calculated.

[0053] ​Optionally, before calculating the hash value of each piece of data in the local cache, further comprising: reading data from the specified hard disk according to the distributed execution plan, wherein the type of the specified hard disk includes a mechanical hard disk, a solid state hard disk and a hybrid hard disk; and writing the read data into the local cache in batches.

[0054] Step S202, an array with the same capacity as the local cache is created, and the hash values of the data are sequentially saved into the array.

[0055] Step S203, a mapping relationship between the array and the local cache is established, and the aggregated data is determined from the local cache by referring to the mapping relationship.

[0056] Optionally, the mapping relationship between the array and the local cache is established, comprising: determining the sequence number position of each data in the local cache and the array position of each hash value in the array; and establishing a corresponding relationship between the sequence number position and the array position to bind the data and the hash value of the data.

[0057] Optionally, the aggregated data is determined from the local cache by referring to the mapping relationship, comprising: determining a target array position with the same hash value from the array, and determining a target sequence number position corresponding to the target data position according to the mapping relationship; and taking the data at the target sequence number position as the aggregated data.

[0058] Step S204, the target hash value corresponding to the aggregated data is obtained, and the aggregated data is copied to the target node cache matching the target hash value for data movement.

[0059] Optionally, the target hash value corresponding to the aggregated data is obtained, and the aggregated data is copied to the target node cache matching the target hash value, comprising: obtaining the target hash value bound to the aggregated data, and querying the node storage list according to the target hash value to obtain the target node, wherein the node storage list stores the correspondence between the hash value and the node; obtaining the target node cache pre-allocated for the target node, and copying the aggregated data to the target node cache once.

[0060] Step S205, when it is determined that the target node cache satisfies the sending condition, the data in the target node cache is compressed to obtain compressed data, the serialized data is obtained after the serialized processing of the compressed data, and the serialized data is sent to the target node for data movement.

[0061] Specifically, in the embodiment, the target node cache is supervised in real time, and when it is determined that the target node cache meets the sending condition, the data in the target node cache is processed and then sent to the target node, where the sending condition can be that the data amount in the target node cache reaches a data amount threshold, for example, when the data amount threshold is 3, as long as three data are stored in the target node cache, the three data are moved to the target node, that is, sent, and therefore, in the embodiment, the data in the target node cache can be sent in the form of allocation.

[0062] It is worth mentioning that in the embodiment, when the data amount in the target node cache reaches the data amount threshold, the data in the target node cache is compressed to obtain compressed data, and at this time, the data to be compressed can be single data independently copied or aggregated data simultaneously copied, and the embodiment does not limit the specific type of data to be compressed. In the embodiment, the data compression can improve the transmission efficiency during data movement, and after the data compression is completed, the compressed data is serialized to obtain serialized data, where data serialization is the process of converting data structure or object state into a format that can be stored or transmitted, such as binary, JSON or XML, so that the original data can be reconstructed when needed, and in the embodiment, after the serialization, the transmission speed of the data can be further improved, and the security of data transmission is improved, and after the serialized data is obtained, the serialized data can be sent to the target node. Correspondingly, the target node receives the serialized data, deserializes and decompresses the data.

[0063] It should be noted that in the embodiment, after the data movement is completed, it is further detected whether the feedback of the target node is received within a specified time range, and when it is determined that the feedback is not received within the specified time range, it means that the target node fails to receive the data, at this time, an alarm prompt information is generated and displayed on the man-machine interaction interface, so that the maintenance personnel can perform corresponding maintenance on the software or hardware device according to the alarm prompt information.

[0064] In the embodiment, a hash array for marking and binding data is created, and the aggregated data is determined from the local cache by referring to the array, and the same aggregated data of adjacent receiving nodes is copied to the target node cache at a time, thereby reducing the number of copying times and relieving the I / O load pressure.

[0065] Embodiment three

[0066] Figure 5 A structure diagram of a data movement device provided for the third embodiment of the application is shown in FIG. 3. Figure 5As shown, the device comprises a hash value calculation module 310, an array creation module 320, an aggregated data determination module 330 and a data replication module 340.

[0067] The hash value calculation module 310 is configured to calculate a hash value of each piece of data in the local cache, wherein the hash value is used to represent the node identity of the received data.

[0068] The array creation module 320 is configured to create an array with the same capacity as the local cache and save the hash values of the data in sequence in the array.

[0069] The aggregated data determination module 330 is configured to establish a mapping relationship between the array and the local cache and determine aggregated data from the local cache according to the mapping relationship, wherein the aggregated data contains multiple pieces of data with the same hash value.

[0070] The data replication module 340 is configured to obtain a target hash value corresponding to the aggregated data and replicate the aggregated data to a target node cache matching the target hash value for data movement.

[0071] Optionally, the device further comprises a data writing module configured to read data from a specified hard disk according to a distributed execution plan, wherein the type of the specified hard disk includes a mechanical hard disk, a solid state hard disk and a hybrid hard disk.

[0072] The read data is written into the local cache in batches.

[0073] Optionally, the aggregated data determination module comprises a mapping unit configured to determine the sequence number position of each piece of data in the local cache and the array position of each hash value in the array.

[0074] The sequence number position and the array position are associated to bind the data and the hash value of the data.

[0075] Optionally, the aggregated data determination module comprises an aggregated data determination unit configured to determine a target array position with the same hash value from the array and determine a target sequence number position corresponding to the target data position according to the mapping relationship.

[0076] The data located at the target sequence number position is taken as the aggregated data.

[0077] Optionally, the data replication module is configured to obtain a target hash value bound to the aggregated data and query a node storage list to obtain a target node according to the target hash value, wherein the node storage list stores the correspondence between the hash value and the node.

[0078] The target node cache pre-allocated for the target node is obtained and the aggregated data is replicated into the target node cache once.

[0079] Optionally, the apparatus further comprises a data moving module configured to compress the aggregated data in the target node cache to obtain compressed data when it is determined that the target node cache satisfies the sending condition, wherein the sending condition comprises that the data amount reaches a data amount threshold.

[0080] The compressed data is serialized to obtain serialized data, and the serialized data is sent to the target node for data moving.

[0081] The data moving apparatus provided by the embodiments of the present application can execute the data moving method provided by any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0082] Embodiment four

[0083] Figure 6 A structural diagram of a mobile device 10 that can be used to implement embodiments of the present application is shown. The mobile device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0084] As shown in Figure 6 The mobile device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., connected to the at least one processor 11 in communication, wherein the memory stores a computer program executable by the at least one processor 11, and the processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the mobile device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0085] A plurality of components in the mobile device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the mobile device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0086] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the data movement method.

[0087] In some embodiments, the data movement method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the mobile device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the data movement method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the data movement method by any other appropriate means, such as by means of firmware.

[0088] The various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0089] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, enables the functions / acts specified in the flowcharts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0090] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0091] To provide for interaction with a user, the systems and techniques described here can be implemented on a mobile device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the mobile device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0092] The systems and techniques described here can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0093] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0094] Embodiment Five

[0095] The embodiment of the application further provides a computer program product, comprising a computer program which, when executed by a processor, implements the data moving method provided in any embodiment of the application.

[0096] The computer program code implementing the application can be written in one or more programming languages or combinations of languages including object oriented languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0097] It should be noted that in the embodiments of the present application, some software, components, models and other prior art solutions can be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but does not mean that the applicant has or will necessarily use the solution.

[0098] It should be noted that the above are only preferred embodiments of the present application and the technical principles applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and those skilled in the art can make various obvious changes, readjustments and substitutions without departing from the scope of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A data movement method, characterized by, The method comprises: calculating the hash value of each piece of data in the local cache, wherein the hash value is used to represent the node identifier receiving the data; creating an array with the same capacity as the local cache and saving the hash values of the data in sequence in the array; establishing a mapping relationship between the array and the local cache, and determining the aggregated data from the local cache by referring to the mapping relationship, wherein the aggregated data contains multiple pieces of data with the same hash value, and the multiple pieces of data in the aggregated data are adjacent; obtaining the target hash value corresponding to the aggregated data, and copying the aggregated data into the target node cache matching the target hash value for data movement; the establishment of the mapping relationship between the array and the local cache comprises: determining the serial number position of each data in the local cache and the array position of each hash value in the array; establishing the corresponding relationship between the serial number position and the array position to bind the data and the hash value of the data.

2. The method of claim 1, wherein, Before the calculation of the hash value of each piece of data in the local cache, the method further comprises: reading data from a specified hard disk according to a distributed execution plan, wherein the types of the specified hard disk include mechanical hard disk, solid state hard disk and hybrid hard disk; writing the read data into the local cache in batches.

3. The method of claim 1, wherein, The determination of the aggregated data from the local cache by referring to the mapping relationship comprises: determining the target array position with the same hash value from the array, and determining the target serial number position corresponding to the target array position according to the mapping relationship; the data located at the target serial number position is taken as the aggregated data.

4. The method of claim 1, wherein, The obtaining of the target hash value corresponding to the aggregated data and the copying of the aggregated data into the target node cache matching the target hash value comprises: obtaining the target hash value bound with the aggregated data, and querying the node storage list to obtain the target node according to the target hash value, wherein the node storage list saves the corresponding relationship between the hash value and the node; obtaining the target node cache pre-allocated for the target node, and copying the aggregated data into the target node cache at a time.

5. The method of claim 1, wherein, After the copying of the aggregated data into the target node cache matching the target hash value, the method further comprises: when it is determined that the target node cache satisfies the sending condition, performing compression processing on the data in the target node cache to obtain compressed data, wherein the sending condition includes that the data amount reaches a data amount threshold; after the serialization processing of the compressed data, obtaining serialized data, and sending the serialized data to the target node for data movement.

6. A data moving device, characterized by The device comprises: a hash value calculation module configured to calculate the hash value of each piece of data in the local cache, wherein the hash value is used to represent the node identifier receiving the data; an array creation module configured to create an array with the same capacity as the local cache and save the hash values of the data in sequence in the array; The aggregation data determination module is configured to establish a mapping relationship between the array and the local cache, and determine aggregation data from the local cache according to the mapping relationship, wherein the aggregation data contains a plurality of data with the same hash value. The data replication module is configured to obtain a target hash value corresponding to the aggregation data, and replicate the aggregation data to a target node cache matching the target hash value, so as to move data. The aggregation data determination module is further configured to determine a serial number position of each data in the local cache and an array position of each hash value in the array. A corresponding relationship between the serial number position and the array position is established to bind the data and the hash value of the data.

7. A terminal device, characterized by comprising: The terminal device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to execute the method of any one of claims 1-5 when executed.

9. A computer program product, characterised in that, The computer program is executed by the processor to implement the method of any one of claims 1-5.

Images