Method and system for moving data in a cloud environment

Abstract

A method for moving data in a cloud environment includes receiving a request to move data from data sources to a target source, generating a sequence of data movement operations with the aid of an API service, and sending said sequence to a task scheduler. The sequence of operations is added to a task queue of a worker node in accordance with a given priority parameter, a set of interfaces for working with a data source is determined, and the possibility of performing the task using the resources available is checked. An optimal route for movement of the data is then determined and the data are read from an initial data source and transferred to a target data source in accordance with the determined route. The technical result is an increase in the reliability of data movement.

Description

METHOD AND SYSTEM FOR MOVING DATA IN A CLOUD ENVIRONMENT AREA OF TECHNOLOGY

[0001] The invention relates generally to the field of data storage and processing, and in particular to a method and system for moving data in a cloud environment, 5 ensuring timely, uninterrupted delivery and movement of large volumes (from several terabytes) of structured and / or unstructured and / or heterogeneous (multimodal) data, including for the purposes of training machine learning models. LEVEL OF TECHNOLOGY 0

[0002] A solution for managing data storage is known from the prior art, disclosed in patent US 8156086 B2, published 10.04.2012. In the known solution for managing data storage, the following is carried out: - performing a first storage operation on a first set of data at a first location to generate a plurality of data blocks, wherein said performing a first storage operation for at least one of the plurality of data blocks further includes the steps of: generating a second set of data associated with the first set of data; creating a first set of metadata associated with the first set of data; and storing the second set of data and the first set of metadata on a storage device; 20 - storing a copy of the first set of metadata in an index remote from the storage device; - generating a second set of metadata associated with a second set of data stored on the storage device; - comparing the first set of metadata in the index with the second set of metadata to check for differences between the first and second sets of data without access to the first and second sets of data; - performing a second operation of storing a second set of data, the second storage operation includes the steps of: generating a third set of data associated with the second set of data; generating a third set of metadata containing information for retrieving the third set of data; and storing the third set of data in a second location; and - compare the first set of metadata in the index with the third set of metadata to check if there are differences between the first and third sets of data.

[0003] Also known are a method and system for multiplexing pipeline data for the purposes of creating backup copies, described in patent US 7315923 B2, published 01.01.2008. In the known solution, a first data stream is received, containing first data, wherein the first data is received by a first data agent specific to a particular application; a second data stream is received, containing second data, wherein the second data is received by a second data agent for a particular application; the first and second data streams are combined into a single stream from one or more data blocks of an archive file, including recording the first data from the first data stream and the second data from the second data stream into the first data block of one or more data blocks; transmitting one or more data blocks via a transport channel to a backup medium; and storing one or more data blocks on a backup medium.

[0004] Also known are a method and an information management system described in application US 2020 / 0319694 A1, published 02 / 14 / 2023. The known system comprises: a network storage system comprising computer hardware configured to: configure power management parameters for a media agent, wherein the media agent is a component in the information management system that interacts with secondary storage devices to perform operations with data stored using the secondary storage devices; identify, using a power management module, a task to be performed in the network storage system, comprising an operation type in which the task is associated with the media agent; based, at least in part, on the operation type of the task, determine the power state for the media agent associated with the task;and upon determining that the power state of the media agent is at or below a specified level: adding a task to a task list associated with the media agent, wherein the task list comprises tasks of the operation type; determining whether the task list exceeds a specified threshold; and based on the power management module determining that the specified threshold is exceeded, directing the media agent to perform one or more tasks in the task list.

[0005] Also known are systems and methods for creating single copies of data blocks in a data storage system, described in patent US 8578120 B2, published 05.11.2013. This document mentions a computing system comprising: one or more storage devices stored on a physical medium; one or more logical containers including several deduplicated data blocks corresponding to data objects; and one or more data structures indicating whether there are references to the data blocks; one or more databases storing information indicating whether there are references to the data blocks; and a secondary storage computing device programmed to: receive instructions for deleting a first set of data blocks from the first logical container; for each of the data blocks in the first set, determine from the databases a reference to the data block;and if the data block is not referenced, updating the data structure to indicate that the data block is not referenced; determining from the data structures that a threshold number of contiguous data blocks in the first logical container that are not referenced has been reached; and providing access for storing a portion of one or more physical media corresponding to contiguous data blocks in the first logical container, wherein the data structures and databases are not part of the native file systems of the storage devices.

[0006] Also known are data storage systems and methods described in patent US 7343453 B2, published 11.03.2008. This document discloses a hierarchical data storage system that provides methods for assessing the state of stored data relative to consumer needs using weighted parameters that can be defined by the user.

[0007] The development of technologies and methods for processing big data on powerful computing complexes and in a cloud environment additionally requires the implementation of new mechanisms for managing large data sets due to the increase in transmission time due to the limited network resource and throughput of data storage, and ensuring the readiness of data at the required point in time when creating continuous data processing pipelines. DISCLOSURE OF INVENTION

[0008] The technical problem or task posed in this technical solution is to create a simple and reliable method and system for moving large volumes of data in a cloud environment.

[0009] The technical result is an increase in reliability when moving data from different data sources in accordance with a specified time by which the said movement should be performed, and the conditions for moving the data.

[0010] The specified technical result is achieved by implementing a method for moving data in a cloud environment, which contains the following stages: - a request is received via API service 20 for moving the first data set from the first data source (ID) 30 and the second data set from the second ID 31 to the target ID 40, containing at least information about the conditions for moving the data; a request is generated via API service 20, containing a sequence of operations for moving the data, as well as a parameter of the priority of moving, determining the order of execution of the moving operation, and the generated request is sent to the address of Task Scheduler 50; - add a new task to move data to the queue using Task Scheduler 50 in accordance with the mentioned priority parameter; - when a task is received from the execution queue, the task is transferred to Work Node 60; - define, by means of Work Node 60, a list of interfaces for working with ID 30, ID 31 and ID 40; - perform, by means of the Work Node 60, an assessment of the possibility of carrying out the data movement task based on the available resources; - determine by means of the Work Node 60 the method of interaction with the target ID 40 based on the volume of data and the format of the data being moved and stored in the said IDs; - determine by means of Work Node 60 the optimal route for moving data from ID 30 and ID 31 to ID 40; - check the operability of the IDs used to move data using the Working Node 60; - by means of the Work Node 60, the first and second sets of data are read from the ID 30 and ID 31 and the said data are transferred to the address of the target ID 40 in accordance with the conditions of data movement.

[0011] In one particular example of implementing the method, ID 30, ID 31 and ID 40 are in the same cloud (i.e., logical pool).

[0012] In another particular example of implementing the method, at least one ID 30, ID 31 or ID 40 is located in a different cloud relative to the said IDs.

[0013] In another particular example of implementing the method, data transmission to the address of the target ID 40 is carried out taking into account the error correction code, data increment or parallel data transmission parameters.

[0014] In another particular example of implementing the method, an additional step is performed in which the registration of all data sources participating in the data movement is checked.

[0015] In another particular example of implementing the method, the request for moving data contains information about at least one additional target ID to which the first and second sets of data should be moved, and the Task Scheduler 50 generates several tasks for moving data with the same priority parameter depending on the number of additional IDs.

[0016] In another particular example of the method implementation, the following steps are additionally performed by means of the Task Scheduler 50: extracting from the data movement request a time value before the expiration of which the data movement should be completed; extracting from memory data characterizing the execution time of similar tasks for movement along a given route; determining, on the basis of the extracted data, the predicted execution time of the movement task; rearranging the data movement task in the queue for its execution at the specified time value, taking into account the predicted execution time of other tasks contained in the queue and the values ​​of their priorities.

[0017] In another particular example of implementing the method for determining the optimal route by means of the Work Node 60, the following steps are performed: determining the route from ID 30 and ID 31 to ID 40, as well as from the IDs in which copies of the data of ID 30 or ID 31 that should be moved are stored to ID 40; select the optimal route for moving data, taking into account the IDs in which copies of the data are stored.

[0018] In another particular example of implementing the method, an additional step of checking the integrity of the transferred data is performed.

[0019] In another particular example of implementing the method, the stage of synchronizing the completion of the process of moving data from IDs 30 and 31 is additionally performed by means of Work Node 60, including taking into account the increment.

[0020] In another particular example of implementing the method, additionally, version control of the moved data is performed by means of the Work Node 60.

[0021] In another particular example of implementing the method, metrics associated with the operation of moving data and using resources are additionally recorded.

[0022] In another particular example of implementing the method, the data transport operation is performed in accordance with a specified schedule.

[0023] In another particular example of implementing the method, the first or second data sets contain structured and / or unstructured data and / or heterogeneous (multimodal) data.

[0024] In another preferred embodiment of the claimed solution, a data movement system in a cloud environment is presented, containing at least three IDs, an API service, a Task Scheduler, at least one Worker Node, and the system is designed with the ability to implement the above-mentioned method. BRIEF DESCRIPTION OF DRAWINGS

[0025] The features and advantages of the present technical solution will become apparent from the following detailed description of the invention and the accompanying drawings, in which: - Fig. 1 shows the general scheme of registration of data sources; - Fig. 2 shows a general diagram of the data movement system; - Fig. 3 and 4 show the general scheme of data processing; - Fig. 5 shows an example of the general appearance of a computing device. IMPLEMENTATION OF THE INVENTION

[0026] Below we will describe the concepts and terms necessary for understanding this technical solution.

[0027] In this technical solution, the term “system” means, among other things, a computer system (in particular, an information system, computing complexes, computing clusters), a computer (electronic computer), a numerical control (CNC), a PLC (programmable logic controller), computerized control systems, specialized devices implementing calculations using FPGA and ASIC technologies, and any other devices capable of performing a given, clearly defined sequence of operations (actions, instructions).

[0028] A command processing unit is an electronic unit, computing device, or integrated circuit (microprocessor) that executes machine instructions (programs).

[0029] The command processing unit reads and executes machine instructions (programs) from one or more storage devices. Storage devices may include, but are not limited to, hard disk drives (HDD), flash memory, ROM (read-only memory), solid-state drives (SSD), and optical drives.

[0030] A program is a sequence of instructions intended for execution by a computer control device or a command processing device.

[0031] A database (DB) is a collection of data organized according to a conceptual structure that describes the characteristics of the data and the relationships between them, and such a collection of data supports one or more application areas (ISO / IEC 2382:2015, 2121423 "database").

[0032] A signal is a material embodiment of a message for use in transmitting, processing and storing information.

[0033] A logical element is an element that implements certain logical dependencies between input and output signals. Logical elements are usually used to construct logical circuits of computers, discrete circuits of automatic control and management. For all types Logical elements, regardless of their physical nature, are characterized by discrete values ​​of input and output signals.

[0034] An automated system (AS) is an organizational and technical system that ensures the development of decisions based on the automation of information processes.

[0035] In accordance with the diagram shown in Fig. 1, at the first stage, the data source 30 is registered in the data movement system in the cloud environment. The operation of registering the data source 30 is performed in advance before the start of the data movement both in relation to the source of the data being moved and in relation to the storage object, i.e. the target data source in which the data will be stored after the movement.

[0036] To register the data source 30, the user fills out a request form to the API service 20 via the user device 10 using the user graphical interface 11, or in the command line mode (user UI). The user device 10 may be a portable or stationary computer, telephone, smartphone, tablet or other computing device equipped with wired and / or wireless communication means.

[0037] The request form contains the storage identifier (name or identifier), data source type (cold storage, hot storage), file system type (HDFS, S3, NFS, etc.), authorization information (token, login-password pair). The completed form can be saved in the user UI module 11 for re-entry purposes, as well as for automated data processing purposes.

[0038] Additionally, the user specifies the parameters for access to the data source 30 - the type of access (permanent, session or scheduled), encryption keys for organizing a secure connection, data transfer speed, data transfer protocol (transfer with or without confirmation of delivery, multi-channel transfer), data transfer mode (synchronous or asynchronous transfer).

[0039] Accordingly, the completed request form is sent by the device 10 to the API service 20, which, upon receiving the request, generates a request to the data source 30 to confirm its availability and sends the generated request to the address of the data source. For receiving and processing data, the API service 20 can be implemented on the basis of at least one server equipped with software and logical elements for performing the functions assigned to it. The API service provides interfaces for registering data sources and initializing the data transport operation, as well as interfaces to relevant metrics of operations performed by the System, maintains an event log, and transmits commands for performing tasks from the User to the Scheduler 50 and the Worker Node 60, as well as reports and notifications on the status of planned and performed operations.

[0040] Upon receipt of a request for availability confirmation, the data source 30 starts a verification procedure, including a source availability check (network availability / network connectivity check), compatibility check (data transfer format and protocol check), user authorization on the data source 30, checking for permission to access the data stored in the data source, specifying the set of access rights (data read only, data write only, data read and write, no permissions). The API service 20 waits for a response from the data source within a waiting period pre-set by the developer in the settings of the API service 20. For example, the waiting period may be from a fraction of a second to tens of seconds.

[0041] Data source 30 returns a response to API service 20 with the results of the check, including confirmation of network availability, confirmation of format and protocol compatibility, confirmation of user authorization, confirmation of permission to access data. API service 20 analyzes the content of the response taking into account the waiting time.

[0042] If the API service 20 has not received a response from the data source confirming the availability of the data source over the network after the specified waiting period, the API service considers the data source unavailable and sends a message to the user device 10 about the unavailability of the data source for displaying it to the user via the UI 11.

[0043] If the API service 20 receives a response confirming incompatibility within a specified waiting period or does not receive a response from the data source 30 confirming compatibility of formats and / or data transfer protocols after the waiting period has expired, the API service 20 considers the data source 30 unavailable and sends a compatibility error message to the user device 10 when establishing a connection to display it to the user via the UI 11.

[0044] If the API service 20 receives a response from the data source 30 within the specified waiting period containing information about the user's authorization refusal, the API service 20 sends a message about the authorization refusal (access refusal) to the user's device 10 for displaying it to the user via the UI 11.

[0045] If, after the specified waiting period, the API service 20 has not received a response from the data source 30 confirming the user's authorization on the data source, the API service 20 considers the data source 30 unavailable and sends a message to the user's device 10 about the unavailability of the data source 30 indicating the status "data source is not responding" for displaying to the user via U1 11.

[0046] If the API service 20 receives a response from the data source 30 within the specified waiting period containing information about the denial of access to the data stored on the source, the API service 20 sends a message to the user device 10 about the denial of access to the data for displaying it to the user via the UI 11.

[0047] If, after a specified waiting period, the API service 20 has not received a response confirming the set of access rights to the data stored on the source, the API service 20 sends a message to the user device 10 about the denial of access to the data for displaying it to the user via the UI 11.

[0048] In case of successful receipt of all requested confirmations from data source 30, API service 20 registers data source 30, after which API service 20 sends confirmation of registration of data source to user device 10 for displaying it to the user via UI 11.

[0049] Once the required number of data sources 30 are registered, the process of moving / transporting data in the cloud environment using the registered sources can be started, and one-time data transport between registered sources or scheduled data transport between registered sources can be performed.

[0050] A one-time transport is a one-time operation in which the user does not plan to repeat the operation without changing the parameters and does not automate the process of repeated execution. A one-time transport is usually the main operation in the development and debugging data processing algorithms, as well as when performing data mining tasks. The data movement process will be described below with reference to Figs. 2, 3 and 4.

[0051] At the first stage (100), the user, by means of the device 10, forms a request for merging / moving the first data set from the first data source (DS) 30 and the second data set from the second DS (31) to the target DS 40, wherein DS 30, DS 31 or DS 40 can be located both in the same cloud (i.e., logical pool) with the other mentioned DS, and in different clouds. In an alternative embodiment of the presented solution, the user can specify several target DS 40 for parallel moving of the first and second data sets to several DS 40.

[0052] To generate a request, the user fills out a request form to the API service 20 using the user UI 11. In the said request form, the user can specify: identifiers ID 30, ID 31 and ID 40 (e.g. ID name, IP address, etc.), type of transportation (one-time or scheduled), authorization information (tokens, login-password pairs), addresses and identifiers of objects containing the moved data, and a list of the moved data, addresses and identifiers of objects to which the data will be moved, parameters of the priority of the operation. The filled out form can be saved in the user module U1 11 for the purposes of scheduled transportation, as well as for the purposes of automated data processing. Information on the data movement condition is also specified (with or without deleting data from the source after movement or saving copies of the data with or without specifying the version of the data set).

[0053] The request form and request may also contain fields for entering and transmitting such parameters as the need to use an error correction code, transmitting an increment (addition) of data, the applied algorithm for checking the integrity of data, conditions in the event of the task being executed by the Scheduler when these conditions are met, information about metadata sources and instructions for moving metadata, parameters for parallel data transfer (moving data with or without consolidation from several sources or moving to several target sources - data recipients).

[0054] For example, a user may request to merge at least one photograph stored in ID 30 and metadata, such as, characterizing the properties of the file, namely - file name, creation date, file size, hash sum, etc., stored in ID 31 in ID 40. Accordingly, the mentioned data merging request will additionally contain: - User ID; - user authorization data; - instructions for merging data: one-time or on a schedule; - the parameter of the move priority, which determines the order of execution of the move operation; - at least one condition selected from: the need to apply an error correction code, for example, to load the first or second data set from an alternative ID in which a copy of the data is stored; the transmission of an increment (addition) of data, for example, indicating the author of the first data set; The ID of the applied data integrity checking algorithm; the condition for performing the data transfer, for example, which may indicate that the data transfer in ID 40 will be performed only if there is free space for storing the data or indicating the time value before which the data transfer should be completed; information about the sources of metadata and instructions for transferring metadata, for example, metadata characterizing the properties of an image, such as the names of those depicted in the photo or tags indicating the location where the photo was taken; parameters for parallel data transfer, indicating the transfer of data with or without consolidation from several sources or transfer to several target sources - data recipients.

[0055] After receiving said request (101), the API service 20 analyzes the instructions for combining data and identifies it, for example, as a request for a one-time data transfer between registered IDs 30, ID 31 and ID 40, after which it checks the registration of all data sources participating in the data transfer. To check the registration, the API service 20 accesses the memory with which it is equipped, which contains information about the registered IDs. If the API service 20 has determined that at least one ID is not registered, then a notification about the presence of an unregistered ID is sent to the user device 10.

[0056] If all IDs are registered, then API service 20 generates a request containing the necessary sequence of data movement operations, in particular a command to move the first data set from ID 30 and the second data set from ID 31 to ID 40, as well as a parameter of movement priority determining the order of execution of the movement operation, and sends the generated request to the Task Scheduler 50, which can be implemented on the basis of at least one computing device equipped with logical elements for performing the functions assigned to it. In addition, other information contained in the data movement request can be sent to the Scheduler 50. Task Scheduler 50 processes the queue of transport tasks, queries the system for the presence of suitable events that will entail the initialization of the sought operations, maintains an event log.

[0057] Upon receipt of the mentioned request for movement, the System Scheduler 50, in accordance with the priority parameter, adds a new task for moving (transporting) data to the queue and sends a confirmation of the placement in the queue to the address of the API service 20. In this case, the Scheduler 50 can also include in the confirmation information about the expected start date of the task, the place of the task in the queue, the priority of the task (if the Scheduler 50 has several queues of different priorities), and other information.

[0058] In the event that a data movement request contains multiple target IDs 40, the Task Scheduler 50 generates multiple data movement tasks with the same priority parameter depending on the number of IDs 40 and adds all the tasks to the queue, with each data transport process in ID 40 being defined as a separate independent process at the execution stage, and all processes in parallel data transport are created with the same priority.

[0059] Upon receiving confirmation from Scheduler 50 about placing the task in the queue, API service 20 sends confirmation about placing the task in the queue to the address of the user device 10 for display to the user via UI 11.

[0060] The system scheduler 50 processes the task queue in accordance with predetermined prioritization algorithms and task-specific priority parameters, including but not limited to prioritization methods: without additional prioritization (FIFO method, first in, first out), with prioritization in the form of allocation of priority queues (each queue has its own priority, tasks from the queue with a higher priority are always executed first), prioritization of proportional resource allocation (in this case, a higher priority queue receives more resources to complete a task, for example, in the case of three queues, four tasks are executed from the highest priority queue, then two tasks from the medium priority queue are executed, then one task from the lowest priority queue, then the cycle repeats), and other methods and algorithms.

[0061] When a task needs to be executed according to a condition, Scheduler 50 suspends processing of a specific task until the condition is met, after which the task is executed with high or highest priority.

[0062] Additionally, the scheduler 50 may be configured to rearrange the task of moving in the queue. For example, the scheduler 50 may be equipped with a memory that contains data describing the execution time of similar tasks of moving, which may be entered by the developer of the scheduler or saved by the scheduler as historical data when executing similar tasks of moving data, taking into account the route of data transmission.

[0063] Accordingly, the Scheduler 50 extracts from the data movement request the time value before which the data movement should be completed, extracts from memory the data characterizing the execution time of similar tasks for movement along the specified route and, on their basis, determines the predicted execution time of the movement task, after which, taking into account the predicted execution time of other tasks contained in the queue and the values ​​of their priorities, it rearranges the movement task in the queue for its execution at the specified time value. The mentioned task can be rearranged both forward in the queue for its execution at the specified time value, and backward so that other tasks are executed at the time value specified for them.

[0064] When a task is received from the execution queue, the Scheduler 50 of the system transfers the task for execution (102) to the Worker Node 60, which, when the task is received for execution, performs the following set of actions - stages.

[0065] At step 103, the Worker Node 60 determines interfaces for working with the ID 30, ID 31 and ID 40, after which it connects to the said ID. The determination of the said interfaces and the establishment of a connection can be carried out by known methods automatically or based on previously made settings of the network and computing environment, taking into account the data transfer protocols implemented at various levels of the OSI network model (https: / / ru.wikipedia.orq / wiki / CeTeBan OSI model), including the Application layer (for example, when taking into account and applying conditions and triggers), the Presentation layer (for example, when applying data encryption when transmitting between data sources), the Session layer (for example, to establish communication sessions when transmitting according to a schedule). In this case, the ID can be connected to the transmission network at the Transport, Network and Data Link layers.

[0066] After the Worker Node 60 has connected to the said IDs, the Worker Node 60 makes a preliminary assessment of the possibility of performing the data movement task based on the available resources. For example, the Worker Node 60 can determine the total volume of data stored in the IDs 30 and 31 intended for movement, and determine the availability of free space for storing data in the ID 40. If the Worker Node determines that the available resources are insufficient to perform the said task, the Worker Node 60 sends a corresponding notification to the user device 20 via the API service 20. If the said resources are sufficient, the Worker Node 60 proceeds to the next step (105).

[0067] At the next step (105), the Worker Node 60 determines the method of interaction with the target ID 40 based on the volume of data and the format of the data being moved and stored in said IDs. For example, the Worker Node 60 can determine that the files in the target ID 40 are stored in the jpeg format or occupy the same amount of data, and based on this information, as a method of interaction with the ID 40, determine that the data transfer should be performed only in a certain format, i.e. jpeg, and the data volume should not exceed the data volume of the files stored in the ID 40, unless the developer has set permissions to save data of any format and volume in the ID 40. The specified file format is given as an example, while the method described in this application can process datasets of structured and unstructured files containing audio and voice information in the vaw, MP3 and other formats, video information in avi, MP4 and other formats, as well as file representations in plain text, csv, tabular representation formats, in encoded form, for example, using Base64 representation, binary representation, representation in JSON and xml formats and in other formats and representations.

[0068] Accordingly, if permissions for saving data of any format and volume in ID 40 are not set and at least one file from the files being moved in ID 30 or ID 31 is saved in a format other than jpeg (for example, in png format), or its volume is several times greater than the volume of files saved in ID 40, then in accordance with the previously determined method of interaction with ID 40, such a file will not be transferred, and the user of device 10 will be sent a corresponding notification.

[0069] After the method of interaction with the target ID 40 is determined, the Worker Node 60 proceeds to the stage 106 of constructing the optimal route for moving data from the ID 30 and ID 31 to the ID 40 in the data transmission network. The following may act as the data transmission network: a local area network; an Intranet network divided into segments and domains; a virtual computing environment in a cloud computing infrastructure; a virtual computing environment that transmits data via dedicated channels via the Internet or via other networks; or combinations of the aforementioned networks.

[0070] The optimal route can be determined by widely known methods, wherein to determine the optimal route the Worker Node 60 determines the route from the ID 30 and the ID 31 to the ID 40, as well as from the IDs in which copies of the data of the ID 30 or the ID 31 that should be moved are stored to the ID 40, after which the optimal route for moving the data is selected, for example, based on an analysis by the Worker Node 60 of the delay time during data transmission along the route from the ID 30 and the ID 31 to the ID 40 and from the IDs in which copies of the data are stored to the ID 40. The ID information with copies of the data stored in the ID 30 and the ID 31 can be stored in advance in the memory of the Worker Node 60 or presented by the user in a request for moving the data.

[0071] After the optimal data transmission route is determined, the Worker Node 60 sends a corresponding signal to check the operability of the IDs involved in the data movement and confirm their operability and readiness to transmit and receive data, respectively, and then, depending on the data integrity verification technology, assigns the first and second sets of data intended for movements, data integrity parameters. The integrity parameter may be a checksum or a hash sum determined by the Worker node 60 using known methods for the first and second data sets, or the data size may be designated as the integrity parameter.

[0072] Then the Worker Node 60 proceeds to the data movement stage 107 in accordance with the data movement conditions, in particular, the Worker Node 60 reads data from the ID 30 and ID 31 or from the IDs in which copies of the data are stored according to the previously determined route, and transfers (transports) the data to the address of the target ID 40, and also taking into account the error correction code, the increment (addition) of the data and the parameters of parallel data transfer. For example, in accordance with the data movement conditions, the first and second sets of data stored in the ID 40 can be supplemented with the specified metadata, or if a plurality of photographs are moved, then in accordance with the data movement conditions, the metadata can be saved both for each photograph and for a specified number of photographs.

[0073] During the transportation, the Worker Node 60 collects and analyzes information about the integrity of the transmitted data 108 by comparing the integrity parameters of the first and second data sets and the integrity parameters of the same data transferred to the target ID 40, determined in the manner described above after their transfer. In an alternative embodiment of the presented solution, the data integrity check can be checked by encoding the first and second data sets received from the ID, transferring them to the target ID 41, and then decoding. If the data received after decoding corresponds to the data from the first or second data set, for example, their size corresponds, then the Worker Node 60 makes a decision on the successful completion of the data transfer.

[0074] Accordingly, the Worker Node 60 makes a similar decision on the successful completion of the data transfer if the said data integrity parameters match. The confirmation of the successful transfer of the corresponding data set and / or fragment is sent by the Worker Node 60 to the Scheduler 50, and in the absence of confirmation of successful transfer, the Scheduler 50 sends a command to the Worker Node 60 for retransmission until success is confirmed. The command for retransmission of the data of the data set and / or fragment is also sent by the Scheduler 50 in the event of problems and failures during the transfer process.

[0075] Also, notifications and reports on completed operations and their status are sent via API service 20 to user device 10, if necessary enriched with notifications with parameters of the task being performed.

[0076] Additionally, the Worker Node 60 is configured to synchronize the completion of the process of moving data from the IDs 30 and 31, taking into account the increment. The synchronization process consists, in particular, in that by the time of completion of the transfer of both photographs and metadata, all pairs are collected, and each photograph is associated with its own metadata, each set of metadata is associated with its own photograph. Accordingly, the Worker Node 60 checks whether the first and second sets of data and the increment have been transferred to the ID 40 before proceeding to the next task or the next stage of the task.

[0077] Also, in addition, before moving the first or second data set, the Worker Node 60 checks whether the ID 40 contains a similar data set. If the ID 40 does not contain a similar data set, then the Worker Node 60 assigns the first version to them after moving the first or second data set. If the ID 40 already contains a similar data set, then the first or second data set being moved is assigned the next version of the data depending on the version of the similar data set.

[0078] After all operations of the assigned tasks are completed, the Scheduler 50, which monitors the movement of data by the Worker Node 60, records metrics 110 related to the operation itself and the use of resources, in particular, the time of movement of the files of the first and second data sets, the amount of memory occupied by them, the number of files transferred, confirmation of the execution of the operation at the specified time, the value of the delay in the transfer of data, etc. Using the API Service 20, the user of the device 10, as well as other modules and systems, can be provided with access to relevant metrics for analysis and preparation of reports.

[0079] If this is provided for by the transportation plan, Task Scheduler 50, upon receiving confirmation, sends to the address of API service 20 a confirmation of the successful transportation of a set and / or fragment of data, on the basis of which API service 20 issues a command to the address of ID to delete the corresponding data in order to prevent the storage of redundant copies of data.

[0080] Scheduled data transport is an operation that is repeated without changing parameters, with the possibility of automating the process of repeated execution. Scheduled data transport is, as a rule, the main operation in the industrial operation of information systems and computing complexes for data processing, including systems operating in real time. If the user, when forming a data movement request, specified the data movement according to a schedule as an instruction for combining data, then the data processing system operates as follows.

[0081] At the first stage (100), the user, in the manner described above, by means of the device 10, forms a request for combining the first data set from the first data source (DS) 30 and the second data set from the second DS (31) in the target DS 40. In this case, the rules for the priority of compliance with the conditions specified in the request include, but are not limited to, checking the readiness of the data for transportation, permission for transportation external to the system (business logic), completion of transportation tasks before a specific moment specified by the user, the start of the task according to the schedule, and other parameters. Specific priorities for the listed rules and the conditions for their application may change depending on the conditions of application of the system that is the subject of the invention.

[0082] Next, using the previously described method 101, a request for data movement is sent to API service 20. Upon receiving the request, API service 20 analyzes the received request, identifies it as a request for scheduled data transport, checks the registration of all data sources participating in the data movement, generates a request containing the necessary sequence of data movement operations, and sends the generated request to Task Scheduler 50.

[0083] When a request for scheduled movement is first received, the Scheduler 50 of the system adds a new transportation task to the beginning of the queue, receives confirmation of the data readiness for transportation from the data source, and sends a confirmation of the placement in the queue to the address of the API service 20. In this case, the Scheduler 50 can also include in the confirmation information about the expected start date of the task, the place of the task in the queue, the priority of the task (if the Scheduler 50 has several queues of different priorities), and other information.

[0084] After this, the Scheduler 50 calculates the time for performing the data transport operation specified by the schedule as the ratio of the data volume to the data transfer rate measured on the basis of previous data transports along a specific route. If the predicted completion time of the transport is after a user-defined time, increases the priority of the task and moves the task forward in the execution queue.

[0085] The scheduler 50 collects, analyzes, and stores the necessary metrics describing the data transportation process, including information about the data transfer rate along specific routes based on the results of measurements during the execution of previous data transportation tasks.

[0086] When the queue for the task execution has arrived, the Scheduler 50 checks the confirmation of the data readiness for transportation from the data source, compares the current time and the scheduled task execution start time, as well as the estimated task completion time and the completion time specified by the user. If the scheduled task execution start time has not arrived, and the transportation task will be completed at the required time, the Scheduler 50 puts the task in the second place in the queue, repeats the calculation of the estimated completion time of the transportation task, compares it with the completion time specified by the User 50, and in case of a positive response about the task completion on time, accepts the next priority task from the queue for which the completion time has not been specified.

[0087] If the predicted completion time of the transportation task, moved to the second place in the queue, occurs after the moment of completion time of the transportation specified by the User, the scheduled transportation task returns to the first place in the queue, and the Scheduler 50 begins its execution.

[0088] If there are two or more data transport tasks in the queue, the tasks are first rearranged in the queue in the order of the start time of execution and taking into account the estimated completion time. If the start time of execution for each of these tasks has not yet occurred, and both tasks are predicted to be completed before the time specified by the User, the Scheduler 50 accepts the task with priority #3 or lower in analogy with the algorithm described earlier in the method.

[0089] If the predicted completion times of two or more transportation tasks require a change in the order of execution in in accordance with the prioritization conditions described earlier, Scheduler 50 makes the appropriate change to the task queue

[0090] At the same time as the first request for a scheduled move is received, the System Scheduler 50 adds a new transportation task to the transportation schedule for future periods, reserving a place in the queue.

[0091] Since the volume of data to be transported may change when operations are repeatedly executed, the Scheduler 50 calculates the forecast time of the start and end of the operation in advance of the execution time according to the schedule, and also checks the availability of data sources and their readiness for transportation. The time period for the advance forecast is set when the system is installed and is adjusted by a fixed or variable value in accordance with the successful or unsuccessful completion of the previous transportation task along the corresponding route in time. In order to execute transportation tasks on time, a threshold interval for the time reserve for compensation of the forecast error can also be set in the system.

[0092] In case of unpreparedness of data sources for transportation (unavailability, lack of authorization on the source, lack of access rights, etc.), the Scheduler 50 sends a corresponding notification to the API service 20, which in turn sends a corresponding warning (Alert) about the risk of failure to perform transportation to the User UI 11 to take the necessary actions to restore the availability of sources.

[0093] If the sources are ready, information about the predicted start and end times of transportation and the readiness status of the sources is stored in the event logs of the Scheduler 50.

[0094] After all the conditions for priorities and conditions for performing transportation are met, the System Scheduler transfers the task for execution to the Work Node 60. The data movement by the work node is performed in the manner described earlier in accordance with steps 102 - 109.

[0095] During the transportation process, the Scheduler 50 collects and analyzes information about the integrity of the transmitted data, receives confirmation of the successful transportation of the corresponding set and / or fragment of data, and in the absence of confirmation of successful transportation, monitors and performs retransmission until success is confirmed.

[0096] API service 20 sends notifications and reports on the operations performed and their status to the User, enriching the notifications with parameters of the task being performed if necessary. In case of problems and failures during the transfer process, Scheduler 60 retransmits the corresponding set and / or fragment of data.

[0097] After the operation of the assigned tasks is completed, the System records metrics related to the operation itself and the resources used. The user and other modules and systems are provided with access to relevant metrics for analysis and reporting.

[0098] Scheduler 60 polls the system for suitable events that will trigger the desired operations.

[0099] The user has the ability to stop a deterministic process (planned or already running) through the user interface. In this case, through the user UI 11, a corresponding request to stop operations is sent to the API service 20, including the operation identifiers, conditions (the period of time in which these operations must be performed) and other necessary parameters. Upon receiving a request to stop, the API service 20 requests confirmation of the stop operation, and after receiving the User's confirmation, sends the corresponding command to the Scheduler and the Worker Node.

[0100] When an operation is interrupted, necessary actions must be taken to preserve the data, including pausing or canceling operations to remove duplicate data.

[0101] In case of parallel data transfer from the data source to the target source, the system treats each transport process as a separate independent process during the execution phase. All processes in parallel data transport must be created with the same priority.

[0102] Deletion of duplicate data is carried out upon completion of the parallel transfer processes of part of the data or the entire set of transferred data.

[0103] When predicting the execution time of parallel data transports, the duration of the process execution with the maximum execution time is taken into account for constructing forecasts.

[0104] If Scheduler 50 and Worker Node 60 detect a large number of transmission errors or do not receive integrity confirmation transmitted data, the Scheduler interrupts the transmission process and sends a notification to the User via API service 20 to take the necessary actions, while simultaneously generating an error report.

[0105] The presented solution provides guaranteed timely controlled secure movement of large and super-large volumes of structured and unstructured data and data streams in cloud environments and in local environments built using cloud technologies, including virtualization, scaling, access restriction, data collaboration, streaming and online data processing, continuous and batch data processing, data consolidation and transformation, data tagging, metadata collection, synchronous and asynchronous processing of data streams, backup, data delivery at the right time or when ready, removal of redundant copies and copies of data and other technologies and methods

[0106] In general (see Fig. 5), the computing device (200) contains one or more processors (201) connected by a common information exchange bus, including computing modules for accelerating computations - GPU (graphics processing unit), TPU (Tensor Processing Unit), IPU (Integrated Processing Unit), as well as modules based on FPGA and ASIC microprocessors, memory means such as RAM (202) and ROM (203), input / output interfaces (204), input / output devices (means) (205), and a device (means) for network interaction (206).

[0107] The processor (201) (or several processors, a multi-core processor, etc.) can be implemented on the basis of CISC, RISC, ARM, etc. architecture, and selected from a range of devices that are widely used at present, for example, from such manufacturers as: Intel™, AMD™, Apple™, Samsung Exynos™, MediaTEK™, Qualcomm Snapdragon™, Huawei Kunpeng™ and Nvidia™, IBM™, Baikal™, IVA™, etc. The processor or one of the processors used in the device (200) must also include a computing accelerator, implemented, for example, in the form of a graphics processor, such as an NVIDIA GPU with a software model compatible with CUDA, Graphcore, TPU (Tensor Processing Unit), IPU (Integrated Processing Unit), as well as modules based on FPGA and ASIC microprocessors, the type of which is also suitable for the full or partial implementation of the method, and can also be used for training and application of machine learning models in various information systems.

[0108] RAM (202) is a random access memory and is intended for storing machine-readable instructions executed by the processor (201) for performing the necessary operations on logical data processing. RAM (202), as a rule, contains executable instructions of the operating system and the corresponding software components (applications, software modules, etc.). In this case, the available memory capacity of the graphic card, graphic processor or other computing accelerators can act as RAM (202).

[0109] ROM (203) is one or more permanent data storage devices, such as a hard disk drive (HDD), a solid-state drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media (CD-R / RW, DVD-R / RW, BlueRay Disc, MD), etc.

[0110] To organize the operation of the device components (200) and to organize the operation of external connected devices, various types of I / O interfaces (204) are used. The choice of the corresponding interfaces depends on the specific design of the computing device, which may be, but are not limited to: PCI, AGP, PS / 2, IrDa, FireWire, COM, SATA, IDE, USB (2.0, 3.0, 3.1, micro, mini, type C), HDMI, DVI, VGA, RJ45, single-mode and multi-mode optical interfaces SC and ST, WiFi, NVLink, InfiniBand, STM, ATM, etc.

[0111] To ensure interaction between the user and the computing device (200), various means (205) of I / O information are used, for example, a keyboard, a display (monitor), a touch display, a touchpad, a joystick, a mouse, a light pen, a stylus, a touch panel, a trackball, speakers, a microphone, augmented reality means, optical sensors, a tablet, light indicators, a projector, a camera, biometric identification means (a retina scanner, a fingerprint scanner, a voice recognition module), etc.

[0112] The network interaction means (206) provides data transmission via an internal or external computer network, such as an Intranet, the Internet, a LAN, etc. One or more means (206) may include, but are not limited to: an Ethernet card, an LTE modem, a 5G modem, a satellite communication module, an NFC module, a Wi-Fi module, single-mode and multi-mode optical interface modules, NVLink, InfiniBand, STM, ATM and DR-

[0113] Additionally, satellite navigation means as part of the device (200) can also be used, for example, GPS, GLONASS, BeiDou, Galileo. Precise time signals received from the mentioned satellite navigation means can be used to synchronize the operation of geographically separated devices.

[0114] The specific selection of device elements (200) for the implementation of various software and hardware architectural solutions may vary while maintaining the required functionality provided.

[0115] Modifications and improvements of the above-described embodiments of the present technical solution will be clear to those skilled in the art. The preceding description is provided only as an example and does not bear any limitations. Thus, the scope of the present technical solution is limited only by the scope of the attached claims.

Claims

CLAUSE OF INVENTION.

1. A method for moving data in a cloud environment, comprising the steps of: - a request is received via API service 20 for moving the first data set from the first data source (ID) 30 and the second data set from the second ID 31 to the target ID 40, containing at least information about the conditions for moving the data; a request is generated via API service 20, containing a sequence of operations for moving the data, as well as a parameter of the priority of moving, determining the order of execution of the moving operation, and the generated request is sent to the address of Task Scheduler 50; - add a new task to move data to the queue using Task Scheduler 50 in accordance with the mentioned priority parameter; - when a task is received from the execution queue, the task is transferred to Work Node 60; - define, by means of Work Node 60, a list of interfaces for working with ID 30, ID 31 and ID 40; - perform, by means of the Work Node 60, an assessment of the possibility of carrying out the data movement task based on the available resources; - determine by means of the Work Node 60 the method of interaction with the target ID 40 based on the volume of data and the format of the data being moved and stored in the said IDs; - determine by means of Work Node 60 the optimal route for moving data from ID 30 and ID 31 to ID 40; - check the operability of the IDs used to move data using the Working Node 60; - by means of the Work Node 60, the first and second sets of data are read from the ID 30 and ID 31 and the said data are transferred to the address of the target ID 40 in accordance with the conditions of data movement.

2. The method according to item 1, characterized in that ID 30, ID 31 and ID 40 are located in the same cloud (i.e., logical pool).

3. The method according to claim 1, characterized in that at least one ID 30, ID 31 or ID 40 is located in a different cloud relative to the said IDs.

4. The method according to item 1, characterized in that the data transmission to the address of the target ID 40 is carried out taking into account the error correction code, data increment or parallel data transmission parameters.

5. The method according to paragraph 1, characterized in that it additionally contains a step in which the registration of all data sources participating in the movement of data is checked.

6. The method according to claim 1, characterized in that the request for moving data contains information about at least one additional target ID to which the first and second sets of data should be moved, and the Task Scheduler 50 generates several tasks for moving data with the same priority parameter depending on the number of additional IDs.

7. The method according to item 1, characterized in that the Task Scheduler 50 additionally performs the following steps: - extract from the data transfer request the time value before which the data transfer must be completed; - extract data from memory that characterize the time it takes to complete similar tasks to move along a given route; - based on the extracted data, the predicted time for completing the movement task is determined; - performs a rearrangement of the data movement task in the queue for its execution at the specified time value, taking into account the predicted execution time of other tasks contained in the queue and the values of their priorities.

8. The method according to item 1, characterized in that in order to determine the optimal route, the following steps are performed by means of the Working Node 60: - determine the route from ID 30 and ID 31 to ID 40, as well as from the IDs in which copies of the data of ID 30 or ID 31 that should be moved are stored, to ID 40; - select the optimal route for moving data, taking into account the IDs in which copies of the data are stored.

9. The method according to item 1, characterized in that an additional step of checking the integrity of the transferred data is performed.

10. The method according to item 1, characterized in that the Work Node 60 additionally performs the step of synchronizing the completion of the process of moving data from ID 30 and 31, including taking into account the increment.

11. The method according to claim 1, characterized in that additionally, the Work Node 60 performs version control of the moved data.

12. The method according to paragraph 1, characterized in that metrics associated with the operation of moving data and the use of resources are additionally recorded.

13. The method according to item 1, characterized in that the execution of the data transportation operation is carried out in accordance with a specified schedule.

14. The method according to claim 1, characterized in that the first or second data sets contain structured and / or unstructured data and / or heterogeneous (multimodal) data.

15. A data movement system in a cloud environment, containing at least three IDs, an API service, a Task Scheduler, at least one A working unit, wherein the system is designed with the ability to implement the method according to any of paragraphs 1-14.

Images