Big data scheduling method and apparatus, computer device, and storage medium
By generating intermediate key-value pairs between the source and target environments and merging and deduplicating them, the problems of cumbersome task data scheduling and poor timeliness are solved, achieving efficient and timely task data synchronization.
Patent Information
- Application Number
- PCT/CN2023/127191
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-10
- Filing Date
- 2023-10-27
- Publication Date
- 2024-11-14
AI Technical Summary
In existing technologies, task data scheduling is cumbersome and complex, has poor timeliness, and is prone to causing duplicate scheduling problems.
By acquiring task information from the source and target environments, intermediate key-value pairs are generated using mapping functions, data tables are merged and deduplicated to generate a task scheduling table, and task data is synchronized to the target environment based on the scheduling table.
It improves the timeliness and efficiency of task data scheduling, reduces the amount of data, lowers the amount of data processing, and avoids redundant scheduling.
Smart Images

Figure CN2023127191_14112024_PF_FP_ABST
Abstract
Description
Big data scheduling method, device, computer equipment and storage medium
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 10, 2023, with application number 202310532665.6 and application name “Big Data Scheduling Method, Device, Computer Equipment and Storage Medium”, all contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of big data, and in particular to a big data scheduling method, apparatus, computer equipment and storage medium. Background Art
[0003] In the field of big data processing, task data published by execution instances is typically stored in a data warehouse, while task data processing is performed by the data platform. Therefore, a system is required to schedule and manage tasks in the data warehouse.
[0004] In the prior art, after task data is collected in a data warehouse, it is necessary to schedule the task data in the data warehouse to the task database for executing the task through a timed synchronization method. The applicant of the present invention found in their research that task data scheduling in the prior art is all performed through manual scripts. When the amount of data exceeds a certain threshold, data scheduling becomes cumbersome and complicated. Manual scheduling has poor timeliness and is prone to duplicate scheduling of task data.
[0005] Application Contents
[0006] The purpose of this application is to provide a task scheduling method, computer equipment, and computer storage medium to improve the timeliness of task scheduling and avoid repeated scheduling of task data.
[0007] In a first aspect, an embodiment of the present application provides a task scheduling method, comprising:
[0008] Obtain the first task information and task synchronization time in the source environment;
[0009] Reading second task information in the set target environment according to the task synchronization time;
[0010] Extracting a first data table corresponding to the first task information, wherein the first data table is composed of a plurality of intermediate key values corresponding to the first task information;
[0011] Extracting a second data table corresponding to the second task information, wherein the second data table is composed of multiple intermediate key values corresponding to the second task information;
[0012] Merging and deduplicating the first data table and the second data table to generate a task scheduling table;
[0013] The task data in the source environment is synchronized to the target environment according to the task scheduling table.
[0014] In a second aspect, an embodiment of the present application provides a task scheduling device, comprising:
[0015] An acquisition module, configured to acquire the first task information and task synchronization time in the source environment;
[0016] A reading module, configured to read the second task information in the set target environment according to the task synchronization time;
[0017] a first extraction module, configured to extract a first data table corresponding to the first task information, wherein the first data table is composed of a plurality of intermediate key values corresponding to the first task information;
[0018] a second extraction module, configured to extract a second data table corresponding to the second task information, wherein the second data table is composed of a plurality of intermediate key values corresponding to the second task information;
[0019] A processing module, configured to merge and remove duplicates from the first data table and the second data table to generate a task scheduling table;
[0020] An execution module is used to synchronize the task data in the source environment to the target environment according to the task scheduling table.
[0021] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor is configured to perform:
[0022] Obtain the first task information and task synchronization time in the source environment;
[0023] Reading second task information in the set target environment according to the task synchronization time;
[0024] Extracting a first data table corresponding to the first task information, wherein the first data table is composed of a plurality of intermediate key values corresponding to the first task information;
[0025] Extracting a second data table corresponding to the second task information, wherein the second data table is composed of multiple intermediate key values corresponding to the second task information;
[0026] Merging and deduplicating the first data table and the second data table to generate a task scheduling table;
[0027] The task data in the source environment is synchronized to the target environment according to the task scheduling table.
[0028] In a fourth aspect, an embodiment of the present application provides a storage medium storing computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors are configured to execute:
[0029] Obtain the first task information and task synchronization time in the source environment;
[0030] Reading second task information in the set target environment according to the task synchronization time;
[0031] Extracting a first data table corresponding to the first task information, wherein the first data table is composed of a plurality of intermediate key values corresponding to the first task information;
[0032] Extracting a second data table corresponding to the second task information, wherein the second data table is composed of multiple intermediate key values corresponding to the second task information;
[0033] Merging and deduplicating the first data table and the second data table to generate a task scheduling table;
[0034] The task data in the source environment is synchronized to the target environment according to the task scheduling table. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0036] FIG1 is a schematic diagram of the basic flow of a big data scheduling method according to a specific embodiment of the present application;
[0037] FIG2 is a schematic diagram of the basic structure of a big data scheduling device according to a specific embodiment of the present application;
[0038] FIG3 is a basic structural block diagram of a computer device according to a specific embodiment of the present application. DETAILED DESCRIPTION
[0039] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.
[0040] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of this application refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.
[0041] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.
[0042] As will be understood by those skilled in the art, the term "terminal" as used herein includes both devices that are wireless signal receivers, i.e., devices that only have wireless signal receivers without transmission capabilities, and devices that have receiving and transmitting hardware capable of performing two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices with single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service) devices that may combine voice, data processing, fax, and / or data communication capabilities; PDAs (Personal Digital Assistants) that may include a radio frequency receiver, a pager, Internet / Intranet access, a web browser, a notepad, a calendar, and / or a GPS (Global Positioning System) receiver; and conventional laptop and / or palmtop computers or other devices that have and / or include a radio frequency receiver. As used herein, a "terminal" may be portable, transportable, installed in a vehicle (air, sea, and / or land), or adapted and / or configured to operate locally, and / or in a distributed manner, at any other location on Earth and / or in space. As used herein, a "terminal" may also be a communication terminal, an Internet access terminal, or a music / video playback terminal, such as a PDA, an MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a device such as a smart TV or a set-top box.
[0043] The hardware referred to by names such as "server", "client", and "service node" in this application is essentially an electronic device with capabilities equivalent to those of a personal computer. It is a hardware device that has the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. Computer programs are stored in its memory, and the central processing unit loads the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.
[0044] It should be noted that the concept of "server" referred to in this application can also be extended to server clusters. Based on the network deployment principles understood by those skilled in the art, the servers described should be logically divided. In physical space, these servers can be independent of each other but callable through interfaces, or integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method of this application.
[0045] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for implementation.
[0046] Unless expressly specified otherwise, the neural network models referenced or may be referenced in this application may be deployed on a remote server and remotely called on the client, or may be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence may be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.
[0047] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as they are suitable for being called by the technical solution of this application.
[0048] Those skilled in the art should be aware that although the various methods of this application are described based on the same concept and thus exhibit commonality, unless otherwise specified, these methods can be independently executed. Similarly, the various embodiments disclosed in this application are all based on the same application concept. Therefore, concepts with the same expression, as well as concepts that are appropriately transformed for convenience despite different expression, should be understood as equivalent.
[0049] Unless expressly stated to be mutually exclusive, the various embodiments disclosed in this application may be cross-combined with the relevant technical features of the various embodiments to flexibly construct new embodiments, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or resolve certain deficiencies in the prior art. Those skilled in the art should be aware of such flexibility.
[0050] Please refer to FIG1 , which is a schematic diagram of the basic flow of the big data scheduling method of this embodiment.
[0051] As shown in FIG1 , a big data scheduling method includes:
[0052] S1100: Acquire first task information and task synchronization time in the source environment;
[0053] In this embodiment, the source environment refers to a database, server, or cloud where task data is collected. Specifically, the source environment can be a test environment or a data warehouse.
[0054] The task data in the source environment is task data published by an execution instance that has an interface connection with the source environment. The source environment collects locally stored task data to generate first task information. The task data in the first task information is formatted as a key-value pair. Therefore, the first task information includes key-value pairs of all task data in the source environment.
[0055] The source environment also includes a synchronization time for task scheduling. The synchronization time is performed at regular intervals. For example, the synchronization time is set to 5 minutes, that is, after the previous task scheduling cycle is completed, a new round of task scheduling will be performed after an interval of 5 minutes. The setting of the synchronization time is not limited to the listed time. According to the specific application scenario, the synchronization time can be (but not limited to): 1S, 5S, 20S, 1min, 50min, 1H or longer and shorter times. In some embodiments, the synchronization time is dynamic. Specifically, the synchronization time is inversely proportional to the amount of task data already in the source environment. When the amount of task data is large, the interval of the synchronization time is shorter; otherwise, it is longer. In some embodiments, the dynamic synchronization time is limited to a set time interval, limiting the shortest time and the longest time of the synchronization task to prevent the synchronization time from being excessively extended when the amount of task data is too small, and to avoid the synchronization time from being excessively shortened when the amount of task data increases sharply, causing greater data synchronization pressure on the source environment and the task environment.
[0056] S1200, reading second task information in a set target environment according to the task synchronization time;
[0057] In this embodiment, the target environment refers to the database, server or cloud where the task data is executed. Specifically, the target environment can be a generation environment or a task platform for executing the task.
[0058] The target environment's task data is synchronized from the source environment to the target environment. The target environment collects locally stored task data to generate secondary task information. The task data in the secondary task information is formatted as key-value pairs. Therefore, the secondary task information includes key-value pairs for all task data in the target environment.
[0059] When the synchronization time point represented by the synchronization time is reached, the second task information in the target environment is collected.
[0060] S1300: Extract a first data table corresponding to the first task information, wherein the first data table is composed of multiple intermediate key values corresponding to the first task information;
[0061] After extracting the first task information, the task data in the first task information needs to be extracted to obtain a first data table. The first data table records all the task data in the first task information, and the task data in the first data table is stored in key-value pairs of the task data.
[0062] In some implementations, in order to further reduce the amount of data in the first data table, it is necessary to perform mapping processing on the task data.
[0063] Specifically, the key-value pairs for each task data in the first task information are extracted. A key-value pair includes key data and value data. The key data represents the identity information of the task data, such as the task data name, the execution instance corresponding to the task data, the task data format, the task data release time, and other information indicating the identity of the task data. The value data represents the specific task content corresponding to the task.
[0064] The key data and value data corresponding to the task data are defined as a first key-value pair, and the first key-value pair is mapped using a predetermined first mapping function. The first mapping function can be a hash function or a hash calculation formula. Through the mapping process of the first mapping function, the first key-value pair containing a large amount of data can be converted into a first intermediate key-value pair containing a smaller amount of data.
[0065] After calculating and generating the first intermediate key-value pair, further data dimensionality reduction is required. Specifically, the first intermediate key-value pair also includes key data and value data. Identical key data in the first intermediate key-value pair is classified and counted to obtain similar key data. The value data corresponding to the similar key data is further mapped, still using a hash function or hash calculation formula. The mapped value data becomes first lightweight value data containing a smaller amount of data.
[0066] After further mapping the value data of the same type of key data, the key data in the first intermediate key-value pair and the first lightweight value data corresponding to the key data are extracted, and a new key-value pair is formed by the key data and the first lightweight value data. The new key-value pair is then statistically analyzed to generate a first data table.
[0067] During the generation of the first data table, data mapping is performed twice for each task information in the first task information. The first mapping involves mapping the entire first key-value pair corresponding to the task data to generate a first intermediate key-value pair. Then, a second mapping is performed for the value data with the same key data in the first intermediate key-value pair to generate even lighter first lightweight value data. Because the value data contains more data, the second mapping can significantly reduce the amount of data in the value data. This minimizes the amount of data contained in the first data table while ensuring that the generated new key-value pairs can still represent the corresponding task data.
[0068] In some embodiments, when performing data mapping on value data with the same key data, it is necessary to generate a corresponding secondary mapping function based on the type of the key data. For example, a mapping function list is provided, which records multiple mapping functions. When performing secondary mapping of the value data, at least one mapping function is randomly selected from the mapping function list based on the type of the key data to form the secondary mapping function. By randomly selecting a mapping function, the possibility of data duplication after secondary mapping of value data with different key data can be avoided.
[0069] S1400: Extract a second data table corresponding to the second task information, wherein the second data table is composed of multiple intermediate key values corresponding to the second task information;
[0070] After the second task information is extracted, the task data in the second task information needs to be extracted to obtain a second data table. The second data table records all the task data in the second task information, and the task data in the second data table is stored in key-value pairs of the task data.
[0071] In some implementations, in order to further reduce the amount of data in the second data table, it is necessary to perform mapping processing on the task data.
[0072] Specifically, the key-value pairs for each task data in the second task information are extracted. A key-value pair includes key data and value data. The key data represents the identity information of the task data, such as the task data name, the execution instance corresponding to the task data, the task data format, the task data release time, and other information indicating the identity of the task data. The value data represents the specific task content corresponding to the task.
[0073] The key data and value data corresponding to the task data are defined as a second key-value pair, and the second key-value pair is mapped using a predetermined first mapping function. The first mapping function can be a hash function or a hash calculation formula. Through the mapping process of the first mapping function, the second key-value pair containing a large amount of data can be converted into a second intermediate key-value pair containing a smaller amount of data.
[0074] After calculating and generating the second intermediate key-value pair, further data dimensionality reduction is required. Specifically, the second intermediate key-value pair also includes key data and value data. Identical key data in the second intermediate key-value pair is classified and counted to obtain similar key data. The value data corresponding to the similar key data is further mapped, still using a hash function or hash calculation formula. The mapped value data becomes the second lightweight value data containing a smaller amount of data.
[0075] After further mapping the value data of the same type of key data, the key data in the second intermediate key-value pair and the second lightweight value data corresponding to the key data are extracted, and a new key-value pair is formed by the key data and the second lightweight value data. The new key-value pair is then statistically analyzed to generate a second data table.
[0076] During the generation of the second data table, data mapping is performed twice for each task information item in the second task information. The second mapping process maps the entire second key-value pair corresponding to the task data to generate a second intermediate key-value pair. Then, the value data with the same key data in the second intermediate key-value pair is mapped again to generate a lighter-weight second lightweight value data item. Because the value data item contains more data, the second mapping process can significantly reduce the amount of data in the value data item. This ensures that the generated new key-value pairs can still represent the corresponding task data, while minimizing the amount of data contained in the second data table.
[0077] In some embodiments, when performing data mapping processing on value data of the same key data, it is necessary to generate a corresponding secondary mapping function according to the category of the key data. For example, a mapping function list is set, in which multiple mapping functions are recorded. When performing secondary mapping of the value data, at least one mapping function is randomly extracted from the mapping function list according to the category of the key data to form a secondary mapping function. By randomly extracting the mapping function, the possibility of data duplication after secondary mapping of value data of different key data can be avoided. It should be pointed out that when performing secondary mapping processing on value data of the same type of key data type, the first intermediate key-value pair and the second intermediate key-value pair have the same key data, and the secondary mapping function used must be the same, so that the value data of the same task data after secondary mapping are the same, which facilitates deduplication when the first data table and the second data table are subsequently combined.
[0078] It should be noted that, in this embodiment, the execution of S1300 and S1400 does not have a clear order, and the execution of the two can be synchronous, or S1300 can be executed before S1400, or S1300 can be executed after S1400.
[0079] S1500: Merge and remove duplicates from the first and second data tables to generate a task scheduling table.
[0080] After the first and second data tables are generated, they need to be merged. During the merging process, duplicate data in the first and second data tables need to be removed.
[0081] When merging tables, if key data exists in the first data table but not in the second, the key data is stored in the newly generated task scheduling table. If key data exists in both the first and second data tables, the first and second lightweight data are compared to see if they are identical. If they are, it indicates that the key data and value data for the task data are identical, indicating that the task data already exists in the target environment and is therefore duplicate data that needs to be deleted to avoid duplication.
[0082] In some implementations, for similar key-value pairs having the same key data in the first data table and the second data table, a simpler and more convenient deduplication method is provided that does not require comparison.
[0083] Extract similar key-value pairs with the same key data from the first data table and the second data table. At this time, the similar key-value pairs include two value data, namely the first lightweight value data and the second lightweight value data. That is to say, for the same key data, there are the first lightweight value data and the second lightweight value data from the first data table.
[0084] Perform a direct product operation on the first lightweight value data and the second lightweight value data in the same key-value pair, and multiply the first lightweight value data and the second lightweight value data in the same key-value pair. For example, when the first lightweight value data of the same key value are two, X and Y, and the second lightweight value data are also two, X and Y. The first lightweight value data and the second lightweight value data are multiplied, and the results are: X squared, XY, XY, Y squared. Among them, X squared and Y squared indicate that both the first data table and the second data table have the task data, which are duplicate data and should be deleted, while XY indicates that the task data does not appear in the second data table and needs to be added to the task scheduling table. The above examples are for better explanation. The deduplication principle of this solution cannot limit the general result of multiplying the first lightweight value data and the second lightweight value data. In actual applications, the number and value of the first lightweight value data of the same key value can be any value, and the number and value of the second lightweight value data of the same key value can also be any value.
[0085] When the result of the direct product operation of the first lightweight value data and the second lightweight value data appears, the task of squaring the product of the first data table and the second data table is repeated according to the operation result.
[0086] S1600: Synchronize the task data in the source environment to the target environment according to the task scheduling table.
[0087] After the first data table and the second data table are merged, the generated task scheduling table includes: task data represented by key data in the first data table but not in the second data table, and task data represented by a value that is not a square when the direct product of the first lightweight value data and the second lightweight value data is calculated. At this point, the task data represented by the key-value pairs included in the task scheduling table are all task data that is not present in the target environment. This unsynchronized task data is synchronized to the target environment according to the task scheduling table, completing task synchronization with the target environment.
[0088] In some embodiments, when generating a task synchronization table, it is necessary to delete task data that has been deleted from the source resource. When task data is stored in the source resource, the execution instance that published the task data can withdraw or delete the task data. The deleted or withdrawn data is considered deleted scheduled data. A task deletion list is formed by pulling the deleted scheduled tasks from the source resource.
[0089] After the task deletion list is generated, it is necessary to delete the above tasks in the task scheduling table according to the task data recorded in the task deletion list. Specifically, perform step S1300 on the task data corresponding to the task deletion list pulled from the source resource to generate a third data table corresponding to the task deletion list. Delete the key data and value data corresponding to the third data table in the task scheduling table to complete the scheduled task deletion.
[0090] In some embodiments, tasks deleted from the task deletion list also include task data with a specific execution time in the source resource. When the execution instance uploads the task data, the specific processing time for the task data is set. The source resource pulls all task data and determines whether the task data's execution time has expired. If the task data is determined to have timed out, the task data is added to the task deletion list.
[0091] When the scheduled tasks that need to be deleted in the task deletion list are deleted by subtracting the task scheduling table from the task deletion list, the unsynchronized task data are scheduled according to the updated task scheduling table and synchronized to the target resource.
[0092] The task deletion list and task scheduling table are used to delete the canceled or timed-out task data by reducing the table, so as to prevent the target resources from executing the task data, reduce the data processing pressure of the target resources, and improve the task processing efficiency.
[0093] In some implementations, when a source resource synchronizes data with a target resource, the synchronized task data must be transmitted across multiple links. Because each link in a multi-link transmission environment has different load information, the amount of data that can be transmitted also varies. To optimize the data transmitted by each link, it is necessary to extract link information for each of the multiple links. The link information contains the load information for the corresponding link, and based on this load information, the optimal data packet size for transmission on that link can be calculated.
[0094] After calculating the packet size for each link, the task data corresponding to the task scheduling table is extracted to form a data set. The task data in the data set is sliced based on the optimal packet size required for transmission on each link. Slicing is performed by combining the task data in the data set based on the packet size corresponding to the link, so that the combined task data is equal to or less than the task packet data size. The task set corresponding to each packet is then treated as a data block. If the data size of a single task exceeds the maximum data size of the packets in multiple links, the single task data needs to be sliced and the resulting data blocks distributed across two or more links for transmission.
[0095] By slicing and combining data, the optimal data block that each link can transmit is formed, so that a one-to-one correspondence is established between the data block and the link. Then, multiple data blocks are transmitted through multiple links respectively, and the task data in the task scheduling table is synchronized to the target resource.
[0096] In this embodiment, the task data in the task scheduling table is combined and sliced through link information, so that each link can transmit data blocks with the optimal data volume, preventing link congestion due to excessive data volume during transmission, and improving transmission efficiency.
[0097] In the above embodiment, when performing data scheduling, the first task information and task synchronization time in the source environment are first obtained. Then, based on the task synchronization time, the second task information in the target environment is obtained. Key-value pairs in the first task information are extracted, and the extracted key-value pairs are further mapped to generate intermediate key values. Based on the intermediate key values, the first data table corresponding to the first task information is obtained. The second data table corresponding to the second task information is obtained using the same method. The first and second data tables are merged, and during the merging process, duplicate tasks in the first and second data tables are deleted, generating a task scheduling table for data scheduling. Finally, the task data in the source environment is scheduled to the target environment using the task scheduling table. This data scheduling allows data in the source environment to be scheduled to the target environment in a timely manner, improving the timeliness of task data scheduling. Furthermore, since both the first and second data tables are composed of intermediate key values, the data tables are more lightweight and smaller than the original data. This reduces the amount of data computation required when merging and deduplicating the data tables, improving the timeliness of task scheduling and avoiding duplicate scheduling of task data.
[0098] In some implementations, in order to further reduce the amount of data in the first data table, it is necessary to perform mapping processing on the task data. Specifically, S1300 includes:
[0099] S1311. Extract multiple first key-value pairs corresponding to the first task information;
[0100] Extract the key-value pairs for each task data in the first task information to generate multiple first key-value pairs. Each first key-value pair includes key data and value data. The key data represents the identity information of the task data, such as the name of the task data, the execution instance corresponding to the task data, the data format of the task data, the release time of the task data, and other information indicating the identity of the task data. The value data represents the specific task content corresponding to the task.
[0101] S1312: Based on the set first mapping function, perform operations on the multiple first key-value pairs to generate multiple first intermediate key-value pairs;
[0102] The key data and value data corresponding to the task data are defined as a first key-value pair, and the first key-value pair is mapped using a predetermined first mapping function. The first mapping function can be a hash function or a hash calculation formula. Through the mapping process of the first mapping function, the first key-value pair containing a large amount of data can be converted into a first intermediate key-value pair containing a smaller amount of data.
[0103] S1313: Perform data mapping processing on the value data having the same key data in the plurality of first intermediate key-value pairs to generate first lightweight value data;
[0104] After further mapping the value data of the same type of key data, the key data in the first intermediate key-value pair and the first lightweight value data corresponding to the key data are extracted, and a new key-value pair is formed by the key data and the first lightweight value data. The new key-value pair is then statistically analyzed to generate a first data table.
[0105] S1314. Generate the first data table according to the key data in the multiple first intermediate key-value pairs and the first lightweight value data.
[0106] During the generation of the first data table, data mapping is performed twice for each task information in the first task information. The first mapping involves mapping the entire first key-value pair corresponding to the task data to generate a first intermediate key-value pair. Then, a second mapping is performed for the value data with the same key data in the first intermediate key-value pair to generate even lighter first lightweight value data. Because the value data contains more data, the second mapping can significantly reduce the amount of data in the value data. This minimizes the amount of data contained in the first data table while ensuring that the generated new key-value pairs can still represent the corresponding task data.
[0107] In some embodiments, when performing data mapping on value data with the same key data, it is necessary to generate a corresponding secondary mapping function based on the type of the key data. For example, a mapping function list is provided, which records multiple mapping functions. When performing secondary mapping of the value data, at least one mapping function is randomly selected from the mapping function list based on the type of the key data to form the secondary mapping function. By randomly selecting a mapping function, the possibility of data duplication after secondary mapping of value data with different key data can be avoided.
[0108] In some implementations, in order to further reduce the amount of data in the second data table, it is necessary to perform mapping processing on the task data. Specifically, S1300 includes:
[0109] S1321. Extract multiple second key-value pairs corresponding to the second task information;
[0110] Extract the second key-value pair for each task data item in the second task information. The second key-value pair includes key data and value data. The key data represents the identity information of the task data item, such as the task data item's name, the execution instance corresponding to the task data item, the task data item's data format, the task data item's release date, and other information indicating the task data item's identity. The value data represents the specific task content corresponding to the task item.
[0111] S1322: Based on the set first mapping function, perform operations on the plurality of second key-value pairs to generate a plurality of second intermediate key-value pairs;
[0112] The key data and value data corresponding to the task data are defined as a second key-value pair, and the second key-value pair is mapped using a predetermined first mapping function. The first mapping function can be a hash function or a hash calculation formula. Through the mapping process of the first mapping function, the second key-value pair containing a large amount of data can be converted into a second intermediate key-value pair containing a smaller amount of data.
[0113] S1323: Perform data mapping processing on the value data having the same key data in the plurality of second intermediate key-value pairs to generate second lightweight value data;
[0114] After calculating and generating the second intermediate key-value pair, further data dimensionality reduction is required. Specifically, the second intermediate key-value pair also includes key data and value data. Identical key data in the second intermediate key-value pair is classified and counted to obtain similar key data. The value data corresponding to the similar key data is further mapped, still using a hash function or hash calculation formula. The mapped value data becomes the second lightweight value data containing a smaller amount of data.
[0115] S1324. Generate the second data table according to the key data in the plurality of second intermediate key-value pairs and the first lightweight value data.
[0116] After further mapping the value data of the same type of key data, the key data in the second intermediate key-value pair and the second lightweight value data corresponding to the key data are extracted, and a new key-value pair is formed by the key data and the second lightweight value data. The new key-value pair is then statistically analyzed to generate a second data table.
[0117] During the generation of the second data table, data mapping is performed twice for each task information in the second task information. The second mapping fully maps the second key-value pairs corresponding to the task data to generate second intermediate key-value pairs. Then, the value data with the same key data in the second intermediate key-value pairs is mapped again to generate even lighter second lightweight value data. Because the value data contains more data, the second mapping can significantly reduce the amount of data in the value data. This ensures that the generated new key-value pairs can still represent the corresponding task data, while minimizing the amount of data contained in the second data table.
[0118] In some embodiments, when performing data mapping processing on value data of the same key data, it is necessary to generate a corresponding secondary mapping function according to the category of the key data. For example, a mapping function list is set, in which multiple mapping functions are recorded. When performing secondary mapping of the value data, at least one mapping function is randomly extracted from the mapping function list according to the category of the key data to form a secondary mapping function. By randomly extracting the mapping function, the possibility of data duplication after secondary mapping of value data of different key data can be avoided. It should be pointed out that when performing secondary mapping processing on value data of the same type of key data type, the first intermediate key-value pair and the second intermediate key-value pair have the same key data, and the secondary mapping function used must be the same, so that the value data of the same task data after secondary mapping are the same, which facilitates deduplication when the first data table and the second data table are subsequently combined.
[0119] In some embodiments, for similar key-value pairs with the same key data in the first data table and the second data table, a more concise and convenient deduplication method is provided that does not require comparison. Specifically, S1500 includes:
[0120] S1510: Extract similar key-value pairs having the same key data from the first data table and the second data table, wherein the similar key-value pairs include the first lightweight value data and the second lightweight value data;
[0121] Extract similar key-value pairs with the same key data from the first data table and the second data table. At this time, the similar key-value pairs include two value data, namely the first lightweight value data and the second lightweight value data. That is to say, for the same key data, there are the first lightweight value data and the second lightweight value data from the first data table.
[0122] S1520: Perform a direct product operation on the first lightweight value data and the second lightweight value data in the same type of key-value pairs;
[0123] A direct product operation is performed on the first lightweight value data and the second lightweight value data in the same type of key-value pairs, and the first lightweight value data and the second lightweight value data in the same type of key-value pairs are multiplied.
[0124] S1530: Merge and remove duplicates based on the results of the direct product operations to generate the task scheduling table.
[0125] When the result of the direct product operation of the first lightweight value data and the second lightweight value data appears, the task of repeating the square of the product of the first data table and the second data table according to the operation result is repeated. Specifically, S1530 also includes:
[0126] S1531, deleting, from the task scheduling table, repeated tasks represented by squared products of direct product operations;
[0127] Perform a direct product operation on the first and second lightweight data in a similar key-value pair, and multiply the first and second lightweight data in the same key-value pair. For example, if the first lightweight data of the same key value is two, X and Y, and the second lightweight data is also two, X and Y, the multiplication of the first and second lightweight data will produce the following results: X², XY, XY, Y².
[0128] S1532: Synchronize the task data in the source environment to the target environment according to the task scheduling table for deleting duplicate tasks.
[0129] X squared and Y squared indicate that both the first data table and the second data table have the task data, which is duplicate data and should be deleted, while XY indicates that the task data does not appear in the second data table and needs to be added to the task scheduling table. The above examples are for better explanation. The deduplication principle of this solution cannot be limited to the general result of multiplying the first lightweight value data and the second lightweight value data. In actual applications, the number and value of the first lightweight value data of the same key value can be any value, and the number and value of the second lightweight value data of the same key value can also be any value.
[0130] In some implementations, when generating a task synchronization table, it is necessary to delete the task data that has been deleted in the source resource. Specifically, S1600 includes:
[0131] S1611. Obtain a task deletion list in the source environment, wherein the deletion list is generated by a revocation request sent by an execution instance;
[0132] When task data is stored in a source resource, the execution instance that published the task data can withdraw or delete the task data. This deleted or withdrawn data is considered deleted scheduled data. A task deletion list is created by pulling deleted scheduled tasks from the source resource.
[0133] S1612. Delete the corresponding scheduled task in the task scheduling table according to the task deletion list;
[0134] After the task deletion list is generated, it is necessary to delete the above tasks in the task scheduling table according to the task data recorded in the task deletion list. Specifically, perform step S1300 on the task data corresponding to the task deletion list pulled from the source resource to generate a third data table corresponding to the task deletion list. Delete the key data and value data corresponding to the third data table in the task scheduling table to complete the scheduled task deletion.
[0135] S1613: Synchronize the task data in the source environment to the target environment according to the task scheduling table of the deleted scheduled task.
[0136] When the scheduled tasks that need to be deleted in the task deletion list are deleted by subtracting the task scheduling table from the task deletion list, the unsynchronized task data are scheduled according to the updated task scheduling table and synchronized to the target resource.
[0137] The task deletion list and task scheduling table are used to delete the canceled or timed-out task data by reducing the table, so as to prevent the target resources from executing the task data, reduce the data processing pressure of the target resources, and improve the task processing efficiency.
[0138] In some implementations, when a source resource synchronizes data with a target resource, the synchronized task data needs to be transmitted via multiple links. Specifically, S1600 includes:
[0139] S1621. Acquire information about multiple links from the source environment to the target environment;
[0140] In a multi-link transmission environment, each link has different load information and can transmit different amounts of data. To optimize the data transmission on each link, it is necessary to extract link information for each of the multiple links. This link information contains the load information for each link, and based on this load information, the optimal data packet size for that link can be calculated.
[0141] S1622: Perform asymmetric data slicing on the task data according to the link parameters represented by the plurality of link information and the task scheduling table, to generate a plurality of data blocks corresponding one-to-one to the plurality of link information;
[0142] After calculating the packet size for each link, the task data corresponding to the task scheduling table is extracted to form a data set. The task data in the data set is sliced based on the optimal packet size required for transmission on each link. Slicing is performed by combining the task data in the data set based on the packet size corresponding to the link, so that the combined task data is equal to or less than the task packet data size. The task set corresponding to each packet is then treated as a data block. If the data size of a single task exceeds the maximum data size of the packets in multiple links, the single task data needs to be sliced and the resulting data blocks distributed across two or more links for transmission.
[0143] S1623: Synchronize the multiple data blocks to the target environment through the links represented by the multiple link information, wherein each link sends a corresponding data block.
[0144] By slicing and combining data, the optimal data block that each link can transmit is formed, so that a one-to-one correspondence is established between the data block and the link. Then, multiple data blocks are transmitted through multiple links respectively, and the task data in the task scheduling table is synchronized to the target resource.
[0145] In this embodiment, the task data in the task scheduling table is combined and sliced through link information, so that each link can transmit data blocks with the optimal data volume, preventing link congestion due to excessive data volume during transmission, and improving transmission efficiency.
[0146] Please refer to FIG2 for details. FIG2 is a schematic diagram of the basic structure of the big data scheduling device of this embodiment.
[0147] As shown in FIG2 , a big data scheduling device includes:
[0148] An acquisition module 1100 is configured to acquire first task information and task synchronization time in a source environment;
[0149] A reading module 1200 is configured to read second task information in a set target environment according to the task synchronization time;
[0150] A first extraction module 1300 is configured to extract a first data table corresponding to the first task information, wherein the first data table is composed of a plurality of intermediate key values corresponding to the first task information;
[0151] A second extraction module 1400 is configured to extract a second data table corresponding to the second task information, wherein the second data table is composed of a plurality of intermediate key values corresponding to the second task information;
[0152] The processing module 1500 is configured to merge and remove duplicates from the first data table and the second data table to generate a task scheduling table.
[0153] The execution module 1600 is configured to synchronize the task data in the source environment to the target environment according to the task scheduling table.
[0154] When performing data scheduling, a big data scheduling device first obtains first task information and task synchronization time from a source environment. Then, based on the task synchronization time, it obtains second task information from a target environment. Key-value pairs are extracted from the first task information, and the extracted key-value pairs are further mapped to generate intermediate key values. Based on the intermediate key values, a first data table corresponding to the first task information is obtained. A second data table corresponding to the second task information is obtained using the same method. The first and second data tables are merged, deleting duplicate tasks from the first and second data tables during the merge to generate a task scheduling table for data scheduling. Finally, the task data from the source environment is scheduled to the target environment using the task scheduling table. This data scheduling allows data from the source environment to be scheduled to the target environment on a regular basis, improving the timeliness of task data scheduling. Furthermore, because both the first and second data tables are composed of intermediate key values, the data tables are more lightweight and smaller than the original data. This reduces the amount of data computation required when merging and deduplicating the data tables, improving the timeliness of task scheduling and avoiding duplicate scheduling of task data.
[0155] Optionally, the first extraction module is further configured to:
[0156] Extracting a plurality of first key-value pairs corresponding to the first task information;
[0157] Based on the set first mapping function, operating the plurality of first key-value pairs to generate a plurality of first intermediate key-value pairs;
[0158] Performing data mapping processing on value data having the same key data in the plurality of first intermediate key-value pairs to generate first lightweight value data;
[0159] The first data table is generated according to the key data in the plurality of first intermediate key-value pairs and the first lightweight value data.
[0160] Optionally, the second extraction module is further configured to:
[0161] Extracting a plurality of second key-value pairs corresponding to the second task information;
[0162] Based on the set first mapping function, operating the plurality of second key-value pairs to generate a plurality of second intermediate key-value pairs;
[0163] Performing data mapping processing on the value data having the same key data in the plurality of second intermediate key-value pairs to generate second lightweight value data;
[0164] The second data table is generated according to the key data in the plurality of second intermediate key-value pairs and the first lightweight value data.
[0165] Optionally, the processing module is further configured to:
[0166] Extracting similar key-value pairs having the same key data from the first data table and the second data table, wherein the similar key-value pairs include the first lightweight value data and the second lightweight value data;
[0167] Performing a direct product operation on the first lightweight value data and the second lightweight value data in the same type of key-value pairs;
[0168] The task scheduling table is generated by merging and removing duplicates according to the operation results of the direct product operation.
[0169] Optionally, the processing module is further configured to:
[0170] Deleting, from the task scheduling table, repeated tasks represented by squared results of direct product operations;
[0171] The task data in the source environment is synchronized to the target environment according to the task scheduling table for deleting duplicate tasks.
[0172] Optionally, the execution module is further configured to:
[0173] Obtaining a task deletion list in the source environment, wherein the deletion list is generated by a revocation request sent by an execution instance;
[0174] Deleting the corresponding scheduled tasks in the task scheduling table according to the task deletion list;
[0175] According to the task scheduling table of deleting the scheduled task, the task data in the source environment is synchronized to the target environment.
[0176] Optionally, the execution module is further configured to:
[0177] Acquire information of multiple links from the source environment to the target environment;
[0178] Performing asymmetric data slicing on the task data according to the link parameters represented by the plurality of link information and the task scheduling table to generate a plurality of data blocks corresponding one-to-one to the plurality of link information;
[0179] The multiple data blocks are synchronized to the target environment through the links represented by the multiple link information, wherein each link sends a corresponding data block.
[0180] To solve the above technical problems, the present application also provides a computer device. Specific reference is made to FIG3 , which is a basic structural block diagram of the computer device of the present embodiment.
[0181] As shown in Figure 3, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected via a system bus. The non-volatile storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a large data scheduling method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can implement a large data scheduling method. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that the structure shown in Figure 3 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation of the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.
[0182] In this embodiment, the processor is used to execute the specific functions of the acquisition module 1100, reading module 1200, first extraction module 1300, second extraction module 1400, processing module 1500, and execution module 1600 in Figure 2. The memory stores the program code and various data required to execute these modules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all submodules in the big data scheduling device. The server can call the server's program code and data to execute the functions of all submodules.
[0183] When performing data scheduling, a computer device first obtains first task information and task synchronization time from a source environment. Then, based on the task synchronization time, it obtains second task information from a target environment. Key-value pairs from the first task information are extracted, and the extracted key-value pairs are further mapped to generate intermediate key values. Based on the intermediate key values, a first data table corresponding to the first task information is obtained. A second data table corresponding to the second task information is obtained using the same method. The first and second data tables are merged, deleting duplicate tasks from the first and second data tables during the merge to generate a task scheduling table for data scheduling. Finally, the task data from the source environment is scheduled to the target environment using the task scheduling table. This data scheduling allows data from the source environment to be scheduled to the target environment on a timely basis, improving the timeliness of task data scheduling. Furthermore, because both the first and second data tables are composed of intermediate key values, the data tables are more lightweight and smaller than the original data. This reduces the amount of data computation required when merging and deduplicating the data tables, improving the timeliness of task scheduling and avoiding duplicate scheduling of task data.
[0184] The present application also provides a storage medium storing computer-readable instructions. The storage medium of the computer-readable instructions can be non-volatile or volatile. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the big data scheduling method of any of the above embodiments.
[0185] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0186] Those skilled in the art will appreciate that the steps, measures, and schemes in the various operations, methods, and processes discussed in this application may be interchanged, modified, combined, or deleted. Furthermore, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be interchanged, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be interchanged, modified, rearranged, decomposed, combined, or deleted.
[0187] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A big data scheduling method, wherein: include: Obtain the first task information and task synchronization time in the source environment; Reading second task information in the set target environment according to the task synchronization time; Extracting a first data table corresponding to the first task information, wherein the first data table is composed of a plurality of intermediate key values corresponding to the first task information; Extracting a second data table corresponding to the second task information, wherein the second data table is composed of a plurality of intermediate key values corresponding to the second task information; Merging and deduplicating the first data table and the second data table to generate a task scheduling table; The task data in the source environment is synchronized to the target environment according to the task scheduling table.
2. The big data scheduling method according to claim 1, wherein: The extracting of the first data table corresponding to the first task information includes: Extracting a plurality of first key-value pairs corresponding to the first task information; Based on the set first mapping function, the plurality of first key-value pairs are operated to generate a plurality of first intermediate key-value pairs; Performing data mapping processing on value data having the same key data in the plurality of first intermediate key-value pairs to generate first lightweight value data; The first data table is generated according to the key data in the plurality of first intermediate key-value pairs and the first lightweight value data.
3. The big data scheduling method according to claim 2, wherein: The extracting of the second data table corresponding to the second task information includes: Extracting a plurality of second key-value pairs corresponding to the second task information; Based on the set first mapping function, the plurality of second key-value pairs are operated to generate a plurality of second intermediate key-value pairs; Performing data mapping processing on value data having the same key data in the plurality of second intermediate key-value pairs to generate second lightweight value data; The second data table is generated according to the key data in the plurality of second intermediate key-value pairs and the first lightweight value data.
4. The big data scheduling method according to claim 3, wherein: The merging and deduplication processing of the first data table and the second data table to generate a task scheduling table includes: Extracting the same type of key-value pairs having the same key data from the first data table and the second data table, wherein the same type of key-value pairs include the first lightweight value data and the second lightweight value data; Performing a direct product operation on the first lightweight value data and the second lightweight value data in the same type of key-value pairs; The task scheduling table is generated by merging and removing duplicates according to the operation results of the direct product operation.
5. The big data scheduling method according to claim 4, wherein: The synchronizing the task data in the source environment to the target environment according to the task scheduling table comprises: Deleting, from the task scheduling table, repeated tasks represented by the squared product of the direct product operation results; According to the task scheduling table for deleting duplicate tasks, the task data in the source environment is synchronized to the target environment.
6. The big data scheduling method according to claim 1, wherein: The synchronizing the task data in the source environment to the target environment according to the task scheduling table comprises: Acquire a task deletion list in the source environment, wherein the deletion list is generated by a revocation request sent by an execution instance; Deleting the corresponding scheduled tasks in the task scheduling table according to the task deletion list; According to the task scheduling table of deleting the scheduled task, the task data in the source environment is synchronized to the target environment.
7. The big data scheduling method according to claim 1, wherein: The synchronizing the task data in the source environment to the target environment according to the task scheduling table comprises: Acquire information of multiple links from the source environment to the target environment; According to the link parameters represented by the multiple link information and the task scheduling table, asymmetric data slicing is performed on the task data to generate multiple data blocks corresponding to the multiple link information one by one; The multiple data blocks are synchronized to the target environment through the links represented by the multiple link information, wherein each link sends a corresponding data block.
8. A big data scheduling device, wherein: include: An acquisition module, used to acquire the first task information and task synchronization time in the source environment; A reading module, used for reading the second task information in the set target environment according to the task synchronization time; A first extraction module, configured to extract a first data table corresponding to the first task information, wherein the first data table is composed of a plurality of intermediate key values corresponding to the first task information; A second extraction module, used to extract a second data table corresponding to the second task information, wherein the second data table is composed of a plurality of intermediate key values corresponding to the second task information; A processing module, used for merging and deduplicating the first data table and the second data table to generate a task scheduling table; An execution module is used to synchronize the task data in the source environment to the target environment according to the task scheduling table.
9. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor is configured to execute: Obtain the first task information and task synchronization time in the source environment; Reading second task information in the set target environment according to the task synchronization time; Extract the first data table corresponding to the first task information, where: The first data table is composed of a plurality of intermediate key values corresponding to the first task information; Extracting a second data table corresponding to the second task information, wherein the second data table is composed of a plurality of intermediate key values corresponding to the second task information; Merging and deduplicating the first data table and the second data table to generate a task scheduling table; The task data in the source environment is synchronized to the target environment according to the task scheduling table.
10. The computer device according to claim 9, wherein: The processor is further configured to execute: Extracting a plurality of first key-value pairs corresponding to the first task information; Based on the set first mapping function, the plurality of first key-value pairs are operated to generate a plurality of first intermediate key-value pairs; Performing data mapping processing on value data having the same key data in the plurality of first intermediate key-value pairs to generate first lightweight value data; The first data table is generated according to the key data in the plurality of first intermediate key-value pairs and the first lightweight value data.
11. The computer device according to claim 10, wherein: The processor is further configured to execute: Extracting a plurality of second key-value pairs corresponding to the second task information; Based on the set first mapping function, the plurality of second key-value pairs are operated to generate a plurality of second intermediate key-value pairs; Performing data mapping processing on value data having the same key data in the plurality of second intermediate key-value pairs to generate second lightweight value data; The second data table is generated according to the key data in the plurality of second intermediate key-value pairs and the first lightweight value data.
12. The computer device according to claim 11, wherein: The processor is further configured to execute: Extracting the same type of key-value pairs having the same key data from the first data table and the second data table, wherein the same type of key-value pairs include the first lightweight value data and the second lightweight value data; Performing a direct product operation on the first lightweight value data and the second lightweight value data in the same type of key-value pairs; The task scheduling table is generated by merging and removing duplicates according to the operation results of the direct product operation.
13. The computer device of claim 12, wherein: The processor is further configured to execute: Deleting, from the task scheduling table, repeated tasks represented by the squared product of the direct product operation results; According to the task scheduling table for deleting duplicate tasks, the task data in the source environment is synchronized to the target environment.
14. The computer device according to claim 9, wherein: The processor is further configured to execute: Acquire a task deletion list in the source environment, wherein the deletion list is generated by a revocation request sent by an execution instance; Deleting the corresponding scheduled tasks in the task scheduling table according to the task deletion list; According to the task scheduling table of deleting the scheduled task, the task data in the source environment is synchronized to the target environment.
15. The computer device of claim 9, wherein: The processor is further configured to execute: Acquire information of multiple links from the source environment to the target environment; According to the link parameters represented by the multiple link information and the task scheduling table, asymmetric data slicing is performed on the task data to generate multiple data blocks corresponding to the multiple link information one by one; The multiple data blocks are synchronized to the target environment through the links represented by the multiple link information, wherein each link sends a corresponding data block.
16. A storage medium storing computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors are configured to perform: Obtain the first task information and task synchronization time in the source environment; Reading second task information in the set target environment according to the task synchronization time; Extract the first data table corresponding to the first task information, where: The first data table is composed of a plurality of intermediate key values corresponding to the first task information; Extracting a second data table corresponding to the second task information, wherein the second data table is composed of a plurality of intermediate key values corresponding to the second task information; Merging and deduplicating the first data table and the second data table to generate a task scheduling table; The task data in the source environment is synchronized to the target environment according to the task scheduling table.
17. The storage medium according to claim 16, wherein: The processor is further configured to execute: Extracting a plurality of first key-value pairs corresponding to the first task information; Based on the set first mapping function, the plurality of first key-value pairs are operated to generate a plurality of first intermediate key-value pairs; Performing data mapping processing on value data having the same key data in the plurality of first intermediate key-value pairs to generate first lightweight value data; The first data table is generated according to the key data in the plurality of first intermediate key-value pairs and the first lightweight value data.
18. The storage medium according to claim 17, wherein: The processor is further configured to execute: Extracting a plurality of second key-value pairs corresponding to the second task information; Based on the set first mapping function, the plurality of second key-value pairs are operated to generate a plurality of second intermediate key-value pairs; Performing data mapping processing on value data having the same key data in the plurality of second intermediate key-value pairs to generate second lightweight value data; The second data table is generated according to the key data in the plurality of second intermediate key-value pairs and the first lightweight value data.
19. The storage medium according to claim 18, wherein: The processor is further configured to execute: Extracting the same type of key-value pairs having the same key data from the first data table and the second data table, wherein the same type of key-value pairs include the first lightweight value data and the second lightweight value data; Performing a direct product operation on the first lightweight value data and the second lightweight value data in the same type of key-value pairs; The task scheduling table is generated by merging and removing duplicates according to the operation results of the direct product operation.
20. The storage medium according to claim 19, wherein: The processor is further configured to execute: Deleting, from the task scheduling table, repeated tasks represented by the squared product of the direct product operation results; According to the task scheduling table for deleting duplicate tasks, the task data in the source environment is synchronized to the target environment.
Citation Information
Patent Citations
Scheduling method and scheduling device of distributed computing system
CN104978228A
Data processing method and system
CN106202092A
Task scheduling method and device, electronic equipment and readable storage medium
CN112306661A
Data synchronization method and device based on update time and primary key and medium
CN114741401A
Data synchronization method and device, equipment and medium
CN116028576A
