A data transmission method, apparatus, device, and storage medium

Abstract

The application discloses a data transmission method and device, equipment and a storage medium, and relates to the technical field of big data. The method comprises the following steps: performing segmentation processing on a received data set to obtain at least one data block, performing key-value pair processing on the data block through a data sending end to obtain key-value pair data, wherein the data sending end is in one-to-one relationship with the data block, determining calculation result data and metadata information of the calculation result data according to the at least one key-value pair data, sending the metadata information of the calculation result data to a task scheduling equipment, and sending the calculation result data to a data receiving equipment, so that the data receiving equipment obtains the metadata information from the task scheduling equipment, and determines target data according to the metadata information and the calculation result data. The above technical scheme is helpful to improving the efficiency and reliability of data transmission by dividing big data into several small data for parallel transmission.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to the field of big data technology, specifically to a data transmission method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development and widespread application of information technology, modern business systems generate a large number of information files, which involve frequent upstream and downstream file transfers.

[0003] Existing technologies typically employ file transfer protocols and corresponding tools to achieve data transfer; for example, the FRP (Fast Reverse Proxy) protocol supports bidirectional file transfer with multiple connections; the SFTP (Secure File Transfer Protocol) protocol uses SSH (Secure Shell) as the underlying protocol for file transfer and supports compression and encryption technologies.

[0004] While the above protocols can meet the needs of frequent upstream and downstream data transmission, they all have certain drawbacks. For example, the FTP protocol does not support data compression, and the transmission time for large files is relatively long. In case of abnormal file transmission due to network jitter or other reasons, although the SFTP protocol supports resuming interrupted transmissions, it cannot support multiple connection transmissions at the same time. Summary of the Invention

[0005] This application provides a data transmission method, apparatus, device, and storage medium to improve the efficiency and reliability of data transmission.

[0006] According to one aspect of this application, a data transmission method is provided, the method being applied to a data transmitting device in a data transmission system, the data transmission system further comprising a task scheduling device and a data receiving device; the data transmitting device and the data receiving device are communicatively connected; the task scheduling device is communicatively connected to both the data transmitting device and the data receiving device; the data transmitting device includes at least one data transmitting end; the method includes:

[0007] The received dataset is segmented to obtain at least one data block, and the data block is processed into key-value pairs through the data sending end to obtain key-value pair data; wherein, the data sending end and the data block have a one-to-one relationship;

[0008] Based on the at least one key-value pair data, determine the calculation result data and the metadata information of the calculation result data;

[0009] The metadata information of the calculation result data is sent to the task scheduling device, and the calculation result data is sent to the data receiving device, so that the data receiving device can obtain the metadata information from the task scheduling device and determine the target data based on the metadata information and the calculation result data.

[0010] According to another aspect of this application, a data transmission apparatus is provided, which is configured in a data transmission system as a data transmitting device. The data transmission system further includes a task scheduling device and a data receiving device; the data transmitting device and the data receiving device are communicatively connected; the task scheduling device is communicatively connected to both the data transmitting device and the data receiving device; the data transmitting device includes at least one data transmitting end; the apparatus includes:

[0011] The first data determination module is used to segment the received dataset to obtain at least one data block, and to perform key-value pair processing on the data block through the data sending end to obtain key-value pair data; wherein, the data sending end and the data block have a one-to-one relationship;

[0012] The second data determination module is used to determine the calculation result data and the metadata information of the calculation result data based on the at least one key-value pair data.

[0013] The data transmission module is used to send the metadata information of the calculation result data to the task scheduling device and to the data receiving device, so that the data receiving device can obtain the metadata information from the task scheduling device and determine the target data based on the metadata information and the calculation result data.

[0014] According to another aspect of this application, an electronic device is provided, the electronic device comprising:

[0015] One or more processors;

[0016] Memory, used to store one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the data transmission methods provided in the embodiments of this application.

[0018] According to another aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements any of the data transmission methods provided in the embodiments of this application.

[0019] According to another aspect of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the data transmission methods provided in the embodiments of this application.

[0020] This application segments the received dataset to obtain at least one data block, and then processes the data block into key-value pairs via a data sending end to obtain key-value pair data; wherein the data sending end and the data block have a one-to-one relationship; based on at least one key-value pair data, it determines the calculation result data and its metadata information; it then sends the metadata information of the calculation result data to a task scheduling device and the calculation result data to a data receiving device, enabling the data receiving device to obtain the metadata information from the task scheduling device and determine the target data based on the metadata information and the calculation result data. This technical solution, by dividing large datasets into several smaller datasets for parallel transmission, helps improve the efficiency and reliability of data transmission. Attached Figure Description

[0021] Figure 1 This is a flowchart of a data transmission method according to Embodiment 1 of this application;

[0022] Figure 2 This is a flowchart of a data transmission method according to Embodiment 2 of this application;

[0023] Figure 3 This is a schematic diagram of a data transmission device according to Embodiment 3 of this application;

[0024] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the data transmission method of the embodiments of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0027] Furthermore, it should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of datasets and data blocks involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0028] Example 1

[0029] Figure 1 This is a flowchart of a data transmission method according to Embodiment 1 of this application. This embodiment is applicable to the transmission of large files in clusters and can be executed by a data transmission device. This data transmission device can be implemented in hardware and / or software and can be configured in a computer device, such as the data sending device of a data transmission system. The data transmission system also includes a task scheduling device and a data receiving device; the data sending device and the data receiving device are communicatively connected; the task scheduling device is communicatively connected to both the data sending device and the data receiving device; the data sending device includes at least one data sending end. Figure 1 As shown, the method includes:

[0030] S110. The received dataset is segmented to obtain at least one data block, and the data block is processed into key-value pairs through the data sending end to obtain key-value pair data; wherein, the data sending end and the data block have a one-to-one relationship.

[0031] In this context, a dataset refers to the raw data that needs to be manipulated during computation or processing; a dataset contains multiple data elements and may be a collection of data in the form of files, records, or other forms. A data block is the smallest unit after the original dataset has been divided, and can be understood as a subset of the original dataset; in big data processing, data is usually divided into multiple data blocks to improve processing efficiency. A data sender is the component responsible for processing data and sending it to subsequent processing stages; typically, the data sender is responsible for dividing, processing, and transmitting data; for example, the data sender in this application can be a Map (Map Phase, mapping function) end. A key-value pair is a data structure composed of a "key" and a "value"; in distributed computing or big data processing, key-value pairs are usually used as the basic unit of data.

[0032] Specifically, the received dataset is segmented to break down a large dataset into smaller data blocks, each of which can be processed independently. Once the data is segmented into smaller blocks, the data sending end processes the blocks and converts them into key-value pairs.

[0033] S120. Based on at least one key-value pair, determine the calculation result data and the metadata information of the calculation result data.

[0034] The calculation result data refers to the final result data generated after a series of processing, calculations, and operations; this data is usually the output of the processing procedure. Metadata information refers to data about the calculation result data, typically describing attributes such as the data's structure, format, source, and storage method; the metadata information of the calculation result data includes relevant descriptions and processing information of the data; for example, the metadata information may include the correspondence between the data sending end and the calculation result file.

[0035] Optionally, the calculation result file includes an index file and a data file; wherein, the index file is used to identify the start and end positions of the data in each partition in the data file; the data file is used to record the data.

[0036] In one alternative implementation, the data sending device further includes a distributed storage terminal; correspondingly, key-value pair data can be stored in the distributed storage system to obtain the calculation result data and the metadata information of the calculation result data.

[0037] Distributed storage refers to a storage system that distributes data across multiple locations. In such a system, data is distributed across multiple physical locations or servers, typically connected together via a network. Each storage node (storage device) can store one or more parts of the data. Distributed storage usually ensures high availability and fault tolerance of data through redundancy mechanisms.

[0038] S130. Send the metadata information of the calculation result data to the task scheduling device and send the calculation result data to the data receiving device, so that the data receiving device can obtain the metadata information from the task scheduling device and determine the target data based on the metadata information and the calculation result data.

[0039] The task scheduling device refers to the system component that manages and schedules computational tasks; it is responsible for assigning tasks to appropriate computing nodes and managing the execution order of tasks. The data receiving device is the system component that receives and processes data, typically used to receive computational result data sent by the task scheduling device. Target data refers to data that has undergone further analysis, processing, or manipulation based on the computational result data and its metadata; it is the final output of the computation or data used for subsequent operations.

[0040] In one optional embodiment, the data receiving device includes at least one data receiving end; the target data is obtained by the data receiving device merging the data to be transmitted; the data to be transmitted is determined by the data receiving end based on the candidate data and the current storage location of the candidate data; the candidate data and the current storage location of the candidate data are determined by the data receiving end based on metadata information and calculation result data.

[0041] In this context, the data receiving end refers to a specific part of the data receiving device, responsible for performing data reception, parsing, and processing tasks. It may involve reading and merging data, and determining how to obtain the target data. For example, the data receiving end in this application could be a Reduce (Reducer) end. Data to be transmitted refers to data that needs to be received and further processed or sent; it is typically extracted from candidate data and serves as the final transmitted content. Candidate data refers to data selected for further processing; it includes multiple data sources or files where data has undergone preliminary screening and is prepared for processing or merging in subsequent steps. The current storage location of the candidate data refers to its location in the storage system; it includes the specific location or path of the data storage, typically the location of a file or in a database.

[0042] Furthermore, the candidate data includes candidate index files and candidate data files; the data to be transmitted is obtained by the data receiving end reading the candidate data files with the start and end positions of the candidate data files as the range; the start and end positions of the candidate data files are determined by the data receiving end by parsing the candidate index files according to the current storage location of the candidate data.

[0043] The candidate index file is a file used to describe the location or structure of candidate data storage; it contains index information about the data file, helping to locate and access specific parts of the data file. The candidate data file is a file that stores candidate data; these files contain data that requires further processing, from which the data receiving end extracts the data to be transmitted. The start and end positions of the candidate data file refer to the specific range of data locations within the file; these positions are determined by the data receiving end based on the parsing results of the candidate index file; the data receiving end can read specific data portions of the file based on the start and end positions of the candidate data file, thereby extracting the data to be transmitted.

[0044] For example, based on the index file and data file included in the computation result file, metadata information of the computation result file can be sent to the task scheduler; the task scheduler can be responsible for maintaining the metadata information uploaded by each data sender. The computation result file is then transferred to the target file system of the data receiving module, so that the corresponding data receiving end can obtain the data in the computation result file through the target file system. Data Reception

[0045] In another optional implementation, the target file system can name the computation result files according to a predetermined naming rule and store them according to a predetermined directory structure; wherein, the naming rule and directory structure can be any pre-specified rule and directory structure. Typically, the target file system can organize files using the aforementioned naming rule and directory structure. The target file system can be any file system pre-specified, distinct from the local disk of the Map server, based on actual application requirements; the target file system can also be a file system defined according to rules, such as a file system capable of providing multiple copies; the predetermined naming rule can include the identifier of the corresponding Map server in the name of the computation result file, and distinguish between data files and index files based on their suffixes.

[0046] For example, the Map nodes identified as "0" and "1" upload their calculation result files to the target file system after writing them to their local disks, with each filename corresponding to the unique identifier of the Map node. For instance, the index file and data file uploaded by the Map node identified as "0" stored in the target file system are named "0.index" and "0.data," respectively. The index file and data file uploaded by the Map node identified as "1" stored in the target file system are named "l.index" and "l.data," respectively.

[0047] Understandably, by naming and storing the computation result files at the data receiving end, the overhead caused by maintaining filename information and the mapping relationship between the Map end and the files can be avoided, thus facilitating the Reduce end to directly access the required files and data; it can also avoid the additional overhead caused by maintaining metadata information such as file storage paths, thereby improving the query efficiency of computation result files in the target file system; moreover, the Reduce end can directly obtain the storage path of the required computation result files and then directly access the aforementioned target file system without first obtaining file path information through other systems.

[0048] Furthermore, the data receiving end can parse the index file in the calculation result file based on the storage path in the metadata information, thereby obtaining the start and end positions of the data of the corresponding partition in the data file corresponding to the index file, and then read the data from the corresponding data file to obtain the target data.

[0049] For example, the data receiving device sends a data acquisition request to at least one Reduce end based on the address of at least one Map end, for acquiring data of the partition corresponding to the Reduce end; the Reduce end can read data belonging to its respective partition from the Map end or the target file system in parallel by multiple Reduce ends; the Reduce task is executed by the data receiving end to generate the target data.

[0050] This application embodiment segments the received dataset to obtain at least one data block, and performs key-value pair processing on the data block through a data sending end to obtain key-value pair data; wherein the data sending end and the data block have a one-to-one relationship; based on at least one key-value pair data, calculation result data and metadata information of the calculation result data are determined; the metadata information of the calculation result data is sent to a task scheduling device, and the calculation result data is sent to a data receiving device, so that the data receiving device can obtain the metadata information from the task scheduling device, and determine the target data based on the metadata information and the calculation result data. The above technical solution, by dividing large data into several smaller data and transmitting them in parallel, helps to improve the efficiency and reliability of data transmission.

[0051] Example 2

[0052] Figure 2This is a flowchart of a data transmission method according to Embodiment 2 of this application. Based on the technical solutions of the above embodiments, this embodiment refines "determining calculation result data based on at least one key-value pair data" into "writing at least one key-value pair data into the memory buffer of the distributed storage terminal, and during the writing process, partitioning and sorting the key-value pair data to obtain first written data; when the amount of key-value pair data written in the distributed storage terminal reaches a data volume threshold, starting an overflow thread to store the unwritten key-value pair data to the local disk of the distributed storage terminal to obtain second written data; integrating the first written data and the second written data to obtain target written data, and writing the target written data to the local disk to obtain calculation result data." It should be noted that for parts not detailed in this embodiment, please refer to the relevant descriptions in other embodiments. Figure 2 As shown, the method includes:

[0053] S210. The received dataset is segmented to obtain at least one data block, and the data block is processed into key-value pairs through the data sending end to obtain key-value pair data; wherein, the data sending end and the data block have a one-to-one relationship.

[0054] S220. Write at least one key-value pair of data into the memory buffer of the distributed storage terminal, and during the writing process, partition and sort the key-value pair of data to obtain the first written data.

[0055] The memory buffer refers to a portion of computer memory that temporarily stores data being processed; it can accelerate data read and write operations and avoid frequent access to slower disk storage. Partition sorting refers to partitioning data according to certain specific rules and sorting the data within each partition; partition sorting can improve the efficiency of subsequent data processing, especially in distributed storage systems. The first data written refers to the data that is written to the memory buffer for the first time; at this stage, the data is in a preliminary, unprocessed state.

[0056] For example, at least one key-value pair of data is written to a memory buffer (e.g., a circular memory buffer) of a distributed storage system; and, during the writing process, partitioning and sorting are performed; thus, the first written data is obtained.

[0057] S230. When it is detected that the amount of key-value pairs written in the distributed storage has reached the data volume threshold, the overflow thread is started to store the unwritten key-value pairs to the local disk of the distributed storage, thus obtaining the second written data.

[0058] The overflow thread is a background thread specifically responsible for handling data overflow operations when the memory buffer is full. That is, when the data in the memory buffer reaches a certain threshold, the overflow thread writes the data to the disk to free up memory space. The local disk refers to the hard drive of a computer or storage node, used for long-term data storage; unlike memory, disk storage is slower but has a larger capacity, suitable for storing large amounts of data. The second write data refers to key-value pairs that have not yet been written to the memory buffer.

[0059] For example, when the amount of key-value pair data written reaches the data volume threshold, a spill thread can be started to spill the data in the memory buffer to the local disk, thereby generating a temporary file, i.e., the second written data.

[0060] S240. Integrate the first written data and the second written data to obtain the target written data, and write the target written data to the local disk to obtain the calculation result data and the metadata information of the calculation result data.

[0061] The target write data refers to the data that will eventually be written to the storage system after a series of processes (such as partition sorting, spilling, etc.); it is data that has been organized and optimized, and is usually persistently stored on the disk.

[0062] Optionally, the first and second written data are partitioned and merged to obtain the third written data; the third written data is partitioned and sorted to obtain the target written data, and the target written data is written to the local disk to obtain the calculation result data.

[0063] Partition merging refers to the process of merging multiple data partitions; it is typically used to combine multiple small data blocks into a larger data block, thereby improving data processing efficiency.

[0064] In one alternative implementation, if it is found that all key-value pair data has been written to the distributed storage terminal and the data volume threshold has not been reached, the first written data is directly written to the local disk to obtain the calculation result data.

[0065] For example, once all key-value pairs corresponding to the data sender have been written, it can be determined whether there is any spilled second write data. If the response determines that there is no spilled data, the data in the memory buffer can be directly written to the local disk to generate the calculation result file. If the response determines that there is spilled data, all the second write data and the first write data in the memory buffer can be merged according to partitions, that is, data with consistent partitions can be merged. Then, the merged data can be sorted again according to partitions, and the sorted data can be written to the local disk to generate the calculation result file.

[0066] It should be noted that partition sorting can be specifically based on the partition to which the hash value of the key belongs; and, key-value pairs with the same key can be combined.

[0067] S250, Send the metadata information of the calculation result data to the task scheduling device and send the calculation result data to the data receiving device, so that the data receiving device can obtain the metadata information from the task scheduling device and determine the target data based on the metadata information and the calculation result data.

[0068] In one optional implementation, the data transmission method provided in this application is implemented based on a MapReduce data transmission system. First, the Map side executes Map tasks to generate computation result files. Then, the Map side uploads the computation result files to a target file system. Upon receiving the computation result files, the target file system names the computation result files according to a predetermined naming rule and stores them according to a predetermined directory structure. Next, the Reduce side obtains the addresses of at least one Map side corresponding to the Reduce side from the task scheduler. Based on the addresses of at least one Map side, the Reduce side sends a data retrieval request to at least one Map side to retrieve data from the partition corresponding to the Reduce side. Then, in response to determining that data retrieval from the Map side corresponding to the Reduce side has failed, the Reduce side retrieves the data from the partition corresponding to the Reduce side from the target file system providing redundant storage. Finally, the Reduce side uses the retrieved data to execute Reduce tasks to generate the final result of the MapReduce process.

[0069] Understandably, the MapReduce cluster transmission mode supports one-to-many, many-to-one, and many-to-many relationships between sending and receiving nodes, avoiding transmission task failures caused by single node failures and increasing transmission high availability. The MapReduce framework splits large files into several smaller files, transmits these smaller files in parallel, and provides multiple replicas of each smaller file, improving transmission efficiency, reducing transmission time, and enhancing file transmission reliability. Furthermore, the splitting of large files into smaller files, each containing indexes and source data information, allows downstream processes to easily query the required data before the smaller files are merged into a larger file, improving downstream call speed.

[0070] This application embodiment segments the received dataset to obtain at least one data block, and processes the data block into key-value pairs via a data sending end to obtain key-value pair data; wherein the data sending end and the data block have a one-to-one relationship; at least one key-value pair data is written into the memory buffer of the distributed storage end, and during the writing process, the key-value pair data is partitioned and sorted to obtain the first written data; when the amount of key-value pair data written in the distributed storage end reaches a data volume threshold, an overflow thread is started to store the unwritten key-value pair data to the local disk of the distributed storage end to obtain the second written data; the first written data and the second written data are integrated to obtain the target written data, and the target written data is written to the local disk to obtain the calculation result data and the metadata information of the calculation result data; the metadata information of the calculation result data is sent to the task scheduling device, and the calculation result data is sent to the data receiving device, so that the data receiving device can obtain the metadata information from the task scheduling device and determine the target data based on the metadata information and the calculation result data. The above technical solution, by dividing big data into several small data and transmitting them in parallel, helps to improve the efficiency and reliability of data transmission.

[0071] Example 3

[0072] Figure 3 This is a schematic diagram of a data transmission device according to Embodiment 3 of this application, applicable to the transmission of large file clusters. The data transmission device can be implemented in hardware and / or software and can be configured in a computer device, such as the data sending device of a data transmission system. The data transmission system also includes a task scheduling device and a data receiving device; the data sending device and the data receiving device are communicatively connected; the task scheduling device is communicatively connected to both the data sending device and the data receiving device; the data sending device includes at least one data sending end. Figure 3 As shown, the device includes:

[0073] The first data determination module 310 is used to segment the received dataset to obtain at least one data block, and to perform key-value pair processing on the data block through the data sending end to obtain key-value pair data; wherein, the data sending end and the data block have a one-to-one relationship.

[0074] The second data determination module 320 is used to determine the calculation result data and the metadata information of the calculation result data based on at least one key-value pair data.

[0075] The data transmission module 330 is used to send the metadata information of the calculation result data to the task scheduling device and to the data receiving device, so that the data receiving device can obtain the metadata information from the task scheduling device and determine the target data based on the metadata information and the calculation result data.

[0076] This application embodiment segments the received dataset to obtain at least one data block, and performs key-value pair processing on the data block through a data sending end to obtain key-value pair data; wherein the data sending end and the data block have a one-to-one relationship; based on at least one key-value pair data, calculation result data and metadata information of the calculation result data are determined; the metadata information of the calculation result data is sent to a task scheduling device, and the calculation result data is sent to a data receiving device, so that the data receiving device can obtain the metadata information from the task scheduling device, and determine the target data based on the metadata information and the calculation result data. The above technical solution, by dividing large data into several smaller data and transmitting them in parallel, helps to improve the efficiency and reliability of data transmission.

[0077] Optionally, the data transmission device further includes a distributed storage terminal; correspondingly, the second data determination module 320 includes:

[0078] The first data writing unit is used to write at least one key-value pair data into the memory buffer of the distributed storage terminal, and during the writing process, the key-value pair data is partitioned and sorted to obtain the first written data;

[0079] The second data writing unit is used to start the spill thread when it is detected that the amount of key-value pair data written in the distributed storage terminal has reached the data amount threshold, so as to store the unwritten key-value pair data to the local disk of the distributed storage terminal and obtain the second written data.

[0080] The data integration unit is used to integrate the first written data and the second written data to obtain the target written data, and write the target written data to the local disk to obtain the calculation result data.

[0081] Optional, data integration unit, specifically used for:

[0082] The first and second written data are partitioned and merged to obtain the third written data;

[0083] The third write data is partitioned and sorted to obtain the target write data, which is then written to the local disk to obtain the calculation result data.

[0084] Optionally, the second data determination module 320 is also used to directly write the first written data to the local disk to obtain the calculation result data when it is found that all key-value pair data has been written to the distributed storage terminal and the data volume threshold has not been reached.

[0085] Optionally, the data receiving device includes at least one data receiving end; the target data is obtained by the data receiving device merging the data to be transmitted; the data to be transmitted is determined by the data receiving end based on the candidate data and the current storage location of the candidate data; the candidate data and the current storage location of the candidate data are determined by the data receiving end based on metadata information and calculation result data.

[0086] Optionally, the candidate data includes a candidate index file and a candidate data file; the data to be transmitted is obtained by the data receiving end reading the candidate data file with the start and end positions of the candidate data file as the range; the start and end positions of the candidate data file are determined by the data receiving end by parsing the candidate index file according to the current storage location of the candidate data.

[0087] The data transmission apparatus provided in this application can execute the data transmission method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing each data transmission method.

[0088] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0089] Example 4

[0090] Figure 4 This is a schematic diagram of the structure of an electronic device 410 implementing the data transmission method of an embodiment of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0091] like Figure 4As shown, the electronic device 410 includes at least one processor 411 and a memory, such as a read-only memory (ROM) 412 or a random access memory (RAM) 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the ROM 412 or loaded from storage unit 418 into the RAM 413. The RAM 413 may also store various programs and data required for the operation of the electronic device 410. The processor 411, ROM 412, and RAM 413 are interconnected via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0092] Multiple components in electronic device 410 are connected to I / O interface 415, including: input unit 416, such as keyboard, mouse, etc.; output unit 417, such as various types of displays, speakers, etc.; storage unit 418, such as disk, optical disk, etc.; and communication unit 419, such as network card, modem, wireless transceiver, etc. Communication unit 419 allows electronic device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0093] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 411 performs the various methods and processes described above, such as data transfer methods.

[0094] In some embodiments, the data transfer method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 410 via ROM 412 and / or communication unit 419. When the computer program is loaded into RAM 413 and executed by processor 411, one or more steps of the data transfer method described above may be performed. Alternatively, in other embodiments, processor 411 may be configured as the data transfer method by any other suitable means (e.g., by means of firmware).

[0095] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0096] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data transmission apparatus, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0097] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0098] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0099] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0100] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0101] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0102] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data transmission method, characterized in that, A data transmitting device is used in a data transmission system, the data transmission system further including a task scheduling device and a data receiving device; the data transmitting device and the data receiving device are communicatively connected. The task scheduling device is communicatively connected to the data sending device and the data receiving device, respectively. The data transmitting device includes at least one data transmitting end; the method includes: The received dataset is segmented to obtain at least one data block, and the data block is processed into key-value pairs through the data sending end to obtain key-value pair data; wherein, the data sending end and the data block have a one-to-one relationship; Based on the at least one key-value pair data, determine the calculation result data and the metadata information of the calculation result data; The metadata information of the calculation result data is sent to the task scheduling device, and the calculation result data is sent to the data receiving device, so that the data receiving device can obtain the metadata information from the task scheduling device and determine the target data based on the metadata information and the calculation result data.

2. The method according to claim 1, characterized in that, The data transmission device further includes a distributed storage terminal; correspondingly, determining the calculation result data based on the at least one key-value pair data includes: The at least one key-value pair data is written into the memory buffer of the distributed storage terminal, and during the writing process, the key-value pair data is partitioned and sorted to obtain the first written data; If the amount of key-value pairs written in the distributed storage terminal is found to have reached the data volume threshold, an overflow thread is started to store the unwritten key-value pairs to the local disk of the distributed storage terminal, thus obtaining the second written data. The first written data and the second written data are integrated to obtain the target written data, and the target written data is written to the local disk to obtain the calculation result data.

3. The method according to claim 2, characterized in that, The first written data and the second written data are integrated to obtain the target written data, and the target written data is written to the local disk to obtain the calculation result data, including: The first and second written data are partitioned and merged to obtain the third written data; The third write data is partitioned and sorted to obtain the target write data, and the target write data is written to the local disk to obtain the calculation result data.

4. The method according to claim 2, characterized in that, The method further includes: If it is found that all the key-value pair data has been written to the distributed storage terminal and the data volume threshold has not been reached, the first written data is directly written to the local disk to obtain the calculation result data.

5. The method according to claim 1, characterized in that, The data receiving device includes at least one data receiving end; the target data is obtained by the data receiving device merging the data to be transmitted; the data to be transmitted is determined by the data receiving end based on candidate data and the current storage location of the candidate data; the candidate data and the current storage location of the candidate data are determined by the data receiving end based on the metadata information and the calculation result data.

6. The method according to claim 5, characterized in that, The candidate data includes a candidate index file and a candidate data file; the data to be transmitted is obtained by the data receiving end reading the candidate data file with the start and end positions of the candidate data file as the range; the start and end positions of the candidate data file are determined by the data receiving end by parsing the candidate index file according to the current storage location of the candidate data.

7. A data transmission device, characterized in that, A data transmitting device configured in a data transmission system, the data transmission system further including a task scheduling device and a data receiving device; the data transmitting device and the data receiving device are communicatively connected. The task scheduling device is communicatively connected to the data sending device and the data receiving device, respectively. The data transmitting device includes at least one data transmitting end; the apparatus includes: The first data determination module is used to segment the received dataset to obtain at least one data block, and to perform key-value pair processing on the data block through the data sending end to obtain key-value pair data; wherein, the data sending end and the data block have a one-to-one relationship; The second data determination module is used to determine the calculation result data and the metadata information of the calculation result data based on the at least one key-value pair data. The data transmission module is used to send the metadata information of the calculation result data to the task scheduling device and to the data receiving device, so that the data receiving device can obtain the metadata information from the task scheduling device and determine the target data based on the metadata information and the calculation result data.

8. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data transmission method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data transmission method as described in any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the data transmission method according to any one of claims 1-6.

Images