Data transmission method and device, equipment and storage medium

Abstract

The invention discloses a data transmission method and device, equipment and a storage medium, and relates to the technical field of big data. The method comprises the following steps: performing segmentation processing on a received data set to obtain at least one data block, and performing key-value pair processing on the data block through a data sending end to obtain key-value pair data; wherein the data sending ends and the data blocks are in a one-to-one relationship; determining calculation result data and metadata information of the calculation result data according to the at least one key value pair data; and sending metadata information of the calculation result data to a task scheduling device, and sending the calculation result data to a data receiving device, so that the data receiving device obtains the metadata information from the task scheduling device, and determines target data according to the metadata information and the calculation result data. According to the technical scheme, the big data is divided into a plurality of small data to be transmitted in parallel, so that the data transmission efficiency and reliability can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, in particular to the field of big data technology, and specifically to a data transmission method, apparatus, device and storage medium. Background Art

[0002] With the rapid development and widespread application of information technology, today's business systems have generated a large number of information files, and these files involve frequent upstream and downstream file transfers.

[0003] The existing technology usually uses file transfer protocols and corresponding tools to realize data transmission; for example, the FRP (Fast Reverse Proxy) protocol that supports multi-connection bidirectional file transfer; the SFTP (Secure File Transfer Protocol) protocol that uses SSH (Secure Shell) as the underlying protocol for file transfer and supports compression and encryption technology.

[0004] Although the above protocols can meet the needs of frequent upstream and downstream data transmission, they all have certain defects. For example, the FTP protocol does not support data compression, and the transmission time for files with large data volumes is longer. In the case of file transfer exceptions caused by network jitter and other reasons, although the SFTP protocol supports breakpoint resumption, it cannot support multi-connection transmission while supporting breakpoint resumption. Summary of the invention

[0005] The present application provides a data transmission method, apparatus, device and storage medium to improve the efficiency and reliability of data transmission.

[0006] According to one aspect of the present application, a data transmission method is provided, which is applied to a data transmission device of a data transmission system, wherein the data transmission system further comprises a task scheduling device and a data receiving device; the data transmission device is communicatively connected to the data receiving device; the task scheduling device is communicatively connected to the data transmission device and the data receiving device respectively; the data transmission device comprises at least one data transmission terminal; the method comprises:

[0007] Segment the received data set to obtain at least one data block, and perform key-value pair processing on the data block through the data sending end to obtain key-value pair data; wherein the data sending end and the data block are in a one-to-one relationship;

[0008] Determine calculation result data and metadata information of the calculation result data according to the at least one key-value pair data;

[0009] The metadata information of the calculation result data is sent to the task scheduling device, and the calculation result data is sent to the data receiving device, so that the data receiving device obtains the metadata information from the task scheduling device and determines the target data according to the metadata information and the calculation result data.

[0010] According to another aspect of the present application, a data transmission device is provided, which is configured in a data transmission device of a data transmission system, wherein the data transmission system further comprises a task scheduling device and a data receiving device; the data transmission device is communicatively connected to the data receiving device; the task scheduling device is communicatively connected to the data transmission device and the data receiving device respectively; the data transmission device comprises at least one data transmission terminal; the device comprises:

[0011] A first data determination module is used to segment the received data set to obtain at least one data block, and perform key-value pair processing on the data block through the data sending end to obtain key-value pair data; wherein the data sending end and the data block are in a one-to-one relationship;

[0012] A second data determination module, used to determine calculation result data and metadata information of the calculation result data according to the at least one key-value pair data;

[0013] A data transmission module is used to send the metadata information of the calculation result data to the task scheduling device, and to send the calculation result data to the data receiving device, so that the data receiving device obtains the metadata information from the task scheduling device and determines the target data based on the metadata information and the calculation result data.

[0014] According to another aspect of the present application, an electronic device is provided, the electronic device comprising:

[0015] one or more processors;

[0016] A memory for storing one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement any one of the data transmission methods provided in the embodiments of the present application.

[0018] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, any one of the data transmission methods provided in the embodiments of the present application is implemented.

[0019] According to another aspect of the present application, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the data transmission methods provided in the embodiments of the present application.

[0020] The present application obtains at least one data block by segmenting the received data set, and obtains key-value pair data by performing key-value pair processing on the data block through the data sending end; wherein the data sending end and the data block are in a one-to-one relationship; according to at least one key-value pair data, the calculation result data and the metadata information of the calculation result data are determined; the metadata information of the calculation result data is sent to the task scheduling device, and the calculation result data is sent to the data receiving device, so that the data receiving device obtains the metadata information from the task scheduling device, and determines the target data according to the metadata information and the calculation result data. The above technical solution helps to improve the efficiency and reliability of data transmission by segmenting large data into several small data for parallel transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a data transmission method provided according to Embodiment 1 of the present application;

[0022] Figure 2 is a flow chart of a data transmission method provided according to Embodiment 2 of the present application;

[0023] Figure 3 is a structural schematic diagram of a data transmission device provided according to Embodiment 3 of the present application;

[0024] Figure 4 It is a structural diagram of an electronic device that implements the data transmission method of an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0027] In addition, it should be noted that the collection, storage, use, processing, transmission, provision and disclosure of relevant data such as data sets and data blocks involved in the technical solution of this application are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0028] Embodiment 1

[0029] Figure 1 This is a flowchart of a data transmission method provided according to the first embodiment of the present application. This embodiment is applicable to the transmission of large file clusters and can be executed by a data transmission device. The data transmission device can be implemented in the form of hardware and / or software. The data transmission device can be configured in a computer device, such as a data sending device in a data transmission system; the data transmission system also includes a task scheduling device and a data receiving device; the data sending device and the data receiving device are communicatively connected; the task scheduling device is communicatively connected to the data sending device and the data receiving device respectively; the data sending device includes at least one data sending terminal. Figure 1 As shown, the method includes:

[0030] S110, segmenting the received data set to obtain at least one data block, and performing key-value pair processing on the data block through the data sending end to obtain key-value pair data; wherein the data sending end and the data block are in a one-to-one relationship.

[0031] Among them, a data set refers to the original data that needs to be operated during the calculation or processing process; a data set contains multiple data elements, which may be a file, record or other form of data collection. A data block refers to the smallest unit after the original data set is segmented, which can be understood as a subset of the original data set; in big data processing, data is usually divided into multiple data blocks to improve processing efficiency. The data sending end refers to the component responsible for processing data and sending it to subsequent processing stages; usually, the data sending end is responsible for segmenting, processing and transmitting data; illustratively, the data sending end of the present application can be a Map (Map Phase, mapping function) end. A key-value pair refers to a data structure consisting of a "key" and a "value"; in distributed computing or big data processing, a key-value pair is usually used as the basic unit of data.

[0032] Specifically, the received data set is segmented to split the large data set into smaller data blocks, each of which can be processed independently; once the data is segmented into smaller data blocks, the data blocks are processed by the data sending end to convert the data blocks into key-value pairs.

[0033] S120. Determine calculation result data and metadata information of the calculation result data according to at least one key-value pair data.

[0034] The calculation result data refers to the final result data generated after a series of processing, calculation and operation; the data is usually the output of the processing process. Metadata information refers to data about the calculation result data, which usually describes the data's structure, format, source, storage method and other attributes; the metadata information of the calculation result data includes the relevant description and processing information of the data; exemplary, the metadata information may include the corresponding relationship between the data sending end and the calculation result file.

[0035] Optionally, the calculation result file includes an index file and a data file; wherein the index file is used to identify the starting position and the ending position of the data of each partition in the data file; and the data file is used to record the data.

[0036] In an optional implementation, the data sending device further includes a distributed storage end; accordingly, the key-value pair data can be stored in the distributed storage system to obtain the calculation result data and metadata information of the calculation result data.

[0037] Among them, distributed storage refers to a storage system that stores data in multiple locations. In this system, data is stored in multiple physical locations or servers, usually connected together through a network; each storage node (storage device) can store one or more parts of the data; distributed storage usually ensures high availability and fault tolerance of data through redundant mechanisms.

[0038] S130, sending metadata information of the calculation result data to the task scheduling device, and sending the calculation result data to the data receiving device, so that the data receiving device obtains the metadata information from the task scheduling device, and determines the target data according to the metadata information and the calculation result data.

[0039] Among them, the task scheduling device refers to the system component that manages and schedules computing tasks; it is responsible for assigning tasks to appropriate computing nodes and managing the execution order of tasks. The data receiving device refers to the system component that receives and processes data, and is usually used to receive the computing result data sent by the task scheduling device. The target data refers to the data that is further analyzed, processed or processed based on the computing result data and its metadata information; it is the final output of the calculation or the data used for subsequent operations.

[0040] In an optional embodiment, the data receiving device includes at least one data receiving end; the target data is obtained by the data receiving device by merging the data to be transmitted; the data to be transmitted is determined by the data receiving end based on the candidate data and the current storage position of the candidate data; the candidate data and the current storage position of the candidate data are determined by the data receiving end based on metadata information and calculation result data.

[0041] Among them, the data receiving end refers to the specific part of the data receiving device, which is responsible for performing the tasks of receiving, parsing and processing data; it may involve reading, merging data, and determining how to obtain the target data; illustratively, the data receiving end of the present application can be a Reduce (Reducer, reduction stage) end. The data to be transmitted refers to the data that needs to be received and further processed or sent; it is usually extracted from the candidate data as the final content of the transmission. Candidate data refers to the data selected for further processing; it includes multiple data sources or files, in which the data has been preliminarily screened and is ready to be processed or merged in subsequent steps. The current storage location of the candidate data refers to the location of the candidate data in the storage system; it includes the specific location or path of the data storage, usually the location of a file or the location in the database.

[0042] Furthermore, the candidate data includes a candidate index file and a candidate data file; the data to be transmitted is obtained by the data receiving end by reading the candidate data file with the starting position and the ending position of the candidate data file as the range; the starting position and the ending position of the candidate data file are determined by the data receiving end by parsing the candidate index file according to the current storage position of the candidate data.

[0043] Among them, the candidate index file refers to a file used to describe the storage location or structure of candidate data; it contains index information about the data file to help locate and access specific parts of the data file. Candidate data files refer to files that store candidate data; these files contain data that need to be further processed, and the data receiving end will extract the data to be transmitted from these files. The starting position and ending position of the candidate data file refer to the specific location range of the data in the file; these positions are determined by the data receiving end based on the parsing results of the candidate index file; the data receiving end can read the specific data part of the file based on the starting and ending positions of the candidate data file, thereby extracting the data to be transmitted.

[0044] Exemplarily, based on the index file and data file included in the calculation result file, metadata information of the calculation result file can be sent to the task scheduling end; the task scheduling end can be responsible for maintaining the metadata information uploaded by each data sending end. The calculation result file is transmitted to the target file system of the data receiving module, so that the corresponding data receiving end obtains the data in the calculation result file through the target file system. Data receiving

[0045] In another optional implementation, the target file system can name the calculation result files according to a predetermined naming rule, and store the calculation result files according to a predetermined directory structure; wherein the naming rule and the directory structure can be any pre-specified rule and standard structure. Generally, the above-mentioned target file system can organize files according to the above-mentioned naming rule and directory structure. The target file system can be any file system that is pre-specified and distinguished from the local disk of the Map end according to actual application requirements; the target file system can also be a file system formulated according to rules, such as a file system that can provide multiple copies; the predetermined naming rule can include the name of the calculation result file including the corresponding Map end identifier, and distinguishing the data file and index file of the above-mentioned calculation result file according to the suffix.

[0046] Exemplarily, the Map ends identified as "0" and "1" upload the calculation result files to the target file system after writing them to the local disk, and the file names correspond to the unique identifiers of the Map ends. For example, the index files and data files uploaded by the Map end identified as "0" stored in the target file system are named "0.index" and "0.data" respectively. The index files and data files uploaded by the Map end identified as "1" stored in the target file system are named "l.index" and "l.data" respectively.

[0047] It can be understood that by naming and storing the calculation result files at the data receiving end, the overhead caused by maintaining the file name information and the mapping relationship between the Map end and the file can be avoided, thereby facilitating the Reduce end to directly access the required files and data; the additional overhead caused by maintaining metadata information such as the file storage path can be avoided, thereby improving the query efficiency of the calculation result files in the target file system; moreover, the Reduce end can directly obtain the storage path of the calculation result file according to the required calculation result file, and then directly access the above-mentioned target file system without having to first obtain information such as the file path through other systems.

[0048] Furthermore, the data receiving end can parse the index file in the calculation result file according to the storage path in the metadata information, thereby obtaining the starting position and ending position of the data of the corresponding partition in the data file corresponding to the index file, and then read the data from the corresponding data file to obtain the target data.

[0049] Exemplarily, the data receiving device sends a data acquisition request for acquiring data of a partition corresponding to the Reduce end to at least one Reduce end according to the address of at least one Map end; the Reduce end can read data belonging to respective partitions from the Map end or the target file system in parallel by multiple Reduce ends; and the Reduce task is executed by the data receiving end to generate target data.

[0050] The embodiment of the present application obtains at least one data block by segmenting the received data set, and obtains key-value pair data by performing key-value pair processing on the data block through the data sending end; wherein the data sending end and the data block are in a one-to-one relationship; according to at least one key-value pair data, the calculation result data and the metadata information of the calculation result data are determined; the metadata information of the calculation result data is sent to the task scheduling device, and the calculation result data is sent to the data receiving device, so that the data receiving device obtains the metadata information from the task scheduling device, and determines the target data according to the metadata information and the calculation result data. The above technical scheme helps to improve the efficiency and reliability of data transmission by segmenting large data into several small data for parallel transmission.

[0051] Embodiment 2

[0052] Figure 2It is a flow chart of a data transmission method provided according to the second embodiment of the present application. Based on the technical solutions of the above embodiments, this embodiment refines "determining the calculation result data according to at least one key-value pair data" into "writing at least one key-value pair data into the memory buffer of the distributed storage end, and in the writing process, partitioning and sorting the key-value pair data to obtain the first write data; when it is identified that the data volume of the key-value pair data written in the distributed storage end reaches the data volume threshold, starting the overflow thread to store the unwritten key-value pair data to the local disk of the distributed storage end to obtain the second write data; integrating the first write data and the second write data to obtain the target write data, and writing the target write data to the local disk to obtain the calculation result data". It should be noted that for the parts not described in detail in the embodiments of the present application, please refer to the relevant statements of other embodiments. Figure 2 As shown, the method includes:

[0053] S210, segmenting the received data set to obtain at least one data block, and performing key-value pair processing on the data block through the data sending end to obtain key-value pair data; wherein the data sending end and the data block are in a one-to-one relationship.

[0054] S220. Write at least one key-value pair data into a memory buffer of a distributed storage end, and during the writing process, partition and sort the key-value pair data to obtain first written data.

[0055] Among them, the memory buffer refers to a part of the computer's memory that temporarily stores the data being processed; it can speed up data reading and writing operations and avoid frequent access to slower disk storage. Partition sorting refers to partitioning data according to certain specific rules and sorting the data within each partition; partition sorting can improve the efficiency of subsequent data processing, especially in distributed storage systems. First write data refers to the data that is first written to the memory buffer; at this stage, the data is only in a preliminary, unprocessed state.

[0056] Exemplarily, at least one key-value pair data is written into a memory buffer (eg, a circular memory buffer) of a distributed storage system; and partition sorting is performed during the writing process; and first written data is obtained.

[0057] S230: When it is identified that the amount of key-value pair data written in the distributed storage end reaches a data amount threshold, an overflow write thread is started to store the unwritten key-value pair data to the local disk of the distributed storage end to obtain second written data.

[0058] The overflow thread refers to a background thread that is responsible for handling data overflow operations when the buffer is full; that is, when the data in the memory buffer reaches a certain threshold, the overflow thread will write the data to the disk to free up memory space. The local disk refers to the hard disk of a computer or storage node, which is used to store data for a long time. Unlike memory, the disk storage speed is slower, but the capacity is larger, which is suitable for storing large amounts of data. The second write data refers to the key-value pair data that has not yet been put into the memory buffer.

[0059] Exemplarily, when the amount of data written to the key-value pair data reaches a data amount threshold, an overflow thread may be started to overflow the data in the memory buffer to the local disk, thereby generating a temporary file, namely, the second written data.

[0060] S240: Integrate the first write data and the second write data to obtain target write data, and write the target write data into a local disk to obtain calculation result data and metadata information of the calculation result data.

[0061] The target write data refers to the data that is ultimately written into the storage system after a series of processing (such as partition sorting, overflow writing, etc.); it is organized and optimized data, and is usually persistently stored on disk.

[0062] Optionally, the first write data and the second write data are partitioned and merged to obtain third write data; the third write data is partitioned and sorted to obtain target write data, and the target write data is written to a local disk to obtain calculation result data.

[0063] Partition merging refers to the process of merging multiple data partitions; it is usually used to merge multiple small data blocks into a larger data block, thereby improving data processing efficiency.

[0064] In an optional implementation, when it is identified that all key-value pair data are written to the distributed storage end and the data volume threshold is not reached, the first write data is directly written to the local disk to obtain calculation result data.

[0065] Exemplarily, when the key-value pair data corresponding to the data sending end have been written, it can be determined whether there is any second write data that overflows. If it is determined that there is no second write data, the data in the above-mentioned memory buffer can be directly written to the above-mentioned local disk, thereby generating a calculation result file; if it is determined that there is, all the above-mentioned second write data and the first write data in the memory buffer can be merged according to partitions, that is, the data with consistent partitions can be merged; then, the merged data can be sorted again according to the partitions, and the sorted data can be written to the local disk, thereby generating a calculation result file.

[0066] It should be noted that the partitions can be sorted specifically according to the partitions to which the hash values ​​of the keys of the key-value pair data belong; and the values ​​of the same key of the key-value pair can be combined.

[0067] S250, sending metadata information of the calculation result data to the task scheduling device, and sending the calculation result data to the data receiving device, so that the data receiving device obtains the metadata information from the task scheduling device, and determines the target data according to the metadata information and the calculation result data.

[0068] In an optional implementation, the data transmission method provided by the present application is implemented by a data transmission system based on MapReduce. First, the Map end executes the Map task to generate a calculation result file. Then, the Map end uploads the calculation result file to the target file system. In response to receiving the calculation result file, the target file system names the calculation result file according to a predetermined naming rule, and stores the calculation result file according to a predetermined directory structure. Next, the Reduce end obtains the address of at least one Map end corresponding to the Reduce end from the task scheduling end. According to the address of at least one Map end, the Reduce end sends a data acquisition request for obtaining data of the partition corresponding to the Reduce end to at least one Map end. Then, in response to determining that the acquisition of data from the Map end corresponding to the Reduce end fails, the Reduce end obtains the data of the partition corresponding to the Reduce end from the target file system that provides redundant storage. Finally, the Reduce end uses the acquired data to execute the Reduce task to generate the final result of the MapReduce process.

[0069] It can be understood that the MapReduce cluster transmission mode can support the sending node and the receiving node to form a one-to-many, many-to-one, and many-to-many form, avoiding the failure of the transmission task caused by the failure of a single node, and increasing the high availability of the transmission; based on the MapReduce framework, large files are divided into several small files, and the small files are transmitted in parallel, and there are multiple copies of small files, which improves the transmission efficiency, reduces the transmission time, and improves the reliability of file transmission; based on the MapReduce framework, large files are divided into several small files, and the small files have indexes and source data information, which can facilitate the downstream to query the required data before the small files are merged into large files, thereby improving the downstream call speed.

[0070] The embodiment of the present application obtains at least one data block by segmenting the received data set, and obtains key-value pair data by performing key-value pair processing on the data block through the data sending end; wherein the data sending end and the data block are in a one-to-one relationship; at least one key-value pair data is written into the memory buffer of the distributed storage end, and during the writing process, the key-value pair data is partitioned and sorted to obtain the first written data; when it is identified that the data volume of the key-value pair data written in the distributed storage end reaches the data volume threshold, the overflow thread is started to store the unwritten key-value pair data to the local disk of the distributed storage end to obtain the second written data; the first written data and the second written data are integrated to obtain the target written data, and the target written data is written to the local disk to obtain the calculation result data and the metadata information of the calculation result data; the metadata information of the calculation result data is sent to the task scheduling device, and the calculation result data is sent to the data receiving device, so that the data receiving device obtains the metadata information from the task scheduling device, and determines the target data according to the metadata information and the calculation result data. The above technical scheme helps to improve the efficiency and reliability of data transmission by dividing the big data into several small data for parallel transmission.

[0071] Embodiment 3

[0072] Figure 3 This is a structural diagram of a data transmission device provided according to the third embodiment of the present application, which can be applied to the case of large file cluster transmission. The data transmission device can be implemented in the form of hardware and / or software. The data transmission device can be configured in a computer device, such as a data transmission device in a data transmission system; the data transmission system also includes a task scheduling device and a data receiving device; the data transmission device and the data receiving device are connected in communication; the task scheduling device is connected in communication with the data transmission device and the data receiving device respectively; the data transmission device includes at least one data transmission terminal. Figure 3 As shown, the device comprises:

[0073] The first data determination module 310 is used to segment the received data set to obtain at least one data block, and perform key-value pair processing on the data block through the data sending end to obtain key-value pair data; wherein the data sending end and the data block are in a one-to-one relationship;

[0074] A second data determination module 320, configured to determine calculation result data and metadata information of the calculation result data according to at least one key-value pair data;

[0075] The data transmission module 330 is used to send metadata information of the calculation result data to the task scheduling device, and send the calculation result data to the data receiving device, so that the data receiving device obtains the metadata information from the task scheduling device and determines the target data based on the metadata information and the calculation result data.

[0076] The embodiment of the present application obtains at least one data block by segmenting the received data set, and obtains key-value pair data by performing key-value pair processing on the data block through the data sending end; wherein the data sending end and the data block are in a one-to-one relationship; according to at least one key-value pair data, the calculation result data and the metadata information of the calculation result data are determined; the metadata information of the calculation result data is sent to the task scheduling device, and the calculation result data is sent to the data receiving device, so that the data receiving device obtains the metadata information from the task scheduling device, and determines the target data according to the metadata information and the calculation result data. The above technical scheme helps to improve the efficiency and reliability of data transmission by segmenting large data into several small data for parallel transmission.

[0077] Optionally, the data sending device further includes a distributed storage terminal; accordingly, the second data determination module 320 includes:

[0078] A first data writing unit is used to write at least one key-value pair data into a memory buffer of a distributed storage end, and in the writing process, partition and sort the key-value pair data to obtain first written data;

[0079] A second data writing unit is used to start an overflow writing thread to store the unwritten key-value pair data to the local disk of the distributed storage end when it is identified that the data amount of the key-value pair data written in the distributed storage end reaches the data amount threshold, so as to obtain second written data;

[0080] The data integration unit is used to integrate the first write data and the second write data to obtain target write data, and write the target write data into a local disk to obtain calculation result data.

[0081] Optional, data integration unit, specifically used for:

[0082] Partition and merge the first write data and the second write data to obtain third write data;

[0083] The third write data is partitioned and sorted to obtain target write data, and the target write data is written to the local disk to obtain calculation result data.

[0084] Optionally, the second data determination module 320 is further used to directly write the first write data into the local disk to obtain calculation result data when it is identified that all the key-value pair data are written into the distributed storage end and the data volume threshold is not reached.

[0085] Optionally, the data receiving device includes at least one data receiving end; the target data is obtained by the data receiving device merging the data to be transmitted; the data to be transmitted is determined by the data receiving end based on the candidate data and the current storage position of the candidate data; the candidate data and the current storage position of the candidate data are determined by the data receiving end based on metadata information and calculation result data.

[0086] Optionally, the candidate data includes a candidate index file and a candidate data file; the data to be transmitted is obtained by the data receiving end by reading the candidate data file with the starting position and the ending position of the candidate data file as the range; the starting position and the ending position of the candidate data file are determined by the data receiving end by parsing the candidate index file according to the current storage position of the candidate data.

[0087] The data transmission device provided in the embodiments of the present application can execute the data transmission method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects for executing each data transmission method.

[0088] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0089] Embodiment 4

[0090] Figure 4 4 is a schematic diagram of the structure of an electronic device 410 that implements the data transmission method of an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present application described and / or required herein.

[0091] like Figure 4As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 to the random access memory (RAM) 413. In RAM413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, ROM412 and RAM413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0092] Multiple components in the electronic device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0093] The processor 411 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 411 performs the various methods and processes described above, such as a data transmission method.

[0094] In some embodiments, the data transmission method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the data transmission method described above may be performed. Alternatively, in other embodiments, the processor 411 may be configured as the data transmission method in any other appropriate manner (e.g., by means of firmware).

[0095] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0096] Computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data transmission device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0097] In the context of the present application, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device or equipment. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable signal medium. A more specific example of a machine readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0098] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0099] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0100] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0101] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document is not limited here.

[0102] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A data transmission method, characterized in that: A data sending device applied to a data transmission system, wherein the data transmission system further comprises a task scheduling device and a data receiving device; the data sending device is in communication connection with the data receiving device; The task scheduling device is communicatively connected with the data sending device and the data receiving device respectively; The data sending device includes at least one data sending terminal; the method includes: Segment the received data set to obtain at least one data block, and perform key-value pair processing on the data block through the data sending end to obtain key-value pair data; wherein the data sending end and the data block are in a one-to-one relationship; Determine calculation result data and metadata information of the calculation result data according to the at least one key-value pair data; The metadata information of the calculation result data is sent to the task scheduling device, and the calculation result data is sent to the data receiving device, so that the data receiving device obtains the metadata information from the task scheduling device and determines the target data according to the metadata information and the calculation result data.

2. The method according to claim 1, characterized in that The data sending device also includes a distributed storage end; accordingly, determining the calculation result data according to the at least one key-value pair data includes: Writing the at least one key-value pair data into the memory buffer of the distributed storage end, and in the writing process, partitioning and sorting the key-value pair data to obtain first written data; When it is identified that the amount of key-value pair data written in the distributed storage end reaches a data amount threshold, an overflow write thread is started to store the unwritten key-value pair data to the local disk of the distributed storage end to obtain second written data; The first write data and the second write data are integrated to obtain target write data, and the target write data is written to the local disk to obtain calculation result data.

3. The method according to claim 2, characterized in that Integrating the first write data and the second write data to obtain target write data, and writing the target write data to the local disk to obtain calculation result data, including: Partitioning and merging the first write data and the second write data to obtain third write data; The third write data is partitioned and sorted to obtain target write data, and the target write data is written to the local disk to obtain calculation result data.

4. The method according to claim 2, characterized in that: The method further comprises: When it is identified that all the key-value pair data are written into the distributed storage end and the data volume threshold is not reached, the first written data is directly written into the local disk to obtain calculation result data.

5. The method according to claim 1, characterized in that The data receiving device includes at least one data receiving end; the target data is obtained by the data receiving device by merging data to be transmitted; the data to be transmitted is determined by the data receiving end based on candidate data and the current storage position of the candidate data; the candidate data and the current storage position of the candidate data are determined by the data receiving end based on the metadata information and the calculation result data.

6. The method according to claim 5, characterized in that The candidate data includes a candidate index file and a candidate data file; the data to be transmitted is obtained by the data receiving end reading the candidate data file with the starting position and the ending position of the candidate data file as the range; the starting position and the ending position of the candidate data file are determined by the data receiving end by parsing the candidate index file according to the current storage position of the candidate data.

7. A data transmission device, characterized in that: A data sending device configured in a data transmission system, wherein the data transmission system further comprises a task scheduling device and a data receiving device; the data sending device is in communication connection with the data receiving device; The task scheduling device is communicatively connected with the data sending device and the data receiving device respectively; The data sending device comprises at least one data sending terminal; the apparatus comprises: A first data determination module is used to segment the received data set to obtain at least one data block, and perform key-value pair processing on the data block through the data sending end to obtain key-value pair data; wherein the data sending end and the data block are in a one-to-one relationship; A second data determination module, used to determine calculation result data and metadata information of the calculation result data according to the at least one key-value pair data; A data transmission module is used to send the metadata information of the calculation result data to the task scheduling device, and to send the calculation result data to the data receiving device, so that the data receiving device obtains the metadata information from the task scheduling device and determines the target data based on the metadata information and the calculation result data.

8. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data transmission method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data transmission method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the data transmission method according to any one of claims 1 to 6.

Images