Data transmission method, device, system, electronic device and storage medium

Abstract

The present application discloses a data transmission method, device, system, electronic device and storage medium, which relate to the technical field of data transmission, including obtaining original data in row format and converting the original data in row format into data in column format, wherein the data in column format have high similarity and are easy to compress, and then compress the data in column format and send it to a distributed message system, thereby solving the technical problem of low network bandwidth utilization caused by sending row format data, and achieving the technical effect of saving network bandwidth and improving data processing capability.

Description

Technical Field

[0001] The present application relates to the technical field of data transmission, and in particular to a data transmission method, device, system, electronic device and storage medium. Background Art

[0002] In related data transmission schemes, a distributed messaging system is usually used for data transmission. In order for the distributed messaging system to transmit structured data, the data format needs to be agreed upon when sending and receiving data.

[0003] In the scenario of related stream computing, data is sent in row format, and stream computing only cares about some fields in the data. Therefore, there is a large amount of redundant data when sending in row format, which easily causes a waste of network bandwidth and data processing capabilities. In addition, the similarity between the values ​​of different fields in the row format data is low, resulting in a low compression rate during data transmission. Summary of the invention

[0004] The present application provides a data transmission method, device, system, electronic device and storage medium to at least solve the problems of waste of network bandwidth and data processing capacity and low compression rate in sending data in row format in stream computing scenarios in related technologies.

[0005] A first aspect of the present application provides a data transmission method, comprising: Acquire a first preset number of original data, where the original data is row format data; Determine the primary key of each row format data from the original data; Based on the primary key of each row format data and the preset number of partitions, determine the partition number corresponding to each row format data; According to the partition number corresponding to each row format data, the row format data is distributed to the partition corresponding to the partition number, and the data in each partition is converted from the row format to the column format to obtain the first data; Selecting data corresponding to a field that meets the data subscription request of a downstream consumer from the first data to obtain second data; The second data is compressed using a preset compression algorithm and sent to the distributed message system.

[0006] A second aspect of the present application provides a data transmission method, comprising: Acquire second data from the distributed messaging system, where the second data is column format data and the second data is data corresponding to a field that meets a data subscription request from a downstream consumer; If the number of the second data is at least two, concatenation is performed on the second data to convert the column format data into row format data.

[0007] A third aspect of the present application provides a data transmission device, including: A first acquisition unit is used to acquire a first preset number of original data, where the original data is row format data; A first determining unit, used to determine the primary key of each row format data from the original data; A second determining unit, configured to determine a partition number corresponding to each row format data based on a primary key of each row format data and a preset number of partitions; a conversion unit, configured to distribute the row format data to partitions corresponding to the partition numbers according to the partition numbers corresponding to each row format data, and convert the data in each partition from the row format to the column format to obtain first data; A selection unit, configured to select data corresponding to a field that meets a downstream consumer's subscription data request from the first data to obtain second data; The sending unit is used to compress the second data using a preset compression algorithm and send the compressed data to the distributed message system.

[0008] A fourth aspect of the present application provides a data transmission device, including: A second acquisition unit is used to acquire second data from the distributed messaging system, where the second data is column format data and is data corresponding to a field that meets a data subscription request from a downstream consumer; The concatenation unit is used for concatenating the second data if there are at least two pieces of second data, so as to convert the column format data into row format data.

[0009] A fifth aspect of the present application provides a data transmission system, the data transmission system comprising a data sending device and a data receiving device, wherein: The data sending device is configured to perform the steps of any one of the data transmission methods in the first aspect above; The data receiving device is configured to execute the steps of any data transmission method in the second aspect above.

[0010] The sixth aspect of the present application provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any one of the above-mentioned data transmission methods when executing the computer program.

[0011] The seventh aspect of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned data transmission methods are implemented.

[0012] Through the present application, since the original data in row format is obtained and converted into data in column format, the similarity between the data in column format is high and easy to compress, and then the data in column format is compressed and sent to the distributed message system, the technical problem of low network bandwidth utilization caused by row format data sending can be solved, and the technical effect of saving network bandwidth and improving data processing capabilities can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0014] Figure 1 A flowchart of a data transmission method provided in an embodiment of the present application; Figure 2 A flowchart of a method for sending data provided in an embodiment of the present application; Figure 3 A flowchart of a method for receiving data provided in an embodiment of the present application; Figure 4 A schematic diagram of a flow chart of a method for calculating the number of partitions provided in an embodiment of the present application; Figure 5 A flowchart of another data transmission method provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of a data transmission system provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of a data transmission device provided in an embodiment of the present application; Figure 8 A schematic diagram of the structure of another data transmission device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0017] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0018] An embodiment of the present application provides a data transmission method, and the method is described in detail in conjunction with the execution flow of the data transmission method.

[0019] Distributed messaging system: is a distributed event stream platform, typically Kafka, which is widely used in high-performance real-time processing of data pipelines, stream analysis, data integration and mission-critical applications. Among them, the topic (Topic) is that the messages in Kafka are classified and stored in different topics, which represents a collection of messages, which usually have similar topics or uses; each topic can be divided into multiple partitions (Partitions), allowing data (messages) to be processed in parallel, improving throughput and fault tolerance; upstream producers (Producers) are client applications or services that publish messages to Kafka topics; downstream consumers (Consumers) are client applications or services that subscribe to and process messages from Kafka topics. Furthermore, multiple downstream message consumers (consumers) can consume data in multiple partitions at the same time. In order to ensure that data is not lost, each partition has multiple copies.

[0020] Stream computing: The main feature of stream computing is that the data stream has no beginning or end, which is different from batch computing which has a beginning and an end. That is, the amount of data in batch computing is limited, while stream computing can be performed continuously. In practical applications, the performance of stream computing can be measured based on throughput and latency.

[0021] Figure 1 A flowchart of a data transmission method provided by an embodiment of the present disclosure is provided.

[0022] like Figure 1 As shown, the method comprises the following steps: Step 101, obtaining a first preset number of original data, where the original data is row format data; In some embodiments, the row format data may include multiple fields. The present application does not limit the number of fields in the row format data, wherein a field is a collection of values ​​of the same column in the original data, that is, one field contains multiple rows of data.

[0023] In some embodiments, a first preset number of raw data may be obtained from a known data source, wherein the data source may be a database, or a file system or other storage system.

[0024] In some embodiments, since the present application is for operations with not too large a data volume and not too sensitive to delays, the first preset number needs to be reasonably selected. For example, if the first preset number is too small, it can be sent directly in row format.

[0025] Step 102, determining the primary key of each row format data from the original data; In some embodiments, as shown in Table 1, Table 1 is the row format data provided in an embodiment of the present application. It can be seen from Table 1 that there are three fields (name, tel, score) in the row format data, wherein the name field includes Peter, Paul, Mary and Kate, the tel field includes 1300888001, 1300888002, 1300888003 and 1300888004, and the score field includes 90, 95, 100 and 85.

[0026]

[0027] In some embodiments, as shown in Tables 2 to 4, they are column format data corresponding to the Name field, column format data corresponding to the Tel field, and column format data corresponding to the Score field, respectively. The first data includes column format data corresponding to the Name field, column format data corresponding to the Tel field, and column format data corresponding to the Score field.

[0028]

[0029] In some embodiments, the primary key is a key field that uniquely identifies each row format data. Taking the row format data in Table 1 as an example, it can be seen that the primary keys of each row format data from top to bottom are 1, 2, 3, and 4 respectively.

[0030] In some embodiments, other fields may be selected as the primary key according to actual needs.

[0031] Step 103, based on the primary key of each row format data and the preset partition quantity, determine the partition number corresponding to each row format data; In some embodiments, Figure 2The flowchart of the method for sending data provided in the embodiment of the present application specifically includes a step of presetting original data, a step of accumulating batches for execution, a step of extracting specified fields, a step of partitioning data, a step of sorting by primary key, a step of converting to column format, a step of adding a checksum, a step of finding the size of the longest field, a step of calculating the number of batch entries, and a step of updating the value of batch entries.

[0032] In some embodiments, Figure 2 In the step of data partitioning, the hash value of the primary key of each row format data can be used to perform a modulo operation on the preset partition number to obtain the partition number corresponding to the row format data. The modulo operation can ensure that the partition number is within the range of the partition number, wherein a partition includes multiple fields, and these fields all belong to the same row range in the original data, that is, the data corresponding to the fields belonging to the same row are in the same partition.

[0033] Step 104 , according to the partition number corresponding to each row format data, distribute the row format data to the partition corresponding to the partition number, and convert the data in each partition from the row format to the column format to obtain the first data.

[0034] In some embodiments, by converting the data in each partition from a row format to a column format, a basic condition can be provided for subsequent data compression and parallel transmission.

[0035] Step 105, selecting data corresponding to the field that meets the downstream consumer's subscription data request from the first data to obtain second data; In some embodiments, a downstream consumer refers to an application or service that subscribes to and consumes data in Kafka.

[0036] In some embodiments, Figure 2 In the step of extracting the specified field, each downstream consumer subscription data request may be the same or different, and the present application does not limit this. Specifically, as in the column format data of the aforementioned step 102 embodiment (column format data corresponding to the Name field, column format data corresponding to the Tel field, and column format data corresponding to the Score field), downstream consumer A may subscribe to the column format data corresponding to the Name field and the column format data corresponding to the Tel field, while downstream consumer B may subscribe to the column format data corresponding to the Tel field and the column format data corresponding to the Score field. For downstream consumer A, the second data includes the column format data corresponding to the Name field and the column format data corresponding to the Tel field, and for downstream consumer B, the second data includes the column format data corresponding to the Name field and the column format data corresponding to the Tel field.

[0037] Step 106: compress the second data using a preset compression algorithm and send the compressed data to the distributed messaging system.

[0038] In some embodiments, the second data may be compressed using a compression algorithm, and the compression algorithm may be a Snappy algorithm, a Gzip algorithm, etc. This application does not limit the specific compression algorithm used.

[0039] In some embodiments, the distributed messaging system in this application takes Kafka as an example, and other distributed messaging systems (such as the message queue system RabbitMQ) may also be used. This application does not limit the specific distributed messaging system used.

[0040] In some embodiments, all the second data in the column format may be sent to Kafka, or only part of the second data in the column format may be sent to Kafka.

[0041] In some embodiments, by compressing the second data using a preset compression algorithm and sending it to a distributed messaging system, column format data corresponding to a specified field can be sent according to the needs of downstream consumers, thereby reducing the redundancy of sent data and improving the utilization rate of network bandwidth.

[0042] Through the present application, since the original data in row format is obtained and converted into data in column format, the similarity between the data in column format is high and easy to compress, and then the data in column format is compressed and sent to the distributed message system, the technical problem of low network bandwidth utilization caused by row format data sending can be solved, and the technical effect of saving network bandwidth and improving data processing capabilities can be achieved.

[0043] In some embodiments, Figure 2 In the step of calculating the number of batches, obtaining the first preset number of raw data can be performed according to the following steps: Get preset raw data; In some embodiments, Figure 2 The preset original data in the data source is the initially determined original data, which can also be obtained from the aforementioned known data source.

[0044] Determine the number of batches of preset raw data, where the number of batches of preset raw data is used to indicate the number of row format data in the preset raw data; In some embodiments, if the preset original data is the row format data in the aforementioned step 102, for example, it can be determined that the batch number of the preset original data is 4 (4 row format data), and each row format data can be confirmed as one item.

[0045] Based on the preset number of batches of raw data and the preset longest field size in the raw data, the average length of the longest field value is determined. The longest field size is determined by the type of the field, and the field is a collection of data in the same column in the raw data. In some embodiments, the values ​​of different fields of the preset original data are of different sizes. Some field values ​​have large data volumes, while other field values ​​have smaller data volumes. The longest field size can be obtained from Kafka. Specifically, the longest field size is determined by the type of the field, and the field type and the length corresponding to the field type can be obtained in Kafka.

[0046] Determine the calculated value of the number of raw data batches based on the average length of the longest field value and the preset upper limit of the size of a single data item; In some embodiments, the preset upper limit of the size of a single data piece refers to the upper limit of the size of a single data piece (message) in Kafka. When the upper limit of the size of a single message is not less than the field size, a message contains only one field, that is, the field is directly sent as a single message; when the upper limit of the size of a single message is less than the field size, the field needs to be split into multiple messages and sent.

[0047] In some embodiments, the calculated value of the number of original data batches can be obtained by comparing the preset upper limit of the size of a single data item times a preset coefficient with the average length of the longest field value. Preferably, the preset coefficient can be determined to be 5 to avoid a field being split into too many messages, which will cause excessive delays in processing this batch of data.

[0048] Based on the calculated value of the number of batches of original data and the preset minimum number of batches of data, a first preset number is determined to obtain a first preset number of original data.

[0049] In some embodiments, Figure 2 In the step of updating the batch number value, by determining the larger value between the calculated value of the original data batch number and the preset data batch number minimum value as the first preset number, it is possible to avoid the calculated batch number being too small due to an unreasonable upper limit on the size of a single data piece.

[0050] In one embodiment, if Figure 2 In the step of batch execution, the preset original data can be batch processed according to the aforementioned update batch item values.

[0051] In some embodiments, the principle of calculating the number of batches of raw data is to balance the size of the batch data and the delay of batch processing. If the number of batch data is too small, the benefit of converting to column format is not high; if the number of batch data is too large, it will take a long time for the downstream to receive the batch of data in full, resulting in increased delay.

[0052] In some embodiments, Figure 2In the step of finding the longest field size, based on the preset number of raw data batches and the preset longest field size in the raw data, the average length of the longest field value can be determined according to the following steps: According to the preset batch data update cycle, extract a second preset number of data from the preset original data, and determine the maximum value of the field length in the second preset number of data as the longest field size in the preset original data; In some embodiments, the preset batch data update cycle is used to determine the cycle for updating the number of original data batches. If the preset batch data update cycle is too short, the number of original data batches is updated too quickly, resulting in excessively high costs. If the preset batch data update cycle is too long, the number of original data batches is updated too slowly, resulting in untimely data updates.

[0053] Compare the preset number of raw data batches with the longest field size to obtain the average length of the longest field value.

[0054] In some embodiments, the average length of the longest field value can be obtained by comparing the longest field size in the preset original data with the number of batches of the preset original data.

[0055] In a possible implementation, according to the partition number corresponding to each row-format data, the row-format data is distributed to the partition corresponding to the partition number, and the data in each partition is converted from the row format to the column format, and after the first data is obtained, the following steps are further included: For the column format data corresponding to each field in the first data, determine the primary key of the corresponding row format data; In some embodiments, by determining the primary key of the corresponding row format data for the column format data corresponding to each field in the first data, it can be determined which row format data in the original data the value corresponds to.

[0056] Sort the column format data in ascending order of the primary key of the row format data.

[0057] In some embodiments, Figure 2 In the step of sorting by primary key, on the basis that the column format data corresponding to each field in the first data has the primary key of the corresponding row format data, the column format data is sorted in ascending order of the primary key of the row format data, so that when splicing different column format data at the receiving end and judging whether the data is completely received, it is not necessary to scan all the data in the field each time, thereby speeding up the data processing speed.

[0058] In a possible implementation, according to the partition number corresponding to each row-format data, the row-format data is distributed to the partition corresponding to the partition number, and the data in each partition is converted from the row format to the column format, and after the first data is obtained, the following steps are further included: Generate a first identifier using a preset verification algorithm, where the first identifier is used to verify the integrity of the column format data corresponding to the field in each partition; Each field in each partition is verified using the first identifier to obtain data of the field containing the first identifier.

[0059] In some embodiments, Figure 2 In the step of adding a checksum, the MD5 algorithm or the SHA-256 algorithm can be used as a preset checksum algorithm in the application to generate the first identifier.

[0060] In some embodiments, the data in the field containing the first identifier is used to verify whether the first data is damaged or lost during transmission.

[0061] In some embodiments, the first identifier can be added at the end of the last field in the partition to obtain data of the field containing the first identifier, wherein the first identifiers added in different partitions can be the same or different, which is not limited in the present application.

[0062] In some embodiments, each column format data can be sent as a separate sub-topic. Specifically, since a topic represents a collection of a type of messages, the topic is used to transmit the fields corresponding to the column format data. This topic is called a sub-topic, that is, all fields in the same sub-topic are values ​​from the same column format data, and the sub-topics of the same group of original data jointly transmit the columns required for the original data. According to this transmission method, different fields (column format data) can be managed separately, and different partition numbers can also be configured for different sub-topics. Different column format data can be transmitted simultaneously, which can reduce data transmission delay.

[0063] In some embodiments, Figure 3 As shown, Figure 3 A flow chart of a method for receiving data provided in an embodiment of the present application specifically includes a step of screening data columns, a step of receiving data, a step of checking segments, a step of confirming that batch data is completely received, and a step of Joining and restoring a wide table.

[0064] In some embodiments, Figure 3 In the step of filtering data columns, multiple downstream consumers can flexibly select sub-topics to subscribe to according to business requirements, or dynamically change the sub-topics to be subscribed, thereby realizing dynamic column trimming of data.

[0065] In some embodiments, the naming method of the subtopic can adopt the "original topic name-BASE64 (column name)" naming specification, which can simultaneously meet the naming specifications of Kafka Topic and solve the problem of special characters in column names.

[0066] In some embodiments, by sending each column format data as a separate sub-topic, the length of the data field can be guaranteed not to exceed the upper limit of the size of a single data piece, thereby ensuring that while sending as much data as possible, the same field does not span multiple messages.

[0067] In some embodiments, different sub-topics can be configured with different numbers of partitions. The characteristics and distribution of data in different column formats are different. For example, the original data has two fields, the phone number and the province. Obviously, the phone number field has more changes and low similarity, while the province field has fewer types of values ​​(enumeration values) and higher similarity. According to the characteristics of data compression, the more similar the data is, the higher the compression rate is, and the smaller the data volume is after compression. In order to save network bandwidth, the sub-topic that transmits such fields does not need to use a high number of partitions for transmission.

[0068] In some embodiments, the specified number of partitions can be obtained based on the configuration calculation number of partitions and the automatic calculation number of partitions, such as Figure 4 As shown, Figure 4 The process of the partition number calculation method provided in the embodiment of the present application can implement the above-mentioned steps of obtaining the specified number of partitions, specifically, including a sampling step, a configuration reading step, a calculation step according to a formula, a step of specifying the number of partitions, and a step of splitting into sub-topics.

[0069] In one possible implementation, Figure 4 In the step of calculating according to the formula, according to the partition number corresponding to each row format data, the row format data is distributed to the partition corresponding to the partition number, and the data in each partition is converted from the row format to the column format. After obtaining the first data, the automatic calculation of the number of partitions can be performed according to the following steps: Obtaining a third preset number of column format data; In some embodiments, Figure 4 In the step of sampling, a third preset number of column format data can be obtained from the aforementioned first data, wherein the third preset number is not greater than the number of column format data in the first data.

[0070] Based on a third preset number of column format data, obtaining a first compression ratio of each column format data by using a preset compression algorithm; In some embodiments, a first compression ratio of each column format data of a third preset number of column format data is obtained.

[0071] If the difference between the first compression ratio of the third preset number of column format data and the average compression ratio is not less than the first threshold, nonlinearly scaling the first compression ratio of the column format data to obtain a second compression ratio corresponding to the column format data; In some embodiments, if the difference between the first compression ratio of the column format data and the average compression ratio is not less than the first threshold, it means that the first compression ratio of the column format data is a compression ratio that deviates relatively far from the average compression ratio.

[0072] In some embodiments, the first compression ratio of the column format data is subjected to nonlinear scaling, and the second compression ratio corresponding to the column format data is obtained, which can be expressed by the following expression:

[0073] in, is the second compression ratio value after scaling corresponding to the nth column data, The first compression ratio of the column format data corresponding to the field. The range of the first compression ratio is [0,1]. is the minimum value of the compression ratios corresponding to the third preset number of column format data, is the maximum value of the compression ratios corresponding to the third preset number of column format data, is the average value of the compression ratios corresponding to the third preset number of column format data, is a constant and tan is the trigonometric function tangent.

[0074] In some embodiments, by performing nonlinear scaling processing on the first compression ratio of the column format data, the deviation of the compression ratio can be further enlarged, thereby increasing the effect of the partition degree adjustment.

[0075] Based on the second compression rate, the preset maximum number of partitions, and the minimum number of partitions, the number of partitions of the subtopic corresponding to the field where the third preset number of column format data is located is determined.

[0076] In some embodiments, based on the second compression rate, the preset maximum number of partitions, and the minimum number of partitions, the number of partitions of the subtopic corresponding to the field where the third preset number of column format data is located can be determined using the following expression:

[0077] Among them, is the number of partitions of the sub-topic corresponding to the field where the n-th column format data is located, is the minimum parallelism allowed, is the maximum degree of parallelism allowed, is the second largest compression ratio after scaling, It is the minimum second compression ratio after scaling. Specifically, the minimum parallelism and maximum parallelism are determined by the configuration.

[0078] In some embodiments, the number of partitions calculated above is rounded to an integer to obtain a final number of partitions.

[0079] In some embodiments, Figure 4 In the step of reading the configuration, you can specify the number of partitions by configuring. Specifically, you can configure the sub-topics corresponding to each field separately according to business requirements. Generally speaking, dimension table fields have limited value changes and a high compression rate, so a small number of partitions can be used. However, fact table field values ​​change more, so a high number of partitions is recommended.

[0080] In one embodiment, if Figure 4 In the step of specifying the number of partitions, the topic can be split into sub-topics with the specified number of partitions according to the number of partitions configured in the above reading and the number of partitions calculated according to the formula (such as Figure 4 steps to split it into subtopics).

[0081] Figure 5 A flowchart of another data transmission method provided by an embodiment of the present disclosure.

[0082] like Figure 5 As shown, the method comprises the following steps: Step 501, obtaining second data from a distributed messaging system, where the second data is column format data and is data corresponding to a field that meets a downstream consumer's subscription data request; In some embodiments, Figure 3 In the step of filtering data columns, according to the subscription data request, when data in different column formats are distributed to the same topic, the column format data corresponding to the fields involved in the calculation can be filtered out.

[0083] In some embodiments, Figure 3 In the step of receiving data, according to the subscription data request, in the case where data in different column formats are distributed to different sub-topics, data of the fields corresponding to the subscription data request can be obtained by selectively subscribing to the sub-topics.

[0084] Step 502: If the second data is at least two, concatenate the second data to convert the column format data into row format data.

[0085] In some embodiments, Figure 3 In the step of restoring the wide table by Join, the data may be spliced ​​using a splicing Join operation to convert the column format data into row format data, wherein the Join operation is used to splice at least two row format data or column format data.

[0086] In some embodiments, for the case where different columns of data are distributed to different sub-topics, a multi-data stream Join operation can be used to speed up the Join process, where one stream corresponds to one sub-topic.

[0087] In one possible implementation, Figure 3 The steps of the verification section, after step 501, can be performed according to the following steps: If the integrity check of the second data fails, a retransmission request is sent, where the retransmission request is used to request retransmission of the second data that failed the integrity check; If the integrity check of the second data is successful, the original data is confirmed.

[0088] In some embodiments, the integrity check of the second data refers to the aforementioned data integrity check using the first identifier. If the check fails, it means that the data is damaged during the transmission process.

[0089] In some embodiments, if the integrity check of the second data is successful, the original data is then confirmed to ensure that no data is missed.

[0090] In some embodiments, Figure 3 The steps to confirm the complete receipt of batch data can be performed according to the following steps: In response to data in different column formats being distributed to the same topic, when the number of fields included in the second data is n, if each partition receives n fields and finds the maximum primary key, a confirmation message is sent to the distributed messaging system to confirm the original data; In some embodiments, data in different column formats are distributed to the same topic. Since there are multiple partitions in the same topic, the original data can only be considered to be completely received when each partition receives the complete original data. Therefore, it is necessary to confirm that each partition has received the complete original data before sending a confirmation message to the distributed messaging system to confirm the original data. Specifically, if all partitions receive the maximum primary key, it means that all segments of the same batch of data are completely received, and a confirmation message can be sent to the distributed messaging system to confirm the original data.

[0091] In some embodiments, data in the same partition will not be out of order, so there is no need to consider the situation where fields of the next batch of data are interspersed in the previous batch of data.

[0092] In response to data in different column formats being distributed to different sub-topics, a maximum primary key is found in each sub-topic, and a confirmation message is sent to the distributed message system to confirm the original data.

[0093] In some embodiments, different column format data are distributed to different sub-topics, and the median value of each sub-topic corresponds to a column format data in the original data. Therefore, it is only necessary to confirm whether the column format data corresponding to each sub-topic is completely received, and then a confirmation message can be sent to the distributed message system to confirm the original data. Specifically, the sending speed of the current column format data corresponding to each sub-topic may be different. Since the data has been sorted, as long as the maximum primary key in each sub-topic is found, it can be determined that the primary key and the data batch before it have been completely received.

[0094] In some embodiments, Figure 6 As shown, Figure 6 A flowchart of a data receiving method provided in an embodiment of the present application.

[0095] Figure 6 A schematic diagram of the structure of a data transmission system 600 provided in an embodiment of the present disclosure.

[0096] like Figure 6 As shown, the data transmission system 600 includes a data sending device 601 and a data receiving device 602 .

[0097] Furthermore, the data transmission system 600 also includes a distributed message system 603 and a partition number calculation device 604 .

[0098] The data sending device 601 is used to obtain a first preset number of original data, where the original data is row format data; determine the primary key of each row format data from the original data; determine the partition number corresponding to each row format data based on the primary key of each row format data and the preset partition number; distribute the row format data to the partition corresponding to the partition number according to the partition number corresponding to each row format data, and convert the data in each partition from row format to column format to obtain first data; select data corresponding to the field that meets the data subscription request of the downstream consumer from the first data to obtain second data; compress the second data using a preset compression algorithm and send it to the distributed message system 603; The data receiving device 602 is used to obtain second data from the distributed message system, where the second data is column format data and the second data is data corresponding to a field that meets the data subscription request of the downstream consumer; if the second data is at least two, the second data is concatenated to convert the column format data into row format data; The distributed message system 603 is used to receive the compressed second data sent by the data sending device 601, and send the compressed second data to the data receiving device 602; The partition number calculation means 604 is used to calculate the number of partitions.

[0099] In some embodiments, for operations with a relatively small amount of data and which are not very sensitive to delay, data sending and receiving can be implemented according to the data sending device 601, the data receiving device 602, and the distributed message system 603 in the above-mentioned data transmission system 600. However, for operations with a relatively large amount of data and which are more sensitive to delay, it is necessary to further calculate the number of partitions in combination with the partition number calculation device 604 in the above-mentioned data transmission system 600, so as to achieve simultaneous transmission of data in different column formats and reduce data transmission delay.

[0100] In some embodiments, based on the above-mentioned data transmission system 600, the number of partitions is calculated by the partition number calculation device 604, and multiple column format data are transmitted by multiple sub-topics. Downstream consumers can subscribe to sub-topics on demand, thereby achieving one-end sending and multiple different consumers using.

[0101] In some embodiments, the present application provides a method applied to a data transmission system, the method is performed by the data transmission system, the data transmission system includes a data sending device and a data receiving device, the method includes: The data sending device obtains a first preset number of original data, where the original data is row format data; The data sending device converts the original data from a row format to a column format to obtain first data; The data sending device compresses the first data and sends it to the distributed message system; The data receiving device obtains second data from the distributed message system, the second data is column format data, and the second data is data corresponding to a field that meets the downstream consumer's subscription data request; If the second data is at least two, the data receiving device performs concatenation processing on the second data to convert the column format data into row format data.

[0102] In some embodiments, the present application further provides a method applied to a data transmission system, the method is performed by the data transmission system, the data transmission system includes a data sending device and a data receiving device, the method includes: The data sending device obtains a first preset number of original data, where the original data is row format data; The data sending device determines the primary key of each row format data from the original data; Based on the primary key of each row format data and the preset partition quantity, the data sending device determines the partition number corresponding to each row format data; According to the partition number corresponding to each row format data, the data sending device distributes the row format data to the partitions corresponding to the partition number, and converts the data in each partition from the row format to the column format to obtain the first data; The data sending device selects data corresponding to the field that meets the downstream consumer's subscription data request from the first data to obtain the second data; The data sending device compresses the second data using a preset compression algorithm and sends the compressed data to the distributed message system; The data receiving device obtains second data from the distributed message system, the second data is column format data, and the second data is data corresponding to a field that meets the downstream consumer's subscription data request; If the second data is at least two, the data receiving device performs concatenation processing on the second data to convert the column format data into row format data.

[0103] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0104] The embodiment of the present application further provides a data transmission device 700, Figure 7 A schematic diagram of a data transmission device provided in an embodiment of the present disclosure is shown in FIG. Figure 7 As shown, including: A first acquisition unit 701 is used to acquire a first preset number of original data, where the original data is row format data; A first determining unit 702 is used to determine the primary key of each row format data from the original data; A second determining unit 703 is used to determine the partition number corresponding to each row format data based on the primary key of each row format data and the preset partition number; A conversion unit 704 is used to distribute the row format data to partitions corresponding to the partition numbers according to the partition numbers corresponding to each row format data, and convert the data in each partition from the row format to the column format to obtain first data; A selection unit 705 is used to select data corresponding to a field that meets the downstream consumer's subscription data request from the first data to obtain second data; The sending unit 706 is used to compress the second data using a preset compression algorithm and send the compressed data to the distributed message system.

[0105] Since the original data in row format is obtained and converted into data in column format, the similarity between the data in column format is high and it is easy to compress. The data in column format is then compressed and sent to the distributed message system. Therefore, the technical problem of low network bandwidth utilization caused by sending row format data can be solved, and the technical effect of saving network bandwidth and improving data processing capabilities is achieved.

[0106] Furthermore, in a possible implementation of the embodiment of the present disclosure, the first acquiring unit 701 is further configured to: Get preset raw data; Determine the number of batches of preset raw data, where the number of batches of preset raw data is used to indicate the number of row format data in the preset raw data; Based on the preset number of batches of raw data and the preset longest field size in the raw data, the average length of the longest field value is determined. The longest field size is determined by the type of the field, and the field is a collection of data in the same column in the raw data. Determine the calculated value of the number of raw data batches based on the average length of the longest field value and the preset upper limit of the size of a single data item; Based on the calculated value of the number of batches of original data and the preset minimum number of batches of data, a first preset number is determined to obtain a first preset number of original data.

[0107] Furthermore, in a possible implementation of the embodiment of the present disclosure, the first acquiring unit 701 is further configured to: According to the preset batch data update cycle, extract a second preset number of data from the preset original data, and determine the maximum value of the field length in the second preset number of data as the longest field size in the preset original data; Compare the preset number of raw data batches with the longest field size to obtain the average length of the longest field value.

[0108] Furthermore, in a possible implementation of the embodiment of the present disclosure, the data transmission device 700 further includes a sorting unit, which is specifically configured to: For the column format data corresponding to each field in the first data, determine the primary key of the corresponding row format data; Sort the column format data in ascending order of the primary key of the row format data.

[0109] Furthermore, in a possible implementation of the embodiment of the present disclosure, the data transmission device 700 further includes a verification unit, which is specifically configured to: Generate a first identifier using a preset verification algorithm, where the first identifier is used to verify the integrity of the column format data corresponding to the field in each partition; Each field in each partition is verified using the first identifier to obtain data of the field containing the first identifier.

[0110] Furthermore, in a possible implementation of the embodiment of the present disclosure, the data transmission device 700 further includes a partition number determination unit, and the partition number determination unit is specifically configured to: Obtaining a third preset number of column format data; Based on a third preset number of column format data, obtaining a first compression ratio of each column format data by using a preset compression algorithm; If the difference between the first compression ratio of the third preset number of column format data and the average compression ratio is not less than the first threshold, nonlinearly scaling the first compression ratio of the column format data to obtain a second compression ratio corresponding to the column format data; Based on the second compression rate, the preset maximum number of partitions, and the minimum number of partitions, the number of partitions of the subtopic corresponding to the field where the third preset number of column format data is located is determined.

[0111] The embodiment of the present application also provides another data transmission device 800, Figure 8 A structural diagram of another data transmission device provided in an embodiment of the present disclosure, such as Figure 8 As shown, including: The second acquisition unit 801 is used to acquire second data from the distributed messaging system, where the second data is column format data and is data corresponding to a field that meets the data subscription request of a downstream consumer; The concatenation unit 802 is configured to concatenate the second data if there are at least two pieces of second data, so as to convert the column format data into row format data.

[0112] Furthermore, in a possible implementation of the embodiment of the present disclosure, the data transmission device 800 further includes an integrity checking unit, which is specifically configured to: If the integrity check of the second data fails, a retransmission request is sent, where the retransmission request is used to request retransmission of the second data that failed the integrity check; If the integrity check of the second data is successful, the original data is confirmed.

[0113] Furthermore, in a possible implementation of the embodiment of the present disclosure, the integrity checking unit is further configured to: In response to data in different column formats being distributed to the same topic, when the number of fields included in the second data is n, if each partition receives n fields and finds the maximum primary key, a confirmation message is sent to the distributed messaging system to confirm the original data; In response to data in different column formats being distributed to different sub-topics, a maximum primary key is found in each sub-topic, and a confirmation message is sent to the distributed message system to confirm the original data.

[0114] For the description of the features in the embodiment corresponding to the data transmission device, reference can be made to the relevant description of the embodiment corresponding to the data transmission method, which will not be repeated here.

[0115] The embodiment of the present application further provides a data transmission system, the data transmission system includes a data sending device and a data receiving device, wherein: The data transmission device is configured to perform the above Figure 1 The steps in the illustrated embodiment; The data receiving device is configured to perform the above Figure 5 Steps in the illustrated embodiment.

[0116] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above data transmission method embodiments.

[0117] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data transmission method embodiments when running.

[0118] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0119] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above data transmission method embodiments are implemented.

[0120] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data transmission method embodiments are implemented.

[0121] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0122] The above is a detailed introduction to a data transmission method, device, system, electronic device and storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data transmission method, characterized in that: The method comprises: Acquire a first preset number of original data, where the original data is row format data; Determine a primary key of each row format data from the original data; Determine the partition number corresponding to each row format data based on the primary key of each row format data and the preset partition quantity; According to the partition number corresponding to each row format data, the row format data is distributed to the partitions corresponding to the partition number, and the data in each partition is converted from the row format to the column format to obtain the first data; Selecting data corresponding to a field that meets the downstream consumer's subscription data request from the first data to obtain second data; The second data is compressed using a preset compression algorithm and sent to the distributed messaging system.

2. The data transmission method according to claim 1, characterized in that: The obtaining of a first preset number of original data comprises: Get preset raw data; Determine the batch number of the preset original data, where the batch number of the preset original data is used to indicate the number of row format data in the preset original data; Determine the average length of the longest field value based on the number of batches of the preset original data and the longest field size in the preset original data, wherein the longest field size is determined by the type of the field, and the field is a collection of data in the same column in the original data; Determine the calculated value of the number of batches of original data based on the average length of the longest field value and a preset upper limit of the size of a single piece of data; Based on the calculated value of the number of batches of original data and a preset minimum number of batches of data, a first preset number is determined to obtain the first preset number of original data.

3. The data transmission method according to claim 2, characterized in that: The determining, based on the number of batches of the preset original data and the longest field size in the preset original data, the average length of the longest field value comprises: According to a preset batch data update cycle, extract a second preset number of data from the preset original data, and determine the maximum value of the field lengths in the second preset number of data as the longest field size in the preset original data; The number of batches of the preset original data is compared with the longest field size to obtain the average length of the longest field value.

4. The data transmission method according to claim 1, characterized in that: After the first data is obtained, the method includes: distributing the row format data to the partitions corresponding to the partition numbers according to the partition numbers corresponding to each row format data, and converting the data in each partition from the row format to the column format. For the column format data corresponding to each field in the first data, determine the primary key of the corresponding row format data; The column format data is sorted in ascending order of the primary keys of the row format data.

5. The data transmission method according to claim 1, characterized in that: After the first data is obtained, the method includes: distributing the row format data to the partitions corresponding to the partition numbers according to the partition numbers corresponding to each row format data, and converting the data in each partition from the row format to the column format. Generate a first identifier using a preset verification algorithm, where the first identifier is used to verify the integrity of the column format data corresponding to the field in each partition; Each field in each partition is verified using the first identifier to obtain data of the field containing the first identifier.

6. The data transmission method according to claim 1, characterized in that: After the first data is obtained, the method includes: distributing the row format data to the partitions corresponding to the partition numbers according to the partition numbers corresponding to each row format data, and converting the data in each partition from the row format to the column format. Obtaining a third preset number of column format data; Based on the third preset number of column format data, obtaining a first compression ratio of each column format data by using a preset compression algorithm; If the difference between the first compression ratio of the third preset number of column format data and the average compression ratio is not less than a first threshold, nonlinearly scaling the first compression ratio of the column format data to obtain a second compression ratio corresponding to the column format data; Based on the second compression rate, a preset maximum number of partitions, and a preset minimum number of partitions, the number of partitions of the subtopic corresponding to the field where the third preset number of column format data is located is determined.

7. A data transmission method, characterized in that: The method comprises: Acquire second data from the distributed messaging system, where the second data is column format data and the second data is data corresponding to a field that meets a data subscription request from a downstream consumer; If the number of the second data is at least two, concatenation processing is performed on the second data to convert the column format data into row format data.

8. The data transmission method according to claim 7, characterized in that: After obtaining the second data from the distributed message system, the method includes: If the integrity check of the second data fails, sending a retransmission request, wherein the retransmission request is used to request retransmission of the second data that failed the integrity check; If the integrity check of the second data is successful, the original data is confirmed.

9. The data transmission method according to claim 8, characterized in that: The confirmation of the original data includes: In response to data in different column formats being distributed to the same topic, when the number of fields included in the second data is n, if each partition receives n fields and finds the maximum primary key, a confirmation message is sent to the distributed messaging system to confirm the original data; In response to data in different column formats being distributed to different sub-topics, a maximum primary key is found in each sub-topic, and the confirmation message is sent to the distributed message system to confirm the original data.

10. A data transmission device, characterized in that: The device comprises: A first acquisition unit, used to acquire a first preset number of original data, wherein the original data is row format data; A first determining unit, configured to determine a primary key of each row format data from the original data; A second determining unit, configured to determine a partition number corresponding to each row format data based on a primary key of each row format data and a preset number of partitions; a conversion unit, configured to distribute the row format data to partitions corresponding to the partition numbers corresponding to each row format data, and convert the data in each partition from the row format to the column format to obtain first data; A selection unit, configured to select data corresponding to a field that meets a downstream consumer's subscription data request from the first data to obtain second data; The sending unit is used to compress the second data using a preset compression algorithm and send the compressed data to the distributed message system.

11. A data transmission device, characterized in that: The device comprises: A second acquisition unit is used to acquire second data from the distributed messaging system, where the second data is column format data and the second data is data corresponding to a field that meets the data subscription request of a downstream consumer; A concatenation unit is used for concatenating the second data if there are at least two of the second data, so as to convert the column format data into row format data.

12. A data transmission system, characterized in that: The data transmission system comprises a data sending device and a data receiving device, wherein: The data sending device is configured to perform the data transmission method according to any one of claims 1 to 6; The data receiving device is configured to execute the data transmission method according to any one of claims 7 to 9.

13. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the data transmission method described in any one of claims 1-6 or claims 7-9.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the data transmission method according to any one of claims 1-6 or claims 7-9.

Images