Data transmission method and data transmission system

Abstract

The application provides a data transmission method and a data transmission system. The application provides a data transmission method suitable for a server, which comprises the following steps: determining a target number of parallel channels according to an available network bandwidth of an industrial site, hardware resources of the server and transmission characteristics of target data; obtaining data blocks of the target number according to the target data; and distributing the data blocks of the target number to the parallel channels of the target number one by one, so that the parallel channels of the target number are used to transmit the data blocks of the target number in parallel. The application sets a channel number control strategy matched with the available network bandwidth of the industrial site, the hardware resources of the server and the transmission characteristics of the target data, so that the balance optimization of transmission efficiency and system resource occupation is realized.

Description

Technical Field

[0001] This application relates primarily to the field of communications, and more particularly to a data transmission method and a data transmission system. Background Technology

[0002] OPC UA (Open Platform Communications Unified Architecture) is an international standard communication protocol in the field of industrial automation. It is widely used in smart manufacturing, energy management and Internet of Things systems, aiming to achieve secure and reliable data exchange between devices.

[0003] However, the demand for transmitting massive amounts of sensor data (such as real-time monitoring streams and batch queries of historical data) has increased dramatically, and existing OPC UA implementations still suffer from problems such as low transmission efficiency, idle resources, or excessive resource consumption. Summary of the Invention

[0004] This application addresses the aforementioned technical problems by providing a data transmission method and a data transmission system that can improve transmission efficiency and reduce resource consumption.

[0005] To address the aforementioned technical problems, this application provides a data transmission method suitable for a server, comprising: determining a target number of parallel channels based on the available network bandwidth at the industrial site, the server's hardware resources, and the transmission characteristics of the target data; obtaining a target number of data blocks based on the target data; and allocating the target number of data blocks one-to-one to the target number of parallel channels, wherein the target number of parallel channels are used for parallel transmission of the target number of data blocks.

[0006] In one embodiment of this application, the server's hardware resources include the number of server CPU logical cores; the transmission characteristics of the target data include the refresh rate of the target data and the data size of the data block; determining the target number of parallel channels based on the available network bandwidth in the industrial site, the server's hardware resources, and the transmission characteristics of the target data includes: obtaining the available network bandwidth in the industrial site, the refresh rate of the target data, and the data size of the data block, wherein the data size of the data block is the average data size of the target number of data blocks; obtaining the bandwidth-based number of parallel channels based on the available network bandwidth in the industrial site, the refresh rate of the target data, and the data size of the data block; obtaining the maximum concurrent session count of the server, the number of server CPU logical cores, and the threshold value of the number of parallel channels; and taking the minimum value among the bandwidth-based number of parallel channels, the maximum concurrent session count, the number of server CPU logical cores, and the threshold value of the number of parallel channels as the target number of parallel channels.

[0007] In one embodiment of this application, the server's hardware resources include the number of server CPU logical cores; the transmission characteristics of the target data include the refresh rate of the target data and the data size of the data block; determining the target number of parallel channels based on the available network bandwidth in the industrial field, the server's hardware resources, and the transmission characteristics of the target data includes: obtaining the target number of parallel channels using the following formula: ,in, The target number of parallel channels. The available network bandwidth at the industrial site. The refresh rate of the target data. The data size of the data block. This represents the maximum number of concurrent sessions on the server. The number of logical CPU cores on the server side. Indicates to Round up.

[0008] In one embodiment of this application, the target data includes multiple first data associated with device identifiers. Obtaining a target number of data blocks based on the target data includes: grouping the multiple first data according to priority to obtain a target number of data groups; and obtaining the target number of data blocks based on the target number of data groups. The priority includes: a first priority: grouping multiple first data with the same device identifier and the same data type into the same group; a second priority: grouping multiple first data with different device identifiers but the same data type into the same group; and a third priority: grouping multiple first data with different device identifiers and different data types into the same group. The first priority is higher than the second priority, and the second priority is higher than the third priority.

[0009] In one embodiment of this application, the data transmission method further includes: in response to obtaining the channel identifier of the target parallel channel, re-obtaining the data group corresponding to the channel identifier of the target parallel channel based on the target data, wherein the target parallel channel is a parallel channel in the target number of parallel channels where the data block transmitted is abnormal; generating a target data block based on the re-obtained data group; and transmitting the target data block to the client through any parallel channel in the target number of parallel channels.

[0010] In one embodiment of this application, obtaining the target number of data blocks based on the target number of data groups includes performing the following steps for each data group in the target number of data groups: standardizing the data groups to obtain the target number of standardized data groups, wherein the data within the standardized data groups are set according to a preset field order and a preset field length; sequentially concatenating the data in the standardized data groups according to the data type identifier, the total length of the data corresponding to the data type identifier, and the order of the data corresponding to the data type identifier to obtain a data body; obtaining a first checksum corresponding to the data body; concatenating the data body with the first checksum corresponding to the data body to obtain a data channel stream; and compressing the data channel stream to obtain the data block.

[0011] In one embodiment of this application, the data type identifier includes a numeric identifier and a string identifier. Compressing the data channel stream to obtain the data block includes: compressing the data channel stream using a first compression strategy in response to the data type identifier of the data body in the data channel stream being the numeric identifier; and compressing the data body and a first checksum corresponding to the data body using a second compression strategy in response to the data type identifier of the data body in the data channel stream being the string identifier, wherein the second compression strategy is different from the first compression strategy.

[0012] This application also provides a data transmission method suitable for a client, comprising: acquiring a target number of data blocks, wherein the target number of data blocks are transmitted in parallel through a target number of parallel channels, wherein the target number of data blocks and the target number of parallel channels correspond one-to-one, and the target number is determined based on the available network bandwidth in the industrial site, the hardware resources of the server, and the transmission characteristics of the target data; and obtaining target data based on the target number of data blocks.

[0013] In one embodiment of this application, the data transmission method further includes: obtaining a data body in the data block, wherein the data body is obtained by sequentially concatenating data according to a data type identifier, the total length of data corresponding to the data type identifier, and the order of data corresponding to the data type identifier; calculating a second check value corresponding to the data body based on the data block; obtaining a first check value in the data block; in response to the first check value and the second check value being different, marking the parallel channel corresponding to the data block as a target parallel channel; returning the channel identifier of the target parallel channel to the server; and obtaining the target data block corresponding to the channel identifier of the target parallel channel.

[0014] In one embodiment of this application, obtaining target data based on the target number of data blocks includes: decompressing the target number of data blocks to obtain a target number of data channel streams, each data channel stream including a data body and a first checksum corresponding to the data body; obtaining the target number of standardized data groups based on the data bodies in the target number of data channel streams, according to the order of data type identifier, the total data length corresponding to the data type identifier, and the data corresponding to the data type identifier; obtaining the target number of data groups based on the target number of standardized data groups, according to a preset field order and a preset field length; and obtaining the target data based on the target number, the source device of the target data, and the data type.

[0015] In one embodiment of this application, the data type identifier includes a numeric identifier and a string identifier. Decompressing the target number of data blocks to obtain the target number of data channel streams includes: in response to the data type identifier of the data body being the numeric identifier, decompressing the data blocks using a first decompression strategy to obtain the data channel stream; and in response to the data type identifier of the data body including the string identifier, decompressing the data blocks using a second decompression strategy to obtain the data channel stream, wherein the second decompression strategy is different from the first decompression strategy.

[0016] This application also proposes a data transmission system, comprising: a server, configured to: determine a target number of parallel channels based on the available network bandwidth at the industrial site, the server's hardware resources, and the transmission characteristics of the target data; obtain a target number of data blocks based on the target data; and allocate the target number of data blocks one-to-one to the target number of parallel channels; the target number of parallel channels are configured to transmit the target number of data blocks in parallel to a client; and the client is configured to obtain the target number of data blocks and obtain the target data based on the target number of data blocks.

[0017] Compared with existing technologies, this application has the following advantages: By determining the target number of parallel channels based on the available network bandwidth, server hardware resources, and transmission characteristics of the target data in the industrial environment, this application enables reasonable allocation of transmission resources, avoiding idle or overloaded bandwidth or server hardware resources due to a fixed number of channels, thereby effectively improving resource utilization and preventing excessive resource consumption. By dividing the target data into a target number of data blocks and using the target number of parallel channels to achieve one-to-one parallel transmission of data blocks, the transmission time for large volumes of data can be shortened. Compared with existing technologies, this application achieves a balanced optimization of transmission efficiency and system resource usage by setting a channel number control strategy that matches the available network bandwidth, server hardware resources, and transmission characteristics of the target data in the industrial environment. Attached Figure Description

[0018] The accompanying drawings are included to provide a further understanding of this application; they are incorporated into and constitute a part of this application. The drawings illustrate embodiments of this application and, together with this specification, serve to explain the principles of this application. In the drawings: Figure 1 This is a block diagram of a data transmission system according to an embodiment of this application; Figure 2 This is a flowchart of a data transmission method applicable to a server according to an embodiment of this application; Figure 3 This is a flowchart illustrating a data transmission method according to an embodiment of this application, in which a target number of data blocks are obtained based on a target number of data groups; Figure 4 This is a block diagram illustrating the data transmission method of an embodiment of this application, in which data blocks are obtained from data groups. Figure 5 This is a flowchart of a data transmission method applicable to a client according to an embodiment of this application; Figure 6 This is a flowchart illustrating the process of obtaining target data based on a target number of data blocks in a data transmission method according to an embodiment of this application. Figure 7 This is a flowchart of data transmission in a data transmission system according to an embodiment of this application; Figure 8 This is a block diagram of a data transmission system according to another embodiment of this application. Detailed Implementation

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this application. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0020] Generally speaking, the terms "comprising" and "including" only indicate that the steps and elements that have been explicitly identified are included, and these steps and elements do not constitute an exclusive list. The method or apparatus may also include other steps or elements.

[0021] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0022] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, these terms have no special meaning and therefore should not be construed as limiting the scope of protection of this application. Moreover, this application should be understood not only through the actual terminology used, but also through the meaning implied by each term.

[0023] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more steps may be removed from these processes.

[0024] Existing OPC UA implementations exhibit significant bottlenecks in large-scale data transmission scenarios: 1) Low transmission efficiency: Traditional OPC UA employs a single-threaded mechanism, meaning a fixed number of data transmission channels and fixed-size data blocks. When processing large-scale data such as GB-level historical data queries, typical operating condition tests show that single-transmission latency is high, bandwidth utilization is low, and system response speed fails to meet industrial real-time requirements. 2) Excessive resource consumption: During data transmission, peak CPU and memory usage rates are high in batch transmission scenarios, easily leading to equipment overload and affecting the normal operation of other industrial control tasks. Therefore, an efficient and adaptive data transmission method and system are urgently needed to support the large-scale application of industrial big data scenarios.

[0025] To address the aforementioned technical problems, this application provides a data transmission method and a data transmission system. The data transmission method and system proposed in this application are applicable to, but not limited to, communication protocols such as OPC UA, User Datagram Protocol (UDP), and Transmission Control Protocol (TCP).

[0026] To facilitate understanding of the data transmission method of this application, this application first introduces the data transmission system of this application.

[0027] Figure 1 A block diagram of a data transmission system 10 according to an embodiment of this application is shown, including a server 11, parallel channels 12, and a client 13. The server 11 performs the following operations: determining the target number of parallel channels 12 based on the available network bandwidth in the industrial field, the hardware resources of the server 11, and the transmission characteristics of the target data; obtaining the target number of data blocks based on the target data; and allocating the target number of data blocks one-to-one to the target number of parallel channels 12. The parallel channels 12 are the target number, used to transmit the target number of data blocks in parallel to the client 13. The client 13 is used to obtain the target number of data blocks and obtain the target data based on the target number of data blocks.

[0028] The data transmission system 10 of this application determines the target number of parallel channels 12 based on the available network bandwidth in the industrial site, the hardware resources of the server 11, and the transmission characteristics of the target data. This enables the rational allocation of transmission resources and avoids the idleness or overload of bandwidth resources or server 11 hardware resources due to a fixed number of channels, thereby effectively improving resource utilization and avoiding excessive resource consumption. The server 11 divides the target data into a target number of data blocks and uses the target number of parallel channels 12 to achieve one-to-one parallel transmission of the data blocks to the client 13. This shortens the transmission time of large amounts of data between the server 11 and the client 13, achieving a balance between transmission efficiency and system resource usage.

[0029] The following section will further describe how the data transmission system 10 of this application achieves a balance between transmission efficiency and system resource consumption, in conjunction with the data transmission method of this application.

[0030] Figure 2 A flowchart illustrating a data transmission method applicable to server 11 according to an embodiment of this application is shown. Figure 2 As shown, the data transmission method 20 proposed in this application for server 11 includes: Step S21: Determine the target number of parallel channels based on the available network bandwidth at the industrial site, the hardware resources of the server, and the transmission characteristics of the target data.

[0031] This application does not impose specific restrictions on the format and content of the target data. In some embodiments, the target data is collected by server 11 at a certain collection frequency. After collecting business data, server 11 converts the collected data into an Industrial Data structure to adapt to diverse industrial data sources. Industrial Data is a custom extended data type that may include the following fields: device identifier (equipment ID), timestamp of collected data, data value, and data type. Industrial Data is used to ensure data standardization and enable time-series traceability of data. The data value can be the value of business parameters such as temperature and pressure. Data types include text and numeric types. Text types include strings, and numeric types include int, float, and Double. For example, the data value can be numeric, and the device identifier and timestamp can be text or numeric.

[0032] In some embodiments, the hardware resources of server 11 include the number of server CPU logical cores, and the transmission characteristics of the target data include the refresh rate of the target data and the data size of the data blocks. Determining the target number of parallel channels 12 based on the available network bandwidth in the industrial site, the hardware resources of server 11, and the transmission characteristics of the target data includes: obtaining the available network bandwidth in the industrial site, the refresh rate of the target data, and the data size of the data blocks, where the data size of the data blocks is the average data size of the target number of data blocks; obtaining the bandwidth-based number of parallel channels based on the available network bandwidth in the industrial site, the refresh rate of the target data, and the data size of the data blocks; obtaining thresholds for the maximum number of concurrent sessions, the number of server CPU logical cores, and the number of parallel channels in server 11; and taking the minimum value among the bandwidth-based number of parallel channels, the maximum number of concurrent sessions, the number of server CPU logical cores, and the thresholds for the number of parallel channels as the target number of parallel channels 12.

[0033] This application determines the number of bandwidth-based parallel channels by comprehensively considering the available network bandwidth in the industrial environment, the target data refresh rate, and the data volume of a single packet. This ensures that the number of channels matches the network transmission capacity, minimizes transmission latency, and avoids bandwidth contention due to too many channels or bandwidth waste due to too few channels, thereby achieving full utilization of network bandwidth resources. By incorporating the maximum number of concurrent sessions, the number of server-side CPU logical cores, and the threshold for the number of parallel channels into the constraints, and taking the minimum value among the four as the target number of parallel channels, the number of channels can be effectively limited. This prevents the non-linear growth of server-side security handshake overhead and session maintenance overhead caused by too many channels, ensuring that the number of channels simultaneously meets the constraints of network transmission capacity, connection resource limits, and data processing capacity limits. The solution in this application avoids the subjectivity and blindness in setting the number of parallel channels. For communication protocols, it is designed by considering the maximum number of concurrent sessions, such as the protocol characteristics of OPC UA protocols where each session or channel has an independent security context and session maintenance overhead. This allows for dynamic adaptation to different system loads and hardware configurations, improving transmission efficiency while ensuring controllable system resource consumption.

[0034] In some embodiments, step S21 includes: The target number of parallel channels is obtained using the following formula (1): (1) in, The target number of parallel channels. For the network bandwidth available in industrial sites, For the refresh rate of the target data, The amount of data in a data block. This represents the maximum number of concurrent sessions on the server. The number of logical CPU cores on the server side. Indicates to Round up.

[0035] In formula (1), the threshold value for the number of parallel channels is 16, which is the number of parallel channels based on bandwidth. In formula (1), 8 is the unit conversion factor, that is, 1 Byte = 8 bits.

[0036] This application does not impose specific restrictions on the method of obtaining the parameters of the parallel channel 12 mentioned above. In some embodiments, the available network bandwidth in the industrial field can be obtained directly from the monitoring of the industrial switch, or through heartbeat packet monitoring. The available network bandwidth in the industrial field is approximately equal to the length of a single heartbeat packet divided by the time difference between adjacent packets at the receiving end. When obtaining the available network bandwidth in the industrial field, the bandwidth occupied by other industrial protocols, such as the bandwidth occupied by the Modbus protocol, must also be deducted. The refresh rate of the target data can be configured according to business requirements. For example, temperature data can be configured to be collected every 100ms, meaning the refresh rate of the target data is once every 100ms. The data volume of a data block is the average of the number of data blocks calculated in real time to the target number. A data block includes a compressed protocol-encapsulated message and the processed data body of the target data. The maximum number of concurrent sessions on the server side is determined according to the communication protocol used for specific data transmission and can be a custom value of the communication protocol. The number of logical CPU cores on the server side can be obtained by reading the hardware parameters of the server 11. The threshold for the number of parallel channels is usually the industrial practice upper limit for the number of parallel channels used by the communication protocol for data transmission. By setting the threshold for the number of parallel channels, excessive parallelism during data transmission can be avoided. The industrial practice upper limit includes the number of parallel channels that can be established or have been established under the current communication protocol.

[0037] The boundary division rules for parallel channels in this application are determined according to the communication protocol. For example, in OPC UA, a parallel channel is a connection channel used for an independent secure session between server 11 and client 13.

[0038] Step S22: Obtain the target number of data blocks based on the target data.

[0039] In some embodiments, the target data includes multiple first data associated with device identifiers. Step S22 includes: grouping the multiple first data according to priority to obtain a target number of data groups; and obtaining a target number of data blocks according to the target number of data groups. The priorities include: a first priority: grouping multiple first data with the same device identifier and the same data type into the same group; a second priority: grouping multiple first data with different device identifiers but the same data type into the same group; and a third priority: grouping multiple first data with different device identifiers and different data types into the same group, wherein the first priority is higher than the second priority, and the second priority is higher than the third priority.

[0040] By grouping multiple pieces of primary data according to priority, prioritizing the grouping of multiple pieces of primary data with the same device identifier and data type into the same group, the data aggregation can be maximized. This allows the client 13 to quickly locate and integrate complete information about the same device after receiving data, such as quickly knowing "which device did what," reducing the complexity of data parsing and retrieval, and improving the response efficiency at the business level. Furthermore, prioritizing the grouping of multiple pieces of primary data with the same data type into the same group supports the subsequent use of differentiated compression strategies for different data types. Leveraging the similar structural characteristics of data of the same type, compression efficiency is effectively improved, and data transmission volume and storage overhead are reduced. Simultaneously, the three-level priority progressive grouping fully utilizes the transmission channel capacity while ensuring the above effects, with the target number of parallel channels 12 completing parallel transmission, achieving multiple optimizations in transmission efficiency, client 13 integration convenience, and compression performance.

[0041] In some embodiments, data can be grouped into fixed-size time windows or based on the value density of the data, such as prioritizing data with high rates of change.

[0042] Next, we will combine Figure 3 and Figure 4 This application describes how to obtain a target number of data blocks from a target number of data groups. Figure 3 A flowchart illustrating a data transmission method according to an embodiment of this application for obtaining a target number of data blocks based on a target number of data groups is shown. Figure 4 A block diagram is shown illustrating a data transmission method according to an embodiment of this application for obtaining data blocks based on data groups.

[0043] like Figure 3 and Figure 4 As shown, in some embodiments, obtaining a target number of data blocks based on a target number of data groups includes performing the following steps for each data group in the target number of data groups: Step S31: Standardize the data groups to obtain the target number of standardized data groups, wherein the data within each standardized data group is set according to the preset field order and preset field length.

[0044] To ensure that subsequent client 13 can decompress the data according to the same rules, the data within the data group is standardized to obtain a standardized data group. This application does not impose specific restrictions on the preset field order and preset field length in the standardization process. For numeric data, the preset field length can be set according to the number of bytes occupied by the data type; for example, data types int or float are generally 4 bytes, and data types Double are generally 8 bytes. For character data, the preset field length is set based on industrial experience, for example, 32 bytes. Data shorter than 32 bytes can be padded with 0s; for characters longer than 32 bytes, the fixed field only stores the character's index and length information. The standardized data group follows a fixed field order, such as "device identifier - acquisition timestamp - data value".

[0045] For example Figure 4 As shown, the IndustrialData type data group at box 401 is standardized to obtain a single standardized data group (numerical type) at box 402: device identifier (1 byte) + collection timestamp (4 bytes) + data value (4 bytes), and a single standardized data group (character type) at box 403: device identifier (1 byte) + collection timestamp (4 bytes) + data value (32 bytes).

[0046] Step S32: Concatenate the data in the standardized data group in the order of data type identifier, total length of data corresponding to data type identifier, and data corresponding to data type identifier to obtain the data body.

[0047] In some embodiments, standardized data of the same data type in a standardized data group are first concatenated, and then concatenated into a single continuous data body in the order of [Data Type Identifier (1 Byte)] [Data Length (2 Bytes)] [Continuous Data of the Same Type (N Bytes)]. The data type identifier is set as needed; for example, int can be set to 0x01, float to 0x02, and string to 0x03. Taking a standardized data group containing int, float, and string data as an example, the format of a single continuous data body can be set as: [0x01] [Int Data Length] [Int Data Content] [0x02] [Float Data Length] [Float Data Content] [0x03] [String Data Length] [String Data Content].

[0048] For example Figure 4 As shown, the standardized bar data groups in boxes 402 and 403 are spliced ​​together to obtain a single data body at box 404: data type identifier (1 Byte) + data length (2 Bytes) + continuous data of the same type (N Bytes).

[0049] Step S33: Obtain the first check value corresponding to the data body.

[0050] This application does not impose specific restrictions on the verification method. For example, hash algorithms such as CRC-32, MD5, or SHA-256 can be used to calculate the verification value of a single data body.

[0051] Step S34: Concatenate the data body with the first check value corresponding to the data body to obtain the data channel stream.

[0052] like Figure 4 As shown, the data body is concatenated with the first check value of the data body to form the data channel stream in box 405.

[0053] Step S35: Compress the data channel stream to obtain data blocks.

[0054] The adaptive compression engine of server 11 traverses each group of data, identifies the data channel stream type as numeric or string based on the data type identifier (1 Byte), and selects the corresponding compression strategy from the built-in compression model library.

[0055] In some embodiments, the data type identifier includes numeric identifiers and string identifiers. Step S35 includes: compressing the data channel stream using a first compression strategy in response to the data type identifier of the data body in the data channel stream being a numeric identifier; and compressing the data channel stream using a second compression strategy in response to the data type identifier of the data body in the data channel stream being a string identifier, wherein the second compression strategy is different from the first compression strategy. The first compression strategy includes the LZ4 compression strategy, and the second compression strategy includes the Zstandard compression strategy. The LZ4 compression strategy is used for numeric data channel streams, and the Zstandard compression strategy is used for string data and mixed-type data channel streams (including both string and numeric types). Each data channel stream is compressed independently. This application improves the compression ratio and processing speed by selecting an appropriate compression algorithm based on the data type.

[0056] In some embodiments, for numerical matrices of a specific format, the first compression strategy also includes DeltaEncoding combined with Zlib compression. In scenarios where bandwidth is extremely limited but real-time requirements are not extremely stringent, the first compression strategy also includes lossy compression algorithms, such as floating-point truncation with specified precision as a configurable option.

[0057] The byte stream formed after the data channel stream is compressed is converted into data blocks of type UA_ByteString, and data nodes are encapsulated according to the OPCUA message specification. Each data block corresponding to a data group is bound to an OPC UA data node.

[0058] For example Figure 4 As shown, the single data body at box 404 and the calculated first check value are concatenated to obtain the data channel stream at box 405: CRC-32 checksum (4 bytes) + single data body (N bytes). The CRC-32 checksum is the first check value. The data channel stream at box 405 is then encapsulated to obtain the OPC UA message at box 406, which includes data blocks obtained after compressing the data channel stream.

[0059] Step S23: Allocate the target number of data blocks one-to-one to the target number of parallel channels, which are used to transmit the target number of data blocks in parallel.

[0060] In some embodiments, the server 11 uses an OPC UA sending queue to distribute a target number of encapsulated messages to a target number of concurrent OPC UA connection channels, and initiates parallel transmission to the client 13 via a target number of parallel channels 12.

[0061] In some embodiments, the data transmission method 20 further includes: in response to obtaining the channel identifier of the target parallel channel 12, re-obtaining the data group corresponding to the channel identifier of the target parallel channel 12 based on the target data, wherein the target parallel channel 12 is the parallel channel 12 among the target number of parallel channels 12 where the transmitted data block is abnormal; generating a target data block based on the re-obtained data group; and transmitting the target data block to the client 13 through any of the target number of parallel channels 12. By re-obtaining the data group corresponding to the channel identifier of the target parallel channel 12 based on the target data, accurate retransmission is achieved. It should be understood that since there is a one-to-one correspondence between data blocks and parallel channels 12, abnormal data blocks can be identified through the channel identifier of the target parallel channel 12. When data transmission fails on a certain parallel channel 12, i.e., the target parallel channel 12, only the data block corresponding to the target parallel channel 12, i.e., the target data block, needs to be retransmitted, rather than initiating a full data retransmission process, which greatly improves the success rate of batch transmission and ensures data integrity.

[0062] In some embodiments, the target data block is generated using steps S31 to S34.

[0063] To address the problems of low transmission efficiency, idle resources, or excessive resource consumption in existing data transmission methods, this application also proposes a data transmission method suitable for client 13. Figure 5A flowchart illustrating a data transmission method applicable to client 13 according to an embodiment of this application is shown. Figure 5 As shown, the data transmission method 50 applicable to client 13 includes: Step S51: Obtain the target number of data blocks. The target number of data blocks are transmitted in parallel through the target number of parallel channels 12. The target number of data blocks and the target number of parallel channels 12 correspond one-to-one. The target number is determined based on the available network bandwidth in the industrial field, the hardware resources of the server, and the transmission characteristics of the target data.

[0064] The method for determining the target quantity based on the available network bandwidth in the industrial field, the server's hardware resources, and the transmission characteristics of the target data has been explained in the previous text and will not be repeated here.

[0065] In some embodiments, the client 13 caches the data blocks received through the parallel channel 12.

[0066] Step S52: Obtain the target data based on the target number of data blocks.

[0067] Figure 6 A flowchart illustrating the process of obtaining target data based on a target number of data blocks in a data transmission method according to an embodiment of this application is shown. Figure 6 As shown, in some embodiments, step S52 includes: Step S61: Decompress the target number of data blocks to obtain the target number of data channel streams, each of the data channel streams including a data body and a first check value corresponding to the data body.

[0068] In some embodiments, after receiving a message including data blocks, the client 13 identifies the data type of the data channel stream corresponding to the data blocks based on the data type identifier and executes the corresponding decompression strategy. In some embodiments, the data type identifier includes numeric identifiers and string identifiers. Decompressing a target number of data blocks to obtain a target number of data channel streams includes: in response to the data type identifier of the data body in the data channel stream being a numeric identifier, decompressing the data blocks using a first decompression strategy to obtain the data channel stream; and in response to the data type identifier of the data body in the data channel stream being a string identifier, decompressing the data blocks using a second decompression strategy to obtain the data channel stream, wherein the second decompression strategy is different from the first decompression strategy.

[0069] In some embodiments, the first decompression strategy includes the LZ4 decompression strategy, and the second decompression strategy includes the Zstandard decompression strategy. The LZ4 decompression strategy is used for numerical data channel streams, while the Zstandard decompression strategy is used for string data and mixed-type data channel streams (including both string and numerical data). Each data channel stream is compressed independently. This application improves the decompression ratio and processing speed by selecting an appropriate decompression algorithm based on the data type.

[0070] In some embodiments, numerical data channel streams are decompressed using the LZ4 algorithm; string and mixed-type data channel streams are decompressed using the Zstandard algorithm. After decompression, the original uncompressed data channel stream is restored, in the format: [CRC-32 checksum (4 bytes)] + [single data block (N bytes)]. The CRC-32 checksum is the first checksum value.

[0071] Step S62: Based on the data volume in the data channel stream of the target number, obtain the standardized data group of the target number in the order of data type identifier, total data length corresponding to the data type identifier, and data corresponding to the data type identifier.

[0072] In some embodiments, client 13 reverse-parses the data according to the assembly rules in step S32, disassembling each data unit segment by segment. The format is as follows: [Data type identifier (1 Byte)] + [Data length (2 Bytes)] + [Continuous data of the same type (N Bytes)]. The specific disassembly process can be as follows: First, read the first byte to determine the data type; read the next 2 bytes to obtain the total length of the data segment; truncate the corresponding number of bytes according to the length to obtain continuous data segments of the same type. Repeat the above disassembly process until the entire data unit is disassembled, obtaining a standardized data group.

[0073] Step S63: Based on the target number of standardized data sets, obtain the target number of data sets according to the preset field order and preset field length.

[0074] In some embodiments, data segments of each type are reverse-disassembled according to the standardization rules in step S31, and restored to a single piece of original industrial data, i.e., the first data, according to the field order of "device identifier-collection timestamp-data value". For numerical data, it is divided into segments according to a fixed byte length. For string data, if it is a string of no more than 32 bytes, it is directly truncated and the 0x00 padding at the end is removed to restore the original short string. If it is a long string of more than 32 bytes, the index and length information are parsed from the fixed fields, the corresponding independent extended data block is located according to the index, and the complete long string is restored by merging.

[0075] Step S64: Obtain the target data based on the target quantity, the source device of the target data, and the data type.

[0076] In some embodiments, the client 13 sorts all the disassembled and restored first data according to the device identifier and the collection timestamp, including reorganizing it according to the time order, source device and data type, and finally restores it to the original business data, i.e. the target data, which is completely consistent with the data sent by the server 11. The client 13 stores the data in the data storage unit.

[0077] In some embodiments, the data transmission method 50 further includes: obtaining a data body from a data block, wherein the data body is obtained by sequentially concatenating data according to a data type identifier, the total length of data corresponding to the data type identifier, and the order of data corresponding to the data type identifier; calculating a first checksum corresponding to the data body based on the data block; obtaining a second checksum from the data block; marking the parallel channel corresponding to the data block as a target parallel channel in response to a difference between the first checksum and the second checksum; returning the channel identifier of the target parallel channel to the server; and obtaining the target data block corresponding to the channel identifier of the target parallel channel. By comparing the first checksum and the second checksum, and re-obtaining the data group corresponding to the channel identifier of the target parallel channel 12 based on the target data when they are different, accurate retransmission is achieved.

[0078] In some embodiments, the data transmission method 20 applicable to the server 11 and the data transmission method 50 applicable to the client 13 described above can be used in combination. Figure 7 A flowchart of data transmission in a data transmission system according to an embodiment of this application is shown. Steps S701-S707 are executed by the server 11, step S708 is executed by the parallel channel 12, and steps S709-S713 are executed by the client 13. Figure 7 As shown, in some embodiments, the data transmission method 70 includes: Step S701: Collect target data; Step S702: Determine the target number of parallel transmission channels, which is equivalent to step S21; Step S703: Divide the first data into a target number of data groups according to priority; Step S704: Concatenate each data group into a data body, which is equivalent to steps S31-S33; Step S705: Determine the data type, that is, determine the data type in the data body according to the data type identifier. If it is a string or mixed type, proceed to step S706; if it is a numeric type, proceed to step S707. Step S706: Zstandard data compression strategy, that is, using the standard data compression strategy to compress the data volume; Step S707: LZ4 data compression strategy, that is, using the LZ4 data compression strategy to compress the data volume; wherein, steps S705-S707 are equivalent to step S34; Step S708: Parallel transmission of data blocks; Step S709: Receive the target number of data blocks, equivalent to step S51; Step S710: Data block parsing and verification, which is equivalent to steps S61-S64; Step S711: Verification passed, meaning the first and second verification values ​​are consistent; if so, proceed to step S713; otherwise, proceed to step S712. Step S712: Data retransmission, that is, returning the channel identifier of the target parallel channel to the server 11; Step S713: Store updated data, i.e., client 13 stores target data.

[0079] Figure 8 A block diagram of a data transmission system according to another embodiment of this application is shown, used to implement the above-described data transmission method 70. For example... Figure 8 As shown, in some embodiments, the server 11 includes a data acquisition and grouping module 111, an adaptive compression engine 112, a data mapping module 113, a sending queue 114, and an error recovery controller 115. The data acquisition and grouping module 111 is used to execute step S701; the adaptive compression engine 112 is used to execute steps S705-S707; the data mapping module 113 is used to encapsulate data blocks to obtain messages; the sending queue 114 is used to execute step S708; and the error recovery controller 115 is used to obtain the data group corresponding to the channel identifier of the target parallel channel based on the target data. The client 13 includes a receiving queue 131, a data decompression module 132, a data parsing module 133, a data verification module 134, and a data storage unit 135. The receiving queue 131 is used to execute step S709; data decompression 132 is used to decompress the received data blocks to obtain the data group and the first check value; data parsing module 133 is used to obtain the second check value, and data decompression 132 and data parsing module 133 are used to jointly execute step S710; data verification module 134 is used to execute steps S711 and S712; and data storage unit 135 is used to execute step S713.

[0080] Similarly, it should be noted that, in order to simplify the description of the present application and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of the embodiments of the present application sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this disclosure method does not imply that the subject matter of the present application requires more features than those mentioned. In fact, the embodiments contain fewer features than all the features of the single embodiments disclosed above.

[0081] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used to describe embodiments are sometimes modified by the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in this application are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of this application are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

Claims

1. A data transmission method suitable for a server, characterized in that, include: The target number of parallel channels is determined based on the available network bandwidth at the industrial site, the hardware resources of the server, and the transmission characteristics of the target data. Obtain the target number of data blocks based on the target data; as well as The target number of data blocks are allocated one-to-one to the target number of parallel channels, which are used to transmit the target number of data blocks in parallel.

2. The data transmission method as described in claim 1, characterized in that, The server's hardware resources include the number of server CPU logical cores; the transmission characteristics of the target data include the refresh rate of the target data and the data size of the data block. The step of determining the target number of parallel channels based on the available network bandwidth at the industrial site, the server's hardware resources, and the transmission characteristics of the target data includes: The available network bandwidth at the industrial site, the refresh rate of the target data, and the data size of the data block are obtained, wherein the data size of the data block is the average data size of the target number of data blocks; The number of parallel channels based on bandwidth is obtained according to the available network bandwidth at the industrial site, the refresh rate of the target data, and the data volume of the data block. Obtain the maximum number of concurrent sessions on the server, the number of CPU logical cores on the server, and the threshold for the number of parallel channels; and The minimum value among the bandwidth-based number of parallel channels, the maximum number of concurrent sessions, the number of server-side CPU logical cores, and the threshold for the number of parallel channels is taken as the target number of parallel channels.

3. The data transmission method as described in claim 1, characterized in that, The server's hardware resources include the number of server CPU logical cores; the transmission characteristics of the target data include the refresh rate of the target data and the data size of the data block. The step of determining the target number of parallel channels based on the available network bandwidth at the industrial site, the server's hardware resources, and the transmission characteristics of the target data includes: The target number of parallel channels is obtained using the following formula: ， in, The target number of parallel channels. The available network bandwidth at the industrial site. The refresh rate of the target data. The data size of the data block. This represents the maximum number of concurrent sessions on the server. The number of logical CPU cores on the server side. Indicates to Round up.

4. The data transmission method as described in claim 1, characterized in that, The target data includes multiple first data associated with device identifiers, and obtaining the target number of data blocks based on the target data includes: The first data is grouped according to priority to obtain a target number of data groups; and Obtain the target number of data blocks based on the target number of data groups; The priorities mentioned include: First priority: group multiple pieces of the first data with the same device identifier and the same data type into the same group; Second priority: group multiple pieces of the first data that have different device identifiers but the same data type into the same group; Third priority: group multiple pieces of the first data that have different device identifiers and different data type identifiers into the same group; The first priority is higher than the second priority, and the second priority is higher than the third priority.

5. The data transmission method as described in claim 4, characterized in that, Also includes: In response to obtaining the channel identifier of the target parallel channel, the data group corresponding to the channel identifier of the target parallel channel is obtained again based on the target data, wherein the target parallel channel is the parallel channel in which the data blocks transmitted in the target number of parallel channels are abnormal; Generate the target data block based on the reacquired data set; and The target data block is transmitted to the client through any of the target number of parallel channels.

6. The data transmission method as described in claim 4, characterized in that, Obtaining the target number of data blocks based on the target number of data groups includes performing the following steps for each data group of the target number of data groups: The data group is standardized to obtain the target number of standardized data groups, wherein the data within the standardized data group is set according to a preset field order and a preset field length. The data in the standardized data group are concatenated sequentially according to the data type identifier, the total length of the data corresponding to the data type identifier, and the order of the data corresponding to the data type identifier to obtain the data body; Obtain the first verification value corresponding to the data body; By concatenating the data body with the corresponding first checksum, a data channel stream is obtained; and The data channel stream is compressed to obtain the data block.

7. The data transmission method as described in claim 6, characterized in that, The data type identifier includes numeric identifiers and string identifiers, and compressing the data channel stream to obtain the data block includes: In response to the data type identifier of the data body in the data channel stream being the numeric identifier, the data channel stream is compressed using a first compression strategy; and In response to the data type identifier of the data body in the data channel stream being a string identifier, the data channel stream is compressed using a second compression strategy, wherein the second compression strategy is different from the first compression strategy.

8. A data transmission method suitable for a client, characterized in that, include: A target number of data blocks are obtained, and the target number of data blocks are transmitted in parallel through a target number of parallel channels. The target number of data blocks and the target number of parallel channels correspond one-to-one. The target number is determined based on the available network bandwidth in the industrial field, the hardware resources of the server, and the transmission characteristics of the target data. as well as The target data is obtained based on the target number of data blocks.

9. The data transmission method as described in claim 8, characterized in that, Also includes: The data body in the data block is obtained by sequentially concatenating the data body according to the data type identifier, the total length of the data corresponding to the data type identifier, and the order of the data corresponding to the data type identifier. The second verification value corresponding to the data body is calculated based on the data block; Obtain the first checksum from the data block; In response to the difference between the first check value and the second check value, the parallel channel corresponding to the data block is marked as the target parallel channel; Return the channel identifier of the target parallel channel to the server; as well as Obtain the target data block corresponding to the channel identifier of the target parallel channel.

10. The data transmission method as described in claim 8, characterized in that, The step of obtaining the target data based on the target number of data blocks includes: Decompress the target number of data blocks to obtain the target number of data channel streams, each of the data channel streams including a data body and a first check value corresponding to the data body; Based on the data body in the data channel stream of the target number, the standardized data group of the target number is obtained in the order of data type identifier, the total length of data corresponding to the data type identifier, and the data corresponding to the data type identifier. Based on the target number of standardized data sets, obtain the target number of data sets according to a preset field order and preset field length; and The target data is obtained based on the target quantity, the source device of the target data, and the data type.

11. The data transmission method as described in claim 10, characterized in that, The data type identifier includes numeric identifiers and string identifiers. Decompressing the target number of data blocks to obtain the target number of data channel streams includes: In response to the data type identifier of the data body in the data channel stream being the numeric identifier, the data block is decompressed using a first decompression strategy to obtain the data channel stream; In response to the data type identifier of the data body in the data channel stream including the string identifier, the data block is decompressed using a second decompression strategy to the data channel stream, wherein the second decompression strategy is different from the first decompression strategy.

12. A data transmission system, characterized in that, include: The server is used to: determine the target number of parallel channels based on the available network bandwidth at the industrial site, the server's hardware resources, and the transmission characteristics of the target data; Obtain the target number of data blocks based on the target data; And to allocate the target number of data blocks one-to-one to the target number of parallel channels; A target number of parallel channels are used to transmit the target number of data blocks to the client in parallel; A client is used to acquire the target number of data blocks and obtain the target data based on the target number of data blocks.

Images