Method, device, storage medium and program product for transmitting unstructured data based on data distribution service

Abstract

The present disclosure relates to the field of data transmission, and provides a transmission method, device, storage medium and program product of unstructured data based on a data distribution service. The transmission method comprises: a publishing end receiving a subscription message of a subscribing end, the publishing end and the subscribing end communicating based on a data distribution service; the publishing end virtually dividing target data corresponding to the subscription message to obtain division information of the target data; and the publishing end sending, to the subscribing end, a plurality of data shards obtained by dividing the target data and shard information of each data shard according to the division information, so as to publish the target data to the subscribing end. The present disclosure can solve the problem that the data distribution service cannot transmit unstructured data, improve the data transmission capability, and meet the transmission requirements of different types of data. In addition, the present disclosure can also dynamically determine the size of the data shard according to the network performance, so as to balance the transmission efficiency and reliability.

Description

Technical Field

[0001] This disclosure relates to the fields of data transmission technology and distributed communication technology, and in particular to a method, device, storage medium and program product for transmitting unstructured data based on data distribution services. Background Technology

[0002] Data Distribution Service (DDS) is a distributed real-time communication middleware standard developed by the Object Management Group (OMG). It uses a publish / subscribe model as its core model and topics as the basic unit of data transmission, providing efficient data transmission services for various application software in distributed systems. Because DDS employs a purely distributed architecture, it boasts high real-time performance and security, and has therefore been widely applied in various industrial IoT fields, including military, energy, healthcare, robotics, and autonomous driving.

[0003] In related technologies, DDS technology is mainly used for the transmission of small, structured data such as control commands, operating parameters, and device status. However, with the continuous upgrading of network resources, the demand for the transmission of large, unstructured files such as configuration files, models, documents, and executable files is becoming increasingly prominent. However, DDS technology only supports the transmission of structured data no larger than 1MB and cannot provide the capability to transmit unstructured data. Summary of the Invention

[0004] This disclosure provides a method, apparatus, storage medium, and program product for transmitting unstructured data based on a data distribution service, to at least solve the problem that DDS technology cannot transmit unstructured data. The technical solution of this disclosure is as follows: According to a first aspect of this disclosure, a method for transmitting unstructured data based on a data distribution service is provided. The method is applied to a publishing end, comprising: the publishing end receiving a subscription message from a subscribing end, wherein the publishing end and the subscribing end communicate based on the data distribution service; the publishing end virtually partitioning target data corresponding to the subscription message to obtain partitioning information for the target data, wherein the target data is unstructured data; and the publishing end sending multiple data fragments obtained from partitioning the target data and fragmenting information for each data fragment to the subscribing end according to the partitioning information, thereby publishing the target data to the subscribing end, wherein the fragmenting information indicates the position of the corresponding data fragment within the target data and the fragmenting length of the corresponding data fragment.

[0005] Optionally, the publishing end virtually partitions the target data to obtain the partitioning information in the following manner: the publishing end determines the fixed overhead of a single data fragment based on the network type used for communication with the subscription end, the transport protocol of the publishing end's transport layer, the real-time publish-subscribe protocol of the data distribution service, and the bytes occupied by recording the fragment information; the publishing end determines the fragment length of a single data fragment and the number of the multiple data fragments based on the total length of the target data and the fixed overhead, in order to obtain the partitioning information.

[0006] Optionally, the publishing end performs multiple virtual partitioning operations to virtually partition the target data to obtain the partitioning information. The virtual partitioning operation includes: determining the data fragment length for this virtual partitioning operation; performing virtual partitioning on the remaining data to be partitioned in the target data according to the data fragment length for this virtual partitioning operation to obtain the data fragment partitioning information for this virtual partitioning operation, and determining the remaining data to be partitioned in the target data for the next time. The publishing end determines the data fragment length for this virtual partitioning operation in the following ways: in response to the absence of at least one performance statistics cycle before this virtual partitioning operation, a preset initial size is determined as the data fragment length for this virtual partitioning operation; in response to the absence of at least one performance statistics cycle before this virtual partitioning operation, the data fragment length for this virtual partitioning operation is determined based on the system performance indicators in the most recent performance statistics cycle before this virtual partitioning operation and the data fragment length of the previous virtual partitioning operation. The system performance indicators characterize the network performance during the process of the publishing end sending the target data to the subscribing end.

[0007] Optionally, the publishing end determines the system performance indicators for the most recent performance statistical period in the following manner: determining multiple network indicators for the most recent performance statistical period, wherein the multiple network indicators include positive indicators and negative indicators, wherein positive indicators refer to network parameters that are positively correlated with network performance, and negative indicators refer to network parameters that are negatively correlated with network performance; for each network indicator, determining the weight corresponding to the network indicator based on the information entropy of the network indicator in the most recent performance statistical period; and determining the system performance indicators based on each network indicator and the weights corresponding to each network indicator.

[0008] Optionally, the publishing end, based on the partitioning information, performs a fragmentation sending operation for each of the plurality of data fragments, sending the plurality of data fragments and the fragmentation information of each data fragment to the subscribing end. For each data fragment, the fragmentation sending operation includes: determining the fragmentation information of the currently to-be-sent data fragment based on the fragmentation information of the previously sent data fragment, wherein the fragmentation information of the currently to-be-sent data fragment is initial information when the data fragment is first sent; partitioning the currently to-be-sent data fragment from the unsent data in the target data according to the partitioning information; encapsulating the fragmentation information of the currently to-be-sent data fragment and the currently to-be-sent data fragment into a current message of the real-time sending subscription protocol of the data distribution service; and sending the plurality of data fragments and the fragmentation information of each data fragment to the subscribing end by sending the current message to the subscribing end.

[0009] According to a second aspect of this disclosure, a method for transmitting unstructured data based on a data distribution service is provided. The method is applied to a subscriber, and includes: the subscriber sending a subscription message to a publisher, wherein the publisher and the subscriber communicate based on the data distribution service; the subscriber receiving from the publisher a plurality of data fragments obtained from target data corresponding to the subscription message, and fragmentation information for each data fragment, to obtain the target data, wherein the plurality of data fragments are obtained by the publisher through partitioning information obtained by virtually partitioning the target data, wherein the target data is unstructured data, and the fragmentation information indicates the position of the corresponding data fragment in the target data and the fragmentation length of the corresponding data fragment.

[0010] Optionally, the subscriber obtains the target data in the following manner: in response to receiving a current message of the real-time publish-subscribe protocol of the data distribution service from the publisher, it parses the current data fragment and its fragment information from the current message; based on the fragment information of the current data fragment, it concatenates the current data fragment to the previously received data fragment to obtain concatenated data, wherein, when a data fragment is received for the first time, the current data fragment is concatenated to the initial position; it updates the fragment information of the current data fragment to the configuration file used for the target data; in response to the current data fragment being the last data fragment in the target data, it obtains the target data based on the current concatenated data and the current configuration file.

[0011] Optionally, the partitioning information indicates the number of the plurality of data shards, and the position of each data shard in the target data is represented by the index value of the corresponding data shard in the sequence of the plurality of data shards. The transmission method further includes: the subscriber receiving the partitioning information from the publisher; the subscriber determining that the current data shard is the last data shard in the target data in response to the partitioning information indicating that the index value of the current data shard is equal to the number of the plurality of data shards indicated by the partitioning information.

[0012] Optionally, the partitioning information includes first verification information, wherein the transmission method further includes: the subscriber receiving the partitioning information from the publisher; the subscriber generating second verification information based on the obtained target data; and the subscriber verifying the integrity of the target data by comparing the first verification information and the second verification information.

[0013] According to a third aspect of this disclosure, an electronic device is provided, the electronic device comprising: a processor; and a memory for storing processor-executable instructions, wherein, when executed by the processor, the processor causes the processor to perform a method for transmitting unstructured data based on a data distribution service according to this disclosure.

[0014] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform a method for transmitting unstructured data based on a data distribution service according to this disclosure.

[0015] According to a fifth aspect of this disclosure, a computer program product is provided, including computer-executable instructions that, when executed by at least one processor, implement the method for transmitting unstructured data based on a data distribution service as described in this disclosure.

[0016] The technical solution provided in this disclosure brings at least the following beneficial effects: According to the data transmission scheme based on data distribution service disclosed herein, unstructured target data can be virtually partitioned at the publishing end to obtain partitioning information. Based on the partitioning information, multiple data fragments are divided from the target data, and each data fragment and its fragment information are sent to the subscription end to publish the target data to the subscription end. In this way, the problem that traditional DDS technology can only support the transmission of small structured data and cannot meet the needs of unstructured data transmission can be solved, thereby improving the data transmission capability of DDS technology and meeting the transmission needs of different types of data.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0019] Figure 1 This is a schematic flowchart of an example of a method for transmitting unstructured data according to an exemplary embodiment of the present disclosure.

[0020] Figure 2 This is a schematic diagram illustrating the data transmission process between a publisher and a subscriber in an unstructured data transmission method according to an exemplary embodiment of the present disclosure.

[0021] Figure 3 This is a schematic diagram illustrating the process of obtaining segmented information at the publishing end in an unstructured data transmission method according to an exemplary embodiment of the present disclosure.

[0022] Figure 4 This is a schematic flowchart of the fragmented transmission operation in an unstructured data transmission method according to an exemplary embodiment of the present disclosure.

[0023] Figure 5 This is a schematic diagram illustrating the principle of data fragmentation and data concatenation in an exemplary embodiment of the present disclosure for transmitting unstructured data.

[0024] Figure 6 This is a schematic diagram of the RTPS message structure in an exemplary embodiment of the present disclosure for transmitting unstructured data.

[0025] Figure 7 This is a flowchart illustrating another example of a method for transmitting unstructured data according to exemplary embodiments of the present disclosure.

[0026] Figure 8 This is a schematic diagram illustrating the process of obtaining target data by a subscriber in an unstructured data transmission method according to an exemplary embodiment of the present disclosure.

[0027] Figure 9 This is a block diagram of an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0028] In order to enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0029] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0030] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0031] As mentioned earlier, among related technologies, DDS technology only supports structured data transmission of no more than 1MB and cannot provide the ability to transmit unstructured data.

[0032] Specifically, in early IoT systems, due to limitations in terminal and network capabilities, the focus was primarily on the real-time transmission of small, structured data, such as control commands, operating parameters, and device status. With the continuous development of intelligent systems, the fundamental resources of IoT systems are constantly being upgraded, and their functions are expanding. The need for transmitting large, unstructured files such as configuration files, models, documents, and executable files is becoming increasingly prominent. However, traditional DDS technology can only support structured data transmission, typically no larger than 1MB, and does not provide file transfer capabilities. Applications need to use DDS technology for real-time data transmission while simultaneously employing other technologies such as File Transfer Protocols (FTP) for file transfer. This dual transmission mechanism significantly increases the burden of system development and maintenance.

[0033] In view of the foregoing, exemplary embodiments of this disclosure provide a method for transmitting unstructured data based on a data distribution service, an electronic device, a computer-readable storage medium, and a computer program product that can solve or at least alleviate the aforementioned problems.

[0034] In a first aspect of an exemplary embodiment of this disclosure, a method for transmitting unstructured data based on a data distribution service is provided.

[0035] The method for transmitting unstructured data based on a data distribution service according to an exemplary embodiment of this disclosure can be applied to a publishing end, which may have a data distribution service loaded on it.

[0036] Here, the publishing end can be any device in an IoT system, such as a tablet, laptop, digital assistant, or wearable device. The publishing end can communicate with the subscribers based on a data distribution service to receive subscription messages from the subscribers, and can determine the data to be published based on the subscription messages and publish that data to the subscribers. In this paper, the concepts of publishing end and subscriber are relative. In some cases, the publishing end can also act as a subscriber to send subscription messages to other publishing ends, and the subscriber can also act as a publishing end to publish data to other subscribers.

[0037] The method for transmitting unstructured data according to the first aspect of the exemplary embodiments of this disclosure can solve the problem that traditional DDS technology can only support the transmission of small-scale structured data and cannot meet the needs of unstructured data transmission, thereby improving the data transmission capability of DDS technology and meeting the transmission needs of different types of data.

[0038] The following will refer to Figures 1 to 6 An example of a method for transmitting unstructured data based on a data distribution service according to embodiments of the present disclosure is described.

[0039] like Figure 1 As shown, the transmission method may include the following steps: In step S110, the publisher can receive subscription messages from the subscriber.

[0040] For example, such as Figure 2 As shown, when creating a DDS system, DDS initialization can be performed on the publisher side to create file publishing / subscription topics on the publisher and subscriber sides. This can include, but is not limited to, at least one of the following: initializing DDS middleware, calling the application programming interface (API) provided by DDS, creating domain participants, creating file transfer-related publishers / subscribers, creating topics, and creating file transfer-related data writers / data readers.

[0041] Here, the publisher and subscriber can communicate based on a Data Distribution Service (DDS). As an example, the publisher and subscriber can each load a Data Distribution Service module to implement DDS. The subscriber's Data Distribution Service module can send subscription messages to the publisher's Data Distribution Service module, and the publisher's Data Distribution Service module can publish data to the subscriber's Data Distribution Service module based on the subscription messages.

[0042] In step S110, the publisher can obtain the subscription message from the subscriber based on the data distribution service. The subscription message can indicate the data that the subscriber needs to obtain from the publisher.

[0043] In step S120, the publishing end can virtually divide the target data corresponding to the subscribed message to obtain the division information of the target data, wherein the target data is unstructured data.

[0044] In this step, the publisher can read target data, such as a file to be sent, based on the subscribed message, and can preprocess the target data to initialize data transmission.

[0045] Here, virtual partitioning can refer to the publishing end determining the partitioning method for the target data, such as determining the total number of data fragments (e.g., file fragments) and the size of each data fragment, or dynamically partitioning the data fragments during data transmission without actually dividing the target data, thus keeping the target data unchanged.

[0046] As an example, such as Figure 3 As shown, the publishing end can virtually partition the target data to obtain partitioning information in the following ways: In step S310, the publisher can determine the fixed overhead of a single data fragment based on the network type used to communicate with the subscriber, the transport protocol of the publisher's transport layer, the real-time publish-subscribe protocol of the data distribution service, and the bytes occupied by the fragment information.

[0047] As an example, the fixed overhead of a single data fragment can be determined based on the maximum transmission unit (MTU) of the network type used to communicate with the subscriber, the bytes occupied by the header of the transport protocol of the publisher's transport layer, the bytes occupied by the header of the real-time publish-subscribe protocol of the data distribution service, and the bytes occupied by the record fragment information.

[0048] For example, the network type used for communication between the publisher and subscriber can be Ethernet, and the Ethernet MTU can be 1500 bytes. The reserved IP header of the publisher's transport layer can occupy 20 bytes, and the transport protocol of the publisher's transport layer can be, for example, User Datagram Protocol (UDP), whose header occupies 8 bytes. The header of the Real-Time Publish-Subscribe Protocol (RTPS) of the data distribution service can occupy 24 bytes; the record fragment information can occupy 4 bytes.

[0049] In this step, the fixed overhead of each data fragment can be determined based on the fixed number of bytes required for the transmission described above.

[0050] In step S320, the publishing end can determine the fragment length of a single data fragment and the number of multiple data fragments based on the total length of the target data and the fixed overhead, in order to obtain the partitioning information.

[0051] Specifically, the publishing end can determine the optimal fragment length (or fragment size) of a single data fragment based on a fixed overhead. For example, the optimal fragment length can be determined by considering a preset margin based on the fixed overhead. For example, the fragment length of a single data fragment can be 1400 bytes.

[0052] The publisher can read the target data to be sent to determine the total length of the target data. For example, it can read the target data corresponding to the subscribed message based on the file path and name set by the user (such as "D:\files\Model.dat"). The publisher can also virtually split the file according to the split length of a single data fragment based on the total length of the target data, calculate the total number of fragments, and write the total number of fragments and the size of each fragment into the partitioning information.

[0053] In this way, the publishing end can virtually fragment the file and determine the partitioning information of the target data for use in the subsequent actual partitioning operation, thereby reducing the memory space occupation caused by pre-fragmentation and realizing real-time data partitioning and transmission.

[0054] As another example, the publishing end can perform multiple virtual partitioning operations to virtually partition the target data and obtain partitioning information. The virtual partitioning operation includes: determining the data fragment length of this virtual partitioning operation; performing virtual partitioning on the remaining data to be partitioned in the target data according to the data fragment length of this virtual partitioning operation, obtaining the partitioning information of the data fragments of this virtual partitioning operation, and determining the remaining data to be partitioned in the target data next time.

[0055] As an example, when performing a virtual partitioning operation for the first time, the remaining data to be partitioned can be the target data, or it can be data to be partitioned that is specified in other ways.

[0056] In this example, the publishing end can determine the data fragment length for this virtual partitioning operation in the following way: Since no performance statistics cycle has been performed before this virtual partitioning operation, the preset initial size is determined as the data fragment length for this virtual partitioning operation; In response to having experienced at least one performance statistics cycle prior to this virtual partitioning operation, the data fragment length for this virtual partitioning operation is determined based on the system performance metrics in the most recent performance statistics cycle prior to this virtual partitioning operation and the data fragment length of the previous virtual partitioning operation.

[0057] Here, system performance metrics can be used to characterize network performance during the process of sending target data from the publisher to the subscriber.

[0058] Specifically, the performance statistics period can be a preset time period, which can be a fixed time interval, such as 60 seconds. Each preset time period can evaluate the network performance of data transmission from the publisher to the subscriber within that period.

[0059] For example, the publishing end can first determine the initial size of the data shards (or the size of the data blocks). X 0 The target data is sliced ​​and transmission begins. During transmission, the publishing end can record network performance data for each performance statistics period T using a sliding window. Here, the initial size... X 0 The value can be set based on historical data or experience; for example, it can be 1KB.

[0060] During the first performance statistics cycle, if a virtual partitioning operation is performed, the data fragment length X remains unchanged and is still the initial size because a complete performance statistics cycle has not yet been completed. In subsequent transmissions, the fragment length X is increased at preset statistical intervals. t Calculate system performance metrics within the most recent performance statistics period (seconds). P When performing this virtual partitioning operation, the data fragment length used in the previous virtual partitioning operation can be adjusted based on the system performance indicators of the most recent performance statistics period to determine the data fragment length for this virtual partitioning operation.

[0061] As an example, the steps to determine the data fragment length for the current virtual partitioning operation based on the system performance metrics in the most recent performance statistics period before the current virtual partitioning operation and the data fragment length of the previous virtual partitioning operation may include: comparing the system performance metrics in the most recent performance statistics period with a preset metric range; and obtaining the data fragment length for the current virtual partitioning operation by increasing, decreasing, or maintaining the data fragment length of the previous virtual partitioning operation based on the comparison results.

[0062] Here, there can be multiple preset indicator ranges. Different indicator ranges correspond to different ways of adjusting the data fragment length of the previous virtual partitioning operation. For example, different ranges correspond to different increases or decreases.

[0063] For example, the system performance metrics in response to the most recent performance statistics period. P satisfy: P If the current network performance is considered to be excellent, the data fragment length X (or data block size) can be increased. For example, the data fragment length of the previous virtual partitioning operation can be increased by a first proportion (e.g., 10%) to obtain the data fragment length of the current virtual partitioning operation.

[0064] Response to system performance metrics in the most recent performance statistics period P Satisfy: Second threshold ≤ P If the data fragment length is less than the first threshold (the second threshold is, for example, 0.6), the current network performance can be considered good. In this case, the current data fragment length X can be kept unchanged, and the data fragment length of the previous virtual partitioning operation can be used as the data fragment length of the current virtual partitioning operation.

[0065] Response to system performance metrics in the most recent performance statistics period P Satisfying: Third threshold ≤ P If the data block size is less than the second threshold (or the third threshold is, for example, 0.4), the current network performance can be considered average, and the data block size can be reduced. X For example, the data fragment length of the current virtual partitioning operation can be obtained by reducing the data fragment length of the previous virtual partitioning operation by a second ratio (e.g., 10%).

[0066] Response to system performance metrics in the most recent performance statistics period P satisfy: P If the third threshold is reached, the current network performance can be considered poor, and the data block size can be significantly reduced. X For example, the data fragment length of the current virtual partitioning operation can be obtained by reducing the data fragment length of the previous virtual partitioning operation by a third ratio (e.g., 20%).

[0067] In addition, as an example, a minimum value for the data shard length can be set. Xmin (For example, it can be 64B) and the maximum value Xmax (For example, it can be 10KB) to prevent over-adjustment. If the adjusted value is greater than the maximum value, the maximum value will be used as the data fragment length for this virtual partitioning operation; if the adjusted value is less than the minimum value, the minimum value will be used as the data fragment length for this virtual partitioning operation.

[0068] As an example, the publishing end can determine the system performance metrics for the most recent performance statistics period in the following way: determine multiple network metrics for the most recent performance statistics period; for each network metric, determine the weight corresponding to the network metric based on the information entropy of the network metric for the most recent performance statistics period; and determine the system performance metrics based on each network metric and the weight corresponding to each network metric.

[0069] Here, multiple network metrics include positive and negative metrics. Positive metrics refer to network parameters that are positively correlated with network performance, while negative metrics refer to network parameters that are negatively correlated with network performance.

[0070] As an example, network parameters can include network latency. D Packet loss rate L ,bandwidth B、 At least one of the following: CPU utilization on the publishing end. Positive metrics may include bandwidth, while negative metrics may include at least one of latency, packet loss rate, and CPU utilization.

[0071] In addition, the network parameters can be normalized (e.g., using Min-Max normalization). Specifically, the normalization of the positive index can be expressed as: The normalization of negative indicators can be expressed as: ,in, Indicates a positive indicator (such as bandwidth). This indicates a negative metric (such as latency, packet loss rate, or CPU utilization, at least one of these). Y These are the original parameter values. Y min and Y max These are the minimum and maximum values ​​of the parameter within historical data or a preset range, respectively. For example, for network latency (a negative indicator), suppose the minimum network latency in historical data is 10. ms The maximum value is 500. ms The current measured network latency is 100. ms Then the normalized network latency D norm =(500 100) / (500 10)≈0.833.

[0072] As an example, the information entropy of each network metric can be determined based on the probability distribution of multiple data points for that network metric within the most recent performance statistics period. For instance, the information entropy of each network metric can be determined within a sliding window performance statistics period T. Y Collect data multiple times (e.g., collect n samples) and calculate their probability distribution, which can be represented by, for example, the following formula (1): (1) in, p ij Indicates the first i Network metrics In the j The probability in a sample, where n is the total number of data points (or samples) collected.

[0073] Based on the above probability distribution, the information entropy of each parameter index can be calculated, for example, it can be expressed by the following formula (2): (2) in, H i Indicates the first i Information entropy of a network indicator.

[0074] Based on the information entropy mentioned above, the weight of each network metric can be calculated, for example, it can be expressed by the following formula (3): (3) Among them, w i Indicates the first i The weights of each network metric, s This represents the total number of network metrics, for example, it could be 4.

[0075] After determining the weight of each network metric, the corresponding network metrics can be weighted based on these weights, and the weighted sum of all network metrics can be determined as a system performance indicator. P For example, it can be represented by the following formula (4): P = w D • D norm + w L • L norm + w B • B norm +w R • R norm (4) in, w D Weights representing network latency w L The weights representing the packet loss rate w B The weight representing bandwidth. w R The weight representing the CPU utilization of the publishing end. D norm This represents the network latency after normalization. L norm This represents the packet loss rate after normalization. B norm This represents the bandwidth after normalization. R norm This represents the CPU utilization rate of the publishing end after normalization. However, the embodiments of this disclosure are not limited to this, and the system performance indicators can also be calculated using network indicators that have not been normalized in the above equation (4).

[0076] Furthermore, as an example, the method may also include: in response to small fluctuations in the overall system performance indicators, such as being less than a preset performance fluctuation threshold, increasing the performance statistics period of the sliding window. T The length of the sliding window can be adjusted, for example, by increasing it in fixed steps, such as +30 seconds; in response to significant fluctuations in overall system performance indicators, such as exceeding or equaling a preset performance fluctuation threshold, the performance statistics period of the sliding window can be reduced. T The length can be reduced in fixed steps, such as -30 seconds. This is used in each calculation of system performance metrics. P Then, you can slide the window forward to move the above statistical intervals. t Remove old data and incorporate new data to calculate subsequent system performance indicators. Here, the statistical interval represents the waiting time after each calculation, for example... t It can be 5 seconds.

[0077] Here, the preset performance fluctuation threshold can be used to determine the critical value for whether the system performance is stable. For example, it can be used to determine whether the fluctuation of the comprehensive system performance index meets the following condition: the rate of change of the system performance index is <10%.

[0078] In addition, performance statistics period T The length can not exceed the preset maximum value. Tmax And not less than the preset minimum value. Tmin Specifically, when the increased cycle length exceeds this maximum value, the performance statistics cycle can be maintained. TThe length is the maximum value Tmax When the reduced cycle length is less than this minimum value, the performance statistics cycle can be maintained. T The length is the minimum value Tmin Here, the minimum value of the performance statistics period. Tmin This can be used to prevent frequent calculations due to excessively small windows; for example, a window size of 30 seconds can be used. It also represents the maximum value of the performance statistics period. Tmax This can be used to prevent excessively large windows from causing slow response times; for example, it can be set to 300 seconds.

[0079] In the above example, when transferring unstructured files based on DDS, compared with the example of fixed-size block transmission, the DDS body dynamically adjusts the data block size by sensing the network and sender / receiver load in real time and performing virtual partitioning based on system performance indicators. In this case, by establishing a dynamic feedback mechanism, network parameters (such as bandwidth, latency, and packet loss rate) and node load parameters (such as CPU utilization) can be used as inputs. After processing by the algorithm, an appropriate data block size is output, thereby achieving a balance between transmission efficiency and reliability.

[0080] Furthermore, in this example, there is no need to pre-determine the total number of data fragments to be transmitted or divided.

[0081] It should be noted that in the embodiments of this disclosure, any of the above examples can be used for virtual partitioning, or a combination of the two can be used. For example, a portion of the target data can be partitioned by determining the fixed overhead of a single data fragment, while another portion of the target data can be partitioned by performing a virtual partitioning operation based on system performance indicators.

[0082] In addition, in the examples above, besides including the fragment length of a single data fragment, the partitioning information may also include first verification information, which can be sent to the subscriber so that the subscriber can compare the first verification information with second verification information generated based on the target data to verify the integrity of the target data.

[0083] Here, the first verification information and the second verification information can be, for example, but not limited to, MD5 values ​​obtained based on the MD5 message-digest algorithm. The MD5 value can be used as the basis for verifying the integrity of the target data. In response to the first verification information matching the second verification information, the subscriber can determine that it has received the complete target data; in response to the first verification information not matching the second verification information, the subscriber can determine that it has not received the complete target data.

[0084] In addition, as an example, the partitioning information may also include, but is not limited to, file name, total file length, etc.

[0085] Furthermore, in this step, such as Figure 2 As shown, the publisher can also send the obtained partitioning information to the subscriber. The subscriber can store the partitioning information and can also determine whether the data fragmentation has been completed based on the partitioning information (which will be described in detail below).

[0086] In step S130, the publisher can send multiple data shards obtained from the target data and the shard information of each data shard to the subscriber based on the partitioning information, so as to publish the target data to the subscriber.

[0087] Here, the fragmentation information can indicate the position of the corresponding data fragment in the target data. For example, the fragmentation information can be the index value indicating the corresponding data fragment in the sequence of the above multiple data fragments.

[0088] As an example, the publisher can send multiple data shards and their shard information to the subscriber by performing a shard sending operation for each data shard in the multiple data shards, based on the partitioning information. For example, Figure 4 As shown, for each data fragment, the fragment transmission operation may include the following steps: In step S410, the fragmentation information of the data fragment to be sent can be determined based on the fragmentation information of the data fragment sent in the previous transmission.

[0089] Here, when sending a data fragment for the first time, the fragment information of the data fragment to be sent can be the initial information, and the fragment information of the data fragments sent before can be determined according to the preset rules and the fragment information of the data fragments sent in the previous time.

[0090] As an example, when sending the first data fragment, the publisher can set the fragment's index to 0. Subsequent data fragments can be sent by incrementing the index of the previously sent fragment by 1. For example, ... Figure 5 As shown, the index value of shard index i1 of data shard 1 can be 1. The index value of shard index i2 of data shard 2 can be obtained by adding 1 to the index value of data shard 1, which is 2. Similarly, the index value of shard index i3 of data shard 3 can be obtained by adding 1 to the index value of data shard 2, which is 3. And so on, the shard index of each data shard in N data shards can be determined.

[0091] Although the above example uses fragment information as the index value, the embodiments of this disclosure are not limited thereto. Fragment information can also be other information, as long as it can indicate the position of the corresponding data fragment in the target data.

[0092] In step S420, data fragments to be sent can be obtained from the data that has not yet been sent in the target data according to the partitioning information.

[0093] During the publication of target data, the publishing end (also known as the sending end) can partition and transmit the target data in real time according to the partitioning information. The publishing end can partition the target data from the beginning position according to the size of each data fragment indicated in the partitioning information. After each partitioning to obtain the current data fragment, the current data fragment can be sent to the subscribing end (also known as the receiving end). In this way, data partitioning and fragment transmission can be implemented in parallel, reducing the memory space occupied by the publishing end and improving the publication efficiency of target data.

[0094] For example, such as Figure 5 As shown, after sending data fragment 2 to the subscriber, the publisher can divide the data fragment 3 to be sent from the data that has not yet been sent in the target data according to the division information.

[0095] In step S430, the fragmentation information of the data fragment to be sent and the data fragment to be sent can be encapsulated into the current message of the real-time sending subscription protocol of the data distribution service.

[0096] In this step, the publishing end can serialize the content of the data fragment to be sent, and can encapsulate the fragmentation information of the data fragment and the serialized data fragment content together into a DDS RTPS message.

[0097] For example, such as Figure 6 As shown, the encapsulated RTPS message, in addition to the flags and globally unique identifier (GUID) specified in the header, can also contain a 4-byte fragmentIndex field and a 1400-byte fragmentContent field in sequence. The fragmentIndex field stores the index of the currently transmitted data fragment, and the fragmentContent field stores the content of the currently transmitted data fragment. Furthermore, in Figure 6In this context, DATA indicates that the currently transmitted message type is a data packet, Flags indicates the flags of the currently transmitted data, octetsToNextHeader indicates the number of bytes from the next header, extraFlags indicates additional flags used for extension, octetsToInlineQos indicates the number of bytes to the inline Quality of Service (QoS), readerID indicates the identity of the data reader, writerID indicates the identity of the data writer, writerSN indicates the sequence number of the data, inlineQos indicates the inline QoS, and 0, 8, 16, and 32 represent the number of bits.

[0098] In step S440, multiple data fragments and fragment information of each data fragment can be sent to the subscriber by sending the current message to the subscriber.

[0099] As an example, the publisher can call the Socket interface to send the above RTPS message to the subscriber over the network, so as to send multiple data fragments and fragment information of each data fragment to the subscriber.

[0100] In addition, as an example, the publisher can also increment the value of the fragment index of the locally maintained data fragment to be sent by 1, so that it can be used for the next data fragment to be sent.

[0101] The publisher can perform the above-described fragment sending operation multiple times, sending each data fragment sequentially to the subscriber until the entire content of the target data has been sent. Correspondingly, the subscriber can receive each data fragment sequentially and can record the fragment information of each data fragment.

[0102] Using the above method, the RTPS protocol based on DDS technology can send each data fragment and its fragment information to the subscriber via message transmission, thereby achieving the transmission of the target data without introducing other data transmission technologies and protocols.

[0103] Furthermore, according to embodiments of this disclosure, after all data fragments have been sent at the sending end, the data transmission can be terminated, and the API interface provided by DDS can be called to destroy local data writers, topics, publishers, and domain participants, thereby realizing DDS resource reclamation.

[0104] In a second aspect of exemplary embodiments of this disclosure, a method for transmitting unstructured data based on a data distribution service is provided.

[0105] The method for transmitting unstructured data based on a data distribution service according to an exemplary embodiment of this disclosure can be applied to a subscription end, which may have a data distribution service loaded on it.

[0106] Here, the subscriber can be any device in the IoT system, such as a tablet, laptop, digital assistant, or wearable device. The subscriber can communicate with the publisher based on the data distribution service to send subscription messages to the publisher and receive data from the publisher.

[0107] The method for transmitting unstructured data according to the second aspect of the exemplary embodiments of this disclosure can solve the problem that traditional DDS technology can only support the transmission of small-scale structured data and cannot meet the needs of unstructured data transmission, thereby improving the data transmission capability of DDS technology and meeting the transmission needs of different types of data.

[0108] The following will refer to Figure 7 Another example of a method for transmitting unstructured data based on a data distribution service according to embodiments of this disclosure is described.

[0109] like Figure 7 As shown, the method for transmitting unstructured data may include the following steps: In step S710, the subscriber can send a subscription message to the publisher.

[0110] As mentioned above, the publisher and subscriber can communicate based on the data distribution service.

[0111] Similarly, such as Figure 2 As shown, when creating a DDS system, DDS initialization can be performed on the subscriber side to create file publishing / subscription topics on the publisher and subscriber sides. For example, it can include, but is not limited to, at least one of the following: initializing DDS middleware, calling the application programming interface (API) provided by DDS, creating domain participants, creating publishers / subscribers related to file transfer, creating topics, and creating data writers / data readers related to file transfer.

[0112] Examples of publishers and subscribers are the same as those described above in the exemplary embodiments according to the first aspect of this disclosure, and therefore will not be repeated here.

[0113] In step S720, the subscriber can receive multiple data fragments obtained from the target data corresponding to the subscription message and the fragment information of each data fragment from the publisher to obtain the target data.

[0114] The aforementioned multiple data shards can be obtained by the publishing end based on the partitioning information obtained by virtually partitioning the target data, and the partitioning method has been described above in the exemplary embodiment according to the first aspect of this disclosure.

[0115] Here, the target data can be unstructured data, and the fragmentation information can indicate the position of the corresponding data fragment in the target data and the fragmentation length of the corresponding data fragment.

[0116] In step S720, as Figure 8 As shown, the subscriber can obtain the target data in the following ways: In step S810, in response to receiving the current message of the real-time publish-subscribe protocol of the data distribution service from the publisher, the current data fragment and the fragment information of the current data fragment can be parsed from the current message.

[0117] After the subscriber begins receiving data from the publisher, it can respond to receiving an RTPS message (as described above) corresponding to a data fragment from the publisher, and parse the content of the data fragment and its fragment information from the received message. Here, the format of the message received by the subscriber can be, for example, as described above. Figure 6 As shown, the subscriber can, for example, remove Figure 6 The 24-byte header of the RTPS message shown provides data fragments and their fragment information (such as index values).

[0118] Here, the subscriber can parse the message according to the DDS-based RTPS protocol, which will not be elaborated further.

[0119] In step S820, the current data fragment can be spliced ​​to the previously received data fragment according to the fragmentation information of the current data fragment to obtain spliced ​​data.

[0120] Here, when a data fragment is received for the first time, the current data fragment can be appended to the initial position. When a data fragment is received again, the current data fragment can be appended to the last received data fragment.

[0121] As an example, the subscriber can write the contents of data shards to disk and append them to previously received and written files. For example, such as... Figure 5 As shown, when the subscriber receives data fragment 3, it can append data fragment 3 to data fragment 2. Furthermore, upon receiving the first data fragment, the subscriber can call the operating system interface to create a file based on the filename and open a data handle, such as a file handle.

[0122] In step S830, the fragmentation information of the current data fragment can be updated to the configuration file used for the target data.

[0123] In this step, the subscriber can update the data shard information (such as index values) to the configuration file to record the latest shard position or shard sequence number.

[0124] In step S840, in response to the current data fragment being the last data fragment in the target data, the target data can be obtained based on the current concatenated data and the current configuration file.

[0125] Furthermore, in response to the current data shard not being the last data shard in the target data, the subscriber can continue to wait for the next data shard to be sent from the publisher.

[0126] Using the above method, the subscriber can piece together the received data fragments one by one to finally obtain the target data.

[0127] As an example, the partitioning information may indicate the number of multiple data fragments, and the position of each data fragment in the target data is represented by the index value of the corresponding data fragment in the sequence of multiple data fragments. In this example, the method for transmitting unstructured data according to an exemplary embodiment of the present disclosure may further include: a subscriber receiving partitioning information from a publisher; and the subscriber determining that the current data fragment is the last data fragment in the target data in response to the partitioning information of the current data fragment indicating that the index value of the current data fragment is equal to the number of multiple data fragments indicated by the partitioning information.

[0128] Specifically, the subscriber can obtain the total number of data fragments to be sent by the publisher through virtual partitioning of the target data from the partitioning information. It can also check whether the number of received fragments reaches the total number of fragments recorded in the partitioning information based on the index value of the current data fragment. If the index value of the current data fragment is equal to the total number of data fragments, it can be determined that the current data fragment is the last data fragment in the target data, and it can be considered that the reception of all data fragments has been completed.

[0129] In this way, the subscriber can determine the timing of the completion of target data reception based on the segmentation information, and promptly execute subsequent data processing to improve the subscriber's processing efficiency of target data.

[0130] Furthermore, according to embodiments of this disclosure, after the subscriber has received all data fragments, or when the number of fragments received by the subscriber reaches the total number of fragments recorded in the partitioning information, the last data fragment can be written and the file can be closed, for example, by calling the operating system interface to close the file handle.

[0131] Furthermore, as mentioned above, the partitioning information may include, for example, first verification information, which can be used to verify the integrity of the target data. In this case, the method for transmitting unstructured data may also include: the subscriber receiving partitioning information from the publisher; the subscriber generating second verification information based on the obtained target data; and the subscriber verifying the integrity of the target data by comparing the first verification information and the second verification information.

[0132] Taking the example where both the first and second verification information are MD5 values, the subscriber can generate an MD5 value for the entire received target data as the second verification information, and compare the generated MD5 value with the MD5 value in the partition information. If they match, the current transmission can be terminated, and the API interface provided by DDS can be called to destroy the local data readers, topics, subscribers, and domain participants to achieve DDS resource reclamation. If they do not match, a data transmission error can be indicated.

[0133] By using the above methods, the integrity of the target data received by the subscriber can be ensured, and data transmission errors caused by missing data fragments during transmission can be avoided.

[0134] The steps in the embodiments of the second aspect here may be the same as or corresponding to the corresponding steps in the embodiments of the first aspect above, and can be understood based on the description in the embodiments of the first aspect above.

[0135] According to embodiments of this disclosure, a DDS-based unstructured data transmission method can be provided. This method does not limit the size of the transmitted data and can compensate for the shortcomings of traditional DDS technology, which can only support small-scale structured data transmission and cannot meet the file transmission needs of IoT systems. It expands the communication content supported by DDS, enabling application systems to achieve unified transmission of various types of data based on a single DDS platform, thereby improving the development and operation efficiency of the system.

[0136] Furthermore, the data transmission method according to the embodiments of this disclosure can transmit files in segments, avoiding the long-term occupation of the network by large amounts of data and improving system availability.

[0137] Furthermore, the data transmission method according to the embodiments of this disclosure can virtually shard files using sharding information such as sharding indexes, thereby reducing the memory space occupied by pre-sharding.

[0138] Figure 9 This is a block diagram of an electronic device according to exemplary embodiments of the present disclosure. Figure 9As shown, the electronic device 10 includes a processor 101 and a memory 102 for storing processor-executable instructions. Here, when executed by the processor, the processor-executable instructions cause the processor to perform a method for transmitting unstructured data based on a data distribution service as described in the exemplary embodiments above.

[0139] As an example, electronic device 10 is not necessarily a single device, but can be a collection of any means or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 10 can also be part of an integrated control system or system manager, or can be configured to interface with a server locally or remotely (e.g., via wireless transmission).

[0140] In electronic device 10, processor 101 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor 101 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0141] The processor 101 can execute instructions or code stored in the memory 102, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transmission protocol.

[0142] The memory 102 may be integrated with the processor 101, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 102 may include a separate device, such as an external disk drive, a storage array, or other storage device that can be used by any database system. The memory 102 and the processor 101 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 101 to read files stored in the memory 102.

[0143] In addition, the electronic device 10 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 10 can be connected to each other via a bus and / or network.

[0144] In an exemplary embodiment, a computer-readable storage medium may also be provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the method for transmitting unstructured data based on a data distribution service as described in the exemplary embodiment above. The computer-readable storage medium may be, for example, a memory including instructions. Optionally, the computer-readable storage medium may be: a read-only memory (ROM), a random access memory (RAM), a random access programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), flash memory, non-volatile memory, a CD-ROM, a CD-R, a CD+R, a CD-RW, a CD+RW, a DVD-ROM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a DVD-RAM, a BD-ROM, a BD-R, or a BD-R. LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0145] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, the computer program product including computer-executable instructions that, when executed by at least one processor, implement a method for transmitting unstructured data based on a data distribution service according to exemplary embodiments of the present disclosure.

[0146] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0147] Furthermore, it should be noted that although several examples of each step have been described above with reference to the specific accompanying drawings, it should be understood that the embodiments of this disclosure are not limited to the combinations given in the examples. The steps appearing in different drawings can be combined, and the execution order of each step can be changed, which will not be exhaustive here.

[0148] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for transmitting unstructured data based on a data distribution service, characterized in that, The transmission method is applied to the publishing end, wherein the transmission method includes: The publisher receives subscription messages from the subscriber, wherein the publisher and the subscriber communicate based on a data distribution service; The publishing end performs a virtual partitioning of the target data corresponding to the subscribed message to obtain partitioning information of the target data. The target data is unstructured data. The virtual partitioning means that the publishing end determines the partitioning method of the target data but does not actually split the target data. During the virtual partitioning process, the target data remains unchanged. The publishing end, based on the partitioning information, sends multiple data fragments obtained from the target data, along with fragment information for each data fragment, to the subscribing end to publish the target data to the subscribing end. The fragment information indicates the position and length of the corresponding data fragment within the target data. During the publication of the target data to the subscribing end, a currently unsent data fragment is partitioned from the unsent data within the target data, and the fragment information of this currently unsent data fragment, along with the fragment itself, is sent to the subscribing end. The publishing end performs multiple virtual partitioning operations to virtually partition the target data, thereby obtaining the partitioning information. The publishing end determines the data fragment length for each virtual partitioning operation in the following way: In response to the fact that at least one performance statistics period has passed before this virtual partitioning operation, the system performance index P in the most recent performance statistics period before this virtual partitioning operation will be compared with the preset index range. The result of the comparison indicates that the system performance index P satisfies: P ≥ first threshold. The data fragment length of the current virtual partitioning operation is obtained by increasing the data fragment length of the previous virtual partitioning operation. In response to the comparison result indicating that the system performance index P satisfies: second threshold ≤ P < first threshold, the data fragment length of the previous virtual partitioning operation is taken as the data fragment length of the current virtual partitioning operation; In response to the comparison result indicating that the system performance index P satisfies: third threshold ≤ P < second threshold, the data fragment length of the current virtual partitioning operation is obtained by reducing the data fragment length of the previous virtual partitioning operation by the second ratio; In response to the comparison result indicating that the system performance index P satisfies: P < the third threshold, the data fragment length for the current virtual partitioning operation is obtained by reducing the data fragment length of the previous virtual partitioning operation by a third ratio, wherein the third ratio is greater than the second ratio. The publishing end determines the system performance metrics for the most recent performance statistics period using the following method: Determine multiple network metrics within the most recent performance statistics period, wherein the multiple network metrics include positive metrics and negative metrics, wherein the positive metrics include bandwidth as a network parameter positively correlated with network performance, and the negative metrics include latency and packet loss rate as network parameters negatively correlated with network performance, and the CPU utilization rate of the publishing end as a node load parameter. For each network metric, the weight corresponding to the network metric is determined based on the information entropy of the network metric in the most recent performance statistical period, wherein the information entropy of each network metric is determined based on the probability distribution of multiple data points of the network metric in the most recent performance statistical period. The system performance metrics are determined based on each network metric and its corresponding weight. The transmission method further includes: increasing the length of the performance statistics period in response to the fluctuation of the system performance index being less than a preset performance fluctuation threshold; and decreasing the length of the performance statistics period in response to the fluctuation of the system performance index being greater than or equal to the performance fluctuation threshold.

2. The transmission method according to claim 1, characterized in that, The publishing terminal performs virtual partitioning of the target data in the following manner to obtain the partitioning information: The publisher determines the fixed overhead of a single data shard based on the network type used to communicate with the subscriber, the transport protocol of the publisher's transport layer, the real-time publish-subscribe protocol of the data distribution service, and the bytes occupied by recording the shard information. The publishing end determines the shard length of a single data shard and the number of multiple data shards based on the total length of the target data and the fixed overhead, in order to obtain the partitioning information.

3. The transmission method according to claim 1, characterized in that, The virtual partitioning operation includes: Determine the data fragment length for this virtual partitioning operation; Based on the data fragment length of this virtual partitioning operation, the remaining data to be partitioned in the target data is virtually partitioned to obtain the data fragment partitioning information of this virtual partitioning operation, and to determine the remaining data to be partitioned in the next step in the target data. The publishing end also determines the data fragment length for each virtual partitioning operation in the following way: Since no performance statistics cycle has been performed prior to this virtual partitioning operation, the preset initial size is determined as the data fragment length for this virtual partitioning operation. The system performance index represents the network performance during the process of sending the target data from the publishing end to the subscribing end.

4. The transmission method according to any one of claims 1 to 3, characterized in that, The publishing end, based on the partitioning information, performs a sharding transmission operation for each of the plurality of data shards, sending the plurality of data shards and the sharding information of each data shard to the subscribing end. The sharding transmission operation for each data shard includes: Based on the fragmentation information of the previously sent data fragment, the fragmentation information of the current data fragment to be sent is determined. In the case of the first data fragment transmission, the fragmentation information of the current data fragment to be sent is the initial information. Based on the partitioning information, data fragments to be sent are obtained from the target data that have not yet been sent. The fragmentation information of the currently to-be-sent data fragment and the currently to-be-sent data fragment are encapsulated into the current message of the real-time sending subscription protocol of the data distribution service; By sending the current message to the subscriber, the multiple data fragments and the fragmentation information of each data fragment are sent to the subscriber.

5. A method for transmitting unstructured data based on a data distribution service, characterized in that, The transmission method is applied to the subscription end, wherein the transmission method includes: The subscriber sends a subscription message to the publisher, wherein the publisher and the subscriber communicate based on a data distribution service. The subscriber receives from the publisher multiple data fragments derived from the target data corresponding to the subscription message, along with fragmentation information for each data fragment, to obtain the target data. The multiple data fragments are obtained by the publisher through partitioning information obtained from a virtual partitioning of the target data. The target data is unstructured data. Virtual partitioning refers to the publisher determining a partitioning method for the target data without actually dividing it. During virtual partitioning, the target data remains unchanged. The fragmentation information indicates the position and length of the corresponding data fragment within the target data. During the process of the subscriber receiving the target data from the publisher, the publisher partitions the target data into currently unsent data fragments from the unsent data and sends the fragmentation information of these currently unsent data fragments, along with the data fragments themselves, to the subscriber. The publishing end performs multiple virtual partitioning operations to virtually partition the target data, thereby obtaining the partitioning information. The publishing end determines the data fragment length for each virtual partitioning operation in the following way: In response to the fact that at least one performance statistics period has passed before this virtual partitioning operation, the system performance index P in the most recent performance statistics period before this virtual partitioning operation will be compared with the preset index range. The result of the comparison indicates that the system performance index P satisfies: P ≥ first threshold. The data fragment length of the current virtual partitioning operation is obtained by increasing the data fragment length of the previous virtual partitioning operation. In response to the comparison result indicating that the system performance index P satisfies: second threshold ≤ P < first threshold, the data fragment length of the previous virtual partitioning operation is taken as the data fragment length of the current virtual partitioning operation; In response to the comparison result indicating that the system performance index P satisfies: third threshold ≤ P < second threshold, the data fragment length of the current virtual partitioning operation is obtained by reducing the data fragment length of the previous virtual partitioning operation by the second ratio; In response to the comparison result indicating that the system performance index P satisfies: P < the third threshold, the data fragment length for the current virtual partitioning operation is obtained by reducing the data fragment length of the previous virtual partitioning operation by a third ratio, wherein the third ratio is greater than the second ratio. The publishing end determines the system performance metrics for the most recent performance statistics period using the following method: Determine multiple network metrics within the most recent performance statistics period, wherein the multiple network metrics include positive metrics and negative metrics, wherein the positive metrics include bandwidth as a network parameter positively correlated with network performance, and the negative metrics include latency and packet loss rate as network parameters negatively correlated with network performance, and the CPU utilization rate of the publishing end as a node load parameter. For each network metric, the weight corresponding to the network metric is determined based on the information entropy of the network metric in the most recent performance statistical period, wherein the information entropy of each network metric is determined based on the probability distribution of multiple data points of the network metric in the most recent performance statistical period. The system performance metrics are determined based on each network metric and its corresponding weight. Specifically, if the fluctuation of the system performance index is less than a preset performance fluctuation threshold, the publishing end increases the length of the performance statistics period; if the fluctuation of the system performance index is greater than or equal to the performance fluctuation threshold, the publishing end decreases the length of the performance statistics period.

6. The transmission method according to claim 5, characterized in that, The subscription client obtains the target data through the following methods: In response to receiving a current message of the real-time publish-subscribe protocol of the data distribution service from the publisher, the current data fragment and the fragment information of the current data fragment are parsed from the current message; Based on the fragmentation information of the current data fragment, the current data fragment is spliced ​​to the previously received data fragment to obtain spliced ​​data. When the data fragment is received for the first time, the current data fragment is spliced ​​to the initial position. Update the sharding information of the current data shard to the configuration file used for the target data; In response to the current data fragment being the last data fragment in the target data, the target data is obtained based on the current concatenated data and the current configuration file.

7. The transmission method according to claim 6, characterized in that, The partitioning information indicates the number of the plurality of data fragments, and the position of each data fragment in the target data is represented by the index value of the corresponding data fragment in the sequence of the plurality of data fragments. The transmission method further includes: The subscriber receives the partitioning information from the publisher. The subscriber responds to the fragmentation information of the current data fragment indicating that the index value of the current data fragment is equal to the number of multiple data fragments indicated by the partitioning information, and determines that the current data fragment is the last data fragment in the target data.

8. The transmission method according to claim 5, characterized in that, The partitioning information includes first verification information, wherein the transmission method further includes: The subscriber receives the partitioning information from the publisher. The subscription client generates second verification information based on the obtained target data; The subscription client verifies the integrity of the target data by comparing the first verification information and the second verification information.

9. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions. Wherein, when the processor executes the processor, it causes the processor to perform a method for transmitting unstructured data based on a data distribution service according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method for transmitting unstructured data based on a data distribution service according to any one of claims 1 to 8.

11. A computer program product comprising computer-executable instructions, characterized in that, When the computer-executable instructions are executed by at least one processor, they implement the method for transmitting unstructured data based on a data distribution service according to any one of claims 1 to 8.

Images