Message transmission method, device and equipment, and storage medium

Abstract

The application provides a message transmission method, device and equipment and a storage medium, and is used for solving the technical problem that collecting messages occupy network bandwidth. According to the frequency of data content change, the data content in the reported collecting message is fragmented. For the content which changes infrequently, a fragmentation flag bit and a unique fragmentation identifier are set in the message. When the fragmentation flag bit indicates that the fragmentation is transmitted for the first time, the message carries the fragmentation content, and the receiving end caches the non-repeated fragmentation received for the first time. Otherwise, the message does not carry the fragmentation content, and only carries the unique fragmentation identifier. The application can effectively improve the transmission efficiency of the collecting message and reduce the bandwidth occupation.

Description

Technical Field

[0001] This application relates to the fields of communication and cloud computing technology, and in particular to a message transmission method, apparatus, device and storage medium. Background Technology

[0002] With the development of cloud computing technology, cloud data centers have become a trend, and the operation and maintenance management of data centers has shifted from managing traditional physical servers to managing virtualized resources such as virtual machines and containers. As virtualization technology advances, data center operation and maintenance management typically requires the support of a data acquisition and analysis system. This system distributes traffic acquisition probes or deploys collectors to each virtual machine or container. The probes collect the traffic sent and received by the virtual machines or containers based on certain policies and rules. The collected traffic is then initially decoded and uploaded to the data acquisition and analysis system for analysis, with the results serving as the basis for operation and maintenance management. However, the uploading of decoded data from each traffic acquisition probe to the system consumes significant network bandwidth, especially when analyzing a large number of virtual machines or containers. This can significantly increase network load and potentially severely impact the normal operation of customer business systems.

[0003] To avoid severely impacting normal business systems through packet collection, the common practice is to limit the data reported by the collection side. This can be done by setting specifications for the collection probes and monitoring the collection traffic. When traffic exceeds limits, packets are dropped according to packet loss rules or randomly to reduce the impact of the collection and analysis system on the business system. While this approach effectively reduces the impact of the collection and analysis system on the business system and can ensure its stability to a certain extent, dropping a large number of packets when the traffic exceeds the specifications may cause bias or even errors in the analysis results, seriously affecting the accuracy of the analysis. Summary of the Invention

[0004] In view of this, this application provides a message transmission method, apparatus, device, and storage medium to solve the technical problem that excessive network bandwidth consumption by the data acquisition probe during data acquisition affects the business system.

[0005] Based on one aspect of the embodiments of this application, this application provides a message transmission method, which is applied to a transmitting end device deployed with a data collector, the method comprising:

[0006] The collected messages are initially decoded, and the decoded data is fragmented and a unique identifier for each fragment is calculated.

[0007] Match the fragment's unique identifier in the local cache;

[0008] If no unique identifier for a fragment is found in the local cache, the unique identifier for the fragment is cached locally, the acquisition message is assembled, the flag corresponding to the fragment in the acquisition message is set as the first transmission flag, the unique identifier field of the fragment is filled with the unique identifier of the fragment, and the fragment data is carried in the non-frequency variable field.

[0009] If a fragment unique identifier is matched in the local cache, the corresponding flag in the collected message will be set to the non-first transmission flag, the fragment unique identifier field will be filled with the fragment unique identifier of that fragment, and no fragment data will be carried.

[0010] Send the collection message.

[0011] Furthermore, the process of segmenting the initially decoded data specifically involves:

[0012] Based on the frequency of change of different fields in the collected messages and a preset frequency of change threshold, the initially decoded data is divided into frequently changing data and infrequently changing data; the infrequently changing data includes session information, or session information and information related to the collector; the frequently changing data includes frequently changing fields in the message header and message payload.

[0013] Divide infrequently changing data into one or more shards, and calculate a unique shard identifier for each shard.

[0014] Furthermore, preliminary decoding of the collected messages refers to parsing the collected original messages according to the protocol to extract the message header information and message payload of the original messages. If the original messages have multi-layer protocol encapsulation, the message header information includes the message header information of the multi-layer protocol.

[0015] Furthermore, the method also includes a cache eviction step: before performing the step of caching the unique identifier of the local shard, it is determined whether the local cache is full; if it is determined that the local cache is full, the Least Recently Used (LRU) algorithm is used to evict the cache space occupied by the least recently hit shard unique identifier.

[0016] Furthermore, the method also includes a cache cleanup step:

[0017] Receive a cache cleanup message and delete the cache space occupied by the fragment unique identifier in the local cache according to the fragment unique identifier carried in the message.

[0018] Based on another aspect of the embodiments of this application, this application also provides a message transmission method, which is applied to a receiving end device deployed with a data acquisition and analysis service, and the method includes:

[0019] Receive and parse the acquisition message, and read the fragmentation flag bit in the acquisition message;

[0020] Determine if the fragmentation flag is for the first transmission;

[0021] When the fragmentation flag indicates that the fragment is being transmitted for the first time, the fragmentation data carried in the acquisition message is cached in the local cache;

[0022] When the fragmentation flag indicates that the fragment is not being transmitted for the first time, the fragmentation data is retrieved from the local cache based on the fragmentation's unique identifier to construct a complete acquisition message.

[0023] The collected messages are provided to the data collection and analysis service.

[0024] Furthermore, the method further includes: before performing the step of caching the fragmented data carried in the collection message in the local cache, determining whether the local cache space is full; if the local cache is full, using the Least Recently Used (LRU) algorithm to release the cache space occupied by the least recently used fragment and sending a cache cleanup message to the collector, wherein the cache cleanup message carries the fragment unique identifier of the released fragment.

[0025] Furthermore, the method also includes: if no fragment data is obtained from the local cache based on the fragment unique identifier, the received collection message is discarded.

[0026] Based on another aspect of the embodiments of this application, this application also provides a message transmission device, which is applied to a transmitting end device on which a data collector is deployed, and the device includes:

[0027] The data acquisition and fragmentation module is used to perform preliminary decoding on the acquired packets, fragment the data after preliminary decoding, and calculate the unique identifier of each fragment.

[0028] The shard identifier caching module is used to match the unique shard identifier in the local cache; if no unique shard identifier is found in the local cache, then the unique shard identifier is cached locally.

[0029] The message assembly module is used to assemble the collected messages. When no fragment unique identifier is matched in the local cache, the corresponding flag of the fragment in the collected message is set to the first transmission flag, the fragment unique identifier field is filled with the fragment unique identifier of that fragment, and the fragment data is carried in the non-frequency variable field. When a fragment unique identifier is matched in the local cache, the corresponding flag of the fragment in the collected message is set to the non-first transmission flag, the fragment unique identifier field is filled with the fragment unique identifier of that fragment, and no fragment data is carried.

[0030] The message sending module is used to send the collected message.

[0031] Furthermore, the data acquisition and fragmentation module specifically divides the initially decoded data into frequently changing data and infrequently changing data based on the change frequency of different fields in the acquired message and a preset change frequency threshold; the infrequently changing data is divided into one or more fragments, and a unique fragment identifier is calculated for each fragment;

[0032] The infrequently changing data includes session information, or session information and information related to the collector; the frequently changing data includes frequently changing fields in the message header and the message payload.

[0033] Furthermore, the preliminary decoding of the collected packets by the acquisition and fragmentation module refers to parsing the original collected packets according to the protocol to extract the packet header information and packet payload of the original packets. If the original packets have multi-layer protocol encapsulation, the packet header information includes the packet header information of the multi-layer protocol.

[0034] Furthermore, the device also includes: the shard identifier cache module, which is further configured to determine whether the local cache is already full before caching the shard unique identifier locally; when the local cache is determined to be full, the Least Recently Used (LRU) algorithm is used to evict the cache space occupied by the least recently hit shard unique identifier.

[0035] Furthermore, the fragment identifier cache module is also used to receive a cache cleanup message and delete the cache space occupied by the fragment unique identifier cached in the local cache according to the fragment unique identifier carried in the message.

[0036] Based on another aspect of the embodiments of this application, this application also provides a message transmission apparatus, which is applied to a receiving end device deployed with a data collection and analysis service, and the apparatus includes:

[0037] The message receiving module is used to receive and parse the collected messages and read the fragmentation flag bits in the collected messages;

[0038] The fragment data caching and acquisition module is used to determine whether the fragment flag is the first transmission; when the fragment flag indicates that the fragment is the first transmission, the fragment data carried in the acquisition message is cached in the local cache; when the fragment flag indicates that the fragment is not the first transmission, the fragment data is retrieved from the local cache according to the fragment unique identifier and a complete acquisition message is constructed.

[0039] The message providing module is used to provide the collected messages to the collection and analysis service.

[0040] Furthermore, the device also includes: a cleanup and synchronization module, used to determine whether the local cache space is full before caching the fragmented data carried in the collection message in the local cache; if the local cache is full, the Least Recently Used (LRU) algorithm is used to release the cache space occupied by the least recently used fragment and send a cache cleanup message to the collector, the cache cleanup message carrying the fragment unique identifier of the released fragment.

[0041] Furthermore, the fragment data caching and acquisition module is also used to discard the received acquisition message when fragment data is not retrieved from the local cache based on the fragment unique identifier.

[0042] The device described in this application can be implemented in software, hardware, or a combination of both. When implemented as a software module, the program code of the software module is loaded into the storage medium of the device, and the processor reads and executes the program code from the storage medium to realize the functions of each component module in the device. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings of the embodiments of this application.

[0044] Figure 1 Example diagram of a network application scenario using the message transmission method provided in this application;

[0045] Figure 2 This is a flowchart illustrating the steps of a data collector performing a message transmission method according to an embodiment of this application.

[0046] Figure 3 A flowchart illustrating the steps of a message transmission method performed by an embodiment of the data acquisition and analysis service provided in this application;

[0047] Figure 4 This is a schematic diagram of an electronic device structure for implementing the message transmission method for a real-time data acquisition device or data acquisition and analysis service provided in this application, as an embodiment of the present application. Detailed Implementation

[0048] The terminology used in the embodiments of this application is for the purpose of describing specific embodiments only and is not intended to limit the embodiments of this application. The singular forms "a," "the," and "the" used in the embodiments of this application are also intended to include the plural forms unless the context clearly indicates otherwise. It should be understood that although the terms first, second, third, etc., may be used to describe various information in the embodiments of this application, this information should not be limited to these terms. These terms are only used to distinguish similar information, entities, or steps, and are not used to describe a specific order or sequence. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Furthermore, the word "if" used can be interpreted as "when," "when," or "in response to a determination." "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. And, in the description of this application, unless otherwise stated, "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can be expressed as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0049] The purpose of this invention is to provide a technical solution for reporting acquisition messages to an acquisition and analysis system, addressing the technical problem that excessive network bandwidth consumption by acquisition messages negatively impacts the performance and stability of the business system. The basic idea of ​​this technical solution is to fragment the data content in the acquisition message based on the frequency of data content changes. For infrequently changing content, a fragmentation flag and a unique fragment identifier are set in the message. The fragmentation flag indicates whether the corresponding fragment content in the message is compressed, and the unique fragment identifier is a unique keyword generated based on the fragment content. When the fragmentation flag indicates that the fragment is being transmitted for the first time (without compression), the message carries the fragment content, and the receiving end caches the first received non-repeating fragment; otherwise, the message does not carry fragment content, only the unique fragment identifier, and the receiving end retrieves the fragment content from the cache based on the unique fragment identifier. This application can transmit more acquisition messages within a certain bandwidth, effectively improving the transmission efficiency of acquisition messages, reducing bandwidth consumption, and effectively improving the performance of the acquisition and analysis system and the stability of the business system.

[0050] Figure 1This diagram illustrates a network application scenario using the message transmission method provided in this application. The message transmission method provided in this application can be widely applied to various application scenarios where network load is to be reduced, especially in network application scenarios where the purpose is to collect messages from the business network and report them to the management end for network security and network operation and maintenance management purposes. Figure 1 In the example, to manage cloud data centers based on cloud computing technology, collectors / collection probes are deployed on devices such as host machines, switching devices, and routing devices of virtual machines or containers in the cloud data center. The collectors collect the sent and received packets on network nodes such as virtual machines, containers, and virtual or physical forwarding devices based on certain policy rules. After preliminary decoding of the packets in the business network, the collected packets are sent to the collection and analysis service. The collection and analysis service parses and analyzes the collected packets to provide decision-making basis for the operation and maintenance management of the cloud data center. When the collector reports the collection message, it uses the message transmission method provided in this application. This message transmission method segments and compresses the content in the collection message according to the frequency of change of different fields of the collected data. The content with a low frequency of change is divided into one or more segments. The receiving end buffers the first non-repeating segment received. For subsequent repetitive segments, the collection message only carries the unique identifier of the segment. The receiving end obtains the segment content from the local machine through the unique identifier of the segment and fills it back into the collection message to recover the complete collection message. This method can effectively reduce the bandwidth occupied by the collection message in the business network, thereby effectively improving the performance of the collection and analysis system and the stability of the business system.

[0051] Figure 2 This is a flowchart illustrating the steps of a message transmission method performed by a data collector according to an embodiment of this application. The illustrated steps are applied to a sending device on which the data collector is deployed, and the steps include:

[0052] S201. Request a cache for storing the unique identifier of each fragment;

[0053] During collector initialization, cache space needs to be allocated for caching shard unique identifiers. For example, an array of preset size can be allocated, with array cells used to store shard unique identifiers (referred to as shard identifiers). For ease of querying, an in-memory key-value database can also be used, where the key is used to store the shard unique identifier, and the key value field can be set to empty. On the collector side, only the shard unique identifiers need to be cached; the actual data content of the shards does not need to be cached.

[0054] S202. Perform preliminary decoding on the collected messages, fragment the data after preliminary decoding, and calculate the unique identifier of each fragment;

[0055] This application refers to the packets collected by the collector as raw packets, and the devices that generate, forward, or send / receive raw packets as packet sources, such as virtual machines, containers, virtual switches, routers, etc. After the collector collects raw packets from the packet source, it needs to perform preliminary decoding on the raw packets to extract the information required for analysis, such as session information, identification information, load information, etc., and package the extracted information into collection packets. Collection packets are the packets used by the collector to report the collected data to the collection and analysis service.

[0056] Preliminary decoding of the collected packets refers to parsing the original collected packets according to the protocol to extract the packet header information and packet payload. If the original packet has multiple layers of protocol encapsulation, the packet header information may include the packet header information of each layer of protocol. For example, if the inner layer of the original packet is TCP / IP protocol encapsulation and the outer layer is General Routing Encapsulation (GRE) or Virtual Extensible Local Area Network (VXLAN) protocol encapsulation, then the packet header information may include in-memory TCP / IP protocol header information and GRE or VXLAN protocol header information.

[0057] The purpose of preliminary decoding is to segment the data after preliminary decoding. The segmentation process is as follows: based on the change frequency of different fields in the collected message and the preset change frequency threshold, the data after preliminary decoding is divided into frequently changing data and infrequently changing data. The infrequently changing data is divided into one or more segments, and a unique segment identifier is calculated for each segment.

[0058] For frequently changing data, it can be directly packaged and carried in the collection message, without the need to add a unique fragment identifier and fragment flag bit to the collection message.

[0059] For example, when transmitting packets between two devices that have established a session connection, the collector collects the packets sent and received by one of the devices (raw packets). Since the session has been established, the content of the session information reflected in the header of the raw packets usually changes very infrequently throughout the session, and can be fragmented as infrequently changing data. Session information may include the source and destination addresses, port numbers, IP protocol versions, tunnel protocol types, tunnel source and destination addresses, etc., in the inner and outer protocol headers of the raw packets.

[0060] Infrequently changing data may also include data related to the collector, such as collector identifier, collector type, packet direction, collected node information, collection time, traffic information, collector MAC and IP information, etc.

[0061] Frequently changing data may include certain frequently changing fields in the message header and message payload, such as the message time to live (TTL), message sequence number, message length, payload length, payload offset, IP data offset, and data source type (virtual machine, container, image, etc.) in the inner and outer protocol headers of the original message.

[0062] One way to calculate the unique identifier of a fragment is by using a hash method. After the infrequently changing data in the initially decoded data is divided into fragments, a hash calculation is performed on the fragments, and the hash value is used as the unique identifier of the fragment.

[0063] S203. Match the fragment unique identifier in the local cache;

[0064] This step matches the unique identifier of the fragment in the local cache to determine whether the corresponding fragment has been sent to the collection and analysis service before. If the unique identifier of the fragment is matched locally, it means that it has been sent before; if it is not matched, it means that it has not been sent before.

[0065] S204. Determine if a unique fragment identifier is matched;

[0066] S205. If no fragment unique identifier is matched, cache the fragment unique identifier locally;

[0067] S206. Assemble the acquisition message, set the corresponding flag of the fragment in the acquisition message to the first transmission flag, fill the fragment unique identifier field with the fragment unique identifier of the fragment, and carry the fragment data in the non-frequency variable field;

[0068] The mandatory fields in the acquisition message format of this application include at least a fragmentation flag, a fragmentation unique identifier, and a frequency-varying field for transmitting frequently changing decoded data. In addition to the aforementioned mandatory fields, a non-frequency-varying field may also be included, which is used to transmit data fragments of the infrequently changing decoded data content described in the preceding steps.

[0069] The fragmentation flag indicates whether the corresponding fragment is being transmitted for the first time. In the case of the first transmission, the corresponding fragmentation flag must be set to the first transmission flag (e.g., set to 0), and the fragment unique identifier field must be filled with the unique identifier of the corresponding fragment. The non-frequency variable field must also be filled with the fragment data of the corresponding fragment. In the case of a fragment not being transmitted for the first time, the corresponding fragmentation flag must be set to the non-first transmission flag (e.g., set to 1), and the fragment unique identifier field must be filled with the unique identifier of the corresponding fragment. The non-frequency variable field does not carry the fragment data of the corresponding fragment; if the non-frequency variable field is empty, it is not carried. In the case of a fragment not being transmitted for the first time, the collector assumes that the receiving end has already cached the fragment from the first transmission. Therefore, subsequent transmissions of the collection message only need to carry the fragment unique identifier. The receiving end can use the fragment unique identifier to read the fragment data from the local cache to recover the complete collection message.

[0070] S207. If a fragment unique identifier is matched in the local cache, the collection message is directly assembled, the fragment corresponding flag in the collection message is set to the non-first transmission flag, the fragment unique identifier field is filled with the fragment unique identifier of the fragment, and the collection message does not carry fragment data.

[0071] When the collector matches the calculated fragment unique identifier in the local cache, it means that the corresponding fragment data has already been transmitted to the collection and analysis service. Therefore, this transmission can only carry the fragment unique identifier, thereby avoiding the repeated transmission of fragment data in the network and reducing the network bandwidth usage.

[0072] S208. Send a collection message to the collection and analysis service.

[0073] In one embodiment of this application, the collector requests a fixed-size cache space for caching fragment unique identifiers. When a new fragment unique identifier needs to be stored in the local cache and the cache area is full, the above steps also include a cache eviction step: using the Last Recently Used (LRU) algorithm to evict the local cache space occupied by the fragment unique identifier that was least recently hit, i.e., matched.

[0074] Figure 3 This is a flowchart illustrating the steps of a message transmission method performed by a data acquisition and analysis service according to an embodiment of this application. The illustrated steps are applied to a receiving device deployed with the data acquisition and analysis service, and include:

[0075] S301. Request a cache area for storing fragments;

[0076] When initializing the data acquisition and analysis service, a cache area needs to be requested for caching fragmented data. This request does not restrict the data structure type of the cache area; for example, an in-memory key-value database can be used, with the key used to store the unique identifier of the fragment and the key value field used to store the fragmented data.

[0077] S302. Receive and parse the acquisition message, and read the fragmentation flag bit in the acquisition message;

[0078] The fragmentation flag in the acquisition message indicates whether it is the first time a fragment is being transmitted. For fragments being transmitted for the first time, the fragment data will be carried in the non-frequency variable field of the acquisition message. For fragments that are not being transmitted for the first time, the acquisition message will not carry fragment data.

[0079] S303. Determine if the fragmentation flag is for the first transmission?

[0080] S304. When the fragmentation flag indicates that the fragment is being transmitted for the first time, the fragmentation data carried in the acquisition message is cached in the local cache;

[0081] The initial transmission refers to the situation where the collector sends a collection message carrying fragmented data to the collection and analysis service when the fragment unique identifier is not found in the collector's buffer. Therefore, the initial transmission includes situations where the collector or the collection and analysis service discards the fragment unique identifier due to a full buffer, then receives the discarded fragment data again and retransmits the fragment to the collection and analysis service.

[0082] When the fragmentation flag indicates that the message is being transmitted for the first time, it means that the acquisition message is a complete message, that is, the acquisition message carries fragmented data, so the acquisition message can be directly provided to the acquisition and analysis service.

[0083] In one embodiment of this application, the following cache cleanup step is further performed before caching the fragmented data locally:

[0084] S3041. Determine if the local cache space is full;

[0085] S3042. When the local cache is full, the Least Recently Used (LRU) algorithm is used to release the cache space occupied by the least recently used fragment and a cache cleanup message is sent to the collector. The cache cleanup message carries the unique fragment identifier of the released fragment.

[0086] After receiving the cache cleanup message from the data collection and analysis service, the collector deletes the cached fragment unique identifier from its local cache based on the fragment unique identifier carried in the message. In this way, the collector and the data collection and analysis service can synchronize the fragment information in their caches.

[0087] S305. When the fragmentation flag indicates that the fragment is not being transmitted for the first time, the fragmentation data is retrieved from the local cache based on the fragmentation unique identifier, and a complete collection message is constructed.

[0088] Under normal network and environmental conditions, if the fragmentation flag indicates that the fragment is not being transmitted for the first time, the inability to retrieve fragment data locally usually will not occur. However, under abnormal network conditions, the data collection and analysis service may fail to send a cache cleanup message to the collector, causing inconsistencies in the fragmentation information cached by the collector and the data collection and analysis service. In such cases, retrieving fragment data from the local cache based on the fragment's unique identifier may fail because the fragment data on the data collection and analysis service's side has already been cleared. When this happens, since the data collection and analysis service cannot recover the complete collection message based on the fragment's unique identifier, it will directly discard the received collection message.

[0089] S306. Provide the complete collection message to the collection and analysis service, which then analyzes the collection message.

[0090] Figure 4 This is a schematic diagram of an electronic device 400 for implementing the message transmission method for a data acquisition device or data acquisition and analysis service provided in this application, according to an embodiment of the present application. The device includes a processor 410 (such as a central processing unit (CPU), a communication bus 420, a communication interface 440, and a memory 430. The processor 410 and the memory 430 can communicate with each other via the communication bus 420. The memory 430 stores a computer program, which, when executed by the processor 410, can implement one or more steps of the message transmission method for a data acquisition device or data acquisition and analysis service provided in this application.

[0091] Memory refers to a device based on a storage medium used to store computer programs and / or data. It can be volatile memory (VM, often called main memory) or non-volatile memory (NVM). Main memory refers to internal storage that directly exchanges data with the processor. It can read and write data at any time and at high speed, serving as a storage medium for temporary data of the operating system and other running programs. Main memory can be Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), etc. Non-volatile memory refers to memory that uses persistent storage media, characterized by large capacity and persistent data retention. It can be Storage Class Memory (SCM), Solid State Drive (SSD), NAND flash memory, hard disks, etc. SCM is an industry term for a new type of storage medium that falls between main memory and flash memory. It is a composite storage technology that combines the characteristics of persistent storage and main memory, with access speeds slower than DRAM but faster than SSDs.

[0092] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0093] It should be understood that embodiments of this application can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in non-transitory (or non-persistent) memory. The methods can be implemented using standard programming techniques, including a non-transitory storage medium configured with a computer program within the computer program, wherein such a storage medium causes the computer to operate in a specific and predefined manner. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit. Furthermore, the operations of the processes described in this application can be performed in any suitable order unless otherwise indicated by this application or otherwise clearly contradicted by the context. The processes described in this application (or variations and / or combinations thereof) can be executed under the control of one or more computer systems configured with executable instructions and can be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program includes a plurality of instructions executable by one or more processors.

[0094] Furthermore, the method can be implemented in any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices. Aspects of this application can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, which, when read by the computer, can be used to configure and operate the computer to perform the processes described herein. Additionally, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention described in this application includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. This application also includes the computer itself when programmed according to the methods and techniques described herein.

[0095] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A message transmission method, characterized in that, This method is applied to a transmitting device with a data collector deployed, and the method includes: The collected messages are initially decoded, and the decoded data is fragmented and a unique identifier for each fragment is calculated. Match the fragment's unique identifier in the local cache; If no unique fragment identifier is found in the local cache, the unique fragment identifier is cached locally, the acquisition message is assembled, the corresponding flag in the acquisition message is set as the first transmission flag, the unique fragment identifier field is filled with the unique fragment identifier of the fragment, and the fragment data is carried in the non-frequently changing field. The non-frequently changing field is used to transmit data fragments of decoded data content that do not change frequently. If a fragment unique identifier is matched in the local cache, the corresponding flag in the collected message will be set to the non-first transmission flag, the fragment unique identifier field will be filled with the fragment unique identifier of that fragment, and no fragment data will be carried. Send the collection message; The specific steps for processing the initially decoded data into fragments are as follows: Based on the frequency of change of different fields in the collected messages and a preset frequency threshold, the initially decoded data is divided into frequently changing data and infrequently changing data. Frequently changing data is packaged and carried in the collected messages without adding a fragment unique identifier or fragment flag to the collected messages. The infrequently changing data includes session information, or session information and information related to the collector. The frequently changing data includes frequently changing fields in the message header and the message payload. Divide infrequently changing data into one or more shards, and calculate a unique shard identifier for each shard.

2. The method according to claim 1, characterized in that, Preliminary decoding of the collected messages refers to parsing the original collected messages according to the protocol to extract the message header information and message payload. If the original message has multiple layers of protocol encapsulation, the message header information includes the message header information of the multiple layers of protocols.

3. The method according to claim 1, characterized in that, The method also includes a cache eviction step: Before performing the step of caching the fragment unique identifier locally, it is determined whether the local cache is full; if the local cache is full, the Least Recently Used (LRU) algorithm is used to evict the cache space occupied by the least recently hit fragment unique identifier.

4. The method according to claim 1, characterized in that, The method also includes a cache cleanup step: Receive a cache cleanup message and delete the cache space occupied by the fragment unique identifier in the local cache according to the fragment unique identifier carried in the message.

5. A message transmission method, characterized in that, This method is applied to receiving devices that are deployed with data acquisition and analysis services, and the method includes: The system receives and parses the acquisition messages, reading the fragmentation flags within them. The transmitting device, equipped with the acquisition unit, performs initial decoding on the acquired messages and divides the initially decoded data into frequently changing and infrequently changing data based on the frequency of change in different fields and a preset frequency threshold. Frequently changing data is packaged into the acquisition messages without adding a unique fragment identifier or fragmentation flag. Infrequently changing data includes session information, or session information combined with information related to the acquisition unit. Frequently changing data includes frequently changing fields in the message header and the message payload. The infrequently changing data is then divided into one or more... Each segment is divided into segments, and a unique segment identifier is calculated for each segment. The unique segment identifier is matched in the local cache. If no unique segment identifier is matched in the local cache, the unique segment identifier is cached locally, the acquisition message is assembled, the segment corresponding flag in the acquisition message is set as the first transmission flag, the unique segment identifier field is filled with the unique segment identifier of that segment, and the segment data is carried in the non-frequency changing field. The non-frequency changing field is used to transmit data segments of decoded data content that do not change frequently. If a unique segment identifier is matched in the local cache, the segment corresponding flag in the acquisition message is set as the non-first transmission flag, the unique segment identifier field is filled with the unique segment identifier of that segment, and no segment data is carried. Determine if the fragmentation flag is for the first transmission; When the fragmentation flag indicates that the fragment is being transmitted for the first time, the fragmentation data carried in the acquisition message is cached in the local cache; When the fragmentation flag indicates that the fragment is not being transmitted for the first time, the fragmentation data is retrieved from the local cache based on the fragmentation's unique identifier to construct a complete acquisition message. The collected messages are provided to the data collection and analysis service.

6. The method according to claim 5, characterized in that, The method further includes: Before executing the step of caching the fragmented data carried in the collected message in the local cache, determine whether the local cache space is full; If the local cache is full, the Least Recently Used (LRU) algorithm is used to release the cache space occupied by the least recently used fragment and a cache cleanup message is sent to the collector. The cache cleanup message carries the unique fragment identifier of the released fragment.

7. The method according to claim 5, characterized in that, The method further includes: If no fragment data is retrieved from the local cache based on the fragment's unique identifier, the received collection message is discarded.

8. A message transmission device, characterized in that, This device is used in transmitting equipment where a data collector is deployed, and the device includes: The data acquisition and fragmentation module is used to perform preliminary decoding on the acquired packets, fragment the data after preliminary decoding, and calculate the unique identifier of each fragment. The shard identifier caching module is used to match the unique shard identifier in the local cache; if no unique shard identifier is found in the local cache, then the unique shard identifier is cached locally. The message assembly module is used to assemble the acquired messages. When no fragment unique identifier is matched in the local cache, the corresponding flag in the acquired message is set to the first transmission flag, the fragment unique identifier field is filled with the fragment unique identifier of that fragment, and the fragment data is carried in the non-frequency changing field. The non-frequency changing field is used to transmit data fragments of decoded data content that do not change frequently. When a fragment unique identifier is matched in the local cache, the corresponding flag in the acquired message is set to the non-first transmission flag, the fragment unique identifier field is filled with the fragment unique identifier of that fragment, and no fragment data is carried. The message sending module is used to send the collected message; The data acquisition and fragmentation module divides the initially decoded data into frequently changing data and infrequently changing data based on the frequency of change of different fields in the acquired message and a preset frequency of change threshold. The frequently changing data is packaged and carried in the acquired message without adding a fragment unique identifier or fragment flag bit to the acquired message. The infrequently changing data is divided into one or more fragments, and a fragment unique identifier is calculated for each fragment. The infrequently changing data includes session information, or session information and information related to the collector; the frequently changing data includes frequently changing fields in the message header and the message payload.

9. The apparatus according to claim 8, characterized in that, The preliminary decoding of the collected packets by the acquisition and fragmentation module refers to parsing the original collected packets according to the protocol to extract the packet header information and packet payload. If the original packet has multiple layers of protocol encapsulation, the packet header information includes the packet header information of the multiple layers of protocol.

10. The apparatus according to claim 8, characterized in that, The device further includes: The shard identifier caching module is also used to determine whether the local cache is full before caching the shard unique identifier locally; when the local cache is full, the Least Recently Used (LRU) algorithm is used to evict the cache space occupied by the least recently used shard unique identifier.

11. The apparatus according to claim 8, characterized in that, The fragment identifier cache module is also used to receive a cache cleanup message and delete the cache space occupied by the unique fragment identifier cached in the local cache according to the unique fragment identifier carried in the message.

12. A message transmission device, characterized in that, This device is used in receiving equipment deployed with data acquisition and analysis services. The device includes: The message receiving module is used to receive and parse the collected messages, and read the fragmentation flag in the collected messages. The sending device, equipped with a data collector, performs preliminary decoding on the collected messages and divides the preliminarily decoded data into frequently changing data and infrequently changing data based on the change frequency of different fields in the collected messages and a preset change frequency threshold. Frequently changing data is packaged and carried in the collected messages without adding a unique fragment identifier or fragmentation flag to the messages. The infrequently changing data includes session information, or session information and information related to the data collector. Frequently changing data includes frequently changing fields in the message header and the message payload. The infrequently changing data is then divided into... For one or more fragments, calculate a unique fragment identifier for each fragment; match the unique fragment identifier in the local cache; if no unique fragment identifier is found in the local cache, then cache the unique fragment identifier locally, assemble the acquisition message, set the corresponding flag of the fragment in the acquisition message to the first transmission flag, fill the unique fragment identifier field with the unique fragment identifier of that fragment, and carry the fragment data in the non-frequency changing field, where the non-frequency changing field is used to transmit data fragments of decoded data content that do not change frequently; if a unique fragment identifier is found in the local cache, then set the corresponding flag of the fragment in the acquisition message to the non-first transmission flag, fill the unique fragment identifier field with the unique fragment identifier of that fragment, and do not carry the fragment data. The fragment data caching and acquisition module is used to determine whether the fragment flag is the first transmission; when the fragment flag indicates that the fragment is the first transmission, the fragment data carried in the acquisition message is cached in the local cache; when the fragment flag indicates that the fragment is not the first transmission, the fragment data is retrieved from the local cache according to the fragment unique identifier and a complete acquisition message is constructed. The message providing module is used to provide the collected messages to the collection and analysis service.

13. The apparatus according to claim 12, characterized in that, The device further includes: The cleanup and synchronization module is used to determine whether the local cache space is full before caching the fragmented data carried in the collection message in the local cache. If the local cache is full, the Least Recently Used (LRU) algorithm is used to release the cache space occupied by the least recently used fragment and send a cache cleanup message to the collector. The cache cleanup message carries the unique fragment identifier of the released fragment.

14. The apparatus according to claim 12, characterized in that, The fragment data caching and acquisition module is also used to discard the received acquisition message when fragment data is not retrieved from the local cache based on the fragment unique identifier.

15. An electronic device, characterized in that, It includes a processor, a communication interface, a storage medium, and a communication bus, wherein the processor, the communication interface, and the storage medium communicate with each other through the communication bus; Storage medium used to store computer programs; A processor, when executing a computer program stored on a storage medium, performs the method steps of any one of claims 1-7.

16. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the method as described in any one of claims 1 to 7.

Images