Fragmented message processing method and device, electronic equipment and computer program product
By analyzing sharded messages in industrial DCS systems and dynamically adjusting the capacity of the adaptive bitmap, the problems of high memory usage and performance in sharded messages are solved, and efficient shard restructuring and memory management are achieved.
Patent Information
- Application Number
- CN202510058077.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-14
- Publication Date
- 2025-05-09
AI Technical Summary
In industrial DCS systems, when network devices process large numbers of sharded packets, they need to perform complex sorting and reorganization, resulting in high memory usage, performance affected, and memory waste.
By analyzing the received shard message, obtaining shard information, and dynamically adjusting the capacity of the adaptive bitmap based on the shard sequence number and the total shard number to avoid memory waste and realize adaptive adjustment of the bitmap capacity.
It significantly reduces processing complexity, improves the efficiency of sharding reorganization, and avoids performance loss and memory waste caused by complex sorting algorithms or fixed bitmaps.
Smart Images
Figure CN119966934A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of industrial control technology, and in particular relates to a method, device, storage medium, electronic equipment and computer program product for processing fragmented messages. Background Art
[0002] Industrial DCS (Distributed Control System) is a distributed automation system consisting of engineer stations, operator stations, field control stations, communication control stations, and data servers connected by Ethernet and a control network using fieldbus technology. In an industrial DCS system, a large amount of data communication is required between stations, and the protocol message application layer used usually contains a large number of fragmented messages. After receiving all the message fragments, the network equipment needs to sort and reassemble them according to the fragment sequence numbers. This process involves a large amount of calculation and memory operations, resulting in a high memory usage rate, which has a great impact on the performance of network equipment.
[0003] How to reduce memory usage and avoid memory waste has become a technical problem that needs to be solved urgently. Summary of the invention
[0004] In order to at least solve the problems existing in the prior art, the present invention provides a processing method based on fragmented messages, comprising:
[0005] The received fragment message is parsed to obtain fragment information, wherein the fragment information includes: a service identifier, a total number of fragments, a fragment sequence number, a fragment length, and an IP address; when the fragment sequence number indicates the first fragment message, if the total number of fragments is not less than the capacity of the adaptive bitmap N, an adaptive bitmap (N+1) is applied to a preset memory pool for replacement, and the capacity of the adaptive bitmap (N+1) is greater than the capacity of the adaptive bitmap N; bits in the corresponding adaptive bitmap are set according to the fragment information; a setting distribution result of the corresponding adaptive bitmap is detected in real time; and according to the fragment information and the corresponding adaptive bitmap, The method further comprises: judging whether the fragment message is completely received according to the setting distribution result of the bitmap; if it is judged that the message is completely received, reorganizing all the fragment messages corresponding to the corresponding adaptive bitmap to obtain the reorganized message; and further comprising: judging whether the bitmap detection period M is arrived; if it is judged that it is arrived, applying to the memory pool for adjusting the capacity of the corresponding adaptive bitmap according to the detection information of the setting distribution result of the corresponding adaptive bitmap within the bitmap detection period M, and the adjusted capacity of the corresponding adaptive bitmap is applied to the bitmap detection period (M+1); if it is judged that it is not arrived, not adjusting the capacity of the corresponding adaptive bitmap.
[0006] Another aspect of the present invention provides a processing device based on fragmented messages, comprising:
[0007] A parsing module, used for parsing the received fragment message to obtain fragment information, wherein the fragment information includes: service identification, total number of fragments, fragment sequence number, fragment length and IP address; a first judgment module, used for applying for adaptive bitmap (N+1) from a preset memory pool for replacement when the fragment sequence number indicates the first fragment message, if the total number of fragments is not less than the capacity of the adaptive bitmap N, and the capacity of the adaptive bitmap (N+1) is greater than the capacity of the adaptive bitmap N; a setting module, used for setting the bits in the corresponding adaptive bitmap according to the fragment information; a distribution detection module, used for real-time detection of the setting distribution result of the corresponding adaptive bitmap; a reorganization module, used for judging whether the fragment message is completely received according to the fragment information and the setting distribution result of the corresponding adaptive bitmap, and if it is judged that the reception is complete, all the fragment messages corresponding to the adaptive bitmap are reorganized to obtain a reorganized message;
[0008] The processing device further comprises:
[0009] The second judgment module is used to judge whether the bitmap detection period M has been reached; the capacity adjustment module is used to, if it is judged that it has been reached, apply to the memory pool for adjusting the capacity of the corresponding adaptive bitmap according to the detection information of the setting distribution result in the corresponding adaptive bitmap within the bitmap detection period M, and the adjusted capacity of the corresponding adaptive bitmap is applied to the bitmap detection period (M+1); if it is judged that it has not been reached, the capacity of the corresponding adaptive bitmap is not adjusted.
[0010] Yet another aspect of the present invention provides an electronic device, comprising: a processor and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the above-mentioned method for processing fragmented messages.
[0011] On the other hand, the present invention provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the above-mentioned method for processing fragmented messages.
[0012] Yet another aspect of the embodiments of the present invention provides a computer program product comprising instructions, and when the computer program product is run on a computer, each step in the aforementioned method and various possible implementations is executed by the computer.
[0013] The technical solution provided by the embodiment of the present invention has the following beneficial effects:
[0014] By parsing the received fragment message, the fragment information is obtained, and the fragment information includes: service identification, total number of fragments, fragment sequence number, fragment length and IP address; when the fragment sequence number indicates the first fragment message, if the total number of fragments is not less than the capacity of the adaptive bitmap N, the adaptive bitmap (N+1) is applied to the preset memory pool for replacement, and the capacity of the adaptive bitmap (N+1) is greater than the capacity of the adaptive bitmap N; the bits in the corresponding adaptive bitmap are set according to the fragment information; the setting distribution result of the corresponding adaptive bitmap is detected in real time; according to the fragment information and the setting distribution result of the corresponding adaptive bitmap, it is judged whether the fragment message is received completely, and if it is judged that the reception is complete, all the fragment messages corresponding to the corresponding adaptive bitmap are processed. The rows are reorganized to obtain a reorganized message; it is also determined whether the bitmap detection period M has been reached; if it is determined to be reached, then the memory pool is applied for adjusting the capacity of the corresponding adaptive bitmap according to the detection information of the setting distribution result of the corresponding adaptive bitmap within the bitmap detection period M, and the adjusted capacity of the corresponding adaptive bitmap is applied to the bitmap detection period (M+1); if it is determined to be not reached, the capacity of the corresponding adaptive bitmap is not adjusted, so that the capacity size of the bitmap can be dynamically adjusted, avoiding the performance loss and memory waste caused by the use of complex sorting algorithms or fixed bitmaps in the prior art, significantly reducing the processing complexity, thereby greatly improving the efficiency of shard reorganization, especially showing excellent performance improvement when processing a large number of shards. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 A schematic diagram of a process for processing a fragmented message provided by an embodiment of the present invention;
[0016] Figure 2 A schematic flow chart of a method for processing fragmented messages provided in another embodiment of the present invention;
[0017] Figure 3 A schematic diagram of the structure of a fragment message processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0018] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.
[0019] See also Figure 1 An embodiment of the present invention provides a method for processing a fragment message, which comprises the following steps:
[0020] Step 101, parse the received fragment message to obtain fragment information.
[0021] The fragment information includes: service ID, total number of fragments, fragment sequence number, fragment length and IP address.
[0022] Fragmented messages are a common data transmission method in network communications. In the process of network communication, the data to be sent is usually divided into multiple smaller fragmented messages for transmission in the network.
[0023] After receiving the fragment message, the fragment message is parsed to obtain the fragment information, which includes: service identification, total number of fragments, fragment sequence number, fragment length and IP address. The service identification is also called business identification, which is used to distinguish different types of services, such as: download engineering service, operator station write value service, read value service to PLC controller, write value service to PLC controller, historical station periodic synchronization data service, etc. The read value service to PLC controller may include point value reading service, trend reading service, etc. The total number of fragments is used to indicate the total number of fragment messages that the data required to be sent by the service corresponding to the service identification is divided into this time. The fragment sequence number is used to indicate the sequence number of the fragment message in the total number of fragments. The fragment length is used to indicate the size of the fragment message. The IP address is used to indicate the IP address related to the fragment message, such as: source address, destination address. The fragment message can be a fragment message in an industrial control protocol, such as Modbus.
[0024] Step 102, when the fragment sequence number indicates the first fragment message, if the total number of fragments is not less than the capacity of the adaptive bitmap N, apply for the adaptive bitmap (N+1) to the preset memory pool for replacement, and the capacity of the adaptive bitmap (N+1) is greater than the capacity of the adaptive bitmap N.
[0025] According to the fragment sequence number, it is determined whether the currently received fragment message is the first fragment message of this service. If it is determined to be yes, it means that the service has started. Then, the total number of fragments and the capacity of the adaptive bitmap N are determined. If the total number of fragments is not less than the capacity of the adaptive bitmap N, the adaptive bitmap (N+1) is applied to the preset memory pool for replacement, that is, the adaptive bitmap N is replaced with the adaptive bitmap (N+1). The capacity of the adaptive bitmap (N+1) is greater than the capacity of the adaptive bitmap N, thereby realizing the adjustment of the bitmap capacity, thereby avoiding the situation where the total number of fragments corresponding to a certain service suddenly increases and the capacity of the adaptive bitmap N cannot meet the requirements. When adjusting the bitmap capacity, the capacity is expanded or reduced after the bitmap address to realize the adaptive adjustment of the bitmap capacity. The capacity of the adaptive bitmap (N+1) after capacity adjustment can be twice the total number of fragments. If the fragment sequence number does not indicate the first fragment message, step 103 is executed; if the fragment sequence number indicates the first fragment message and the total number of fragments is less than the capacity of the adaptive bitmap N, step 103 is executed.
[0026] The memory pool is to apply for memory allocation of memory blocks for backup before actually using the memory. When there is a new memory demand, the memory blocks are allocated from the memory pool to avoid frequent kernel system calls, so that the memory allocation efficiency can be improved. For the adjustment of the bitmap capacity, the bitmap before the capacity adjustment is called the adaptive bitmap N, and the capacity of the bitmap before the capacity adjustment is called the capacity of the adaptive bitmap N; if the bitmap capacity adjustment occurs, the bitmap after the capacity adjustment is called the adaptive bitmap (N+1), and the capacity of the bitmap after the capacity adjustment is called the capacity of the adaptive bitmap (N+1). In the current bitmap detection cycle, if the adaptive bitmap N appears, if the capacity does not need to be adjusted, the capacity of the adaptive bitmap in the next bitmap detection cycle is still the capacity of the adaptive bitmap N, that is, the capacity of the adaptive bitmap that continues to be used is the same as the capacity of the adaptive bitmap corresponding to the current bitmap detection cycle, until the capacity of the adaptive bitmap needs to be adjusted in a certain bitmap detection cycle or when a certain bitmap detection cycle arrives. In the same bitmap detection cycle, each service has the same adaptive bitmap capacity.
[0027] Step 103: Set the bits in the corresponding adaptive bitmap according to the fragmentation information.
[0028] The corresponding adaptive bitmap refers to adaptive bitmap N or adaptive bitmap (N+1). When the fragment sequence number indicates the first fragment message and the total number of fragments is not less than the capacity of the adaptive bitmap N, the bit in the adaptive bitmap (N+1) is set according to the fragment information; when the fragment sequence number indicates the first fragment message and the total number of fragments is less than the capacity of the adaptive bitmap N, the bit in the adaptive bitmap N is set according to the fragment information, or when the fragment sequence number does not indicate the first fragment message, the bit in the adaptive bitmap N is set according to the fragment information. When setting, the position bits in the corresponding adaptive bitmap are usually set to 1 in order from low to high according to the fragment sequence number.
[0029] Step 104: Detect the placement distribution result of the corresponding adaptive bitmap in real time.
[0030] In the bitmap detection cycle, the placement distribution result of the corresponding adaptive bitmap is detected in real time. When the bitmap detection cycle arrives, the detection information of the placement distribution result of the corresponding adaptive bitmap in the bitmap detection cycle is obtained. When applied, the real-time detection can be performed based on the distribution detection technology. This step can be performed independently of the aforementioned steps 101, 102, 103 and 105, that is, the execution of this step can be parallel to the execution of steps 101, 102, 103 and 105.
[0031] Step 105, judging whether the fragment message is completely received according to the fragment information and the setting distribution result of the corresponding adaptive bitmap, if it is judged that it is completely received, reassembling all the fragment messages corresponding to the corresponding adaptive bitmap to obtain a reassembled message.
[0032] Specifically, whether the fragment message is completely received is determined according to the total number of fragments and the setting distribution result of the corresponding adaptive bitmap. If it is determined to be completely received, all the fragment messages corresponding to the corresponding adaptive bitmap are reassembled to obtain a reassembled message.
[0033] The method further includes:
[0034] Step 106, determining whether the bitmap detection period has arrived.
[0035] Determine whether the bitmap detection period has arrived. The bitmap detection period is used to indicate whether the capacity of the corresponding adaptive bitmap is adjusted, that is, whether it is necessary to apply to the memory pool for replacement of the adjusted bitmap, that is, whether it is necessary to use the bitmap after the adjusted capacity to replace the bitmap before adjustment (or current) capacity. In other words, whether it is necessary to use the adaptive bitmap (N+1) to replace the adaptive bitmap N. The bitmap detection period can also be called the bitmap adaptive period, and the size can be multiple service periods. The service period refers to the time period from the start to the end of the service. The number of service cycles can be 10 or other values. Since the downloading engineering service period is longer than other service periods, the service period is usually set to the downloading engineering service period (or downloading engineering business period). The number of data fragments caused by the downloading engineering business is more than that of other businesses. The downloading engineering business fragments generally do not exceed 1000. Therefore, the capacity of the bitmap is initially set to 1000, that is, the adaptive bitmap. Figure 1 The capacity is 1000.
[0036] Step 107, if it is determined to have been reached, then an application is made to the memory pool to adjust the capacity of the corresponding adaptive bitmap according to the detection information of the setting distribution result in the corresponding adaptive bitmap within the bitmap detection period M, and the adjusted capacity of the corresponding adaptive bitmap is applied to the bitmap detection period (M+1).
[0037] When the bitmap detection period M arrives, an application is made to the memory pool to adjust the capacity of the corresponding adaptive bitmap according to the detection information of the setting distribution result in the corresponding adaptive bitmap within the bitmap detection period M, that is, within the bitmap detection period (M+1), the adaptive bitmap with adjusted capacity is used, and the same capacity size as the adaptive bitmap corresponding to the bitmap detection period M is no longer used.
[0038] Specifically, one or more services may appear in a bitmap detection cycle. When a service appears, the set distribution result of the corresponding adaptive bitmap of the service is detected to obtain the corresponding detection information, that is, one service corresponds to one detection information, that is, the adaptive bitmap corresponding to each service is used The principle of distributed detection is used to obtain the detection information. Usually, the detection is based on the preset high and low lines. Based on this, when the corresponding adaptive bitmap in the previous bitmap detection cycle is the adaptive bitmap N, the detection information of the corresponding adaptive bitmap of each service in the previous bitmap detection cycle is obtained, and each service has the same service identifier; if there is detection information in each detection information that is set to 1 within the high line, then apply for a new adaptive bitmap capacity from the memory pool for replacement, that is, use the adaptive bitmap (N+1) to replace the corresponding adaptive bitmap N, the capacity of the new adaptive bitmap is greater than the capacity of the corresponding adaptive bitmap, and the capacity of the new adaptive bitmap is used in the current bitmap detection cycle, that is, the capacity of the new adaptive bitmap is used in the current bitmap detection cycle. The capacity of the adaptive bitmap used is the same as the capacity of the new adaptive bitmap; if the low-order lines in each detection information are not set to 1, a new adaptive bitmap is applied to the memory pool for replacement, and the capacity of the new adaptive bitmap is smaller than the corresponding adaptive bitmap capacity (i.e., the capacity of the aforementioned adaptive bitmap N). The capacity of the new adaptive bitmap is used in the current bitmap detection cycle, that is, the adaptive bitmap capacity used in the current bitmap detection cycle is the same as the capacity of the new adaptive bitmap; otherwise, the adaptive bitmap capacity used in the current bitmap detection cycle is still the same as the adaptive bitmap capacity used in the previous bitmap detection cycle.
[0039] Among them, when the capacity is adjusted to a large capacity, the capacity of the adjusted adaptive bitmap is twice the maximum total number of shards in each service within the bitmap detection cycle. The high line belongs to the high range, which can be 70% to 80% of the bitmap index in the corresponding adaptive bitmap, which can be 70%, 75%, 80%, etc.; the low line belongs to the low range, which can be 20% to 30% of the bitmap index in the corresponding adaptive bitmap, which can be 20%, 25%, 30%, etc.
[0040] For example, in the first bitmap detection cycle (or bitmap detection cycle 1), the bitmap capacity to be used by each service is 1000, which is used as the adaptive bitmap capacity of the first bitmap detection cycle. Figure 1 capacity, that is, the adaptive bit Figure 1 Refers to the bitmap that each service will use in the current bitmap detection cycle. There are two download engineering services (or download engineering services), with the number of fragmented messages being 5 and 12 respectively. Since neither exceeds the adaptive bitmap, Figure 1If the existing capacity is 1000, there is no need to adjust the capacity of the adaptive bitmap in the manner of step 102. In the first bitmap detection cycle, the bitmap distribution detection is performed on each download engineering service. The 5 and 12 corresponding to the two download engineering services are both within the low bit line of the adaptive bitmap size of 1000, so the bitmap capacity is reduced. The reduced adaptive bitmap capacity is 24, which is called a new adaptive bitmap with a capacity smaller than the corresponding adaptive bitmap capacity. Specifically, the adjusted bitmap capacity is twice the maximum number of fragmented messages (i.e., 12) in the first bitmap detection cycle.
[0041] In the second bitmap detection cycle (or bitmap detection cycle 2), the adaptive bitmap capacity that each service is about to use is 24, which is used as the capacity of the adaptive bitmap of the second bitmap detection cycle, that is, the capacity of the adaptive bitmap used in the first bitmap detection cycle is adjusted based on the detection information of each service in the first bitmap detection cycle, that is to say: the capacity of the bitmap is the same as the capacity of the bitmap adjusted based on the first bitmap detection cycle, and a download engineering service appears, and the number of fragmented messages is 500. Since it exceeds the size of the adaptive bitmap at this time (i.e. 24), the capacity of the bitmap is expanded, and the expanded bitmap capacity is 1000. Specifically, the adjusted bitmap capacity is twice the maximum number of fragmented messages (i.e. 500) in the second bitmap detection cycle.
[0042] In the third bitmap detection cycle (or bitmap detection cycle 3), the adaptive bitmap capacity that each service is about to use is 1000, which is used as the adaptive bitmap capacity of the third bitmap detection cycle. A downloading engineering service appears, and the number of fragmented messages is 917. In the third bitmap detection cycle, the bitmap distribution detection is performed on the downloading engineering service, resulting in the bits within 70% of the high bit line of the adaptive bitmap size 1000 being set to 1, so the bitmap capacity is expanded, and the expanded bitmap capacity is 1834. Specifically, the adjusted bitmap capacity is twice the maximum number of fragmented messages (i.e., 917) in the third bitmap detection cycle.
[0043] Step 108: If it is determined that the capacity has not been reached, the capacity of the corresponding bitmap is not adjusted.
[0044] The judgment process of this step can refer to the relevant description process of step 105, which will not be repeated here. When it is judged that the bitmap detection cycle has not arrived, the capacity of the corresponding adaptive bitmap is not adjusted, that is, the adjusted adaptive bitmap is not applied to the memory pool for replacement, that is, in the current bitmap detection cycle (that is, bitmap detection cycle (M+1)), the capacity size of the corresponding adaptive bitmap corresponding to the previous bitmap detection cycle (that is, bitmap detection cycle M) is still used.
[0045] Through the above method, the bitmap size can be dynamically adjusted according to the collection of fragmented messages. When the high-order bits in the bitmap reach the set high-order line, the bitmap automatically expands; when the setting of the low-order bits shows that the memory utilization rate is low, the bitmap shrinks. Therefore, this method can be called an adaptive size bitmap method based on distribution detection, which avoids the performance loss and memory waste caused by the use of complex sorting algorithms or fixed bitmaps in the prior art, significantly reduces the processing complexity, and thus greatly improves the efficiency of fragment reorganization, especially when processing a large number of fragments. Excellent performance improvement is shown. It should be noted that steps 106-108 can be performed independently of the aforementioned steps 101, step 102, step 103 and step 105, that is, the execution of these steps can be parallel to the execution of steps 101, step 102, step 103 and step 105.
[0046] The specific implementation of step 106 may also be:
[0047] When the time corresponding to the bitmap detection period arrives, determine whether the service start signal and the service end signal are received. If it is determined that the service start signal is received but the service end signal is not received, wait until the service end signal is received and determine that the bitmap detection period has arrived.
[0048] In other embodiments, the specific process of the method for processing fragmented messages is as follows:
[0049] Parse the received fragment message to obtain fragment information, which includes: service identification, total number of fragments, fragment sequence number, fragment length and IP address; set the bit in the corresponding adaptive bitmap according to the fragment information; judge whether the fragment message is received completely according to the fragment information and the setting distribution result of the corresponding adaptive bitmap, if it is judged that the reception is complete, reorganize all the fragment messages corresponding to the corresponding adaptive bitmap to obtain the reorganized message. Before setting the bit in the corresponding adaptive bitmap according to the fragment information, it also includes: comparing the total number of fragments with the capacity of the adaptive bitmap N, when the total number of fragments is not less than the capacity of the adaptive bitmap N, generating the first capacity adjustment information, the first capacity adjustment information is used to apply to the preset memory pool for the adaptive bitmap (N+1) for replacement, the capacity of the adaptive bitmap (N+1) is greater than the capacity of the adaptive bitmap N.
[0050] Detect the position distribution result of the corresponding adaptive bitmap in real time; determine whether the first capacity adjustment information is received, and adjust the bitmap capacity accordingly in response to the first capacity adjustment information; determine whether the bitmap detection period M has arrived; if it is determined to be arrived, apply to the memory pool for adjusting the capacity of the corresponding adaptive bitmap according to the detection information of the position distribution result of the corresponding adaptive bitmap within the bitmap detection period M, and the adjusted capacity of the corresponding adaptive bitmap is applied to the bitmap detection period (M+1); if it is determined to be not arrived, do not adjust the capacity of the corresponding adaptive bitmap.
[0051] After step 101, the method further includes:
[0052] According to the service identifier and the IP address, it is determined whether there is a hash node corresponding to the service identifier and the IP address in the preset hash table; if it is determined to exist, jump to step 102; if it is determined not to exist, create an adaptive bitmap. Figure 1 and create new hash nodes in the hash table, adaptive bits Figure 1 The address of the new hash node corresponds to the adaptive bit. Figure 1 It can be the initial bitmap. The size of the initial bitmap capacity can be set to 1000, which is determined based on the number of data slices generated by the download engineering business. When the work starts, the initial bitmap is the first bitmap.
[0053] Specifically, the hash table hash_table is initialized, and the bucket content of hash_table is initialized to NULL.
[0054] After receiving the fragment message, the fragment header of the fragment message is parsed. The parsed information includes: service identifier (or service ID) requestID, fragment size (or fragment length) frag_len, total number of fragments frag_number, fragment sequence number frag_index and IP address. The fragment size frag_len can be used to determine how much memory to apply for when caching this fragment message. After applying for memory, the memory pointer is placed in the corresponding position of the fragment pointer array. For example, the currently received fragment is the third fragment, that is, the fragment sequence number frag_index = 3, and the fragment length frag_len = 1200, then 1200 bytes of memory will be applied for caching this fragment, and then the pointer of this memory will be placed in the third position of the fragment pointer array. The total number of fragments frag_number is used to detect whether the fragment message is received completely when the fragment message is reassembled.
[0055] Calculate hash_key with IP address and service ID as keywords, and use hash function to calculate hash value. Double hashing structure can be used for calculation. Double hashing structure can avoid hash conflicts and ensure that each bucket stores the address of bitmap with the same service ID. That is, when the fragment message of a service ID is collected and passed in, the IP address and service ID are first hashed for search. It should be noted that the hash function can be: (hash1(key)+i*hash2(key))%TABLE_SIZE, hash1(key) represents the first hash function, hash2(key) represents the second hash function, TABLE_SIZE represents the size of the hash table, i is an iteration number, usually starting from 0, and incremented each time a hash conflict occurs.
[0056] Look for a hash node for this service ID in the preset hash table. If no hash node is found, it means that the fragment message of this service ID is received for the first time. At this time, you should try to insert the hash node in the hash table. When trying to insert a hash node, first determine whether a hash conflict occurs. Use the hash value calculated above to find the data in the bucket in the hash table hash_table. If the value is NULL, it means that this hash node is not used and no hash conflict occurs. If the search data is not NULL, it means that a hash conflict has occurred. At this time, you need to reselect the i value in the hash function, recalculate the hash value hash_value, and then insert it. It should be noted that when the hash node is inserted, the insertion time of the hash node will be recorded to form the hash node timestamp last_update_time. If it is found, you need to update the hash node timestamp last_update_time in the hash node, that is, update the hash node timestamp to the time of the query at that time, which can also be called the time of querying this hash node. This timestamp is used for the hash aging thread, that is, for the timeout aging release of this hash structure.
[0057] When a hash node needs to be inserted, the address of the bitmap bit_map (i.e. the adaptive bitmap) is requested from the memory pool. Figure 1The address of the initial bitmap and the address of the fragment pointer array fragmentArray are used to apply for the capacity of 1000 fragment messages. The bitmap capacity is size_t*bitmap. Define a structure named hash_node, fill it with the bitmap address, the address of the fragment pointer array fragmentArray, the service ID and the hash node timestamp last_update_time, and insert hash_node into the hash table. That is, the inserted content of the hash node is a structure named hash_node. The bitmap is used to record the collection of fragment messages for each service ID. The address of the fragment pointer array fragmentArray is used to correspond to the memory storage location of each fragment message.
[0058] It should be noted that the fragment pointer array is used to cache fragment messages, and the bitmap is used to determine which fragment message has been received. For example, if the third fragment message in a data packet is currently received, this fragment message is stored in the third position of the fragment pointer array, and at the same time, the third bit of the bitmap is set to 1. In this way, before the fragments are reassembled, it can be determined whether all the fragment messages have been received by checking whether each bit of the bitmap is set to 1.
[0059] According to the fragment sequence number frag_index of the current fragment message, the frag_index position starting from the low bit in the bitmap is set to 1, and a piece of memory is requested from the memory pool to cache the current fragment message, and the pointer of this memory is placed in the corresponding position of the fragment pointer array fragmentArray[frag_index].
[0060] Whether the fragment message is received completely. The judgment process can be: use the bitmap bit_map to judge whether the fragment message is received completely, specifically, whether the loop judgment condition bitmap[i]==(size_t)(-1) is satisfied. If it is judged that the fragment message is received completely, the memory addresses in the fragment pointer array fragmentArray are combined together to complete the fragment reassembly, and this process can be executed by the fragment reassembly thread.
[0061] Sometimes failures occur during business reorganization, so see Figure 2 The embodiment of the present invention further provides a method for processing fragmented messages, which, based on the above steps, further includes: a service status monitoring step, specifically including the following contents:
[0062] Step 201, parse the received fragment message to obtain fragment information, when the fragment sequence number indicates the first fragment message.
[0063] At this time, the fragment sequence number in the fragment information indicates the first fragment message, corresponding to the service start flag in the service start and end flag in step 203.
[0064] Step 202, judging whether the fragment message is completely received according to the fragment information and the setting distribution result of the corresponding adaptive bitmap, if it is judged that it is completely received, reassembling all the fragment messages corresponding to the corresponding adaptive bitmap to obtain a reassembled message.
[0065] At this time, the fragmented messages are reassembled to obtain a reassembled message, which corresponds to the service end flag in the service start and end flag in step 203.
[0066] Step 203, generate a reorganization status message, which includes: a service identifier, an index identity identifier corresponding to the service identifier, and service start and end flags. The service start and end flags include: a service start flag or a service end flag. The service start flag is used to indicate that the first fragment message of the service corresponding to the service identifier has been received, and the service end flag is used to indicate that all fragment messages of the service have been received completely.
[0067] Step 204: When the service start and end mark is a service start mark, the reassembly timestamp corresponding to the reassembly state information is updated.
[0068] Step 205: monitor the reassembly process according to the service start and end flags, the preset service status monitoring period and the reassembly timestamp.
[0069] Specifically, the index identity UniqueRequestID corresponding to the service identifier is used to represent the index of the service identifier when it is stored. It can be incremented from 0. The first service inserted into the hash_table has a UniqueRequestID of 0. The UniqueRequestID of the services inserted later is 1, 2, 3, etc. You can apply for a structure instance, fill it with the service ID, UniqueRequestID, and the service start and end flags, and then transmit it to the business monitoring thread through the message queue. The business monitoring thread updates the service details mapping table according to the message content in the message queue. Different message contents can reflect the status of the service in real time. The business monitoring thread is used to perform the above-mentioned service status monitoring steps. The index identity UniqueRequestID corresponding to the service identifier can be used to store the service status in the service details mapping table, and update the service start and end flags according to whether the service starts or ends. The index identity UniqueRequestID corresponding to the service identifier is used to act as an index of the array to ensure that each service identifier is correctly stored in the corresponding position.
[0070] In the application, first take out the reorganization status message sent from the message queue, check the message, and locate the item with the value of UniqueRequestID as the index in the service details mapping table. If the message contains content that represents the start of the business, the aforementioned service start and end flags are set to 1, which is used to indicate that the first shard has been received, indicating that the shard reorganization thread has started to reorganize the shards. This hash node needs to be monitored and the reorganization timestamp is updated with the current time. If the message contains content that represents the end of the business, the aforementioned service start and end flags are set to 0, indicating that the business reorganization is over and there is no need to pay attention to this hash node.
[0071] The business monitoring thread periodically traverses the service details mapping table to determine whether the service reorganization time corresponding to each service identifier has timed out. If it is determined to be timed out, it indicates that the business reorganization corresponding to this service identifier has failed. For timed-out services, a fault log is reported to the upper computer. The log contains the service identifier, which indicates which service has failed. As for how to determine whether the reorganization timestamp of the service identifier has timed out, it can be: if the service start and end marks are the start marks, and the difference between the current time and the reorganization timestamp is greater than the service status monitoring period, then it is determined that the service reorganization time corresponding to the service identifier has timed out.
[0072] Due to the complexity of the industrial control DCS system, a large number of different services will be communicating at the same time. During the fragmentation reorganization process, if a service fragment message is incomplete due to network reasons, the system will show multiple related errors. By periodically checking the fragmentation reorganization status mentioned above, it is possible to quickly detect abnormalities in the fragmentation reorganization process, such as fragment loss or timeout. When a fault is found, fault information will be generated, which includes detailed logs such as the service ID and timestamp of the fault, and then the information will be reported to the host computer, so as to facilitate engineers to locate and analyze the fault, which is conducive to the fault service to be handled in the first time, and can effectively improve the maintainability and fault handling efficiency of the industrial control DCS system.
[0073] The implementation method of monitoring the message reassembly process according to the service start and end marks, the preset service status monitoring cycle and the reassembly timestamp includes:
[0074] When the service start and end mark is the service start mark, determine whether the difference between the current time and the reassembly timestamp is greater than the service status monitoring period; if it is determined to be greater, a fault message is issued, and the fault message is used to indicate that a fault has occurred in the message reassembly process; when the service start and end mark is the service end mark, the monitoring of the message reassembly process is terminated. In other words, when the service start and end mark is the service end mark, the monitoring of the message reassembly process of the service is ignored. The service status monitoring period can be 1 download engineering service period or 1.5 times the download engineering service period.
[0075] In practical applications, it is possible that a certain slice in a service corresponding to a certain service identifier is lost due to a network failure, so the method further includes: a hash aging step, which includes the following contents:
[0076] According to the aging cycle, hash node timestamp and current time, determine whether to release the memory corresponding to the hash node.
[0077] Specifically, when a fragment message is received, the hash node timestamp is updated; it is determined whether the difference between the current time and the hash node timestamp is greater than the aging cycle. If it is determined to be greater, the memory corresponding to the hash node is released, otherwise it is not released, that is, the hash storage node is aged based on the aging cycle. The memory corresponding to the hash node can be: bitmap memory, memory for storing fragment messages, memory in the hash node, etc. The hash node timestamp is updated each time a fragment message is received.
[0078] The hash node timestamp refers to the time when the hash node was inserted or the time when the hash node was last queried. When the hash node is inserted, the timestamp will record the time when the hash node was inserted. Each subsequent query to this hash node will update the timestamp to the time when the query was made. Based on the aging cycle, each hash node is traversed, and the time difference is calculated based on the current time and the hash node timestamp. If the preset aging cycle is exceeded, the hash node will be released.
[0079] For example: take index as the index, locate the index-th node in hash_table, read the hash node timestamp last_update_time in the node, and then compare it with the current time current_time. If it exceeds the set aging period, it is determined that aging is required, that is, the memory pointed to by the pointer in the node and the structure instance in the node are released; if it does not exceed the set aging period, index index+1 and continue to search for the next node in hash_table.
[0080] See also Figure 3 An embodiment of the present invention provides a method and device for processing fragmented messages, which is used to execute the method for processing fragmented messages provided by the above embodiments, and includes: a parsing module 301, a first judgment module 302, a setting module 303, a distribution detection module 304, a reorganization module 305, a second judgment module 306 and a capacity adjustment module 307.
[0081] Among them, the parsing module 301 is used to parse the received fragment message to obtain the fragment information, and the fragment information includes: service identification, total number of fragments, fragment sequence number, fragment length and IP address. The first judgment module 302 is used to apply for adaptive bitmap (N+1) to the preset memory pool for replacement when the fragment sequence number indicates the first fragment message, if the total number of fragments is not less than the capacity of the adaptive bitmap N, and the capacity of the adaptive bitmap (N+1) is greater than the capacity of the adaptive bitmap N. The setting module 303 is used to set the bit in the corresponding adaptive bitmap according to the fragment information. The distribution detection module 304 is used to detect the setting distribution result of the corresponding adaptive bitmap in real time. The reorganization module 305 is used to judge whether the fragment message is received completely according to the fragment information and the setting distribution result of the corresponding adaptive bitmap. If it is judged that the reception is complete, all the fragment messages corresponding to the adaptive bitmap are reorganized to obtain the reorganized message. Before parsing the received fragment message, the processing device further includes: a second judgment module 306 and a capacity adjustment module 307. The second judgment module 306 is used to judge whether the bitmap detection period M has arrived. The capacity adjustment module 307 is used to apply to the memory pool for adjusting the capacity of the corresponding adaptive bitmap according to the detection information of the setting distribution result in the corresponding adaptive bitmap in the bitmap detection period M if it is judged that it has arrived, and the adjusted capacity of the corresponding adaptive bitmap is applied to the bitmap detection period (M+1); if it is judged that it has not arrived, the capacity of the corresponding adaptive bitmap is not adjusted.
[0082] Optionally, after parsing the received fragment message, the processing device further includes: a service storage module, which is used to determine whether there is a hash node corresponding to the service identifier and the IP address in the preset hash table according to the service identifier and the IP address; if it is determined to exist, jump to the first judgment module 302 for execution; if it is determined not to exist, create an adaptive bit Figure 1 and create a new hash node in the hash table, and the address of adaptive problem 1 corresponds to the new hash node.
[0083] Optionally, the capacity adjustment module 307 is used to obtain detection information of the corresponding adaptive bitmap of each service within the bitmap detection cycle based on preset high and low lines, and each service has the same service identifier; if there are high lines in each detection information that are all set to 1, then a new adaptive bitmap with a capacity greater than the capacity of the corresponding adaptive bitmap is applied to the memory pool for replacement, and the new adaptive bitmap with a capacity greater than the capacity of the corresponding adaptive bitmap is applied to the bitmap detection cycle (M+1); otherwise, if the low lines in each detection information are not set to 1, then a new adaptive bitmap with a capacity less than the capacity of the corresponding adaptive bitmap is applied to the memory pool for replacement, and the new adaptive bitmap with a capacity less than the capacity of the corresponding adaptive bitmap is applied to the bitmap detection cycle (M+1).
[0084] Optionally, the capacity of a new adaptive bitmap that is larger than the capacity of a corresponding adaptive bitmap is twice the maximum total number of shards in each service within the bitmap detection period; the capacity of a new adaptive bitmap that is smaller than the capacity of a corresponding adaptive bitmap is twice the maximum total number of shards in each service within the bitmap detection period.
[0085] Optionally, the device also includes: a service status monitoring module, which is used to generate a reassembly status message, the message including: a service identifier, an index identity identifier corresponding to the service identifier, and a service start and end flag, the service start and end flags including a service start flag and a service end flag, the service start flag is used to indicate that the first fragment message has been received, and the service end flag is used to indicate that all fragment messages have been received completely; when the service start and end flags are service start flags, the reassembly timestamp corresponding to the reassembly status information is updated; the message reassembly process is monitored according to the service start and end flags, the preset service status monitoring period and the reassembly timestamp.
[0086] Among them, the service status monitoring unit is used to monitor the message reassembly process according to the service start and end flags, the preset service status monitoring cycle and the reassembly timestamp, specifically: when the service start and end flags are the service start flags, it is used to judge whether the difference between the current time and the reassembly timestamp is greater than the service status monitoring cycle; if it is judged to be greater, a fault message is issued, and the fault message is used to indicate that a fault occurs in the message reassembly process; when the service start and end flags are the service end flags, the monitoring of the message reassembly process is ended.
[0087] Optionally, the device also includes: a hash aging module, which is used to update the hash node timestamp when receiving a fragmentation message; and determine whether to release the memory corresponding to the hash node based on the aging period, the hash node timestamp and the current time.
[0088] In other embodiments, the fragment message processing device further includes: a fragment reassembly module and an adaptive bitmap adjustment module. The fragment reassembly module is used to: reassemble the fragment message, and the process of reassembling the fragment message can be as follows: parse the received fragment message to obtain fragment information, and the fragment information includes: service identification, total number of fragments, fragment sequence number, fragment length and IP address; set the bit in the corresponding adaptive bitmap according to the fragment information; detect the setting distribution result of the corresponding adaptive bitmap in real time; judge whether the fragment message is received completely according to the fragment information and the setting distribution result of the corresponding adaptive bitmap, and if it is judged that the fragment message is received completely, then reassemble all the fragment messages corresponding to the corresponding adaptive bitmap to obtain a reassembled message. The shard reorganization module is also used to compare the total number of shards with the capacity of the adaptive bitmap N. When the total number of shards is not less than the capacity of the adaptive bitmap N, first capacity adjustment information is generated. The first capacity adjustment information is used to instruct the adaptive bitmap adjustment module to apply to the preset memory pool for adaptive bitmap (N+1) for replacement. The capacity of the adaptive bitmap (N+1) is greater than the capacity of the adaptive bitmap N.
[0089] The adaptive bitmap adjustment module is used to receive the first capacity adjustment information sent by the shard reorganization module, and then adjust the bitmap capacity accordingly. The adaptive bitmap adjustment module is also used to: detect the position distribution result of the corresponding adaptive bitmap in real time; determine whether the bitmap detection period M has arrived; if it is determined to have arrived, apply to the memory pool for adjusting the capacity of the corresponding adaptive bitmap according to the detection information of the position distribution result of the corresponding adaptive bitmap within the bitmap detection period M, and the adjusted capacity of the corresponding adaptive bitmap is applied to the bitmap detection period (M+1); if it is determined to have not arrived, the capacity of the corresponding adaptive bitmap is not adjusted.
[0090] It should be noted that: the fragmented message processing device provided in the above embodiment only uses the division of the above functional modules as an example when processing fragmented messages. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the fragmented message processing device provided in the above embodiment and the fragmented message processing method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0091] An embodiment of the present invention provides an electronic device, comprising: a memory and a processor. The processor is connected to the memory and is configured to execute the above-mentioned method for processing fragmented messages based on instructions stored in the memory. The number of processors may be one or more, and the processor may be single-core or multi-core. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as a read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory may be an example of the following computer-readable medium.
[0092] An embodiment of the present invention provides a computer-readable storage medium, on which at least one instruction, at least one program, code set or instruction set is stored, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the above-mentioned method for processing fragmented messages. Computer-readable storage media include: permanent and non-permanent, removable and non-removable media can implement information storage by any method or technology. Information can be a computer-readable instruction, a data structure, a module of a program or other data. Examples of computer storage media include, but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk-read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassette, disk storage or other magnetic storage device or any other non-transmission medium, which can be used to store information that can be accessed by a computing device.
[0093] An embodiment of the present invention provides a computer program product including instructions. When the computer program product is run on a computer, the above-mentioned method for processing fragmented messages is executed by the computer.
[0094] It is known from common technical knowledge that the present invention can be implemented by other embodiments that do not deviate from its spirit or essential features. Therefore, the above disclosed embodiments are only illustrative in all respects and are not exclusive. All changes within the scope of the present invention or within the scope equivalent to the present invention are included in the present invention.
Claims
1. A method for processing fragmented messages, characterized in that: include: Parse the received fragment message to obtain fragment information, which includes: service identification, total number of fragments, fragment sequence number, fragment length and IP address; When the fragment sequence number indicates the first fragment message, if the total number of fragments is not less than the capacity of the adaptive bitmap N, apply to the preset memory pool for adaptive bitmap (N+1) for replacement, and the capacity of the adaptive bitmap (N+1) is greater than the capacity of the adaptive bitmap N; Setting a bit in a corresponding adaptive bitmap according to the slicing information; Real-time detection of the placement distribution results of the corresponding adaptive bitmap; Determining whether the fragment message is completely received according to the fragment information and the setting distribution result of the corresponding adaptive bitmap, and if it is determined that the fragment message is completely received, reassembling all the fragment messages corresponding to the corresponding adaptive bitmap to obtain a reassembled message; Also includes: Determine whether the bitmap detection period M has arrived; If it is determined to be reached, then applying to the memory pool for adjusting the capacity of the corresponding adaptive bitmap according to the detection information of the setting distribution result of the corresponding adaptive bitmap in the bitmap detection period M, and the adjusted capacity of the corresponding adaptive bitmap is applied to the bitmap detection period (M+1); If it is determined that the capacity is not reached, the capacity of the corresponding adaptive bitmap is not adjusted.
2. The processing method according to claim 1, characterized in that: After parsing the received fragment message, the method further includes: According to the service identifier and the IP address, determining whether there is a hash node corresponding to the service identifier and the IP address in a preset hash table; If it is judged to exist, jump to the step when the fragment sequence number indicates the first fragment message; If it is determined that it does not exist, an adaptive bitmap 1 is created and a new hash node is created in the hash table, and the address of the adaptive bitmap 1 corresponds to the new hash node.
3. The processing method according to claim 1, characterized in that: The applying to the memory pool for adjusting the capacity of the corresponding adaptive bitmap according to the detection information of the setting distribution result in the corresponding adaptive bitmap within the bitmap detection period M includes: Based on the preset high bit line and low bit line, acquiring the detection information of the corresponding adaptive bitmap of each service within the bitmap detection period M, each of the services having the same service identifier; If all bits within the high bit lines in each of the detection information are set to 1, a new adaptive bitmap with a capacity greater than the capacity of the corresponding adaptive bitmap is requested from the memory pool for replacement, and the new adaptive bitmap with a capacity greater than the capacity of the corresponding adaptive bitmap is applied in the bitmap detection cycle (M+1); If none of the low bit lines in the detection information is set to 1, a new adaptive bitmap with a capacity smaller than the corresponding adaptive bitmap capacity is requested from the memory pool for replacement, and the new adaptive bitmap with a capacity smaller than the corresponding adaptive bitmap capacity is applied to the bitmap detection cycle (M+1).
4. The processing method according to claim 3, characterized in that: The capacity of the new adaptive bitmap, which is larger than the capacity of the corresponding adaptive bitmap, is twice the maximum total number of slices in each service within the bitmap detection period; The capacity of the new adaptive bitmap that is smaller than the capacity of the corresponding adaptive bitmap is twice the maximum total number of fragments in each service within the bitmap detection period M.
5. The processing method according to claim 1, characterized in that: Also includes: Generate a reassembly status message, the reassembly status message including: the service identifier, an index identity identifier corresponding to the service identifier, and service start and end flags, the service start and end flags including a service start flag and a service end flag, the service start flag is used to indicate that the first fragment message is received, and the service end flag is used to indicate that all fragment messages are completely received; When the service start and end mark is the service start mark, updating the reorganization timestamp corresponding to the reorganization state information; The message reassembly process is monitored according to the service start and end marks, the preset service status monitoring cycle and the reassembly timestamp.
6. The processing method according to claim 5, characterized in that: The monitoring of the message reassembly process according to the service start and end flags, the preset service status monitoring period and the reassembly timestamp comprises: When the service start and end mark is the service start mark, determining whether the difference between the current time and the reorganization timestamp is greater than the service status monitoring period; If it is judged to be greater than, a fault message is issued, wherein the fault message is used to indicate that a fault occurs in the message reassembly process; When the service start and end mark is the service end mark, the monitoring of the message reassembly process ends.
7. The processing method according to claim 2, characterized in that: Also includes: When receiving a fragment message, update the hash node timestamp; According to the aging cycle, the hash node timestamp and the current time, it is determined whether to release the memory corresponding to the hash node.
8. A fragment message processing device, characterized in that: include: The parsing module is used to parse the received fragment message to obtain fragment information, wherein the fragment information includes: service identification, total number of fragments, fragment sequence number, fragment length and IP address; A first judgment module is used for applying to a preset memory pool for an adaptive bitmap (N+1) for replacement when the fragment sequence number indicates the first fragment message and if the total number of fragments is not less than the capacity of the adaptive bitmap N, and the capacity of the adaptive bitmap (N+1) is greater than the capacity of the adaptive bitmap N; A setting module, used for setting the bits in the corresponding adaptive bitmap according to the slice information; A distribution detection module, used for detecting the placement distribution result of the corresponding adaptive bitmap in real time; a reassembly module, configured to determine whether the fragment message is completely received according to the fragment information and the setting distribution result of the corresponding adaptive bitmap, and if it is determined that the fragment message is completely received, reassembling all the fragment messages corresponding to the adaptive bitmap to obtain a reassembled message; The processing device also includes: The second judgment module is used to judge whether the bitmap detection period M has arrived; The capacity adjustment module is used to apply to the memory pool for adjusting the capacity of the corresponding adaptive bitmap according to the detection information of the setting distribution result in the corresponding adaptive bitmap within the bitmap detection period M if it is judged that the capacity has been reached, and the adjusted capacity of the corresponding adaptive bitmap is applied to the bitmap detection period (M+1); if it is judged that the capacity has not been reached, the capacity of the corresponding adaptive bitmap is not adjusted.
9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory for storing executable instructions of the processor; The processor is configured to execute the method for processing fragmented messages described in any one of claims 1 to 7.
10. A computer program product comprising instructions, characterized in that When the computer program product runs on a computer, the method for processing fragmented messages according to any one of claims 1 to 7 is executed by the computer.