Message deduplication method and device based on 5G servitization interface

Abstract

The invention provides a message deduplication method and device based on a 5G service interface, and the method comprises the steps: carrying out the initialization of flow node information and the initialization of a TCP bidirectional serial number according to an initial message in a first data stream under the condition that the first data stream is transmitted through the 5G service interface according to an HTTP / 2 protocol; for the to-be-transmitted message and the recorded message under the same flow ID, performing target comparison on the to-be-transmitted message and the recorded message, and updating the first data flow according to a comparison result to obtain a second data flow; and identifying the second data stream by adopting a deep message processing system based on a stream ID identification technology, and rejecting repeatedly identified messages in the second data stream according to an identification result to obtain a third data stream. According to the method provided by the invention, the memory occupied in the de-duplication process is smaller, the consumed performance is lower, the division granularity of the de-duplication area is lower, and the de-duplication efficiency and accuracy of the message are improved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a message deduplication method and device based on a 5G service interface. Background Art

[0002] The fifth generation mobile communication technology (5G) adopts the service-based architecture (SBA), and the main signaling interfaces (N11 / N16 / N8 / N12, etc.) all use the HTTP / 2 protocol. When performing in-depth processing on these interfaces, it is necessary to face the problem of duplicate packets caused by the protocol's own retransmission mechanism and repeated collection. In particular, the large number of duplicate packets generated by repeated collection will affect the processing flow and increase unnecessary memory and performance overhead.

[0003] In the prior art, the deduplication process is generally implemented based on TCP streams, and the specific implementation methods are divided into: message content hash table building and matching method, TCP stream hash table building and matching method, and five-tuple zoning and timestamp verification and sequence number matching method; these methods generally require independent message deduplication processing, which occupies a certain amount of memory and performance overhead. In addition, these methods have a large matching granularity and a high probability of deduplication misjudgment, which reduces the efficiency and accuracy of message deduplication, and thus reduces the quality of communication transmission.

[0004] In response to the above problems, how to make the deduplication process more lightweight and smaller in granularity is a problem that needs to be solved in this technical field in order to address the impact of duplicate messages generated by retransmission or repeated collection using the 5G service-oriented interface protocol. Summary of the invention

[0005] The present invention provides a message deduplication method and device based on a 5G service interface, which are used to solve the defects of the prior art in that when processing a large number of duplicate packets generated by 5G communication, the memory usage and performance overhead are large, and the probability of misjudgment of deduplication is high, resulting in low message deduplication efficiency and accuracy, thereby improving the message deduplication efficiency and accuracy.

[0006] The present invention provides a message deduplication method based on a 5G service-oriented interface, comprising: When the first data stream is transmitted through the 5G service-oriented interface according to the HTTP / 2 protocol, the stream node information and the TCP bidirectional sequence number are initialized according to the initial message in the first data stream; the stream node information includes a stream ID and a TCP bidirectional sequence number, and different messages correspond to different sequence numbers and different stream IDs; For the message to be transmitted and the recorded message under the same flow ID, target comparison is performed on the message to be transmitted and the recorded message, and the first data flow is updated according to the comparison result to obtain the second data flow; the target comparison includes at least one of sequence number comparison and direction comparison of the message; the message to be transmitted belongs to the first data flow; The second data stream is identified by a deep message processing system based on stream ID identification technology, and repeated identified messages in the second data stream are removed according to the identification result to obtain a third data stream.

[0007] According to a message deduplication method based on a 5G service-oriented interface provided by the present invention, the target comparison includes sequence number comparison and direction comparison; The step of comparing the message to be transmitted with the recorded message and updating the first data stream according to the comparison result to obtain the second data stream includes: Compare the sequence number of the first transmission message under the same flow ID with the sequence number of the recorded message, and update the first data flow according to the sequence number comparison result to obtain a new first data flow; the first transmission message belongs to the first data flow; The direction of the second message to be transmitted under the same flow ID is compared with the direction of the recorded message, and the new first data flow is updated according to the direction comparison result to obtain the second data flow; the second message to be transmitted belongs to the second data flow.

[0008] According to a message deduplication method based on a 5G service-oriented interface provided by the present invention, the sequence number of the first transmission message under the same flow ID is compared with the sequence number of the recorded message, and the first data flow is updated according to the sequence number comparison result to obtain the second data flow, including: Parsing the uplink and downlink information of the first transmission message according to the HTTP / 2 protocol; Determine the recorded message according to the uplink and downlink information; When the sequence number of the first transmission message is the same as the sequence number of the recorded message and the first transmission message is not misjudged as a single message being carried by multiple streams, the first transmission message is removed from the first data stream to obtain the second data stream.

[0009] According to a message deduplication method based on a 5G service-oriented interface provided by the present invention, the direction of the second message to be transmitted under the same flow ID is compared with the direction of the recorded message, and the new first data flow is updated according to the direction comparison result to obtain the second data flow, including: Determine a transmission type of the second message to be transmitted; the transmission type includes a request message and a response message; When the sequence number of the second message to be transmitted is smaller than the sequence number of the recorded message and the new first data stream satisfies one of the following conditions, the second message to be transmitted is removed from the new first data stream, and the sequence numbers corresponding to the messages not removed in the new first data stream under the flow nodes are updated according to the uplink and downlink information to obtain the second data stream: The new first data stream does not receive a response message; The new first data flow has received a response message and a request message.

[0010] According to a message deduplication method based on a 5G service-oriented interface provided by the present invention, the deep message processing system based on the stream ID identification technology identifies the second data stream, and removes the duplicated identified messages in the second data stream according to the identification result, so as to obtain a third data stream including: Performing flow ID information comparison and timestamp comparison processing on the second data flow using specific identity information and identification fields through the 5G service-oriented interface; If the stream ID information corresponding to the third message to be transmitted in the second data stream is the same as the stream ID information of the recorded message, and the timestamp corresponding to the third message to be transmitted and the timestamp of the recorded message are within the target time range, the third message to be transmitted is removed from the second data stream to obtain the third data stream.

[0011] According to a message deduplication method based on a 5G service interface provided by the present invention, the 5G service interface adopts a TCP long link, and the 5G service interface is used to determine the uplink and downlink directions of the message to be transmitted through the HTTP / 2 protocol.

[0012] The present invention also provides a message deduplication device based on a 5G service-oriented interface, comprising: An initialization module, configured to respectively initialize the flow node information and the TCP bidirectional sequence number according to the initial message in the first data flow when the first data flow is transmitted through the 5G service interface according to the HTTP / 2 protocol; the flow node information includes a flow ID and a TCP bidirectional sequence number, and different messages correspond to different sequence numbers and different flow IDs; A first deduplication module is used for performing a target comparison on the to-be-transmitted message and the recorded message under the same flow ID, and updating the first data flow according to the comparison result to obtain a second data flow; the target comparison includes at least one of a sequence number comparison and a direction comparison of the message; the to-be-transmitted message belongs to the first data flow; The second deduplication module is used to identify the second data stream using a deep message processing system based on stream ID identification technology, and to remove duplicated messages in the second data stream according to the identification result to obtain a third data stream.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for deduplicating a message based on a 5G service interface as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the XXXX method as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned message deduplication methods based on the 5G service interface.

[0016] The message deduplication method and device based on 5G service interface provided by the present invention, when sampling the 5G service interface to transmit the first data stream according to the HTTP / 2 protocol, initialize the flow node information and the TCP bidirectional sequence number according to the initial message; for the messages to be transmitted and the recorded messages under the same flow ID, target comparison is performed on the messages to be transmitted and the recorded messages, and the first data stream is updated to obtain the second data stream, and finally the second data stream is identified by a deep message processing system based on the flow ID identification technology, and the repeated identified messages in the second data stream are eliminated according to the identification result to obtain the third data stream, which can deduplicate independent messages in each data stream passing through the 5G service interface, so as to realize that the deduplication process occupies less memory, consumes less performance, and has a lower granularity of deduplication area division, while ensuring the deduplication readiness, realizing a lighter and smaller granularity message deduplication process, and improving the efficiency and accuracy of message deduplication. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the message deduplication method based on 5G service interface provided by the present invention.

[0019] Figure 2It is a structural schematic diagram of the message deduplication device based on the 5G service interface provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] It should be noted that in 5G networks, service-oriented interfaces are used to realize communication between different network functions (NFs). As an efficient transmission protocol, HTTP / 2 protocol has been widely used in 5G service-oriented interfaces.

[0023] The following describes in detail how to initialize the stream node information (including stream ID and TCP bidirectional sequence number) through the data stream transmitted by the HTTP / 2 protocol and how to process different messages: In this embodiment, before starting to transmit the data stream, an HTTP / 2 connection is established first; specifically, a TCP connection is established between the sender (client or server) and the receiver (server or client), and encryption is performed through TLS (if secure transmission is required).

[0024] In this embodiment, one connection in HTTP / 2 can carry multiple concurrent streams, each stream is identified by a unique stream ID, and HTTP / 2 protocol transmission messages generally include a header frame (HEADERS frame) and a data frame (DATA frame), etc.

[0025] In this embodiment, the HTTP / 2 protocol assigns a unique stream ID to each stream. The stream ID initiated by the client is usually an odd number, and the stream ID initiated by the server is an even number.

[0026] In this embodiment, if the initial message is the first message in the first data stream, its stream ID is known (usually starts from 1, and the ID can also be set according to specific user requirements).

[0027] In this embodiment, when a TCP connection is established, an initial sequence number is assigned to each direction; when transmitting according to the HTTP / 2 protocol, these sequence numbers are used to ensure reliable transmission and retransmission of data; It should be noted that the TCP sequence number is shared for each stream, but when processing specific streams, they can be distinguished by the stream ID; each message has a unique sequence number in HTTP / 2 (relative at the stream level, based on the TCP sequence number) to ensure the order and integrity of the data; for each received message, it can be associated with a specific stream based on its stream ID.

[0028] In this embodiment, each message contains its own sequence number; for example, in the HTTP / 2 frame header, the sequence number of the message does not belong to the sequence number at the TCP level, and the sequence number of the message can be used for frame sorting and confirmation.

[0029] Assume that there is a 5G service-oriented interface scenario, in which the client sends a request to the server via HTTP / 2, and the client and the server establish an HTTP / 2 connection via TCP / TLS; the client sends a HEADERS frame with a stream ID of 1 to request a resource; the server receives the HEADERS frame and identifies that the stream ID is 1, then the server initializes or updates the stream node information, records the stream ID = 1 and its corresponding TCP sequence number; the server sends a HEADERS frame in response, with a stream ID of 1 (for the same request response), and contains information such as the response status code; the server may subsequently send a DATA frame containing the actual request data content, also using stream ID = 1; the client receives the server's HEADERS frame and DATA frame, and associates them with the same request / response cycle based on stream ID = 1; the client processes the response data and sends subsequent messages as needed (such as an ACK frame to confirm the received data).

[0030] Combine the following Figure 1-Figure 2 The present invention describes a message deduplication method and device based on a 5G service-oriented interface.

[0031] Figure 1 It is a flow chart of the message deduplication method based on the 5G service interface provided by the present invention, such as Figure 1 As shown, the message deduplication method based on the 5G service-oriented interface includes the following steps: Step 110: When transmitting the first data stream through the 5G service interface in accordance with the HTTP / 2 protocol, the flow node information and the TCP bidirectional sequence number are initialized according to the initial message in the first data stream; the flow node information includes the flow ID and the TCP bidirectional sequence number, and the first message is obtained through the 5G service interface. Different messages correspond to different sequence numbers and different flow IDs.

[0032] In this step, the first data stream includes multiple messages or data packets, and the messages in the first data stream are all transmitted according to the HTTP / 2 protocol.

[0033] In this step, the initial message can be the first message processed by the 5G service interface, or it can be a message or data packet specified by the user.

[0034] In this embodiment, the 5G service-oriented interface adopts a TCP long link, and the 5G service-oriented interface is used to determine the uplink and downlink directions of the message to be transmitted through the HTTP / 2 protocol.

[0035] In this embodiment, the 5G service interface uses a TCP long link to maintain the basic information of HTTP / 2 it carries; for example, the uplink and downlink directions are determined through HTTP / 2 node initialization information. As a specific service on HTTP / 2, the data stream can be fragmented and identified with a stream ID, and the associated deduplication areas are divided. In the single data stream interaction process, the deduplication process of each deduplication area is performed independently.

[0036] In this embodiment, according to the flow management mechanism of signaling message processing of the 5G service-oriented interface, when the initial message triggers the flow node to establish initialization, the uplink and downlink are determined by the HTTP / 2 basic information, and the bidirectional sequence number under the flow node is initialized by obtaining the TCP sequence number and the confirmation sequence number.

[0037] Specifically, first, an HTTP / 2 node is established for the TCP long link, and the five-tuple information is initialized. The initialization information of the first packet is used as the anchor point for uplink judgment, and subsequent messages are identified based on IP or port directions.

[0038] In this embodiment, the data streams carried on HTTP / 2 are distinguished by stream IDs and belong to different sub-service processes. The processing of the 5G service-oriented interface requires the establishment of a node management process based on the stream ID, which includes the creation, maintenance, deletion and aging of stream nodes.

[0039] In this embodiment, the message deduplication process is performed within a single data stream; when the initial message triggers the flow node to establish initialization, the upstream and downstream are identified by comparing the HTTP / 2 basic information with the current message IP or port information, and the TCP sequence number and confirmation sequence number of the current message are extracted to initialize the upstream sequence number and downstream sequence number of the flow node.

[0040] It should be noted that the interaction process of a data stream in the 5G service-oriented interface is extremely short, and the maintenance cycle of the stream node is generally within 1s and no more than 3s; this ensures that the deduplication process embedded in a single data stream will not be misjudged. This embodiment uses the HTTP / 2 protocol to effectively manage multiple concurrent streams in the 5G service-oriented interface, and ensures the correct transmission and processing of data through the stream ID and TCP bidirectional sequence number.

[0041] Step 120: For the messages to be transmitted and the recorded messages under the same flow ID, target comparison is performed on the messages to be transmitted and the recorded messages, and the first data stream is updated according to the comparison result to obtain the second data stream; the target comparison includes at least one of the sequence number comparison and direction comparison of the messages; the message to be transmitted belongs to the first data stream.

[0042] In this step, multiple flows with the same flow ID generally refer to the Header and Data type flows under the same message.

[0043] In this step, the first data stream can be updated according to the comparison result. For example, it can be determined that the current message to be transmitted in the first data stream is not a duplicate message compared to the message already recorded in the system, and the current message to be transmitted can be removed from the first data stream and the message sequence number can be updated. Alternatively, it can be determined that the current message to be transmitted is not a duplicate message, and the sequence number corresponding to the new message can be added to the original sequence number.

[0044] In this embodiment, whether the message to be transmitted is a duplicate message may be determined by at least one of sequence number comparison and message direction comparison.

[0045] Specifically, the target comparison includes sequence number comparison and direction comparison; target comparison is performed on the message to be transmitted and the recorded message, and the first data stream is updated according to the comparison result to obtain the second data stream, including: comparing the sequence number of the first transmission message under the same flow ID with the sequence number of the recorded message, and updating the first data stream according to the sequence number comparison result to obtain a new first data stream; the first transmission message belongs to the first data stream; the direction of the second message to be transmitted under the same flow ID is compared with the direction of the recorded message, and the new first data stream is updated according to the direction comparison result to obtain the second data stream; the second message to be transmitted belongs to the second data stream.

[0046] Step 130: Use a deep message processing system based on stream ID identification technology to identify the second data stream, and remove duplicated messages in the second data stream according to the identification result to obtain a third data stream.

[0047] In this step, the deep message processing system based on flow ID identification technology is widely used in network security monitoring, traffic optimization, network troubleshooting and other fields; for example, in network security monitoring, the system can monitor abnormal behaviors in network traffic in real time, and promptly detect and defend against network attacks; in traffic optimization, the system can analyze the distribution and characteristics of network traffic, and optimize the allocation and utilization of network resources; in network troubleshooting, the system can locate the cause and location of network failures, and provide support for the rapid recovery of network failures.

[0048] Specifically, the deep message processing system has the following functions: (1) Traffic classification and identification: Classify and identify network traffic to distinguish normal traffic from abnormal traffic; this helps to promptly detect potential threats and security issues in the network; through machine learning algorithms such as clustering and deep learning, the system can automatically identify and classify network traffic, improving the accuracy and efficiency of traffic identification.

[0049] (2) Message content analysis: In-depth analysis of the content of the data packet to extract useful information. For example, in network security monitoring, the system can detect whether the data packet contains malicious code or attack behavior; by analyzing the content of the data packet, the system can also achieve real-time monitoring and early warning of network behavior, providing strong protection for network security.

[0050] (3) Real-time processing and response: Process network traffic in real time and respond to abnormal events in the network in a timely manner. For example, when the system detects a network attack, it can immediately trigger an alarm mechanism and take corresponding defensive measures.

[0051] In this embodiment, a deep message processing system based on stream ID identification technology is used to identify the second data stream, and according to the identification result, repeatedly identified messages in the second data stream are eliminated to obtain the third data stream, including: using the 5G service interface to compare the stream ID information and the timestamp of the second data stream using specific identity information and identification fields; if the stream ID information corresponding to the third message to be transmitted in the second data stream is the same as the stream ID information of the recorded message, and the timestamp corresponding to the third message to be transmitted is within the target time range, the third message to be transmitted is eliminated from the second data stream to obtain the third data stream.

[0052] In this embodiment, the target time range can be set according to user needs.

[0053] Specifically, the 5G service-oriented interface uses specific identity information and identification fields to perform further self-association processing on different HTTP data streams; during the self-association processing, the message of the stream ID information is compared. If the stream ID information corresponding to the current message is consistent with the historical stream ID information, it can be further determined whether the timestamps corresponding to the two stream IDs are within the threshold range. If they are within the threshold range, the message is determined to be a duplicate message and is discarded.

[0054] In this embodiment, the 5G service-oriented interface processes specific business data according to the request and response message interaction traffic, and uses IMSI, smContextRef, etc. to perform self-association processing on different messages, wherein the entire process performs session management on a user basis; for the very few duplicate packets that arrive after being deleted during the flow node management process, the matching identification of the flow ID is embedded here.

[0055] Specifically, when a request or response message under the same data flow enters the business processing flow for the first time, the request flow ID, response flow ID and update timestamp of the session management node are updated; when a subsequent request or response message enters the business processing flow, the flow ID is judged again. If the flow ID is the same and the timestamp is within the threshold range, the message is judged as a duplicate packet and no further processing is performed. The threshold setting is generally not more than 3s.

[0056] The message deduplication method based on 5G service interface provided by the embodiment of the present invention, when sampling the 5G service interface to transmit the first data stream according to the HTTP / 2 protocol, initializes the flow node information and the TCP bidirectional sequence number according to the initial message; for the messages to be transmitted and the recorded messages under the same flow ID, target comparison is performed on the messages to be transmitted and the recorded messages, and the first data stream is updated to obtain the second data stream, and finally the second data stream is identified by a deep message processing system based on the flow ID identification technology, and the repeated identified messages in the second data stream are eliminated according to the identification result to obtain the third data stream, which can deduplicate independent messages in each data stream passing through the 5G service interface, so as to realize that the deduplication process occupies less memory, consumes less performance, and has a lower granularity of deduplication area division, while ensuring the deduplication readiness, realizing a lighter and smaller granularity message deduplication process, and improving the efficiency and accuracy of message deduplication.

[0057] The following specifically describes the message deduplication method based on the 5G service-oriented interface when the target comparison includes sequence number comparison and direction comparison.

[0058] (1) Comparing the sequence number of the first transmission message under the same flow ID with the sequence number of the recorded message, and updating the first data flow according to the sequence number comparison result to obtain the second data flow includes: parsing the uplink and downlink information of the first transmission message according to the HTTP / 2 protocol; determining the recorded message according to the uplink and downlink information; when the sequence number of the first transmission message is the same as the sequence number of the recorded message, and the first transmission message does not belong to the erroneous judgment that a single message is carried by multiple flows, removing the first transmission message from the first data flow to obtain the second data flow.

[0059] It should be noted that the HTTP / 2 protocol supports multiplexing, that is, multiple requests and responses can be transmitted in parallel on the same connection, and each request or response is assigned a unique stream ID.

[0060] In this embodiment, an HTTP / 2 protocol parser is used to parse the to-be-transmitted messages in the received first data stream; during the parsing process, for messages under the same stream node that are identified to have the same stream ID, the stream ID, sequence number, and uplink and downlink information (i.e., whether the message is uplink or downlink) of each message is extracted.

[0061] For example, the client sends an upstream message to the server according to the HTTP / 2 protocol; the server sends a downstream message to the client according to the HTTP / 2 protocol.

[0062] In this embodiment, if the sequence number of the current message is the same as the sequence number of the historical message under the flow node, it is further determined whether it is a duplicate of a single message carried by multiple flows with the same flow ID. If not, the current message is determined to be a duplicate message and is discarded. If so, subsequent verification is performed.

[0063] It should be noted that multiple flows with the same flow ID generally refer to the Header and Data type flows under the same message.

[0064] Specifically, the message associated with the initialized flow node has completed the flow ID verification, and the uplink or downlink direction of the message is identified by comparing the HTTP / 2 basic information with the current message IP or port information; if it is an uplink message, the sequence number (Sequence Number) extracted from the current message is compared with the uplink sequence number in the flow node; if it is a downlink message, the sequence number (Sequence Number) extracted from the current message is compared with the downlink sequence number in the flow node; if the comparison results are the same, it is necessary to further determine whether the current processing process is a repeated judgment of multiple flows carrying the same message and the same flow ID. If it is not a repeated judgment of the same message, the current message is determined to be a duplicate message and is discarded. This step can filter out most of the duplicate packets, and in other cases, subsequent verification is continued.

[0065] In this embodiment, after the preliminary duplicate packet deduplication determination is completed through sequence number comparison, the message with normally accumulated sequence number is determined to be a non-duplicate message, and the upstream and downstream directions of the current message are tracked. If it is an upstream message, the upstream sequence number in the flow node is updated, and if it is a downstream message, the downstream sequence number in the Liu node is updated.

[0066] In some embodiments, direction checking is performed on messages under the same flow ID in the second data flow, messages with existing direction records are discarded, and TCP sequence numbers under flow nodes are updated for messages that are not discarded, including: (2) Compare the direction of the second message to be transmitted under the same flow ID with the direction of the recorded message, and update the new first data flow according to the direction comparison result, so as to obtain the second data flow including: determining the transmission type of the second message to be transmitted; the transmission type includes a request message and a response message; when the sequence number of the second message to be transmitted is smaller than the sequence number of the recorded message, and the new first data flow satisfies one of the following conditions, remove the second message to be transmitted from the new first data flow, and update the corresponding sequence number of the message not removed in the new first data flow under the flow node according to the uplink and downlink information, so as to obtain the second data flow: the new first data flow has not received the response message; the new first data flow has received the response message and the request message.

[0067] In this embodiment, the situations where further supplementary direction verification is required are the situations where the current message sequence number is out of order when compared with the corresponding sequence number in the flow node, and the situations where the sequence numbers are the same and supplementary direction verification is required; the supplementary direction verification is mainly aimed at identifying and filtering duplicate messages in a small number of out-of-order situations.

[0068] In this embodiment, the flow node divides the request and response message types according to the business logic of the flow data, and identifies the received request or response message and updates it to the flow node; the supplementary direction verification specifically determines whether the current data flow has not received a response message or both the request / response message have been received. If so, it determines that the current message is a duplicate message and discards it.

[0069] In this embodiment, further verification is performed on the situation where the sequence number of the to-be-transmitted message of the second data stream is the same as the sequence number of the historical message under the flow node and requires additional direction verification, as well as the situation where the sequence number is out of order; if the current data stream has not received a response message or both the request / response messages have been received, it is determined that the current message is still a duplicate message and is discarded; for the messages that are not discarded, the sequence numbers under the flow node are updated according to the uplink and downlink directions of the messages, providing a basis for judgment for subsequent messages.

[0070] The message deduplication method based on the 5G service interface provided in the embodiment of the present invention performs preliminary deduplication on the data stream by comparing the sequence numbers of the messages to be transmitted and the existing messages in the data stream, and then performs additional verification on the direction of the received message according to the specific HTTP / 2 data stream request / response under the 5G service interface, and filters a small number of out-of-order duplicate packets. The deduplication area is divided at a lower granularity, thereby achieving lighter and smaller granular message deduplication and further improving the accuracy of deduplication.

[0071] The following is a description of the message deduplication device based on the 5G service-oriented interface provided by the present invention. The message deduplication device based on the 5G service-oriented interface described below and the message deduplication method based on the 5G service-oriented interface described above can be referenced to each other.

[0072] Figure 2 : is a structural diagram of a message deduplication device based on a 5G service-oriented interface provided by the present invention, such as Figure 2 As shown, the message deduplication device based on the 5G service interface includes: an initialization module 210, a first deduplication module 220 and a second deduplication module 230.

[0073] The initialization module 210 is used to initialize the flow node information and the TCP bidirectional sequence number according to the initial message in the first data flow when the first data flow is transmitted through the 5G service interface according to the HTTP / 2 protocol; the flow node information includes the flow ID and the TCP bidirectional sequence number, the first message is obtained through the 5G service interface, and different messages correspond to different sequence numbers and different flow IDs; The first deduplication module 220 is used for performing a target comparison on the to-be-transmitted message and the recorded message under the same flow ID, and updating the first data flow according to the comparison result to obtain the second data flow; the target comparison includes at least one of a sequence number comparison and a direction comparison of the message; the to-be-transmitted message belongs to the first data flow; The second deduplication module 230 is used to identify the second data stream using a deep message processing system based on a stream ID identification technology, and remove duplicated messages in the second data stream according to the identification result to obtain a third data stream.

[0074] The message deduplication device based on 5G service interface provided by the embodiment of the present invention, when sampling the 5G service interface to transmit the first data stream according to the HTTP / 2 protocol, initializes the flow node information and the TCP bidirectional sequence number according to the initial message; for the messages to be transmitted and the recorded messages under the same flow ID, target comparison is performed on the messages to be transmitted and the recorded messages, and the first data stream is updated to obtain the second data stream, and finally the second data stream is identified by a deep message processing system based on the flow ID identification technology, and the repeated identified messages in the second data stream are eliminated according to the identification result to obtain the third data stream, which can deduplicate independent messages in each data stream passing through the 5G service interface, so as to realize that the deduplication process occupies less memory, consumes less performance, and has a lower granularity of deduplication area division, while ensuring the deduplication readiness, realizing a lighter and smaller granularity message deduplication process, and improving the efficiency and accuracy of message deduplication.

[0075] Figure 3 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 3As shown, the electronic device may include: a processor 310 , a communications interface 320 , a memory 330 and a communication bus 340 , wherein the processor 310 , the communications interface 320 , and the memory 330 communicate with each other via the communication bus 340 . The processor 310 can call the logic instructions in the memory 330 to execute a message deduplication method based on the 5G service interface, the method including: when transmitting the first data stream through the 5G service interface according to the HTTP / 2 protocol, initializing the flow node information and the TCP bidirectional sequence number according to the initial message in the first data stream; the flow node information includes the flow ID and the TCP bidirectional sequence number, the first message is obtained through the 5G service interface, and different messages correspond to different sequence numbers and different flow IDs; for the message to be transmitted and the recorded message under the same flow ID, the message to be transmitted and the recorded message are compared by target, and the first data stream is updated according to the comparison result to obtain the second data stream; the target comparison includes at least one of the sequence number comparison and direction comparison of the message; the message to be transmitted belongs to the first data stream; the second data stream is identified by a deep message processing system based on the flow ID identification technology, and the duplicated identified messages in the second data stream are eliminated according to the identification result to obtain the third data stream.

[0076] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0077] On the other hand, the present invention also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the message deduplication method based on the 5G service interface provided by the above methods, the method comprising: in the case of transmitting a first data stream through a 5G service interface according to the HTTP / 2 protocol, initializing the flow node information and the TCP bidirectional sequence number according to the initial message in the first data stream; the flow node information includes a flow ID and a TCP bidirectional sequence number, the first message is obtained through the 5G service interface, and different messages correspond to different sequence numbers and different flow IDs; for the message to be transmitted and the recorded message under the same flow ID, the message to be transmitted and the recorded message are compared by target, and the first data stream is updated according to the comparison result to obtain a second data stream; the target comparison includes at least one of the sequence number comparison and the direction comparison of the message; the message to be transmitted belongs to the first data stream; the deep message processing system based on the flow ID identification technology is used to identify the second data stream, and the duplicated identified messages in the second data stream are eliminated according to the identification result to obtain a third data stream. 。

[0078] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the message deduplication method based on the 5G service interface provided by the above-mentioned methods, the method comprising: when transmitting a first data stream through a 5G service interface in accordance with the HTTP / 2 protocol, initializing the flow node information and the TCP bidirectional sequence number according to the initial message in the first data stream; the flow node information includes a flow ID and a TCP bidirectional sequence number, the first message is obtained through the 5G service interface, and different messages correspond to different sequence numbers and different flow IDs; for the message to be transmitted and the recorded message under the same flow ID, a target comparison is performed on the message to be transmitted and the recorded message, and the first data stream is updated according to the comparison result to obtain a second data stream; the target comparison includes at least one of a sequence number comparison and a direction comparison of the message; the message to be transmitted belongs to the first data stream; a deep message processing system based on a flow ID identification technology is used to identify the second data stream, and the repeatedly identified messages in the second data stream are eliminated according to the identification result to obtain a third data stream.

[0079] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0080] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A message deduplication method based on 5G service interface, characterized in that: include: When the first data stream is transmitted through the 5G service-oriented interface according to the HTTP / 2 protocol, initializing the stream node information and the TCP bidirectional sequence number according to the initial message in the first data stream; The flow node information includes a flow ID and a TCP bidirectional sequence number, and different messages correspond to different sequence numbers and different flow IDs; For the messages to be transmitted and the recorded messages under the same flow ID, target comparison is performed on the messages to be transmitted and the recorded messages, and the first data flow is updated according to the comparison result to obtain a second data flow; The target comparison includes at least one of a sequence number comparison and a direction comparison of the message; The message to be transmitted belongs to the first data flow; The second data stream is identified by a deep message processing system based on stream ID identification technology, and repeated identified messages in the second data stream are removed according to the identification result to obtain a third data stream.

2. The message deduplication method based on 5G service interface according to claim 1 is characterized in that: The target comparison includes sequence number comparison and direction comparison; The step of comparing the message to be transmitted with the recorded message and updating the first data stream according to the comparison result to obtain the second data stream includes: Compare the sequence number of the first transmission message under the same flow ID with the sequence number of the recorded message, and update the first data flow according to the sequence number comparison result to obtain a new first data flow; the first transmission message belongs to the first data flow; The direction of the second message to be transmitted under the same flow ID is compared with the direction of the recorded message, and the new first data flow is updated according to the direction comparison result to obtain the second data flow; the second message to be transmitted belongs to the second data flow.

3. The message deduplication method based on 5G service interface according to claim 2 is characterized in that: The step of comparing the sequence number of the first transmission message under the same flow ID with the sequence number of the recorded message, and updating the first data flow according to the sequence number comparison result to obtain the second data flow comprises: Parsing the uplink and downlink information of the first transmission message according to the HTTP / 2 protocol; Determine the recorded message according to the uplink and downlink information; When the sequence number of the first transmission message is the same as the sequence number of the recorded message and the first transmission message is not misjudged as a single message being carried by multiple streams, the first transmission message is removed from the first data stream to obtain the second data stream.

4. The message deduplication method based on 5G service interface according to claim 2 is characterized in that: The step of comparing the direction of the second message to be transmitted under the same flow ID with the direction of the recorded message, and updating the new first data flow according to the direction comparison result to obtain the second data flow comprises: Determine a transmission type of the second message to be transmitted; the transmission type includes a request message and a response message; When the sequence number of the second message to be transmitted is smaller than the sequence number of the recorded message and the new first data stream satisfies one of the following conditions, the second message to be transmitted is removed from the new first data stream, and the sequence numbers corresponding to the messages not removed in the new first data stream under the flow nodes are updated according to the uplink and downlink information to obtain the second data stream: The new first data stream does not receive a response message; The new first data flow has received a response message and a request message.

5. The message deduplication method based on 5G service interface according to claim 2 is characterized in that: The deep message processing system using the stream ID identification technology identifies the second data stream, and removes the repeatedly identified messages in the second data stream according to the identification result, so as to obtain a third data stream including: Performing flow ID information comparison and timestamp comparison processing on the second data flow using specific identity information and identification fields through the 5G service-oriented interface; If the stream ID information corresponding to the third message to be transmitted in the second data stream is the same as the stream ID information of the recorded message, and the timestamp corresponding to the third message to be transmitted and the timestamp of the recorded message are within the target time range, the third message to be transmitted is removed from the second data stream to obtain the third data stream.

6. The message deduplication method based on 5G service interface according to claim 5 is characterized in that: The 5G service-oriented interface adopts a TCP long link, and the 5G service-oriented interface is used to determine the uplink and downlink directions of the message to be transmitted through the HTTP / 2 protocol.

7. A message deduplication device based on 5G service interface, characterized in that: include: An initialization module, configured to respectively initialize the flow node information and the TCP bidirectional sequence number according to the initial message in the first data flow when the first data flow is transmitted according to the HTTP / 2 protocol through the 5G service interface; The flow node information includes a flow ID and a TCP bidirectional sequence number, and different messages correspond to different sequence numbers and different flow IDs; A first deduplication module is used for comparing the to-be-transmitted messages and the recorded messages under the same flow ID, and updating the first data flow according to the comparison result to obtain a second data flow; The target comparison includes at least one of a sequence number comparison and a direction comparison of the message; The message to be transmitted belongs to the first data flow; The second deduplication module is used to identify the second data stream using a deep message processing system based on stream ID identification technology, and to remove duplicated messages in the second data stream according to the identification result to obtain a third data stream.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the message deduplication method based on the 5G service interface as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the method for deduplicating packets based on the 5G service interface as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by the processor, the method for deduplicating packets based on the 5G service interface as described in any one of claims 1 to 6 is implemented.

Images