Intelligent gateway multi-protocol routing optimization method based on reinforcement learning

Abstract

The invention discloses an intelligent gateway multi-protocol routing optimization method based on reinforcement learning. The method comprises the following steps: constructing a gateway observation state vector containing a multi-protocol interface link quality parameter, a queue state parameter and a forwarding consistency parameter in a preset sampling period; generating a routing credibility score vector by adopting a weighted score deduction strategy, and splicing the routing credibility score vector with the gateway observation state vector to obtain an adversarial perception state vector; inputting the adversarial perception state vector into a route optimization model based on a deep Q network to obtain a candidate route action set, and performing action screening based on a route credibility score vector; in the execution stage, protocol switching is triggered according to the condition that the credibility of the current protocol is lower than a switching threshold value and the credibility of the alternative protocol is higher than an access threshold value, and the next hop with the minimum end-to-end hop count is selected to complete forwarding during triggering. Therefore, the misleading probability of the abnormal measurement to the learning decision is reduced, and the continuity and the security robustness of the industrial Internet of Things service are improved.

Description

Technical Field

[0001] This invention relates to the technical field of multi-protocol routing optimization for communication networks and industrial IoT edge gateways, and particularly to a multi-protocol routing optimization method for smart gateways based on reinforcement learning. Background Technology

[0002] With the integrated deployment of industrial IoT, edge computing, and communication infrastructure such as 5G / TSN, smart gateways for field devices are gradually taking on key roles in multi-protocol access, heterogeneous forwarding, and edge collaboration. Industrial fields simultaneously contain multiple protocol stacks and multi-link interfaces, including Ethernet / Industrial Ethernet, Wi-Fi, cellular, LDP, and fieldbus. Service traffic exhibits characteristics of strong bursts, strong time-varying patterns, and multiple constraints (latency, jitter, packet loss, reliability). Meanwhile, security events such as falsified link quality parameters, congestion induction, and forwarding anomalies can cause routing metric distortion and policy misselection, leading to frequent protocol switching, queue backlog propagation, and end-to-end performance disasters. While existing routing optimization research has evolved from static rules and heuristic algorithms to adaptive decision-making based on reinforcement learning, it still generally lacks modeling of the inherent constraints between interface-level link quality, queue status, and forwarding consistency, especially lacking mechanisms to constrain the reliability of routing metrics under attack and anomaly conditions and deeply couple this reliability with the learning decision link. This results in policies being easily misled in adversarial scenarios, leading to unstable forwarding and switching behaviors.

[0003] CN108900419A discloses a routing decision method and device based on deep reinforcement learning under the SDN architecture. This scheme focuses on traffic engineering optimization from the global perspective of the controller. The routing decision is mainly driven by factors such as real-time traffic / link bandwidth occupancy. It lacks a structured state expression for the gateway side for the joint observation of multi-interface link quality parameters, queue state parameters, and forwarding consistency parameters of multi-protocol smart gateways. At the same time, it does not provide a credibility constraint and action screening mechanism for routing metric anomalies (forgery / spoofing / congestion inducement), nor does it involve protocol stack-level switching trigger thresholds and minimum hop count forwarding constraints after switching. Therefore, it is difficult to cover the continuous forwarding and security robustness requirements of industrial IoT gateways in adversarial scenarios.

[0004] CN109547351A discloses a routing method based on Q-learning and trust models in Ad Hoc networks. This scheme focuses on node-level trust and Q-table updates in Ad Hoc scenarios, and its trust calculation and interactive message mechanism is closer to the maintenance of adjacency relationships in ad hoc networks. However, its state and reward design does not address the heterogeneous interfaces and protocol stack selection of multi-protocol smart gateways, and lacks a framework that incorporates gateway internal operating states such as queue occupancy, link quality fluctuations, and the proportion of missing forwarding receipts into a unified observation vector and couples them with learning decisions. At the same time, this scheme does not provide a constraint process for screening candidate actions based on routing credibility scoring vectors, nor does it cover the threshold triggering of industrial gateways under protocol switching conditions and the end-to-end hop count constraint forwarding rules after switching. Therefore, in scenarios with deceptive metrics and congestion-induced concurrency, problems such as misled learning strategies, discontinuous forwarding, or switching oscillations may still occur.

[0005] In summary, existing reinforcement learning / trust-based routing technologies generally suffer from the following problems: insufficient interface-level observation of multi-protocol gateways, lack of trustworthiness constraints on links for routing metric anomalies, and lack of operable threshold triggering and forwarding rules for protocol switching. This invention addresses these issues by proposing a reinforcement learning-based intelligent gateway multi-protocol routing optimization method: A gateway observation state vector containing multi-protocol interface link quality parameters, queue state parameters, and forwarding consistency parameters is constructed under a preset sampling period. A weighted deduction strategy is used to generate a routing trustworthiness score vector, which is then concatenated with the gateway observation state vector to obtain an adversarial perception state vector. This adversarial perception state vector is input into a routing optimization model based on a deep Q-network to obtain a set of candidate routing actions, and actions are selected based on the routing trustworthiness score vector. During the execution phase, protocol switching is triggered when the current protocol trustworthiness is below the switching threshold and the candidate protocol trustworthiness is above the access threshold. Upon triggering, the next hop with the smallest end-to-end hop count is selected for forwarding, thereby reducing the probability of anomaly metrics misleading learning decisions and improving the continuity and security robustness of industrial IoT services. Summary of the Invention

[0006] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section, as well as in the abstract and title of the present application, to avoid obscuring the purpose of this section, the abstract and title of the invention. Such simplifications or omissions shall not be used to limit the scope of the present invention.

[0007] In view of the aforementioned existing problems, the present invention is proposed.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: As a preferred embodiment of the reinforcement learning-based intelligent gateway multi-protocol routing optimization method of the present invention, wherein: multi-protocol interface link quality parameters, queue state parameters and forwarding consistency parameters are collected and preprocessed under a preset sampling period to form a gateway observation state vector; A weighted deduction strategy is used to calculate the route credibility score vector on the gateway observation state vector, and the route credibility score vector is concatenated and fused with the gateway observation state vector to obtain the adversarial perception state vector; The adversarial perception state vector is input into the routing optimization model built on a deep Q-network, and a set of candidate routing actions is output. The routing actions are then filtered according to the routing credibility score vector to obtain the routing execution actions. According to the routing execution action, the target service packets are forwarded and the protocol is switched. The protocol switching is triggered when the current protocol credibility is lower than the switching threshold and the alternative protocol credibility is higher than the access threshold. When triggered, the next hop with the minimum end-to-end hop count is selected to complete the forwarding.

[0009] The beneficial effects of this invention are as follows: By collecting and preprocessing multi-protocol interface link quality parameters, queue state parameters, and forwarding consistency parameters under a preset sampling period, this invention forms a gateway observation state vector, enabling routing decisions to have a unified and alignable operational input, reducing routing bias caused by link fluctuations and queue backlogs; by generating a route credibility score vector using a weighted deduction strategy and concatenating it with the gateway observation state vector to obtain an adversarial awareness state vector, abnormal forwarding and measurement distortion risks are incorporated into the decision input, suppressing deception interference; by outputting a set of candidate route actions based on a deep Q network and combining it with the route credibility score vector to filter actions and obtain route execution actions, the probability of high Q-value but low credibility actions being selected is reduced; by triggering protocol switching according to a threshold and selecting the next hop with the smallest end-to-end hop count to complete forwarding, switching oscillations and detour overhead are reduced, thereby improving service continuity and transmission stability under attack and abnormal conditions. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating the reinforcement learning-based intelligent gateway multi-protocol routing optimization method of the present invention. Figure 2 This is a schematic diagram of routing entry information as shown in the present invention; Figure 3This is a schematic diagram illustrating the business reward tier information of the present invention. Detailed Implementation

[0011] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0012] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort should fall within the scope of protection of this invention.

[0013] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0014] According to an embodiment of the present invention, in combination Figure 1 The flowchart shown illustrates a multi-protocol routing optimization method for intelligent gateways based on reinforcement learning, which specifically includes the following steps: S1. Collect and preprocess multi-protocol interface link quality parameters, queue status parameters, and forwarding consistency parameters within a preset sampling period to form a gateway observation state vector. Note that the following points should be noted in this step: S1.1 At the beginning of each preset sampling period, the smart gateway reads the link quality parameters, queue status parameters and forwarding consistency parameters from each multi-protocol interface, and writes each parameter into the preset sampling period index to form the original parameter set.

[0015] In this embodiment, the smart gateway is the IoT edge computing gateway in the prior art, and it will not be listed here.

[0016] As an example, a multi-protocol interface includes at least: a first multi-protocol interface (Ethernet interface, carrying a TCP / IP protocol stack), a second multi-protocol interface (cellular interface / Wi-Fi interface, carrying a cellular protocol stack or a wireless LAN protocol stack), and a third multi-protocol interface (low-power wide-area network interface, carrying an LPWAN protocol stack).

[0017] As an example, the preset sampling period is a fixed time window, which is preferably 200ms to 1000ms in this embodiment; 500ms can be selected in the home / building gateway scenario, and 200ms can be selected in the industrial IoT gateway scenario.

[0018] Specifically, at the beginning of each preset sampling period, the smart gateway generates a preset sampling period index based on the unified clock module. The preset sampling period index is a monotonically increasing integer count value and is stored corresponding to the sampling start time. Subsequently, the smart gateway reads the link quality parameters, queue status parameters, and forwarding consistency parameters from the driver layer or protocol stack statistics module of each multi-protocol interface, and writes the preset sampling period index into each parameter record to form the original parameter set.

[0019] In a preferred embodiment, the link quality parameters include at least: interface received signal strength indicator (RSSI), interface packet loss rate, interface round-trip time (RTD), and interface jitter; wherein, the RSSI of the Ethernet interface can be replaced by link negotiation status and bit error statistics, and the RSSI of the wireless interface is taken from the RSSI or equivalent indicators provided by the wireless driver; the queue status parameters include at least: the number of bytes occupied by the interface transmit queue, the number of packets occupied by the interface transmit queue, and the incremental count of dropped packets in the interface queue; the forwarding consistency parameters include at least: the proportion of missing receipts, the proportion of out-of-order receipts, and the proportion of duplicate receipts; wherein the receipts come from one of the three types: link layer acknowledgment, transport layer acknowledgment, or application layer receipts, preferably end-to-end acknowledgment of the target service packet as the source of receipts to reduce cross-layer statistical bias.

[0020] For example, when the smart gateway enables both Ethernet and cellular multi-protocol interfaces, the preset sampling period is 500ms, and the preset sampling period index is 31 in a certain period; the Ethernet interface has a packet loss rate of 0.2%, a round-trip latency of 8ms, and a packet count in the interface sending queue of 12 in this period; the cellular interface has a packet loss rate of 1.8%, a round-trip latency of 45ms, and a packet count in the interface sending queue of 38 in this period; each record in the original parameter set is written with the preset sampling period index = 31, and the interface identifier and parameter type identifier are written.

[0021] It should be noted that the reason for reading link quality parameters, queue status parameters, and forwarding consistency parameters from each multi-protocol interface through the smart gateway in this embodiment is as follows: the link quality parameters reflect the instantaneous reachability of the transmission medium and the wireless environment; the queue status parameters reflect the congestion and queuing delay trends within the gateway; and the forwarding consistency parameters reflect the reliability of the end-to-end acknowledgment link and the consistency of the protocol stack state. These three types of parameters are complementary in terms of time scale and distortion sources. After being aligned according to the preset sampling period index, they can form comparable cross-interface observations at the same time, thereby providing interpretable input for subsequent candidate route entry-level scoring and action selection. Compared with the common practice of relying solely on routing control message metric fields or solely on link layer signal indicators, this embodiment incorporates the acknowledgment missing ratio into the forwarding consistency parameters and uses an acknowledgment matching window for statistics. This makes the scoring independent of a single-level metric field, reducing the impact of abnormal routing metric fields, statistical jitter, and short-term false alarms on the strategy. At the same time, missing parameter completion and writing conservative values ​​for long missing parameters make parameter missing traceable, and when the missing parameters are severe, the scoring naturally biases towards conservative paths, thereby reducing service packet retransmissions and link oscillations caused by unstable forwarding selection in scenarios with frequent multi-protocol switching.

[0022] S1.2. Based on the preset sampling period index, perform timestamp alignment on the original parameter set, and fill in the missing items with the most recent valid value of the same interface to obtain the aligned parameter set.

[0023] In a preferred embodiment, timestamp alignment is performed using a preset sampling period index as the primary key: the smart gateway groups the original parameter set by interface identifier and sorts it by the preset sampling period index within each group; when multiple records of the same type exist under the same preset sampling period index for the same interface, the most recent record at the end of the sampling window is used as the representative value of that type of parameter for that interface, and its original sampling time is retained for audit traceability; when a record of a certain parameter type is missing under the same preset sampling period index for the same interface, it is filled in with the most recent valid value for the same interface, and the filling behavior is written into the missing marker field.

[0024] As an example, within the window of the preset sampling period index=31, the round-trip time statistics module of the cellular interface experiences a reporting delay, causing the round-trip time record to be written to index=32. When the smart gateway is aligned, it finds that the cellular interface is missing a round-trip time record at index=31. Therefore, it reads the round-trip time of the cellular interface at index=30 as the most recent valid value to fill in the gap, and sets the missing flag field of the parameter record as the filling flag so that the source of the filling can be identified in subsequent consistency analysis and scoring mapping.

[0025] It should be noted that the missing data completion methods include: when the consecutive missing data does not exceed the consecutive missing data threshold (e.g., 3 sampling periods), the most recent valid value is used for completion; when the consecutive missing data exceeds the consecutive missing data threshold, the missing data flag field corresponding to the parameter type is set to a long missing data flag, and the parameter value is written to a preset conservative value; the preset conservative value of the link quality parameter can be selected from the degraded end value of the range, the preset conservative value of the queue status parameter can be selected from the upper limit value of the queue, and the preset conservative value of the forwarding consistency parameter can be selected from the upper limit value of the missing data ratio, so that subsequent scoring is more inclined to suppress unreliable routing entries.

[0026] S1.3. Perform normalization processing on the alignment parameter set according to parameter type. Among them, the link quality parameter is mapped to [0,1] according to the preset range interval, the queue status parameter is mapped to [0,1] according to the upper limit of the interface queue, and the forwarding consistency parameter is mapped to [0,1] according to the missing ratio in the receipt matching window, so as to obtain the standardized parameter set.

[0027] It should be noted that the preset range intervals in this embodiment are pre-stored in the gateway configuration table according to parameter type, and can be configured separately according to interface type. For link quality parameters, the following ranges are defined: interface packet loss rate range is 0% to 10%, interface round-trip time range is 0ms to 200ms, and interface jitter range is 0ms to 50ms. For interface received signal strength indication value, the wireless interface range is defined as −100dBm to −40dBm. For queue status parameters, the upper limit of the interface queue is the maximum available capacity of the transmission queue, with one of the upper limit of the maximum number of packets and the upper limit of the maximum number of bytes as a unified benchmark. To facilitate consistency across interfaces, this embodiment preferably uses the upper limit of the maximum number of packets as the upper limit of the interface queue, and obtains it by reading the driver layer queue configuration during the initialization of each interface.

[0028] In a preferred embodiment, the receipt matching window is a statistical window that slides according to a preset sampling period index. The window length is a fixed number of periods, such as 5 to 20 sampling periods. If the preset sampling period is 500ms and the window length is 10, the receipt matching window covers a range of 5 seconds. The receipt missing ratio is calculated as the ratio of the difference between the sent count and the matched receipt count of the target service group within the receipt matching window to the sent count. The counts are also calculated separately according to the interface identifier and the protocol stack selection identifier to avoid aliasing caused by differences in the confirmation mechanisms of different protocols.

[0029] For example, when the target service packet transmission count for a cellular interface is 200 and the matched receipt count is 186 within a certain receipt matching window, the receipt missing rate is 7%; when the target service packet transmission count for an Ethernet interface is 200 and the matched receipt count is 198 within the same window, the receipt missing rate is 1%; after the link quality parameters are mapped according to the preset range interval, the round-trip time of the wireless interface of 45ms is mapped to the low-to-medium value range of 0-1; after the queue status parameters are mapped according to the upper limit of the interface queue, the number of packets occupied by the interface transmission queue of 38 is mapped to the high-to-medium value range of 0-1 under the upper limit of 64.

[0030] S1.4. Concatenate the standardized parameter set in a fixed order of interface identifier, parameter type, and sampling period index, and write the state vector length identifier to generate the gateway observation state vector.

[0031] In a preferred embodiment, the interface identifier includes at least: Ethernet interface identifier, cellular interface identifier, Wi-Fi interface identifier, and LPWAN interface identifier; the parameter types include at least: link quality parameter type, queue status parameter type, and forwarding consistency parameter type; the fixed order in this embodiment is a pre-agreed splicing sequence, that is, a three-layer order of ascending order by interface identifier, order by parameter type within the interface, and fixed order by parameter item within the parameter type; wherein the fixed order of parameter items is, for example, for link quality parameters, arranged by interface received signal strength indication value, interface packet loss rate, interface round-trip time delay, and interface jitter; for queue status parameters, arranged by the number of packets occupied in the interface sending queue and the incremental count of dropped packets in the interface sending queue; and for forwarding consistency parameters, arranged by the proportion of missing receipts, the proportion of out-of-order receipts, and the proportion of duplicate receipts.

[0032] It should be noted that the state vector length identifier is an integer field that records the number of elements in the gateway observation state vector. When the number of interfaces or the number of parameter items changes, the state vector length identifier is updated accordingly, so that subsequent model inputs can be checked for fixed length. When storing the gateway observation state vector, a structure of length identifier field + vector body is adopted. The length identifier field is placed before the vector body and is bound to the same preset sampling period index for writing.

[0033] As an example, when this embodiment enables two types of multi-protocol interfaces and each type of interface corresponds to 4 link quality parameters, 2 queue status parameters, and 3 forwarding consistency parameters, then each interface has a total of 9 standardized parameter elements; the two interfaces have a total of 18 standardized parameter elements. The state vector length identifier is 18, and the gateway observation state vector is concatenated in the following order: first write 9 elements of the Ethernet interface, then write 9 elements of the cellular interface.

[0034] S2. A weighted scoring strategy is applied to the gateway observation state vector to calculate the route credibility score vector. The route credibility score vector is then concatenated and fused with the gateway observation state vector to obtain the adversarial awareness state vector. Note that the following points should be noted in this step: S2.1. Based on the gateway observation state vector, enumerate the next hop identifier and protocol stack selection identifier to obtain a set of candidate route entries, and write the index positions of the corresponding link quality parameters, queue state parameters and forwarding consistency parameters for each candidate route entry.

[0035] In a preferred embodiment, the candidate route entry set is generated by a Cartesian enumeration of the next-hop identifier × protocol stack selection identifier, and constrained filtering is performed in conjunction with the gateway's current routing table and neighbor table; the next-hop identifier is a unique identifier of the neighbor node, such as the link layer address of the neighbor node or the internal neighbor index number of the gateway; the protocol stack selection identifier includes at least: a first protocol stack selection identifier (Ethernet protocol stack), a second protocol stack selection identifier (cellular protocol stack), a third protocol stack selection identifier (Wi-Fi protocol stack), and a fourth protocol stack selection identifier (LPWAN protocol stack); when a protocol stack is configured as unavailable or the interface is in a disconnected state in the current period, the protocol stack selection identifier does not participate in the candidate enumeration.

[0036] Specifically, in this embodiment, the method for writing the index position for each candidate route entry is as follows: In the gateway observation state vector, the vector index range corresponding to each interface, each parameter type, and each parameter item is known in the aforementioned fixed order; for each candidate route entry, the smart gateway maps its protocol stack selection identifier to the corresponding interface identifier, and writes the starting position of the link quality parameter index, the starting position of the queue status parameter index, and the starting position of the forwarding consistency parameter index corresponding to the interface identifier into the candidate route entry structure to form a directly accessible index position record.

[0037] For example, when a candidate route entry contains a next-hop identifier = neighbor 7 and a protocol stack selection identifier = cellular protocol stack, the entry is written to the starting position of the link quality parameter subscript, the starting position of the queue status parameter subscript, and the starting position of the forwarding consistency parameter subscript corresponding to the cellular interface; when another candidate route entry contains a next-hop identifier = neighbor 3 and a protocol stack selection identifier = Ethernet protocol stack, the entry is written to the starting position of the three types of parameters corresponding to the Ethernet interface.

[0038] In a preferred embodiment, the routing control message is a control message exchanged by the protocol stack corresponding to the multi-protocol interface during the routing maintenance process. The message contains fields such as routing metric field value and next-hop identifier. In this embodiment, the routing control message types include at least: First, neighbor discovery / maintenance messages and link state advertisement messages from link-state routing protocols, the former carrying a neighbor reachability summary and interface state, and the latter carrying routing metric field value and path information; Second, routing update messages from distance-vector routing protocols, carrying a destination prefix, next-hop identifier, and routing metric field value; Third, routing request messages and routing response messages from on-demand routing protocols, carrying cost information of candidate paths and routing metric field value of return paths, respectively; Fourth, routing advertisement messages and routing destination advertisement messages from low-power network routing, carrying path cost and reachability information from the current node to the root, respectively; Fifth, session maintenance / bearer update control messages from cellular or carrier-side bearer networks, carrying a bearer quality summary and a set of available next-hop identifiers.

[0039] All of the above message types can be used as data sources for generating candidate route entry sets and reading route metric field values. They are also classified and stored according to the protocol stack selection identifier under the same preset sampling period index, so as to make consistent entry-level associations with link quality parameters, queue status parameters and forwarding consistency parameters.

[0040] In a preferred embodiment, each candidate route entry in the candidate route entry set is recorded using a structured field, with field names remaining consistent throughout the entire scheme and matching the fixed-order concatenation rule of the gateway observation state vector; refer to Figure 2 Here is an example of the field representation of a candidate route entry. The index positions of the link quality parameter, queue status parameter, and forwarding consistency parameter correspond one-to-one with the concatenation order of the gateway observation state vector. This allows each candidate route entry to locate its corresponding standardized parameter element range under the same preset sampling period index, avoiding mixed reading across interfaces and parameter types.

[0041] S2.2 Calculate a deviation deduction item for each candidate route entry, wherein the deviation deduction item is obtained by converting the difference between the measured route metric value determined by the link quality parameters and the route metric field value carried in the route control message corresponding to the candidate route entry through a preset mapping table; calculate a consistency deduction item for each candidate route entry, wherein the consistency deduction item is obtained by converting the acknowledgment missing ratio corresponding to the forwarding consistency parameters through a preset segmentation threshold set; calculate a congestion deduction item for each candidate route entry, wherein the congestion deduction item is obtained by converting the queue occupancy rate corresponding to the queue status parameters through a preset upper limit threshold.

[0042] It should be noted that the deviation deduction item is used to characterize the degree of deviation between the value of the routing control message metric field corresponding to the candidate route entry and the measured routing metric value converted from the link quality parameters; the consistency deduction item is used to characterize the penalty intensity when the proportion of missing receipts is in different risk ranges; the congestion deduction item is used to characterize the penalty intensity when the queue occupancy rate is close to the queue limit; all three types of deduction items use non-negative values ​​and are limited to the preset deduction range range to facilitate subsequent weighted summation.

[0043] In a preferred embodiment, the measured routing metric is obtained by converting link quality parameters. Interface packet loss rate, interface round-trip time, and interface jitter are combined into a single measured routing metric by applying a preset conversion weight. The routing metric field value carried in the routing control message is taken from the routing protocol control message associated with the candidate route entry, such as the link cost or path cost field. The deviation deduction item first calculates the metric difference and then converts it into a deduction value according to a preset mapping table. Its mathematical expression is as follows: in, For the first The metric difference between candidate route entries; For the first The measured routing metric values ​​of each candidate route entry; For the first The routing metric field value carried in the routing control message corresponding to each candidate route entry; For the first Deviation deduction items for each candidate route entry; This is the conversion function corresponding to the preset mapping table.

[0044] Specifically, the default mapping table is set according to the following rules: segments are divided according to the metric difference and assigned scores in a monotonically increasing manner, with the segment boundaries related to the business tolerance; for example, for latency-sensitive businesses, smaller difference segments can be assigned a higher slope; for example: when When the deviation deduction item is in the range of 0 to 5, it takes a linear mapping from 0 to 10. When the deviation deduction item is in the range of 5 to 20, a linear mapping of 10 to 40 is taken. When the deviation deduction item is greater than 20, the saturation mapping of 40 to 60 is taken; the above segment boundaries can be determined by the historical statistical measurement fluctuation quantile, such as taking the 80th and 95th quantiles as boundaries.

[0045] In a preferred embodiment, the consistency deduction is calculated from the proportion of missing receipts using a preset set of segmented thresholds. The preset set of segmented thresholds consists of three or four incremental thresholds and is set together with the receipt matching window length; its mathematical expression is as follows: in, For the first Consistency deductions for each candidate route entry; For the first The percentage of missing receipts for each candidate route entry; It is a conversion function based on a preset set of segmented thresholds.

[0046] For example, the preset segmented threshold set includes: a first threshold of 1%, a second threshold of 5%, and a third threshold of 10%; when the proportion of missing receipts is less than 1%, the consistency deduction item is 0 to 5; when the proportion of missing receipts is between 1% and 5%, the consistency deduction item is 5 to 20; when the proportion of missing receipts is between 5% and 10%, the consistency deduction item is 20 to 40; and when the proportion of missing receipts is not less than 10%, the consistency deduction item is 40 to 60.

[0047] Furthermore, the threshold setting rule is as follows: based on the retransmission cost of the target business group and the stability of the receipt count within the window, first calculate the long-term average and fluctuation range of the missing proportion using historical data, then set the threshold as a combination of the long-term average plus a certain multiple of the fluctuation amount, and review and update it at fixed intervals during the online period.

[0048] In a preferred embodiment, the queue occupancy rate is determined by the number of packets occupied by the interface sending queue and the upper limit of the interface queue, and the congestion deduction item is calculated from the queue occupancy rate using a preset upper limit threshold; its mathematical expression formula is as follows: in, For the first Congestion deduction for each candidate route entry; For the first Queue occupancy rate of the interface corresponding to each candidate route entry; This is a conversion function based on a preset upper limit threshold.

[0049] It should be noted that in this embodiment, the preset upper limit thresholds are 0.80 and 0.95: when the queue occupancy rate is less than 0.80, the congestion deduction item is 0 to 10; when the queue occupancy rate is between 0.80 and 0.95, the congestion deduction item is 10 to 35; and when the queue occupancy rate is not less than 0.95, the congestion deduction item is 35 to 60. The setting rule for the preset upper limit threshold is: based on the average sending rate of the interface and the burstiness of the target service packets, first determine the maximum queue length corresponding to the acceptable queuing delay, then convert it into a queue occupancy rate threshold, and reserve 0.95 as the strong penalty trigger point when the queue is close to full.

[0050] S2.3. The deviation deduction item, consistency deduction item and congestion deduction item are weighted and summed according to the preset weight vector to obtain the route credibility score, and the candidate route entries are arranged in order to form the route credibility score vector.

[0051] In a preferred embodiment, the preset weight vector includes three weights, corresponding to deviation deduction, consistency deduction, and congestion deduction, respectively, and the sum of the three weights is 1 (e.g., deviation deduction weight 0.30, consistency deduction weight 0.45, and congestion deduction weight 0.25). The route credibility score is formed as follows: for the same candidate route entry, the three types of deductions are weighted and summed according to the preset weight vector to obtain the total deduction value, and then converted into a route credibility score according to the rule that the larger the deduction, the lower the credibility.

[0052] Specifically, in this embodiment, the preset weight vector setting rules are as follows: when the target service group is a reliability-priority service, increase the weight of the consistency deduction item; when the target service group is a low-latency-priority service, increase the weight of the congestion deduction item; when the network has a risk of vulnerability or distortion in the routing control message metric field, increase the weight of the deviation deduction item to suppress abnormal metric paths; the weight vector can be preset in the gateway configuration table according to the service category and called according to the service identifier.

[0053] S2.4. Determine the vector length of the gateway observation state vector based on the state vector length identifier recorded in the gateway observation state vector, and determine the vector length of the route credibility score vector based on the number of entries in the candidate route entry set. Map the route credibility score vector to [0,1] according to the preset score range interval, and write missing flag values ​​for missing entries with insufficient vector length to obtain the standardized score vector.

[0054] In a preferred embodiment, the vector length of the gateway observation state vector is determined by reading the state vector length identifier; the vector length of the route confidence score vector is determined by the number of entries in the candidate route entry set, and is kept in the same order as the candidate route entry set; for the normalization of the route confidence score vector, a preset score range interval is determined first, and then the score is mapped to 0 to 1 according to the interval; the preset score range interval is 0 to 100, where 0 represents the lowest confidence and 100 represents the highest confidence; if a candidate route entry does not have complete scoring conditions in the current period (for example, its corresponding interface is in a long missing marker state), then a missing marker value is written at the corresponding position of the score vector; the missing marker value is -1 to distinguish it from the normalized interval of 0 to 1, and the missing marker value is retained in subsequent splicing for model identification.

[0055] For example, when the number of candidate route entries is 6 and the scoring range is 0 to 100, the scores of 5 of the entries are calculated to be 88, 76, 63, 55 and 40 respectively, and the score of the other entry is written as missing value -1 due to long missing tag; then the standardized scoring vector is 0.88, 0.76, 0.63, 0.55, 0.40 and -1.

[0056] S2.5. Concatenate the standardized scoring vector and the gateway observation state vector in a fixed order with the gateway observation state vector first and the standardized scoring vector last, and write the adversarial perception vector length identifier to generate the adversarial perception state vector.

[0057] The length identifier of the adversarial perception vector is an integer field, including a first length field and a second length field; wherein, the first length field is the vector length of the gateway observation state vector, and the second length field is the vector length of the route trustworthiness score vector.

[0058] Preferably, the gateway observation state vector provides cross-interface raw environment observations, and the standardized scoring vector provides item-level interpretable credibility priors. When the two are combined in a fixed order, the model input includes both the environment state and the item credibility constraints, and the input structure remains verifiable when the number of candidate items changes through the adversarial perception vector length identifier. Compared with common reinforcement learning routing schemes that drive action output solely based on observation state, this embodiment introduces a scoring vector based on a deduction strategy, which makes the model action space subject to credibility threshold filtering constraints at the output stage, thereby reducing the probability of degraded actions caused by abnormal measurement fields, short-term congestion bursts, or missing receipts being executed.

[0059] S3. Input the adversarial awareness state vector into the routing optimization model built on a deep Q-network, output a set of candidate routing actions, and filter the candidate routing actions according to the routing credibility score vector to obtain the routing execution actions. Note that the following should be noted in this step: S3.1 Write the adversarial perception state vector into the policy input buffer, and perform a fixed-length check on the adversarial perception state vector according to the adversarial perception vector length identifier to obtain the model input vector.

[0060] In a preferred embodiment, the policy input buffer is a circular buffer structure, and the most recent adversarial perception state vectors are written according to a preset sampling period index for online decision-making and offline training sampling reading; when writing the adversarial perception state vectors into the policy input buffer, the first length field and the second length field in the adversarial perception vector length identifier are written at the same time.

[0061] Specifically, the method for performing fixed-length verification in this embodiment is as follows: the expected input length of the model is pre-stored in the gateway configuration table; during writing, the adversarial perception vector length identifier is read and the actual input length is calculated; when the actual input length is less than the expected input length of the model, the vector is padded at the end with a preset padding value, and the source of the padding is written to the verification mark field; the preset padding value is preferably 0 for the gateway observation state vector part and preferably -1 for the standardized scoring vector part to maintain semantic consistency; when the actual input length is greater than the expected input length of the model, it is truncated according to the rule of retaining the full gateway observation state vector plus retaining the first N terms of the scoring vector, and the truncation start position is written to the verification mark field; where N is determined by the maximum number of candidate entries supported by the model.

[0062] As an example, when the model expects the input length to be 256, and the actual length of the adversarial perception state vector in a certain period is 240, then 16 padding values ​​are added to the end; when the number of candidate route entries in a certain period increases to the actual length of 300, then the end of the scoring vector is truncated according to the preset maximum number of candidate entries, and the truncation marker field is written.

[0063] S3.2 Input the model input vector into the online network of the deep Q network, and output the action Q value vector that corresponds one-to-one with the candidate route entry set, wherein the index order of the action Q value vector is consistent with the order of the entries in the candidate route entry set.

[0064] It should be noted that the deep Q-network in this embodiment includes an online network and a target network. The online network is used for action evaluation in the current cycle, and the target network is used to stabilize the target value during the training phase. The input of the online network is the model input vector, and the output is an action Q-value vector corresponding one-to-one with the candidate route entry set, and the output index order is consistent with the entry order of the candidate route entry set. The structure of the online network is as follows: after the input layer, two fully connected layers and one output layer are connected in sequence. The number of neurons in each fully connected layer can be selected as 256 or 128, and the number of neurons in the output layer is equal to the maximum number of candidate entries supported by the model. When the number of candidate route entries is less than the maximum number of candidate entries, the action Q-values ​​at the redundant positions corresponding to the output layer are ignored by the masking mark in the subsequent selection phase.

[0065] The calculation of the action Q-value vector can be expressed by the following mathematical expression: in, The input vector for the model; This is the action Q-value vector; Forward mapping functions for online networks; This is the set of parameters for an online network.

[0066] During the training phase, the online network parameters are updated based on the following formula: in, The target Q value; This is the reward value; Discount factor; This serves as the model input vector for the next cycle. Candidate actions for the next cycle; This is used to estimate the Q-value of the target network for the state and action in the next cycle. This is the set of parameters for the target network.

[0067] In this embodiment, the reward value The composition of the reward is related to forwarding latency, packet loss, receipt matching results, and queue occupancy changes. For example, positive rewards are given to actions with successful receipts and small end-to-end hop counts, while negative rewards are given to actions with missing receipts or increased congestion. The reward value ranges from -1 to +1 and is configured according to the business category.

[0068] In a preferred embodiment, the reward value ranges from -1 to +1, and the reward components and value levels are configured according to the business category. The reward value is obtained by summing up several components according to preset weights. The components include: receipt matching results, end-to-end hop count changes, queue occupancy changes, interface packet loss rate changes, interface round-trip latency changes, and protocol switching event records.

[0069] Reference Figure 3 Here is a set of example mapping tables, where the values ​​are for illustrative purposes only. The gateway can load these values ​​from the configuration table by service identifier. Figure 3 The diagram illustrates that successful or missing receipt matching is determined by the statistical results of the receipt matching window; changes in the end-to-end hop count are obtained by comparing the end-to-end hop count field with the corresponding entry of the previous preset sampling period index; changes in queue occupancy are determined by the difference in queue occupancy rate corresponding to the queue status parameter between adjacent sampling periods; and handover is determined by the protocol handover event record.

[0070] S3.3. Based on the action Q-value vector, select the corresponding entries from largest to smallest according to the preset selection number to obtain the candidate route action set. Each action in the candidate route action set contains a next-hop identifier and a protocol stack selection identifier, and the index order of each action is consistent with the order of the entries in the candidate route entry set.

[0071] In a preferred embodiment, the preset selection number is the number of candidate action entries selected from the action Q-value vector, preferably 2 to 5; 2 can be selected in scenarios with a small number of interfaces and a small number of candidate entries, and 5 can be selected in scenarios with a large number of candidate entries and large fluctuations in link status; when selecting corresponding entries based on the action Q-value vector from large to small, it is preferable to first perform masking processing on the masked position, and then select the aforementioned preset selection number of entries according to the Q-value; each entry corresponds to a candidate routing action, and the candidate routing action includes a next-hop identifier and a protocol stack selection identifier, and inherits the index order of the candidate routing entry set.

[0072] It should be noted that in this embodiment, by pre-setting the number of selections, redundant candidates are reserved for subsequent credibility screening while keeping the search space controllable, so as to avoid having no action to choose from due to a single high Q-value action being eliminated after credibility screening.

[0073] S3.4. Based on the index order of the candidate route action set, read the route credibility score corresponding to each candidate route action in the route credibility score vector, generate a candidate score set, compare the candidate score set with the preset credibility threshold, write the removal mark for candidate route actions with route credibility scores lower than the preset credibility threshold, and retain the route credibility score higher than the preset credibility threshold, and finally obtain the filtered candidate route action set.

[0074] It should be noted that when a certain score in the candidate score set is a missing tag value, it is considered as low confidence and a removal tag is written to the corresponding candidate route action.

[0075] In this embodiment, the preset credibility threshold is a normalized scoring threshold, preferably 0.60 to 0.80; 0.75 can be selected in reliability-priority services, and 0.65 can be selected in low-latency-priority services; the setting rule for the preset credibility threshold is: based on the scoring distribution of samples whose receipt missing ratio does not exceed the second threshold and whose queue occupancy rate does not exceed the first upper limit threshold, the lower quantile is taken as the threshold so that the threshold has the same distinguishability as the actual operating data.

[0076] S3.5 Select the first item from the candidate route action set in descending order of the action Q value vector as the route execution action; when the candidate route action set is empty, select the candidate route action corresponding to the first item from descending order of the route credibility score vector as the route execution action.

[0077] It should be noted that the action Q-value reflects the strategy's estimate of long-term returns in the current state. The filtered set has eliminated low-confidence candidates, so selecting the action with the highest Q-value under the confidence constraint can balance long-term benefits and operational reliability. This fallback path does not rely on Q-value estimation, but directly sorts the entries by their prior confidence, so that even when all model outputs do not meet the threshold, relatively more reliable routing entries can still be selected, reducing the risk of delays and drops in service packets due to the lack of routing actions. Compared to the method of directly determining the route by reinforcement learning, this embodiment adds a confidence threshold screening and fallback mechanism before execution, making the strategy more robust to responses to metric anomalies, missing receipts, and congestion bursts, and reducing routing oscillations and frequent protocol switching.

[0078] S4. Based on the routing execution action, perform forwarding and protocol switching on the target service packets. Protocol switching is triggered when the current protocol's reliability is below the switching threshold and the alternative protocol's reliability is above the access threshold. Upon triggering, the next hop with the minimum end-to-end hop count is selected to complete the forwarding. Note that the following points should be noted in this step: S4.1 Determine the current protocol stack selection identifier and the next hop identifier based on the routing execution action, and write the target service packet into the sending queue corresponding to the current protocol stack selection identifier.

[0079] It should be noted that the action index in the route execution action is read, and the corresponding entry is accessed using the action index as the entry number in the candidate route entry set. The accessed entry is then identified as the target route entry. When the action index exceeds the number of entries in the candidate route entry set, a modulo operation is performed based on the action index and the number of entries to obtain a corrected action index. The corrected action index is then used as the entry number in the candidate route entry set to access the corresponding entry. The accessed entry is then identified as the target route entry. The value of the protocol stack selection field of the target route entry is used as the current protocol stack selection identifier, and the value of the next hop field of the target route entry is used as the next hop identifier.

[0080] Specifically, the method for writing the target service packet into the sending queue is as follows: write the next hop identifier and the queue identifier corresponding to the current protocol stack selection identifier into the header of the target service packet, and enqueue the target service packet into the sending queue corresponding to the current protocol stack selection identifier according to the queue identifier; when the sending queue reaches the interface queue limit, write the enqueue failure event into the congestion event record, and trigger the queue status parameter update in the next sampling period.

[0081] For example, when the routing action is action index=5 and the number of candidate route entries is 6, the target route entry is obtained by accessing the 5th entry; its protocol stack selection field is cellular protocol stack and its next hop field is neighbor 7; then the current protocol stack selection identifier is cellular protocol stack, the next hop identifier is neighbor 7, and the target service packet is enqueued into the cellular transmission queue.

[0082] S4.2. Based on the routing trustworthiness score vector, read the current protocol trustworthiness corresponding to the current protocol stack selection identifier, and read the trustworthiness of each candidate protocol in the pre-selected protocol set to obtain the protocol trustworthiness comparison set; wherein, the pre-selected protocol set is the protocol stack set corresponding to the multi-protocol interface enabled by the smart gateway.

[0083] In a preferred embodiment, the method for reading the current protocol credibility is as follows: based on the current protocol stack selection identifier, filter the set of entries corresponding to the protocol stack selection identifier in the candidate route entry set, read the score of the corresponding entry from the route credibility score vector, and obtain the current protocol credibility by taking the maximum score within the same protocol.

[0084] In a preferred embodiment, the method for reading the reliability of candidate protocols is as follows: for each candidate protocol in the pre-selected candidate protocol set, the entries are filtered in the candidate route entry set according to the same rules and the score is read from the route reliability score vector. The reliability of the candidate protocol is obtained by taking the maximum score within the same protocol. Finally, a protocol reliability comparison set is formed, which records the correspondence between the protocol stack selection identifier and the protocol reliability.

[0085] For example, when the pre-configured candidate protocol set includes an Ethernet protocol stack, a cellular protocol stack, and a Wi-Fi protocol stack, the current protocol stack selection identifier is the cellular protocol stack; if the highest score for the cellular protocol stack entry in the candidate route entry set is 0.63, then the current protocol's credibility is 0.63; if the highest score for the Ethernet protocol stack entry is 0.88, then the Ethernet candidate protocol's credibility is 0.88; if the highest score for the Wi-Fi protocol stack entry is 0.70, then the Wi-Fi candidate protocol's credibility is 0.70; and a protocol credibility comparison set is formed accordingly.

[0086] S4.3 When the current protocol's credibility is lower than the handover threshold and the candidate protocol's credibility is higher than the access threshold, select the candidate protocol with the highest credibility from the candidate protocols that meet the access threshold as the target protocol stack selection identifier, and replace the current protocol stack selection identifier with the target protocol stack selection identifier.

[0087] In a preferred embodiment, both the switching threshold and the access threshold are normalized scoring thresholds, and the access threshold is not lower than the switching threshold to avoid frequent switching when the credibility is similar. In this embodiment, the values ​​of the switching threshold and the access threshold are 0.60 and 0.75, respectively. The selection rule for the switching threshold is: take the lowest tolerance lower bound of the protocol credibility for the current service category. The selection rule for the access threshold is: take the minimum value of the alternative protocol that is significantly better than the current protocol, and set it as the switching threshold + a fixed margin, where the fixed margin is 0.10 to 0.20.

[0088] Specifically, when the current protocol's credibility is lower than the switching threshold and the credibility of the alternative protocol is higher than the access threshold, the method for selecting the target protocol stack selection identifier is as follows: among the alternative protocols that meet the access threshold, first sort them by credibility from highest to lowest, and take the first one as the target protocol stack selection identifier; when alternative protocols have the same credibility, prioritize the one with the smaller end-to-end hop count, and then prioritize the one with the lower queue occupancy rate to break the tie; then, replace the current protocol stack selection identifier with the target protocol stack selection identifier, and write it into the protocol switching event record. The protocol switching event record should at least include the preset sampling period index, the original protocol stack selection identifier, and the set of comparisons between the target protocol stack selection identifier and the protocol credibility at the time of triggering.

[0089] For example, if the current protocol confidence level is 0.55, which is less than the switching threshold of 0.60; the Ethernet alternative protocol confidence level is 0.88, which is greater than the access threshold of 0.75; and the Wi-Fi alternative protocol confidence level is 0.70, which does not meet the access threshold, then the Ethernet protocol stack is selected as the target protocol stack selection identifier and the current protocol stack selection identifier is replaced.

[0090] S4.4. Based on the target protocol stack selection identifier, filter the candidate route entry set to obtain a subset of target route entries, and select the next hop identifier with the smallest end-to-end hop number from the subset of target route entries to replace the next hop identifier. Then send the target service packet according to the target protocol stack selection identifier and the next hop identifier.

[0091] In a preferred embodiment, the method for filtering a subset of target route entries from the candidate route entry set based on the target protocol stack selection identifier is as follows: traverse the candidate route entry set, retain entries whose protocol stack selection field is equal to the target protocol stack selection identifier, and form a subset of target route entries; for each entry in the subset of target route entries, read its end-to-end hop number segment; the end-to-end hop number segment may be derived from the path information of the routing table entry or calculated by the gateway based on the neighbor table and upstream routing information, and written when the candidate route entry is generated.

[0092] Furthermore, the method for selecting the next-hop identifier with the smallest end-to-end hop count from the subset of target routing entries to replace the previous next-hop identifier is as follows: sort the subset of target routing entries in ascending order of end-to-end hop count, and take the field value of the next-hop field of the first entry as the next-hop identifier after replacement; when the end-to-end hop counts are tied, select the next-hop identifier with the higher routing reliability score according to the rule; after the replacement is completed, send the target service packet with the next-hop identifier selected according to the target protocol stack and the next-hop identifier after replacement; when sending, attach the next-hop identifier to the target service packet for forwarding by the link layer or network layer, and record the sending count of the packet for statistics in the acknowledgment matching window.

[0093] For example, the target protocol stack is identified as the Ethernet protocol stack. There are three entries in the target routing entry subset, with end-to-end hop counts of 3, 2, and 2, respectively; and corresponding scores of 0.80, 0.88, and 0.76, respectively. The entry with 2 hops and a score of 0.88 is selected first, and its next hop field is neighbor 3. After replacing the next hop identifier with neighbor 3, the target service packet is sent according to the Ethernet protocol stack.

[0094] It should be noted that, provided the access threshold is met, paths with fewer end-to-end hops typically correspond to lower cumulative queuing and forwarding overhead, and the acknowledgment loop within the acknowledgment matching window is faster, facilitating the updating of forwarding consistency parameters and scores in the next cycle, thereby reducing the convergence time after protocol switching. Compared to simply replacing the next hop with a single highest-scoring entry, this embodiment introduces a minimum end-to-end hop count constraint within the target protocol, which can suppress latency inflation and jitter accumulation caused by similar scores but lengthy paths, reducing performance fluctuations of service packets during switching.

[0095] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-protocol routing optimization method for intelligent gateways based on reinforcement learning, characterized in that, include: Under a preset sampling period, multi-protocol interface link quality parameters, queue status parameters, and forwarding consistency parameters are collected and preprocessed to form a gateway observation status vector; A weighted deduction strategy is used to calculate the route credibility score vector on the gateway observation state vector, and the route credibility score vector is concatenated and fused with the gateway observation state vector to obtain the adversarial perception state vector; The adversarial perception state vector is input into the routing optimization model built on a deep Q-network, and a set of candidate routing actions is output. The routing actions are then filtered according to the routing credibility score vector to obtain the routing execution actions. According to the routing execution action, the target service packets are forwarded and the protocol is switched. The protocol switching is triggered when the current protocol credibility is lower than the switching threshold and the alternative protocol credibility is higher than the access threshold. When triggered, the next hop with the minimum end-to-end hop count is selected to complete the forwarding.

2. The reinforcement learning-based intelligent gateway multi-protocol routing optimization method according to claim 1, characterized in that, The gateway observation state vector is formed by: At the beginning of each preset sampling period, the smart gateway reads the link quality parameters, queue status parameters and forwarding consistency parameters from each multi-protocol interface, and writes each parameter into the preset sampling period index to form the original parameter set. Based on the preset sampling period index, the original parameter set is timestamped and aligned, and missing items are filled with the most recent valid value of the same interface to obtain the aligned parameter set. The alignment parameter set is normalized according to parameter type, wherein the link quality parameter is mapped to [0,1] according to a preset range interval, the queue status parameter is mapped to [0,1] according to the upper limit of the interface queue, and the forwarding consistency parameter is mapped to [0,1] according to the missing ratio within the receipt matching window, thus obtaining a standardized parameter set; The standardized parameter set is concatenated in a fixed order of interface identifier, parameter type, and sampling period index, and then written into the state vector length identifier to generate the gateway observation state vector.

3. The multi-protocol routing optimization method for intelligent gateways based on reinforcement learning according to claim 2, characterized in that, A weighted deduction strategy is used to calculate the route reliability score vector based on the gateway observation state vector, including: Based on the gateway observation state vector, a set of candidate route entries is obtained by enumerating the next hop identifier and the protocol stack selection identifier, and the index positions of the corresponding link quality parameters, queue state parameters and forwarding consistency parameters are written for each candidate route entry. For each candidate route entry, a deviation deduction item is calculated, wherein the deviation deduction item is obtained by converting the difference between the measured route metric value determined by the link quality parameters and the route metric field value carried in the route control message corresponding to the candidate route entry through a preset mapping table; for each candidate route entry, a consistency deduction item is calculated, wherein the consistency deduction item is obtained by converting the acknowledgment missing ratio corresponding to the forwarding consistency parameters through a preset segmentation threshold set; for each candidate route entry, a congestion deduction item is calculated, wherein the congestion deduction item is obtained by converting the queue occupancy rate corresponding to the queue status parameters through a preset upper limit threshold. The deviation deduction item, the consistency deduction item, and the congestion deduction item are weighted and summed according to a preset weight vector to obtain the route credibility score, and then arranged in the order of the candidate route entries to form a route credibility score vector.

4. The multi-protocol routing optimization method for intelligent gateways based on reinforcement learning according to claim 3, characterized in that, The route credibility score vector is concatenated and fused with the gateway observation state vector to obtain the adversarial awareness state vector, including: The vector length of the gateway observation state vector is determined based on the state vector length identifier recorded in the gateway observation state vector, and the vector length of the route credibility score vector is determined by the number of entries in the candidate route entry set. The route credibility score vector is mapped to [0,1] according to a preset score range interval, and missing entries that are less than the vector length are written with missing flag values ​​to obtain a standardized score vector. The standardized scoring vector and the gateway observation state vector are concatenated in a fixed order, with the gateway observation state vector first and the standardized scoring vector second, and an adversarial perception vector length identifier is written in to generate the adversarial perception state vector.

5. The multi-protocol routing optimization method for intelligent gateways based on reinforcement learning according to claim 4, characterized in that, The adversarial awareness vector length identifier is an integer field, including a first length field and a second length field; wherein, the first length field is the vector length of the gateway observation state vector, and the second length field is the vector length of the route trustworthiness score vector.

6. The multi-protocol routing optimization method for intelligent gateways based on reinforcement learning according to claim 4, characterized in that, The adversarial perception state vector is input into the routing optimization model, which outputs a set of candidate routing actions, including: Write the adversarial perception state vector into the policy input buffer, and perform a fixed-length check on the adversarial perception state vector according to the adversarial perception vector length identifier to obtain the model input vector; The model input vector is input into the online network of the deep Q-network, and the output is an action Q-value vector that corresponds one-to-one with the candidate route entry set, wherein the index order of the action Q-value vector is consistent with the entry order of the candidate route entry set; Based on the action Q-value vector, corresponding entries are selected from largest to smallest according to a preset selection number to obtain a candidate routing action set. Each action in the candidate routing action set includes a next-hop identifier and a protocol stack selection identifier, and the index order of each action is consistent with the order of the entries in the candidate routing entry set.

7. The reinforcement learning-based intelligent gateway multi-protocol routing optimization method according to claim 1 or 6, characterized in that, The route execution actions are obtained by filtering the candidate route actions, including: Based on the index order of the candidate routing action set, read the routing credibility score corresponding to each candidate routing action in the routing credibility score vector, generate a candidate score set, compare the candidate score set with a preset credibility threshold, write a removal mark for candidate routing actions whose routing credibility score is lower than the preset credibility threshold, and obtain a filtered candidate routing action set. In the set of candidate route actions, the first item is selected as the route execution action in descending order of the action Q value vector; when the set of candidate route actions is empty, the candidate route action corresponding to the first item is selected as the route execution action in descending order of the route credibility score vector.

8. The multi-protocol routing optimization method for intelligent gateways based on reinforcement learning according to claim 7, characterized in that, According to the routing execution action, the target service packets are forwarded and the protocol is switched, including: Based on the routing execution action, the current protocol stack selection identifier and the next hop identifier are determined, and the target service packet is written into the sending queue corresponding to the current protocol stack selection identifier. Based on the routing reliability score vector, the current protocol reliability corresponding to the current protocol stack selection identifier is read, and the reliability of each candidate protocol in the pre-selected protocol set is read to obtain the protocol reliability comparison set; wherein, the pre-selected protocol set is the protocol stack set corresponding to the multi-protocol interface enabled by the smart gateway; When the credibility of the current protocol is lower than the switching threshold and the credibility of the alternative protocol is higher than the access threshold, the alternative protocol with the highest credibility among the alternative protocols that meet the access threshold is selected as the target protocol stack selection identifier, and the current protocol stack selection identifier is replaced by the target protocol stack selection identifier. Based on the target protocol stack selection identifier, a subset of target route entries is obtained by filtering from the candidate route entry set. Then, the next-hop identifier with the smallest end-to-end hop count is selected from the subset of target route entries to replace the next-hop identifier. The target service packet is then sent according to the target protocol stack selection identifier and the next-hop identifier.

9. The multi-protocol routing optimization method for intelligent gateways based on reinforcement learning according to claim 8, characterized in that, The method for determining the current protocol stack selection identifier and the next-hop identifier based on the routing execution action includes: reading the action index in the routing execution action, and accessing the corresponding entry using the action index as the entry number of the candidate route entry set, and determining the accessed entry as the target route entry; wherein, when the action index exceeds the number of entries in the candidate route entry set, a modulo operation is performed based on the action index and the number of entries to obtain a corrected action index, and the corresponding entry is accessed using the corrected action index as the entry number of the candidate route entry set, and the accessed entry is determined as the target route entry.

10. The multi-protocol routing optimization method for intelligent gateways based on reinforcement learning according to claim 9, characterized in that, Also includes: The value of the protocol stack selection field of the target route entry is used as the current protocol stack selection identifier, and the value of the next hop field of the target route entry is used as the next hop identifier.

Images