Reinforcement learning-based signaling recombination method, apparatus and device, and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a signaling recombination method and device based on reinforcement learning, equipment and a medium, and the method comprises the steps: receiving a signaling forwarded by an IP multimedia subsystem; extracting a target signaling feature in the target signaling; sending the target signaling feature to a reinforcement learning agent so as to judge whether the target signaling is the first message and predict the arrival time of the next signaling of the target signaling; and if the target signaling is not the first message, the next signaling of the target signaling is stored in the alternative message area based on the arrival time, and when the target signaling is successfully matched with the alternative signaling in the alternative message area, the target signaling is forwarded. According to the invention, based on the reinforcement learning algorithm, the next predicted signaling is maintained by adopting a linked list insertion sorting mode, so that the signaling matching speed is improved, the processing delay is reduced, the accuracy and efficiency of message recombination are improved, and the stability and reliability of the system are ensured.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a signaling reorganization method, device, equipment and medium based on reinforcement learning. Background Art

[0002] In modern communication networks, IP Multimedia Subsystem (IMS) has become an important infrastructure for providing high-quality multimedia communication services. IMS manages and controls multimedia sessions through the Session Initiation Protocol (SIP), supporting multiple communication forms such as voice, video, and instant messaging. However, when IMS signaling is transmitted between networks of different operators, network fluctuations and delays not only affect the communication quality, but may also cause serious signaling errors, which in turn affect the user experience and the normal operation of the business.

[0003] In cross-operator scenarios, IMS signaling needs to pass through multiple network nodes and border gateways, which are managed and maintained by different operators. The network architecture, equipment performance, and management strategies of each operator may differ significantly, which often manifests as network latency and jitter. These factors can cause varying degrees of delay or even misordering of signaling messages during transmission. In actual deployment applications, the performance of network equipment used by different operators also differs significantly. High-performance equipment can quickly process and forward signaling messages, while low-performance equipment may cause message backlogs and delays. The inconsistency and differences in equipment performance often lead to signaling misordering.

[0004] Existing solutions are mainly based on protocol stack implementation. Although the message reordering mechanism based on the protocol stack solves the problem of misordered signaling to a certain extent, it still has significant defects in cross-operator scenarios. The misordered message discovery mechanism based on CSeq is only effective when the CSeq is less than the previous message. When messages are missed or missing, the mechanism cannot solve the problem and will still cause session abnormalities due to processing messages in the wrong sequence. Summary of the invention

[0005] In view of this, the present invention provides a signaling reorganization method, device, equipment and medium based on reinforcement learning to solve the problem that the existing message reordering mechanism has poor message sorting effect in cross-operator application scenarios.

[0006] In a first aspect, the present invention provides a signaling reorganization method based on reinforcement learning, the method is executed by a central controller, and the method includes:

[0007] Receive signaling forwarded by the IP multimedia subsystem;

[0008] Extracting target signaling features from the target signaling; the target signaling features include a timestamp and a message header;

[0009] Send the target signaling features to the reinforcement learning agent to determine whether the target signaling is the first message and predict the arrival time of the next signaling of the target signaling;

[0010] If the target signaling is not the first message, the next signaling of the target signaling is stored in the alternative message area based on the arrival time, otherwise the target signaling is forwarded; the alternative message area includes a signaling linked list method, which is used to represent the set of next predicted signaling of each signaling;

[0011] When the target signaling successfully matches the candidate signaling in the candidate message area, the target signaling is forwarded.

[0012] The signaling reorganization method based on reinforcement learning provided in the embodiment of the present invention predicts the arrival time of the next information according to the target signaling characteristics after the central controller receives the signaling, and inserts it into the alternative message area according to the arrival time. When the target signaling successfully matches other information in the alternative message area, it is forwarded to avoid processing delays and resource waste caused by signaling backlog, provide more reliable protection for signaling transmission in IMS across networks, and ensure high-quality multimedia communication services.

[0013] In an optional implementation, if the target signaling is not the first message, the next signaling of the target signaling is stored in the candidate message area based on the arrival time, further comprising:

[0014] Based on the arrival time, the next signaling is inserted into the signaling linked list to ensure that the signaling linked list sorts the signals according to the arrival time.

[0015] The signaling reorganization method based on reinforcement learning provided by the embodiment of the present invention is that when the reinforcement learning agent predicts the next signaling according to the algorithm, the alternative message area is inserted into the existing signaling linked list according to its predicted arrival time by means of linked list insertion sorting, thereby ensuring that the linked list is always arranged in order, thereby improving the signaling matching speed and processing efficiency.

[0016] In an optional embodiment, the method further includes:

[0017] If the target signaling and the candidate signaling in the candidate message area are not matched successfully, the target signaling that is not successfully matched is stored in the waiting buffer;

[0018] When the next signal predicted by the reinforcement learning agent based on the first signal matches the buffered signal in the waiting buffer, the buffered signal is forwarded, otherwise the next signal of the first signal is stored in the candidate message area.

[0019] The signaling reorganization method based on reinforcement learning provided in an embodiment of the present invention sets a waiting buffer. When the target signaling fails to match the alternative signaling in the alternative message area, storing it in the waiting buffer can provide an additional processing opportunity. That is, even if the initial match fails, the signaling will not be immediately discarded or ignored, but will be re-evaluated and processed in the subsequent process.

[0020] In an optional implementation, the target signaling feature is sent to the reinforcement learning agent to determine whether the target signaling is the first message and predict the arrival time of the next signaling of the target signaling, further comprising:

[0021] Send the signaling features to the reinforcement learning agent to determine whether the signaling is a redundant message;

[0022] If the signaling is a redundant message, the redundant message is discarded.

[0023] The signaling reorganization method based on reinforcement learning provided in the embodiment of the present invention can effectively avoid introducing network congestion caused by too many retransmitted messages into the IMS network by judging whether it is a redundant message and discarding the redundant message.

[0024] In an optional embodiment, the method further includes:

[0025] If the next signaling of the target signaling has not arrived after the preset time, the next signaling of the target signaling is deleted from the signaling linked list.

[0026] The signaling reorganization method based on reinforcement learning provided in the embodiment of the present invention sets a message management mechanism. For signals that have not arrived within a preset time, the system will automatically delete them from the linked list, thereby avoiding long-term occupation of memory resources and improving system operation efficiency.

[0027] In an optional embodiment, sending the target signaling feature to the reinforcement learning agent further includes:

[0028] Extracting pre-signaling features of several pre-signalings before the target signaling;

[0029] Generate a feature tensor based on the target signaling feature and the pre-signaling feature;

[0030] The feature tensor and the network state vector are sent to the reinforcement learning agent; the network state vector is determined by the proxy server based on the target signaling features and the pre-signaling features.

[0031] The reinforcement learning-based signaling reorganization method provided in an embodiment of the present invention is a central controller that extracts features of preceding signaling and combines them with features of target signaling to generate richer feature tensors, which helps the reinforcement learning agent to more accurately understand the context and pattern of signaling and better adapt to different network environments and communication modes, thereby improving the accuracy of predicting the arrival time of the next signaling.

[0032] In an optional implementation, the target signaling includes session initiation protocol signaling.

[0033] In an optional implementation, before receiving the signaling forwarded by the IP multimedia subsystem, the method further includes:

[0034] Obtain historical decision data;

[0035] Generate strategy analysis vectors based on historical decision data;

[0036] Input the analysis vector into the reinforcement learning agent to obtain the decision result;

[0037] Update the neural network parameters of the reinforcement learning agent based on the decision results.

[0038] The signaling reorganization method based on reinforcement learning provided by the embodiment of the present invention realizes the functions of self-repair and self-optimization by continuously learning and optimizing its own decision-making process, thereby improving the accuracy and efficiency of message reorganization. This allows the intelligent agent to gradually understand the actual business process autonomously without the need to train the business process in advance, and to reasonably distribute signaling, greatly reducing the signaling process maintenance work of operation and maintenance personnel.

[0039] In a second aspect, the present invention provides a signaling reorganization device based on reinforcement learning, the device comprising:

[0040] A receiving module, used for receiving signaling forwarded by the IP multimedia subsystem;

[0041] An extraction module, used to extract target signaling features in the target signaling; the target signaling features include a timestamp and a message header;

[0042] A judgment module is used to send the target signaling feature to the reinforcement learning agent to determine whether the target signaling is the first message and predict the arrival time of the next signaling of the target signaling;

[0043] A storage module, used for storing the next signaling of the target signaling in the alternative message area based on the arrival time if the target signaling is not the first message, otherwise forwarding the target signaling; the alternative message area includes a signaling linked list method, which is used to represent the set of next predicted signaling of each signaling;

[0044] The forwarding module is used to forward the target signaling when the target signaling successfully matches the candidate signaling in the candidate message area.

[0045] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the signaling reorganization method based on reinforcement learning of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0046] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the signaling reorganization method based on reinforcement learning of the above-mentioned first aspect or any corresponding embodiment thereof.

[0047] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the signaling reorganization method based on reinforcement learning of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 is a system schematic diagram of a signaling reorganization system based on reinforcement learning according to an embodiment of the present invention;

[0050] Figure 2 is a schematic diagram of the architecture of a signaling reorganization server of a signaling reorganization system based on reinforcement learning according to an embodiment of the present invention;

[0051] Figure 3 is a flow chart of a signaling reorganization method based on reinforcement learning according to an embodiment of the present invention;

[0052] Figure 4 is a flow chart of another signaling reorganization method based on reinforcement learning according to an embodiment of the present invention;

[0053] Figure 5 is a structural block diagram of a signaling reorganization device based on reinforcement learning according to an embodiment of the present invention;

[0054] Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0056] In modern communication networks, IP Multimedia Subsystem (IMS) is a key infrastructure for providing high-quality multimedia communication services. It implements effective management and control of multimedia sessions through the Session Initiation Protocol (SIP). The SIP protocol supports multiple communication forms such as voice, video, and instant messaging, enabling IMS to provide users with a rich communication experience in cross-operator scenarios. However, when IMS signaling is transmitted between networks of different operators, network fluctuations, latency, and differences in device performance become particularly prominent. These challenges not only affect the quality of communication, but may also cause signaling misordering, loss, or duplication, which in turn causes session anomalies, affecting user experience and normal business operation.

[0057] In cross-operator docking, the SIP protocol stack is the core component of IMS signaling processing, and its built-in CSeq field is used to identify and sort each request message. However, the current misordered message discovery mechanism based on the CSeq field is powerless in the face of complex and changing network environments. On the one hand, the CSeq mechanism assumes that all messages are transmitted through the same path, but in cross-operator scenarios, this assumption is often not true, resulting in message sorting errors; on the other hand, when the CSeq mechanism finds misordered messages, it only temporarily stores them, lacking an effective sorting and cleaning mechanism, resulting in a backlog of internally stored signaling, affecting system performance. In addition, different operators have different specific implementations of the SIP protocol, further exacerbating the complexity of signaling compatibility and sorting issues.

[0058] Based on this, the inventors discovered that a cross-network message reorganization mechanism based on reinforcement learning can solve the deficiencies in signaling misorder processing in cross-operator scenarios. By dynamically adapting to changes in network status and business processes, and using reinforcement learning algorithms to predict the next signaling and its estimated arrival time, corresponding processing preparations can be made before the message arrives, effectively solving the signaling backlog problem.

[0059] According to an embodiment of the present invention, an embodiment of a signaling reorganization method based on reinforcement learning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0060] In this embodiment, a signaling reorganization method based on reinforcement learning is provided, which can be used in a signaling reorganization system based on reinforcement learning. Figure 1 As shown, the structure of the signaling reorganization server (ASRS) in the system is as follows Figure 2 As shown, it aims to solve the problem of IMS signaling reorganization across different operator networks. It provides a complete design framework, including six modules: PROXY, central controller, alternative message area, waiting buffer area, training center, and experience database. The method is executed by the central controller. Figure 3 is a flow chart of a signaling reorganization method based on reinforcement learning according to an embodiment of the present invention, such as Figure 3 As shown, the process includes the following steps:

[0061] Step S301: receiving a signaling forwarded by an IP multimedia subsystem.

[0062] Specifically, when the IMS network receives signaling, it will record the current network status information according to the received signaling data packet and in combination with the current network environment.

[0063] Furthermore, the central controller in the signaling reassembly server receives the signaling content sent by the IMS network and the recorded network status information.

[0064] Step S302: extract target signaling features from the target signaling.

[0065] Specifically, the received target signaling is usually a structured data packet, so it is necessary to first parse and time-sequence the target signaling to extract the target signaling features, including the timestamp and message header of the target signaling. The timestamp is a key field that records the generation or sending time of the signaling, and the message header contains key information such as the signaling type, destination address, and source address.

[0066] Furthermore, optionally, the extracted timestamp and message header fields are combined into a feature vector, which is subsequently used as an input for the reinforcement learning agent.

[0067] Step S303, sending the target signaling feature to the reinforcement learning agent to determine whether the target signaling is the first message and predict the arrival time of the next signaling of the target signaling.

[0068] Specifically, the feature vector obtained in step S302 is sent to the reinforcement learning agent for analysis and decision-making to obtain the analysis results, including whether there is a preceding message (i.e., whether the target signaling is the first message) and the next message data of the target signaling and the predicted arrival time.

[0069] Step S304: if the target signaling is not the first message, the next signaling of the target signaling is stored in the candidate message area based on the arrival time, otherwise the target signaling is forwarded.

[0070] For example, if target signaling A is received at this time and target signaling A is not the first message, then after obtaining the estimated arrival time of the next signaling B of signaling A based on step S303, signaling B is stored in the alternative message area according to the arrival time. If target signaling A is the first message, it is forwarded directly.

[0071] Furthermore, the candidate message area includes a signaling linked list method, which is used to represent a set of next predicted signalings of each signaling.

[0072] In some optional implementations, the above step S304 further includes:

[0073] Based on the arrival time, the next signaling is inserted into the signaling linked list to ensure that the signaling linked list sorts the signals according to the arrival time.

[0074] The signaling linked list is a storage method that uses the time of arrival of the next signal predicted by the reinforcement learning agent, and then inserts it into the storage using the insertion sort method. It can be seen that this signaling linked list method can change the default storage method of signaling first arriving and existing, and make the signaling sorted according to the preset arrival time, avoiding the extreme situation that the target signaling may be at the end of the signaling queue in the first arriving and existing mode, thereby improving the query efficiency of the signaling.

[0075] Step S305: when the target signaling successfully matches the candidate signaling in the candidate message area, the target signaling is forwarded.

[0076] For example, assuming that there is signaling A before, which is not the first signaling, and the next signaling B is predicted based on signaling A, after signaling B is stored in the alternative message area, the target signaling B arrives at this time, and when it matches the alternative message area, the target signaling B is forwarded.

[0077] In summary, the signaling reassembly method based on reinforcement learning in the embodiment of the present invention, when receiving signaling in the IMS network, is preferentially processed by the adaptive signaling reassembly server. The adaptive signaling reassembly server analyzes the currently received signaling based on the business process through reinforcement learning to determine whether a message disorder occurs.

[0078] When the IMS network receives a signal, it first passes it to the adaptive signaling reassembly server. After receiving the signal, the server immediately performs timing analysis, and predicts the arrival time of the next signal by analyzing the timestamp, message header and other key information of the signal. At the same time, the server determines whether the current signal is the first message. If the current signal is not the first message, the next predicted information of the current target signal is sent to the alternative message area for matching, and the current target signal is matched with the existing signal in the alternative message area. If the match is successful, it means that the current signal sequence is correct and can be directly processed and forwarded. In this way, the server can maintain good performance, avoid processing delays and resource waste caused by signaling backlogs, provide more reliable guarantees for IMS signaling transmission across networks, and ensure high-quality multimedia communication services.

[0079] In some optional embodiments, the method further comprises:

[0080] If the target signaling and the candidate signaling in the candidate message area are not matched successfully, the target signaling that is not successfully matched is stored in the waiting buffer;

[0081] When the next signal predicted by the reinforcement learning agent based on the first signal matches the buffered signal in the waiting buffer, the buffered signal is forwarded, otherwise the next signal of the first signal is stored in the candidate message area.

[0082] The process of this embodiment is as follows Figure 4 As shown, the target signaling includes the session initialization protocol signaling. If the target signaling does not match the existing signaling in the alternative message area, the signaling is considered to be a misordered message and is sent to the waiting buffer area to wait for the arrival of the preceding message. In the waiting buffer area, the arrival of the signaling will continue to be monitored. When the server predicts the next signaling of the first signaling, that is, the second signaling message, based on the first signaling, the second signaling message will be sent to the waiting buffer area for matching. If the match is successful, it means that the post-signaling arrives in advance. At this time, the signaling that successfully matches can be forwarded to ensure the correctness of the message sequence; on the other hand, if the second signaling does not match the existing signaling in the waiting buffer area, the second signaling will be stored in the alternative message area.

[0083] Taking a specific example for illustration, assuming that the signaling process of a desired service is signaling A-signaling B-signaling C-signaling D, but the actual arrival of the signaling is signaling B-signaling A-signaling D-signaling C, then the process of the signaling reorganization method based on reinforcement learning according to an embodiment of the present invention is:

[0084] First, signal B arrives first. Based on the reinforcement learning agent, signal C is predicted and stored in the candidate message area. Signal B cannot match the existing signal in the candidate message area, so signal B is placed in the waiting buffer.

[0085] Next, signaling A arrives and signaling B is predicted. Since signaling A will be judged as the first message, signaling A can be forwarded directly. Signaling B successfully matches the message in the waiting buffer and signaling B is forwarded in the order of signaling A-signaling B.

[0086] Then, signaling D arrives, the predicted signaling is empty, signaling D fails to match the existing signaling in the candidate message area, and is stored in the waiting buffer;

[0087] Finally, signaling C arrives, and signaling D is predicted. Signaling C successfully matches the existing signaling C in the alternative message area, and signaling C is forwarded. Signaling D successfully matches the signaling D in the waiting buffer, and signaling D is forwarded, resulting in the final forwarding order: signaling A-signaling B-signaling C-signaling D.

[0088] In some optional implementations, the target signaling feature is sent to the reinforcement learning agent to determine whether the target signaling is the first message and predict the arrival time of the next signaling of the target signaling, further comprising:

[0089] Send the signaling features to the reinforcement learning agent to determine whether the signaling is a redundant message;

[0090] If the signaling is a redundant message, the redundant message is discarded.

[0091] The server is also used to determine whether the target signaling is a redundant message. If the current signaling is a redundant message, it will be discarded. Since the proxy server (PROXY) can monitor the current network status, such as bandwidth usage, delay, etc., and send this information to the signaling reassembly server (ASRS), so that ASRS can better reassemble and predict signaling. At the same time, a warning is issued to the PROXY to warn of external network congestion and take corresponding measures as needed.

[0092] In an optional embodiment, the method further includes:

[0093] If the next signaling of the target signaling has not arrived after the preset time, the next signaling of the target signaling is deleted from the signaling linked list.

[0094] This embodiment uses a message management mechanism to automatically delete the signaling from the linked list for the signaling that has not arrived after the scheduled time, avoiding long-term occupation of memory resources and improving the system operation efficiency. This is achieved by integrating multiple timers in the central controller. Specifically, the experience cleaning timer and the alarm signal cleaning timer regularly clean up the message data that exceeds the time window, so that the ASRS can monitor the network status in real time and learn the data in the recent period of time, so that the intelligent agent can adapt to the changes in the signaling process in time and optimize its decision-making strategy. At the same time, the central controller also includes an alternative message cleaning timer and a waiting message cleaning timer to regularly and asynchronously clean up the abnormal signaling that has timed out, effectively avoiding the decision-making errors and performance degradation caused by the backlog of signaling inside the ASRS, and ensuring the efficient operation of the system. In addition, the intelligent agent training timer controls the intelligent agent to learn the historical decision-making strategy by regularly controlling the intelligent agent, so that the intelligent agent parameters can be adjusted in time with the changes in the business process, avoiding the accumulation of a large number of decision-making errors caused by changes in the signaling process, so that the intelligent agent can autonomously learn and adapt to the signaling sequence relationship in the latest business process. Through the coordinated work of these timers, the central controller not only ensures the stability and efficiency of the system, but also significantly improves the adaptability and decision-making accuracy of ASRS in complex network environments.

[0095] In summary, the central controller integrates multiple timers to achieve comprehensive management of each module and data. In the decision-making process, the central controller ensures that the adaptive signaling reassembly server (ASRS) can obtain the current network performance status in a timely and accurate manner and effectively clean up user data.

[0096] In an optional embodiment, sending the target signaling feature to the reinforcement learning agent further includes:

[0097] Extracting pre-signaling features of several pre-signalings before the target signaling;

[0098] Generate a feature tensor based on the target signaling feature and the pre-signaling feature;

[0099] The feature tensor and the network state vector are sent to the reinforcement learning agent; the network state vector is determined by the proxy server based on the target signaling features and the pre-signaling features.

[0100] The central controller extracts key feature data, such as message category, timestamp, etc., and integrates the feature data vector of the previous N messages currently received with the current vector to construct a feature tensor, which is used to comprehensively predict future signaling features for the agent based on the previous N signals. If the pre-arrival vector is less than N, it is filled with an all-0 feature vector. At the same time, the message feature vector and the network state vector transmitted by PROXY are preprocessed and provided to the agent for decision-making.

[0101] In an optional implementation, the target signaling includes session initiation protocol signaling.

[0102] In an optional implementation, before receiving the signaling forwarded by the IP multimedia subsystem, the method further includes:

[0103] Obtain historical decision data;

[0104] Generate strategy analysis vectors based on historical decision data;

[0105] Input the analysis vector into the reinforcement learning agent to obtain the decision result;

[0106] Update the neural network parameters of the reinforcement learning agent based on the decision results.

[0107] The specific training process is as follows, taking the SIP protocol as an example: 1. Obtain the data set: load historical decision data from the historical experience database for intelligent agent training and learning.

[0108] 2. Get the actual strategy analysis vector: Calculate whether there is a pre-signaling: According to the actual business, extract the signaling at the beginning of each business, and set the vector to 0 if there is a pre-signaling, and set the vector to 1 for the remaining signaling.

[0109] Calculate the next signaling feature: extract the [from, to, cseq, call-id, type] of the next signaling of each signaling.

[0110] Among them: [from, to, cseq, call-id] are all corresponding header field data, time is the arrival timestamp, type is the message type, and the corresponding mapping relationship is shown in Table 1:

[0111] Table 1: Mapping relationship table:

[0112]

[0113]

[0114] Calculate the deletion time: Calculate the time difference Δt between the current signaling and its post-vector, and take twice the time as the deletion time.

[0115] Calculate the estimated arrival time: Calculate the time difference Δt between this signaling and its post-vector as the estimated arrival time.

[0116] Calculate whether it is a redundant signaling message: Determine whether retransmission occurs within the time window of this signaling message. If this signaling message is a retransmission message, it is regarded as redundant signaling and the vector is set to 1, otherwise it is set to 0.

[0117] Calculate whether signaling reassembly failure occurs: Read the IMS network alarm signal transmitted in PROXY. When a message is discarded in the IMS network, it is considered that signaling reassembly failure occurs, and an alarm is sent to ASRS. If this signaling is discarded by the alarm, the vector is set to 1, otherwise it is set to 0.

[0118] The decision analysis vector is obtained by the training center through the analysis of historical data and the actual network transmission and signaling analysis, and is used to evaluate the intelligent agent strategy in step 5.

[0119] 3. Input state vector processing: Process this vector data: Extract the features [from, to, cseq, call-id, type, time] in the SIP signaling message header, where [from, to, cseq, call-id, type] is consistent with 2, and time is the relative arrival time of the signaling.

[0120] Collect the previous signaling within the time window: Take the feature vectors of the first N arriving signals, integrate them with the current vector, and construct a feature tensor to help the agent comprehensively predict the features of future signals with reference to the first N signals. If the previous arrival vector is less than N, fill it with all-0 feature vectors.

[0121] Collect the network status vector of PROXY statistics: [total number of candidate message timeouts, total number of cached message timeouts, total number of signaling discards, total number of cumulative received signaling];

[0122] Construct state tensor: Merge the feature tensor with the network state vector to form a state tensor.

[0123] 4. Obtain the next state of each piece of data in a dislocated manner: In order to avoid the inability to form a complete Markov state transition sequence due to missing data, the training center will traverse the read historical records one by one. When encountering data with an empty state transition result, the training center will adopt a dislocated acquisition method, that is, using the state of the next record as the state transition result of the current record, thereby ensuring that each piece of data can participate in model training and improve data utilization.

[0124] 5. Strategy evaluation: Extract the agent’s historical decisions from historical experience and compare them with the decision analysis vector obtained by the training center to obtain the reward value:

[0125]

[0126] in, is the action predicted by the agent, is the corresponding strategy analysis vector, where This is the signaling reassembly failure vector. punish i For each penalty, the value is a negative number.

[0127] 6. The agent uses the DDPG algorithm to update the neural network parameters and learn from historical experience: In order to control the agent's strategy to approach the empirical results, additional strategy convergence penalties are added as rewards:

[0128] r=r acc -|a-π(s k |θ)|2

[0129] Among them, a is the empirical strategy, π(s k |θ) is the agent strategy under the current network parameters, r acc is the scoring result among 5, r is the actual score, |·|2 is the two-norm,

[0130] TD algorithm updates the value network:

[0131]

[0132] The gradient ascent algorithm calculates the policy network gradient:

[0133]

[0134] Update network parameters:

[0135]

[0136] in, θ is the policy network and value network parameters, α is the learning rate. The value network scores the current action selected by the agent in the current state, and the policy network is the neural network ultimately used for agent decision-making. Its input is the current state, and its output is the best policy.

[0137] In this embodiment, a signaling reorganization device based on reinforcement learning is also provided, which is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0138] This embodiment provides a signaling reorganization device based on reinforcement learning, such as Figure 5 As shown, including:

[0139] The receiving module 501 is used to receive the signaling forwarded by the IP multimedia subsystem.

[0140] The extraction module 502 is used to extract target signaling features in the target signaling; the target signaling features include a timestamp and a message header.

[0141] The judgment module 503 is used to send the target signaling feature to the reinforcement learning agent to determine whether the target signaling is the first message and predict the arrival time of the next signaling of the target signaling.

[0142] Storage module 504 is used to store the next signaling of the target signaling in the alternative message area based on the arrival time if the target signaling is not the first message, otherwise forward the target signaling; the alternative message area includes a signaling linked list method, which is used to represent the set of next predicted signaling of each signaling.

[0143] The forwarding module 505 is used to forward the target signaling when the target signaling successfully matches the candidate signaling in the candidate message area.

[0144] In some optional implementations, the storage module 504 includes:

[0145] The linked list submodule is used to insert the next signaling into the signaling linked list based on the arrival time to ensure that the signaling linked list sorts the signals according to the arrival time.

[0146] In some optional embodiments, the device further comprises:

[0147] The waiting buffer module is used for storing the unsuccessfully matched target signaling into the waiting buffer if the target signaling fails to match the candidate signaling in the candidate message area.

[0148] The matching signaling module is used to forward the buffered signaling when the next signaling predicted by the reinforcement learning agent based on the first signaling matches the buffered signaling in the waiting buffer, otherwise the next signaling of the first signaling is stored in the alternative message area.

[0149] In some optional implementations, the storage module 503 includes:

[0150] The redundancy judgment submodule is used to send the signaling features to the reinforcement learning agent to determine whether the signaling is a redundant message.

[0151] The discarding submodule is used to discard the redundant message if the signaling is a redundant message.

[0152] In some optional embodiments, the device further comprises:

[0153] The deleting module is used to delete the next signaling of the target signaling from the signaling linked list if the next signaling of the target signaling has not arrived after a preset time.

[0154] In some optional implementations, the storage module 503 further includes:

[0155] The feature extraction submodule is used to extract the pre-signaling features of several pre-signalings before the target signaling.

[0156] The tensor generation submodule is used to generate feature tensors based on target signaling features and pre-signaling features.

[0157] The sending tensor submodule is used to send the feature tensor and the network state vector to the reinforcement learning agent; the network state vector is determined by the proxy server based on the target signaling characteristics and the pre-signaling characteristics.

[0158] In some optional embodiments, the device further comprises:

[0159] The data acquisition module is used to obtain historical decision data.

[0160] The vector generation module is used to generate strategy analysis vectors based on historical decision data.

[0161] The decision module is used to input the analysis vector into the reinforcement learning agent to obtain the decision result.

[0162] The update module is used to update the neural network parameters of the reinforcement learning agent based on the decision results.

[0163] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0164] The signaling reorganization device based on reinforcement learning in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0165] The embodiment of the present invention also provides a computer device having the above Figure 5 The signaling reorganization device based on reinforcement learning is shown.

[0166] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0167] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0168] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0169] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0170] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0171] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0172] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0173] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0174] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0175] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A signaling reorganization method based on reinforcement learning, characterized in that: The method is executed by a central controller, and comprises: Receive signaling forwarded by the IP multimedia subsystem; Extracting target signaling features from the target signaling; the target signaling features include a timestamp and a message header; Sending the target signaling feature to the reinforcement learning agent to determine whether the target signaling is the first message and predicting the arrival time of the next signaling of the target signaling; If the target signaling is not the first message, the next signaling of the target signaling is stored in the alternative message area based on the arrival time, otherwise the target signaling is forwarded; the alternative message area includes a signaling linked list method, which is used to represent the set of next predicted signalings of each signaling; When the target signaling successfully matches the candidate signaling in the candidate message area, the target signaling is forwarded.

2. The method according to claim 1, characterized in that If the target signaling is not the first message, storing the next signaling of the target signaling in the candidate message area based on the arrival time, further comprising: Based on the arrival time, the next signaling is inserted into the signaling linked list to ensure that the signaling linked list sorts the signalings according to the arrival time.

3. The method according to claim 2, characterized in that The method further comprises: If the target signaling and the candidate signaling in the candidate message area are not matched successfully, the target signaling that is not successfully matched is stored in a waiting buffer; When the next signaling predicted by the reinforcement learning agent based on the first signaling matches the buffered signaling in the waiting buffer, the buffered signaling is forwarded; otherwise, the next signaling of the first signaling is stored in the candidate message area.

4. The method according to any one of claims 1 to 3, characterized in that The step of sending the target signaling feature to the reinforcement learning agent to determine whether the target signaling is the first message and predicting the arrival time of the next signaling of the target signaling further includes: Sending the signaling feature to the reinforcement learning agent to determine whether the signaling is a redundant message; If the signaling is a redundant message, the redundant message is discarded.

5. The method according to claim 2, characterized in that: The method further comprises: If the next signaling of the target signaling has not arrived after the preset time, the next signaling of the target signaling is deleted from the signaling linked list.

6. The method according to claim 4, characterized in that The sending of the target signaling feature to the reinforcement learning agent further includes: Extracting pre-signaling features of several pre-signalings before the target signaling; Generate a feature tensor based on the target signaling feature and the pre-signaling feature; The feature tensor and the network state vector are sent to the reinforcement learning agent; the network state vector is determined by the proxy server based on the target signaling feature and the pre-signaling feature.

7. The method according to claim 4, characterized in that The target signaling includes session initiation protocol signaling.

8. The method according to claim 7, characterized in that Before receiving the signaling forwarded by the IP multimedia subsystem, the method further includes: Obtain historical decision data; generating a strategy analysis vector based on the historical decision data; Inputting the analysis vector into the reinforcement learning agent to obtain a decision result; The neural network parameters of the reinforcement learning agent are updated based on the decision result.

9. A signaling reorganization device based on reinforcement learning, characterized in that: The device comprises: A receiving module, used for receiving signaling forwarded by the IP multimedia subsystem; An extraction module, used to extract target signaling features in the target signaling; the target signaling features include a timestamp and a message header; A judgment module, used to send the target signaling feature to the reinforcement learning agent to determine whether the target signaling is the first message and predict the arrival time of the next signaling of the target signaling; A storage module, used for storing the next signaling of the target signaling in the alternative message area based on the arrival time if the target signaling is not the first message, otherwise forwarding the target signaling; the alternative message area includes a signaling linked list method, which is used to represent the set of next predicted signaling of each signaling; A matching module, configured to forward the target signaling when the target signaling successfully matches the candidate signaling in the candidate message area; The forwarding module is used to forward the target signaling when the target signaling in the candidate message area successfully matches the remaining signaling in the candidate message area.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the signaling reorganization method based on reinforcement learning as described in any one of claims 1 to 8 by executing the computer instructions.

Images