VoIP and eLink conversation seamless switching decision-making method based on reinforcement learning

Abstract

The invention relates to the technical field of communication, and discloses a VoIP and eLink conversation seamless switching decision-making method based on reinforcement learning, which comprises the following steps of: when a new incoming call event is received, comprehensively evaluating a real-time workflow state of a user and contact values of a current conversation and a new incoming call through a high-level decision-making module so as to determine a macroscopic strategy target; the method comprises the following steps of: selecting a strategy target, preferentially processing a new incoming call or protecting a current call, and under the guidance of the strategy target, further combining real-time network quality, terminal equipment resources and other technical conditions by a low-layer decision module, and deciding and executing an atomic operation, such as keeping state or direct sound mixing. According to the method, a layered reinforcement learning framework is adopted, the problem that interests of multiple parties cannot be balanced due to the adoption of a static rule is solved by dynamically sensing and reasoning multi-dimensional contexts such as user states, communication values and technical resources, the balance is carried out between the guarantee of communication opportunities and the protection of user working focuses, and the user experience is improved. And the user experience and the working efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of communication technology, specifically to a decision-making method for seamless switching between VoIP and eLink calls based on reinforcement learning. Background Technology

[0002] With the development of modern communication technologies, integrating and running multiple communication systems on a single terminal device has become commonplace. For example, VoIP technology, due to its flexibility and low cost, is widely used in enterprise and personal communications. Meanwhile, many enterprises also deploy proprietary communication systems based on different protocols (collectively referred to here as eLink) for internal high-security or high-priority communication. Therefore, users often face scenarios where a VoIP call is in progress while a new eLink call is coming in, or vice versa, when using a PC client or mobile device. This presents a technical requirement for smooth, efficient, and intelligent switching and management between these concurrent, heterogeneous call events.

[0003] To handle such concurrent call scenarios, existing technologies typically employ static strategies based on preset rules. These strategies are usually configured by the administrator during system deployment or hard-coded during software development. For example, the system is configured so that eLink calls always have higher priority than VoIP calls. When a VoIP call is in progress, any incoming eLink call will automatically trigger the current VoIP call to be put on hold and the eLink call to be connected. Alternatively, the system is configured not to make any automatic decisions, but simply to display notifications and option buttons on the user interface, such as "Hold and answer" or "Hang up and answer," leaving the entire decision-making burden to the user manually. The underlying technical implementation of these methods is relatively straightforward: it triggers a fixed signaling interaction flow by monitoring the type of incoming call, such as sending an RFC-compliant SIPre-INVITE request to control whether the call is held or hung up.

[0004] However, this decision-making mechanism, which relies entirely on preset static rules, reveals its limitations when faced with the complex, dynamic, and ever-changing real-world scenarios of modern office environments. A core problem is that this mechanism lacks the ability to perceive and understand context, and cannot intelligently weigh conflicting decision-making objectives. For example, if a user is on a routine VoIP call with a regular contact but is in a highly focused workflow (e.g., coding or writing documents), an incoming eLink call, which is judged to be high priority according to static rules but is actually not urgent, will cause the system to interrupt the user's current call and workflow without consideration, resulting in unnecessary interference and severely impacting work efficiency and user experience. Conversely, if a user is on a non-essential VoIP call and is idle, an overly conservative static strategy (such as only providing a pop-up notification) might cause the user to fail to respond promptly when an important, commercially valuable eLink call comes in from a client, leading to a missed opportunity. This mechanical approach cannot dynamically measure and compare the maintenance value of the current call, the potential value of a new call, and the opportunity cost of interrupting the user's current workflow, thus failing to make optimal decisions that align with the user's true intentions and maximize overall benefits. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a seamless switching decision method for VoIP and eLink calls based on reinforcement learning. This method solves the problem that existing communication terminals often use preset fixed rules for switching or management when handling multiple concurrent VoIP and eLink calls. This lacks the ability to perceive the user's current work status and call context, and cannot dynamically prioritize different calls based on their potential importance. This results in unnecessary interruptions to the user's workflow or missed important calls.

[0006] To address the aforementioned technical problems, this invention provides a reinforcement learning-based decision-making method for seamless switching between VoIP and eLink calls.

[0007] This method constructs a hierarchical decision-making system, which decomposes complex call management tasks into two stages: high-level strategy decision-making and low-level resource execution.

[0008] The high-level policy agent is responsible for selecting the optimal policy objective from the perspective of user intent and macro-value, while the low-level resource agent is responsible for executing the policy objective with the best service quality under the current technological conditions.

[0009] The method includes the following steps: S1. Monitor the communication interface in real time to receive VoIP or eLink new call events that trigger decisions; S2. Obtain a high-level state consisting of the user workflow state and a communication session summary containing the new incoming call event; The high-level state A vector can be represented as a multidimensional feature vector: ; in, For the user workflow state vector, This is a communication session summary vector. For environment and schedule vectors, For historical decision preference vectors, For time.

[0010] S3. The high-level policy agent, based on the high-level state, selects a set of preset policy objectives. In the middle, determine a strategic objective. ; S4. Obtain the low-level state consisting of the policy objective, network quality, and device resources; The lower-level state A vector can be represented as a multidimensional feature vector: ; in, The current strategy objective to be executed. For network quality vectors, For device resource vectors, An audio stream attribute vector. For time.

[0011] S5. The low-level resource agent, based on the low-level state, performs a set of preset atomic operations. In this process, an atomic operation is determined for performing the policy objective. ; S6. Convert the atomic operation into a control command and send the control command to the hardware audio processing unit or software interface of the VoIP phone to change the audio stream path or connection status of the current call, thereby realizing the switching or management of VoIP and eLink calls.

[0012] Preferably, the user workflow state includes a probability distribution of whether the user is currently in a focused state, a collaborative state, or an idle state. This probability distribution serves as an input dimension of the high-level state and is used by the high-level policy agent to perform decision calculations to assess the timing of executing the policy objective.

[0013] In a specific embodiment, the step of obtaining the user workflow status in step S2 specifically includes: Analyze the user's behavior sequence to determine the user's workflow status; The behavioral sequence includes keyboard input frequency, mouse movement pattern, and currently active application window.

[0014] The analysis processes the behavioral sequence using a sequence model, such as a long short-term memory network model, to output the probability distribution.

[0015] Preferably, the communication session summary includes the call type of the current call and the new incoming call, as well as the contact's value level. The contact value level is used by the high-level policy agent to evaluate the priority of different calls.

[0016] The value level of the contact can be obtained from the customer relationship management system or calculated based on historical interaction frequency data.

[0017] Furthermore, in step S3, the high-level policy agent is based on a high-level reward function. Determine the objectives of the strategy.

[0018] The high-level reward function is based on opportunity value reward. User veto punishment The weighted summation is used to construct the formula.

[0019] in, Calculated based on the contact's value level. It is a preset negative reward value applied when a system decision is manually rejected by the user.

[0020] The formula for calculating the high-level reward function is as follows: ; in, and These are the weighting coefficients for opportunity value rewards and user veto penalties, respectively. Represents the value of communication. This represents the opportunity cost of interruption.

[0021] In a specific embodiment, step S5, which involves determining the atomic operation, specifically includes: First, assess whether the network quality and device resources meet a quality of service threshold. Then, the atomic operation is determined only if the quality of service threshold is met; The service quality thresholds include the minimum mean opinion score (MOS), the maximum network latency, and the maximum CPU load.

[0022] Furthermore, in step S5, the low-level resource agent is based on a low-level reward function. Given the atomic operations, the formula for calculating the low-level reward function is as follows: ; in, It is an intrinsic reward used to evaluate the performance of the atomic manipulation technique. It is a pre-set positive extrinsic reward obtained upon successfully achieving the stated strategic objective. .

[0023] Preferably, the intrinsic reward is based on the delay generated after the atomic operation is performed. Resource consumption and estimated average opinion score Determine which atomic operation has the least impact on call quality.

[0024] The intrinsic reward is determined by a preset quality of service function. The calculation yields the following result: ; Preferably, in step S3, the strategy objective is selected from: Combine calls into a conference, prioritize new calls, protect the current call, or process calls in sequence; The purpose of the high-level policy agent in selecting the policy objective is to achieve a seamless or low-interference transition between multiple call requests.

[0025] Preferably, in step S5, the atomic operation is selected from: Direct mixing, audio stream transcoding followed by mixing, holding in hold mode, or playing a notification tone; The low-level resource agent selects the atomic operations to maintain the continuity of the audio stream during the switching or management process, in order to achieve seamless switching.

[0026] This invention provides a reinforcement learning-based decision-making method for seamless handover between VoIP and eLink calls, which has the following beneficial effects: 1. This invention obtains and utilizes the contact value level in the user's workflow state and communication session summary as a high-level state, enabling the high-level policy agent to make decisions based on the user's real-time work context and call priority. This makes call management decisions no longer based on fixed, undifferentiated rules, but can adapt to the user's current work needs, avoiding interruptions from low-priority calls when the user is focused on work, while ensuring that high-value communication requests are prioritized, thus improving the accuracy and context adaptability of decisions.

[0027] 2. This invention adopts a hierarchical decision-making architecture with high-level policy agents and low-level resource agents, decomposing the complex call management task into two independent sub-problems: determining the policy objective and executing atomic operations. This hierarchical structure reduces the complexity of the state space and action space that each agent needs to process when making decisions, thereby accelerating the convergence speed of the reinforcement learning model and reducing the computational overhead of the system in real-time decision-making, resulting in a more stable and efficient decision-making model.

[0028] 3. This invention assesses the current network quality and equipment resources before making decisions by having a low-level resource agent evaluate them. It then selects specific atomic operations based on an intrinsic reward function that includes latency, resource consumption, and average opinion score. This ensures that any call switching or management operation is performed under technically permissible conditions and in a manner that minimizes the impact on call quality, providing a technical guarantee for achieving continuous and uninterrupted audio streaming and high-quality seamless switching. Attached Figure Description

[0029] Figure 1 This is a structural block diagram of a seamless call handover decision system according to an embodiment of the present invention; Figure 2 A flowchart illustrating a reinforcement learning-based decision-making method for seamless switching between VoIP and eLink calls according to an embodiment of the present invention; Figure 3 This is a diagram illustrating the internal structure and data flow of a high-level decision-making module according to an embodiment of the present invention. Figure 4 This is a diagram illustrating the internal structure and data flow of a low-level decision module according to an embodiment of the present invention. Figure 5 This is a schematic diagram of a seamless call handover decision-making process in a specific scenario, according to an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] See attached document Figure 1 , Figure 1 This is a structural block diagram of a seamless call handover decision system according to an embodiment of the present invention.

[0032] The seamless call handover decision system can be implemented in a terminal device 100, which includes, but is not limited to, a personal computer, a workstation, or a dedicated communication terminal.

[0033] The terminal device 100 includes one or more processors 110, a memory 120, and one or more communication interfaces 130.

[0034] The processor 110, memory 120 and communication interface 130 are interconnected and communicate with each other through a bus system.

[0035] The processor 110 is responsible for executing the program instructions stored in the memory 120 to control the overall operation of the system.

[0036] Communication interface 130 is responsible for handling data transmission and reception for communication protocols such as VoIP and eLink.

[0037] The memory 120 stores computer program instructions, which, when executed by the processor 110, are used to implement a reinforcement learning-based decision-making method for seamless switching between VoIP and eLink calls.

[0038] Specifically, the program instructions stored in memory 120 can be divided into multiple functional modules, including: data monitoring module 121, high-level decision-making module 122, low-level decision-making module 123, and instruction execution module 124.

[0039] The data monitoring module 121 is used to monitor the communication interface 130 in real time to receive VoIP or eLink new call events that trigger decisions.

[0040] In addition, the data monitoring module 121 is also responsible for collecting the data required to construct the decision state, including user behavior sequences and communication session information for constructing the high-level state, as well as network quality and device resource information for constructing the low-level state.

[0041] The high-level decision-making module 122 has an embedded high-level policy agent.

[0042] The high-level decision-making module 122 obtains the user workflow status and communication session summary from the data monitoring module 121, which together constitute the high-level status. vector.

[0043] Subsequently, the high-level policy agent determines a policy objective based on the high-level state. .

[0044] The high-level state can be represented as: ; in, For the user workflow state vector, This is a communication session summary vector. For environment and schedule vectors, For historical decision preference vectors, For time.

[0045] The low-level decision-making module 123 has an embedded low-level resource intelligence agent.

[0046] The lower-level decision-making module 123 obtains the determined strategic objectives from the higher-level decision-making module 122. It also obtains network quality and device resource information from the data monitoring module 121, which together constitute the low-level status. vector.

[0047] Subsequently, the low-level resource agent, based on this low-level state... A vector determines an atomic operation used to perform a policy objective. .

[0048] Low-level state A vector can be represented as: ; in, The current strategy objective to be executed. For network quality vectors, For device resource vectors, An audio stream attribute vector. For time.

[0049] Instruction execution module 124 is responsible for executing the atomic operations determined by lower-level decision module 123. This is converted into specific hardware or software control instructions.

[0050] Subsequently, the instruction execution module 124 sends control instructions to the hardware audio processing unit of the VoIP phone or its software interface to change the audio stream path or connection status of the current call, ultimately realizing the switching or management of VoIP and eLink calls.

[0051] See attached document Figure 2 , Figure 2 This is a flowchart of a reinforcement learning-based decision-making method for seamless handover between VoIP and eLink calls according to an embodiment of the present invention.

[0052] This method can be implemented by the processor 110 in the aforementioned terminal device 100 executing program instructions in the memory 120, and its overall process specifically includes the following steps: S1. After the system starts, the data monitoring module 121 continuously monitors the communication interface 130.

[0053] When a new VoIP or eLink call event occurs, such as receiving a SIPINVITE request message or an eLink call setup signaling, the event serves as the trigger signal for the decision-making process.

[0054] The data monitoring module 121 analyzes the event and extracts initial information such as the caller ID and call type.

[0055] S2. Upon receiving the trigger signal, the data monitoring module 121 actively obtains the high-level construction status from multiple data sources. The original data required for the vector.

[0056] This includes: calling operating system interfaces to obtain information about currently active application windows, keyboard input frequency, and mouse movement patterns, as well as other user behavior sequences; querying local or remote contact databases to obtain the caller's value level; and obtaining the status information of currently existing calls.

[0057] This raw data is preprocessed and integrated into a standardized vector form, namely the high-level state. The vector is then sent to the high-level decision module 122.

[0058] S3, High-level decision-making module 122 receives high-level status. After the vector is applied, its embedded high-level policy agent will transfer the high-level state. Vectors are used as input.

[0059] In a specific model implementation, the high-level policy agent is a deep Q-network.

[0060] As a function approximator, the goal of a deep Q-network is to learn an optimal action-value function. Used to evaluate in any state Next action Its long-term value.

[0061] Deep Q network receives high-level state After the vector is processed, a forward propagation calculation is performed, and a Q-value vector is output.

[0062] The Q-value vector is a list of values, where each element corresponds to the expected cumulative reward for a preset strategy goal (such as merging calls into conferences, prioritizing new incoming calls, etc.), which is the Q-value.

[0063] Specifically, if the system has Optional strategic objectives The network will then output a Q-value vector. .

[0064] The high-level decision-making module 122 then determines an optimal policy objective from the Q-value vector based on a preset action selection strategy. .

[0065] For example, during the exploitation phase, a greedy strategy can be directly adopted to select the action with the maximum Q value: During the exploration phase, an ε-greedy strategy can be adopted, that is, using... The action corresponding to the maximum Q value is chosen with a probability of [a certain percentage], while [another action] is chosen with a very small probability. Choose an action randomly to explore more possibilities.

[0066] S4, Strategic Objectives Once confirmed, it is sent to the lower-level decision-making module 123.

[0067] At the same time, the data monitoring module 121 is triggered to perform a real-time resource probe.

[0068] This includes: sending probe packets to specific network nodes to measure the current network's round-trip latency and packet loss rate; and calling the operating system API to query the real-time load of the CPU and memory.

[0069] These real-time resource data and strategy objectives They were collectively integrated into a lower-level state. Vectors serve as inputs for lower-level decisions.

[0070] S5, Low-level decision module 123 receives low-level status. After the vector, its embedded low-level resource agent first compares the network quality and device resource indicators in the state with a preset quality of service threshold.

[0071] The decision-making process will only continue when all indicators meet the service quality threshold requirements.

[0072] Once the conditions are met, the low-level resource agent (which can also be a DQN model) will change the low-level state. The vector is used as input for forward propagation calculation, and the output is a Q-value vector corresponding to each atomic operation (such as direct mixing, audio stream transcoding and post-mixing, etc.).

[0073] The low-level decision module 123 ultimately selects the atomic operation with the largest Q value. As output.

[0074] S6. Finalized Atomic Operations It is sent to instruction execution module 124.

[0075] The function of instruction execution module 124 is to perform this atomic operation as a logical instruction. Translate into one or more specific, executable control instructions.

[0076] For example, if the atomic operation is placed in a hold state, the instruction execution module 124 will generate a SIPre-INVITE request conforming to RFC3264, which includes the SDP attribute that sets the media stream to inactive.

[0077] The generated instructions are then sent to the communication interface 130 or the associated hardware audio processing unit for execution, thereby completing the change of the call state.

[0078] The decision-making process is now complete.

[0079] The following section will elaborate on the internal implementation principles of the high-level decision-making module 122, and how it executes steps S2 and S3 of the method. (See appendix for details.) Figure 3 , Figure 3 This is an internal structure and data flow diagram of a high-level decision-making module according to an embodiment of the present invention.

[0080] In step S2, the high-level decision module 122 is responsible for constructing the high-level state. Vector. High-level state The vector is formed by integrating multi-source heterogeneous data provided by the data monitoring module 121.

[0081] In one specific embodiment, high-level state A vector contains the following parts: First is the user workflow state vector .

[0082] The data monitoring module 121 collects user behavior sequences within a preset time window (e.g., the most recent 60 seconds). The behavior sequences include: keyboard input frequency measured in keystrokes per second, mouse movement distance measured in pixels per second, and number of clicks per second, as well as a one-hot code representing the type of the currently active application window.

[0083] The sequence of behaviors is fed into a pre-trained long short-term memory network model.

[0084] Long Short-Term Memory (LSTM) network models process time-series data, and their output is a three-dimensional probability distribution vector, such as [0.8, 0.1, 0.1], which corresponds to the probability that the user is currently in a focused state, a collaborative state, or an idle state, respectively.

[0085] This vector is .

[0086] The pre-training process of the Long Short-Term Memory Network model includes: First, collecting a large number of user behavior sequence samples, and simultaneously labeling these samples with corresponding real workflow state labels by user manual labeling or according to specific application scenario rules (e.g., labeling a code editor in full-screen mode as a focused state when no instant messaging application is running).

[0087] Then, these labeled datasets are divided into training and validation sets. Supervised learning is used to iteratively train the LSTM model using the backpropagation algorithm and cross-entropy loss function until the classification accuracy of the Long Short-Term Memory network model on the validation set reaches a preset threshold (e.g., 95%).

[0088] Secondly, there is the communication session summary vector. .

[0089] Summary Vector It describes the current overall communication situation, mainly including the call type of the current call and new incoming call (e.g., 0 represents VoIP, 1 represents eLink) and the contact value level.

[0090] In one specific embodiment, contact value level It is quantified into an integer value from 1 to 5. The integer value can be obtained by the data monitoring module 121 by querying a local or remote customer relationship management database, or calculated based on historical communication frequency, duration and other data, according to a preset scoring rule.

[0091] In a specific implementation, the scoring rules calculated based on historical data can be given by the following formula: .

[0092] in, Total number of historical calls Average call duration The time interval since the last call (e.g., in days). , , For their respective preset weight coefficients, Represents the natural logarithm function, that is, expressed in terms of the natural constant. Logarithm with base 0.

[0093] Use here This is a common smoothing technique used to reduce the excessive impact of extremely high call counts and to ensure that historical call counts are minimized. Even when the value is 0, the function is still defined.

[0094] It represents the natural constant, an irrational number approximately equal to 2.71828.

[0095] In this formula, it functions as an exponential function. The base is used to construct an exponentially decaying term. The value of this term increases with the time interval since the last call. The value rapidly approaches 0 as the number of contacts increases, which allows recently contacted contacts to receive higher scores, consistent with the recency effect principle of communication importance.

[0096] The calculation results were then normalized and mapped to integer levels from 1 to 5.

[0097] For example, 5 represents the highest value contact, and 1 represents an unknown or low value contact.

[0098] In step S3, the high-level decision module 122 makes decisions based on the constructed high-level state. A vector determines an optimal policy objective. .

[0099] Strategic objectives are derived from a pre-defined set of strategic objectives. The selection includes: merging calls into a conference, prioritizing new incoming calls, protecting the current call, or processing calls in sequence.

[0100] These objectives correspond to macro-level operations such as mixing the audio streams of two calls, putting the current call on hold and answering a new call, rejecting a new call or transferring it, putting a new call in the waiting queue and playing a notification tone to the user.

[0101] The high-level policy agent embedded in the high-level decision-making module 122 can be implemented as a deep Q-network.

[0102] The number of neurons in the input layer and the high-level states of a deep Q-network The dimensions of the vectors match.

[0103] Number of output layer neurons and policy target set of the network The number of targets is equal, and each output neuron corresponds to the Q-value of a policy target.

[0104] A high-level policy agent uses a high-level reward function. Conduct training to learn the optimal decision-making strategy.

[0105] The formula for calculating the high-level reward function is: ; in, As an opportunity value reward, in one embodiment, it can be calculated as the difference between the value level of the new caller and the value level of the current caller. .

[0106] To penalize user vetoes, when a policy objective executed by the system is vetoed by the user through manual action (e.g., manually hanging up or switching) within a preset time, It is assigned a fixed negative value, such as -10.

[0107] and It is a preset weighting coefficient used to adjust the relative importance of the two rewards.

[0108] Represents the value of communication. This represents the opportunity cost of interruption.

[0109] During decision-making, the high-level policy agent will consider the current high-level state. Inputting the DQN network, the Q-values ​​of all possible policy objectives are obtained through a single forward propagation calculation.

[0110] The high-level decision module 122 then selects the strategy objective with the largest Q value as the result of this decision. And pass it to the lower-level decision module 123.

[0111] The following section will elaborate on the internal implementation principles of the low-level decision module 123, and how it executes steps S4 and S5 of the method. (See appendix.) Figure 4 , Figure 4 This is an internal structure and data flow diagram of a low-level decision module according to an embodiment of the present invention.

[0112] In step S4, the low-level decision module 123 is responsible for constructing the low-level state. vector.

[0113] The lower-level decision module 123 first receives the determined strategy objective from the higher-level decision module 122. .

[0114] At the same time, the data monitoring module 121 is triggered to obtain real-time data to describe the current technical environment.

[0115] These real-time data collectively constitute the lower-level state. A vector, in one specific embodiment, comprises the following parts: First, the network quality vector .

[0116] The data monitoring module 121 measures and quantifies network performance in real time by sending ICMP probe packets or parsing RTCP reports to relevant network nodes (e.g., SIP proxy servers or media gateways for VoIP services).

[0117] Network Quality Vector It can be specifically expressed as [latency, jitter, packet loss rate], for example [80,25,0.5], which represents a round-trip latency of 80 milliseconds, jitter of 25 milliseconds, and a packet loss rate of 0.5%.

[0118] Secondly, there is the device resource vector. .

[0119] The data monitoring module 121 obtains the current resource load of the terminal device 100 by calling the application programming interface provided by the operating system.

[0120] Device resource vector It can be specifically expressed as [CPU load, memory load], for example [0.65, 0.50], which represent 65% CPU utilization and 50% memory utilization, respectively.

[0121] In step S5, the low-level decision module 123 makes decisions based on the constructed low-level state. A vector determines an optimal atomic operation. .

[0122] The execution process of step S5 is divided into two stages.

[0123] The first phase is the service quality threshold assessment.

[0124] Before making model-based decisions, the low-level decision module 123 first performs a deterministic pre-check.

[0125] It will be the lower-level state The network quality and device resource metrics in the vector are compared with a set of preset service quality thresholds.

[0126] The set of service quality thresholds includes: a minimum mean opinion score (MOS) estimate (e.g., 3.5), a maximum network latency (e.g., 150 milliseconds), and a maximum CPU load (e.g., 80%).

[0127] The decision-making process only proceeds to the next stage when all real-time metrics are within the allowable range of the service quality threshold.

[0128] If any metric fails to meet the requirements, the system will execute a preset, safe default action, such as rejecting new calls or playing a busy tone, to avoid a decrease in call quality due to insufficient resources.

[0129] The second stage is model-based selection of atomic operations.

[0130] Once the service quality threshold is met, the low-level resource agent embedded in the low-level decision module 123 begins to work.

[0131] The low-level resource agent can be implemented as a deep Q-network, with the low-level state as its input. Vector, output as a set of atomic operations Q-value evaluation for each operation.

[0132] Atomic operations are specific technical actions that control audio streams, such as direct mixing, audio stream transcoding followed by mixing, putting the stream in hold mode, or playing a cue tone.

[0133] Low-level resource agents use a low-level reward function Training is conducted to learn how to select operations that have the least impact on technical performance while meeting strategic objectives.

[0134] The training process for low-level resource agents can be carried out in an offline environment based on historical data or simulation.

[0135] During training, the system randomly samples a batch of transition samples from an experience replay pool, which includes (current low-level state, atomic operation performed, intrinsic reward obtained, next low-level state, and whether the policy objective has been achieved).

[0136] Based on these samples, the weights of the deep Q-network are updated using optimization algorithms such as Adam, with the goal of minimizing the loss function defined by the Bellman equation.

[0137] Through extensive iterative training, low-level resource agents can learn which atomic operation can maximize the cumulative Q-value, including both immediate intrinsic rewards and long-term extrinsic rewards, given a low-level state and policy objective.

[0138] The formula for calculating the low-level reward function is: ; in, As an external reward, it is a reward for ultimately achieving high-level strategic goals. A fixed, large, positive constant that is assigned only at certain times. (For example, +50), while in all other intermediate steps it is 0.

[0139] As an intrinsic reward, it is used to evaluate the technical performance of a single atomic operation itself.

[0140] It uses a preset quality of service function. The calculation yields the following result: ; in, The additional processing delay introduced to perform this atomic operation The CPU and memory resources consumed to perform this operation This is the estimated average opinion score for the call after performing this operation.

[0141] In one embodiment, the function can be defined as a weighted sum: .

[0142] in , , These are the preset weighting coefficients.

[0143] This inherent reward mechanism guides agents to choose atomic operations that have lower latency, consume fewer resources, and have a greater positive impact on call quality (measured by MOS score).

[0144] During decision-making, the low-level resource agent will consider the current low-level state. The vector is input into the DQN network, and the Q-values ​​of all optional atomic operations are obtained through one forward propagation.

[0145] The low-level decision module 123 ultimately selects the atomic operation with the largest Q value as the result of this decision. And pass it to instruction execution module 124.

[0146] See attached document Figure 5 , Figure 5 This is a schematic diagram of a seamless call handover decision-making process in a specific scenario according to an embodiment of the present invention.

[0147] The following will describe the complete workflow of the reinforcement learning-based VoIP and eLink call seamless switching decision method provided by this invention through a specific implementation scenario.

[0148] In an initial state, suppose a user is using terminal device 100 to make a call to a contact via VoIP.

[0149] Meanwhile, the data monitoring module 121 analyzes the user's behavioral sequence (e.g., continuous keyboard input and concentrated application window switching) and calculates through its internal LSTM model that the probability that the user is currently in a focused state is 0.9.

[0150] The value level of this VoIP call contact was confirmed as Level 2 (ordinary contact) by querying the CRM database.

[0151] S1. At this time, the communication interface 130 receives a new eLink call event.

[0152] The data monitoring module 121 analyzes the new call event and finds that the caller is a contact with a value level of 5 (high-value contact).

[0153] This event triggered the entire decision-making process.

[0154] S2, Data Monitoring Module 121 immediately constructs high-level status. vector.

[0155] High-level status The vector specifically includes: user workflow state vector Its value is [0.9, 0.1, 0.0], representing the probabilities of being focused, collaborative, and idle, respectively; Communication session summary vector It contains information that the current call value is 2 and the new call value is 5.

[0156] High-level status The vector is transmitted to the high-level decision module 122.

[0157] S3, High-level decision-making module 122 receives high-level status. vector.

[0158] Its embedded high-level policy agent performs forward computation.

[0159] In the forward calculation, since the value of a new incoming call is much higher than the value of a current call, the opportunity value reward... It produces a large positive value.

[0160] Despite the user's focused state, the weighting of high-value communications resulted in the Q-value of the strategy objective of prioritizing new calls being calculated to be the highest.

[0161] Therefore, the high-level decision-making module 122 determines the strategic objectives. To prioritize handling new incoming calls and output them.

[0162] S4, Low-level decision-making module 123 receives the strategy objective. .

[0163] Simultaneously, the data monitoring module 121 performs a rapid technical environment probe to obtain the network quality vector. The device resource vector is [45, 15, 0.1] (representing a latency of 45ms, jitter of 15ms, and packet loss rate of 0.1%). The value is [0.40, 0.35] (representing a CPU load of 40% and a memory load of 35%).

[0164] These data and strategic objectives Together they constitute the lower-level state vector.

[0165] S5, Low-level decision-making module 123 first checks the low-level state The technical indicators in the vector are used to assess service quality thresholds.

[0166] Since the 45ms latency and 40% CPU load were all within the preset acceptable range, the evaluation was passed. Subsequently, the low-level resource agent evaluated the available atomic operations to achieve the goal of prioritizing the processing of new incoming calls.

[0167] Among the optional atomic operations, placing the call in a hold state introduces less additional resource consumption and causes less disruption to the original call compared to directly ending the current call, thus its intrinsic reward is lower. The value is higher.

[0168] Ultimately, the low-level resource agent outputs atomic operations with the maximum Q value. That is, it is placed in the hold state.

[0169] The result of low-level decision module 123 is the inevitable choice made by the low-level resource agent based on its training objectives.

[0170] During its training, the low-level resource agent has learned that, given sufficient network and device resources, placing the atomic operation of maintaining a state is the intrinsic reward for prioritizing the processing of new incoming calls. (due to its lower resource consumption) And the original call MOS division The zero negative impact of this decision is far higher than other operations (such as direct mixing, which leads to chaotic calls and a lower MOS score; or hanging up directly, which results in a very low intrinsic reward due to a complete interruption of communication).

[0171] Therefore, it has the highest long-term cumulative Q value.

[0172] S6. The instruction execution module 124 receives the atomic operation and puts it in a holding state.

[0173] The instruction execution module 124 converts it into two specific control instruction sequences: Instruction 1 generates a SIPRE-INVITE request with the a=sendonly media attribute for the current VoIP call to put the VoIP call on hold; Instruction 2 generates a signaling to accept new eLink calls. These two instructions are sent to the communication interface 130 for execution.

[0174] The final result is that the user's current VoIP call is smoothly placed on hold, while the system automatically connects to an eLink call from a high-value contact, thus preserving the user's original work context while ensuring the priority of high-value communications.

Claims

1. A seamless switching decision method for VoIP and eLink calls based on reinforcement learning, characterized in that, Includes the following steps: S1. Monitor the communication interface in real time and receive VoIP or eLink new call events that trigger decision-making. S2. Obtain a high-level state consisting of the user workflow state and a communication session summary containing the new incoming call event; S3. The high-level policy agent determines a policy objective based on the high-level state. S4. Obtain the low-level state consisting of the policy objective, network quality, and device resources; S5. The low-level resource agent determines an atomic operation to execute the policy objective based on the low-level state. S6. Convert the atomic operation into a control command and send the control command to the hardware audio processing unit or software interface of the VoIP phone to change the audio stream path or connection status of the current call, thereby realizing the switching or management of VoIP and eLink calls.

2. The seamless handover decision-making method for VoIP and eLink calls based on reinforcement learning according to claim 1, characterized in that, The user workflow state includes a probability distribution of whether the user is currently in a focused state, a collaborative state, or an idle state. This probability distribution is used by the high-level policy agent to evaluate the timing of executing the policy objective.

3. The seamless handover decision-making method for VoIP and eLink calls based on reinforcement learning according to claim 2, characterized in that, The step of obtaining the user workflow status in step S2 specifically includes: Analyze the user's behavior sequence to determine the user's workflow status; The behavioral sequence includes keyboard input frequency, mouse movement pattern, and currently active application window.

4. The seamless handover decision-making method for VoIP and eLink calls based on reinforcement learning according to claim 1, characterized in that, The communication session summary includes the call type of the current call and the new incoming call, as well as the contact value level. The contact value level is used by the high-level policy agent to evaluate the priority of different calls.

5. The seamless handover decision-making method for VoIP and eLink calls based on reinforcement learning according to claim 4, characterized in that, In step S3, the high-level policy agent determines the policy objective based on a high-level reward function, which includes: An opportunity value reward calculated based on the contact's value level, and a preset negative reward value applied when the system decision is manually rejected by the user.

6. The seamless handover decision-making method for VoIP and eLink calls based on reinforcement learning according to claim 1, characterized in that, Step S5, which determines the atomic operation, specifically includes: First, assess whether the network quality and device resources meet a quality of service threshold. Then, the atomic operation is determined only if the quality of service threshold is met; The service quality thresholds include the minimum mean opinion score (MOS), the maximum network latency, and the maximum CPU load.

7. The seamless handover decision-making method for VoIP and eLink calls based on reinforcement learning according to claim 1, characterized in that, In step S5, the low-level resource agent determines the atomic operation based on a low-level reward function, which includes: An intrinsic reward used to evaluate the performance of the atomic operation technique, and a pre-defined extrinsic reward positive value obtained when the strategy objective is successfully achieved.

8. The seamless handover decision-making method for VoIP and eLink calls based on reinforcement learning according to claim 7, characterized in that, The intrinsic reward is determined based on the latency, resource consumption, and estimated mean opinion score (MOS) value generated after the atomic operation is executed, and is used to select the atomic operation with the least impact on call quality.

9. The seamless handover decision-making method for VoIP and eLink calls based on reinforcement learning according to claim 1, characterized in that, In step S3, the strategy objective is selected from: Combine calls into a conference, prioritize new calls, protect the current call, or process calls in sequence; The purpose of the high-level policy agent in selecting the policy objective is to achieve a seamless or low-interference transition between multiple call requests.

10. The seamless handover decision-making method for VoIP and eLink calls based on reinforcement learning according to claim 1, characterized in that, In step S5, the atomic operation is selected from: Direct mixing, audio stream transcoding followed by mixing, holding in hold mode, or playing a notification tone; The low-level resource agent selects the atomic operations to maintain the continuity of the audio stream during the switching or management process, achieving seamless switching.

Images