Reinforcement learning based seamless handover decision method for VoIP and eLink call

Abstract

The application relates to the field of communication technology and discloses a VoIP and eLink call seamless switching decision method based on reinforcement learning, which comprises the following steps: when a new incoming call event is received, a high-level decision module is used to comprehensively evaluate the real-time work flow state of a user, the current call and the contact value of the new incoming call, so that a macroscopic strategy target is determined, and the new incoming call is preferentially processed or the current call is protected; under the guidance of the strategy target, a low-level decision module is used to further combine real-time network quality and terminal equipment resources and other technical conditions to make a decision and execute an atomic operation, such as being placed in a hold state or directly mixing sound. The application adopts a layered reinforcement learning framework, dynamically perceives and reasons on multi-dimensional contexts such as user states, communication values and technical resources, solves the problem that multiple interests cannot be weighed due to the use of static rules, balances between guaranteeing communication opportunities and protecting user work focus, and improves user experience and work efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of communication technology, and particularly to a seamless switching decision method for VoIP and eLink calls based on reinforcement learning. BACKGROUND

[0002] With the development of modern communication technology, it has become a common practice to integrate and run multiple communication systems on a single terminal device. For example, VoIP technology is widely used in enterprise and personal communications due to its flexibility and low cost, while at the same time, many enterprises also deploy proprietary communication systems based on different protocols (hereinafter collectively referred to as eLink) for internal high-security or high-priority contact. Therefore, when using PC clients or mobile devices, users often face a scenario where a VoIP call is in progress while an eLink incoming call is incoming, or vice versa. This raises technical requirements for how to smoothly, efficiently and intelligently switch and manage between these concurrent and heterogeneous call events.

[0003] To handle such concurrent call scenarios, existing technologies usually adopt static strategies based on preset rules, which are usually configured by administrators when the system is deployed or hard-coded during software development. For example, the system is set to have the priority of eLink calls always higher than that of VoIP calls, and when a VoIP call is in progress, any eLink incoming call will automatically trigger an operation to put the current VoIP call on hold and connect the eLink incoming call. Or the system is configured not to make any automatic decision, but only to pop up a notification and option button, such as hold and answer, hang up and answer, etc., on the user interface, completely transferring the burden of decision-making to the user for manual processing. The underlying technical implementation of these methods is relatively straightforward, i.e., triggering a fixed signaling interaction process by monitoring the type of incoming call, such as sending a SIP re-INVITE request conforming to the RFC standard to control the hold or hang up of the call.

[0004] However, this decision mechanism that relies entirely on preset static rules has limitations when faced with the complex and dynamic real scenarios of modern office environments. A core problem is that this mechanism lacks the ability to perceive and understand the context and cannot intelligently weigh conflicting decision goals. For example, a user is having a regular VoIP call with a normal contact, but he is in a highly focused work flow state (e.g., coding or document writing), at this time, an eLink incoming call that is judged to be high priority but is not actually urgent according to static rules, the system will interrupt the user's current call and work flow without thinking, causing unnecessary disturbance and seriously affecting work efficiency and user experience. On the contrary, the user is having a non-core VoIP call and is in an idle state, when an eLink incoming call from an important customer arrives, which is of great commercial value, a too conservative static strategy (such as only pop-up notification) will cause the user to fail to respond in time, thus missing important opportunities. This mechanical processing method cannot dynamically measure and compare the maintenance value of the current call, the potential value of the new incoming call, and the opportunity cost of interrupting the user's current work flow, and cannot make the optimal decision that meets the user's real intention and maximizes the overall benefit. SUMMARY

[0005] To overcome the deficiencies of the prior art, the present application provides a seamless switching decision method for VoIP and eLink calls based on reinforcement learning, which solves the problem that existing communication terminals use preset fixed rules for switching or management when handling multiple concurrent calls such as VoIP and eLink, lack the ability to perceive the user's current work state and call context, and cannot dynamically prioritize different calls according to their potential importance, resulting in unnecessary interruption of the user's work flow or missing important calls.

[0006] To solve the above technical problems, the present application provides a seamless switching decision method for VoIP and eLink calls based on reinforcement learning.

[0007] This method divides the complex call management task into two stages: high-level policy decision and low-level resource execution by building a hierarchical decision system.

[0008] The high-level policy agent is responsible for selecting the optimal policy goal from the user's intention and macro value, and the low-level resource agent is responsible for executing the policy goal with the optimal service quality under the current technical conditions.

[0009] The method comprises the following steps:

[0010] S1, real-time monitoring of the communication interface to receive a new VoIP or eLink incoming call event that triggers decision making;

[0011] S2, obtaining a high-level state composed of a user workflow state and a communication session summary containing the new incoming call event;

[0012] The high-level state The vector can be represented as a multi-dimensional feature vector:

[0013] ;

[0014] wherein, is a user workflow state vector, is a communication session summary vector, is an environment and schedule vector, is a historical decision preference vector, is time.

[0015] S3, determining a policy target from a preset policy target set by a high-level policy agent according to the high-level state ; ;

[0016] S4, obtaining a low-level state composed of the policy target, network quality and device resources;

[0017] The low-level state The vector can be represented as a multi-dimensional feature vector:

[0018] ;

[0019] wherein, is a current policy target to be executed, is a network quality vector, is a device resource vector, is an audio stream attribute vector, is time.

[0020] S5, determining an atomic operation for executing the policy target from a preset atomic operation set by a low-level resource agent according to the low-level state ; ;

[0021] S6, converting the atomic operation into a control instruction and sending the control instruction to a hardware audio processing unit or a software interface of the VoIP phone to change the audio stream path or connection state of the current call, realizing the switching or management of VoIP and eLink calls.

[0022] Preferably, the user workflow state comprises a probability distribution of the user being in a focused state, a collaborative state or an idle state, which is an input dimension of the high-level state for the high-level policy agent to make decision calculation to evaluate the timing of executing the policy target.

[0023] In one specific embodiment, the step of obtaining the user workflow state in step S2 specifically comprises:

[0024] analyzing a behavior sequence of the user to determine the user workflow state;

[0025] The behavior sequence comprises keyboard input frequency, mouse movement pattern and currently active application window.

[0026] The analysis is performed by a sequence model, such as a long short-term memory network model, to process the behavior sequence to output the probability distribution.

[0027] Preferably, the communication session summary comprises the call type of the current call and new incoming call and the contact value level The contact value level is used by the high-level policy agent to evaluate the priority of different calls.

[0028] The contact value level can be obtained from a customer relationship management system or calculated based on historical interaction frequency data.

[0029] Further, in step S3, the high-level policy agent determines the policy target based on a high-level reward function

[0030] The high-level reward function is composed of a weighted sum of opportunity value reward and user veto penalty .

[0031] wherein, is calculated based on the contact value level, is a preset negative reward value applied when the system decision is manually vetoed by the user.

[0032] The calculation formula of the high-level reward function is:

[0033] ;

[0034] wherein, and are the weight coefficients corresponding to the opportunity value reward and the user veto penalty respectively, represents the value of the communication, represents the opportunity cost of interruption.

[0035] ​In one embodiment, the step of determining the atomic operation in step S5 comprises:

[0036] First, the network quality and the device resources are evaluated to see if they meet a service quality threshold;

[0037] Then, only when the service quality threshold is met, the atomic operation is determined;

[0038] The service quality threshold includes a minimum mean opinion score (MOS) value, a maximum network delay, and a maximum central processing unit (CPU) load.

[0039] Further, in step S5, the low-layer resource agent determines the atomic operation based on a low-layer reward function The low-layer reward function is calculated as follows:

[0040] ;

[0041] wherein, is an intrinsic reward for evaluating the technical performance of the atomic operation, is a preset positive extrinsic reward obtained when the policy goal is successfully completed .

[0042] Preferably, the intrinsic reward is determined according to the delay , resource consumption , and estimated mean opinion score after the atomic operation is executed, for selecting an atomic operation that has the least impact on call quality.

[0043] The intrinsic reward is calculated by a preset service quality function , that is,

[0044] ;

[0045] Preferably, in step S3, the policy goal is selected from:

[0046] merging calls into a conference, prioritizing new incoming calls, protecting current calls, or processing calls in sequence;

[0047] The high-layer policy agent selects the policy goal to achieve seamless or low-interference transition among multiple call requests.

[0048] Preferably, in step S5, the atomic operation is selected from:

[0049] direct mixing, audio stream transcoding and mixing, placing in a hold state, or playing a prompt tone;

[0050] The low-layer resource agent selects the atomic operation aiming to maintain audio stream continuity in the switching or management process, so as to realize seamless switching.

[0051] The application provides a seamless switching decision method for VoIP and eLink calls based on reinforcement learning, which has the following beneficial effects:

[0052] 1. The application uses the user workflow state and the contact value level in the communication session summary as high-level states, so that the high-level strategy agent can make decisions according to the real-time work situation of the user and the priority of the call, and the call management decision is no longer based on fixed and indiscriminate rules, but can adapt to the current work needs of the user, avoiding interruption of low-priority calls when the user is concentrating on work, while ensuring that high-value communication requests are given priority, improving the accuracy and situational adaptability of the decision.

[0053] 2. The application adopts a hierarchical decision architecture of high-level strategy agents and low-level resource agents, which divides the complex call management task into two independent sub-problems of strategy target determination and atomic operation execution. This hierarchical structure reduces the complexity of the state space and action space that each agent needs to process when making decisions, thereby accelerating the convergence speed of the reinforcement learning model and reducing the computational overhead of the system when making real-time decisions, obtaining a more stable and efficient decision model.

[0054] 3. The application evaluates the current network quality and device resources before the low-level resource agent executes the decision, and selects specific atomic operations according to an internal reward function containing delay, resource consumption and average opinion score evaluation, ensuring that any call switching or management operation is performed in the premise of technical conditions and in the way that minimally affects the call quality, providing technical support for realizing continuous and uninterrupted audio stream and high-quality seamless switching. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 The structural block diagram of the call seamless switching decision system of an embodiment of the application;

[0056] Figure 2 The flowchart of the call seamless switching decision method based on reinforcement learning of an embodiment of the application;

[0057] Figure 3 The internal structure and data flow diagram of the high-level decision module of an embodiment of the application;

[0058] Figure 4 The internal structure and data flow diagram of the low-level decision module of an embodiment of the application;

[0059] Figure 5 A flowchart of a call seamless handover decision-making process in a specific scenario according to an embodiment of the present application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0061] Referring to the accompanying drawings Figure 1 , Figure 1 FIG. 1 is a structural block diagram of a call seamless handover decision-making system according to an embodiment of the present application.

[0062] The call seamless handover decision-making system can be implemented in a terminal device 100, which includes but is not limited to a personal computer, a workstation or a dedicated communication terminal.

[0063] The terminal device 100 includes one or more processors 110, a memory 120 and one or more communication interfaces 130.

[0064] The processor 110, the memory 120 and the communication interface 130 are connected and communicate with each other through a bus system.

[0065] The processor 110 is responsible for executing program instructions stored in the memory 120 to control the overall operation of the system.

[0066] The communication interface 130 is responsible for processing data transmission and reception of communication protocols such as VoIP and eLink.

[0067] The memory 120 stores computer program instructions which, when executed by the processor 110, implement a VoIP and eLink call seamless handover decision-making method based on reinforcement learning.

[0068] Specifically, the program instructions stored in the memory 120 can be divided into a plurality of functional modules, including a data monitoring module 121, a high-level decision-making module 122, a low-level decision-making module 123 and an instruction execution module 124.

[0069] The data monitoring module 121 is responsible for real-time monitoring of the communication interface 130 to receive VoIP or eLink new incoming call events triggering decision-making.

[0070] In addition, the data monitoring module 121 is also responsible for collecting data required for constructing the decision state, including user behavior sequence for constructing high-level state, communication session information, and network quality and device resource information for constructing low-level state.

[0071] A high-level decision module 122, which embeds a high-level policy agent.

[0072] The high-level decision module 122 obtains user workflow state and communication session summary from the data monitoring module 121, which jointly constitute a high-level state vector.

[0073] Subsequently, the high-level policy agent determines a policy target according to the high-level state.

[0074] The high-level state can be represented as:

[0075] ;

[0076] wherein, is a user workflow state vector, is a communication session summary vector, is an environment and schedule vector, is a historical decision preference vector, is time.

[0077] A low-level decision module 123, which embeds a low-level resource agent.

[0078] The low-level decision module 123 obtains the determined policy target from the high-level decision module 122, and obtains network quality and device resource information from the data monitoring module 121, which jointly constitute a low-level state vector.

[0079] Subsequently, the low-level resource agent determines an atomic operation for executing the policy target according to the low-level state vector.

[0080] The low-level state vector can be represented as:

[0081] ;

[0082] wherein, is a current policy target to be executed, is a network quality vector, is a device resource vector, is an audio stream attribute vector, is time.

[0083] an instruction execution module 124, which is responsible for converting the atomic operation determined by the low-level decision module 123 into specific hardware or software control instructions.

[0084] Subsequently, the instruction execution module 124 sends the control instructions to the hardware audio processing unit or its software interface of the VoIP phone to change the audio stream path or connection state of the current call, and finally realize the switching or management of the VoIP and eLink calls.

[0085] Referring to the accompanying drawings Figure 2 , Figure 2 is a flowchart of a reinforcement learning-based VoIP and eLink call seamless switching decision method according to an embodiment of the present application.

[0086] The method can be implemented by the processor 110 in the terminal device 100 executing the program instructions in the memory 120, and the overall flow specifically includes the following steps:

[0087] S1, after the system is started, the data monitoring module 121 continuously monitors the communication interface 130.

[0088] When a new VoIP or eLink incoming call event occurs, for example, an SIP INVITE request message or an eLink call setup signaling is received, the event is taken as a trigger signal of the decision flow.

[0089] The data monitoring module 121 analyzes the event and extracts initial information such as the caller identification and call type.

[0090] S2, after receiving the trigger signal, the data monitoring module 121 actively acquires the original data required for constructing the high-level state vector from multiple data sources.

[0091] This includes: calling the operating system interface to obtain the current active application window information, keyboard input frequency, and mouse movement pattern and other user behavior sequences; querying the local or remote contact database to obtain the value level of the caller; and obtaining the state information of the current existing call.

[0092] These original data are preprocessed and integrated into a standardized vector form, i.e., the high-level state vector, and are sent to the high-level decision module 122.

[0093] S3, after the high-level decision module 122 receives the high-level state vector, the high-level strategy agent embedded therein takes the high-level state vector as input.

[0094] ​In one specific model implementation, the high-level policy agent is a deep Q-network.

[0095] As a function approximator, the goal of a deep Q-network is to learn an optimal action-value function that evaluates the long-term value of performing an action in any state .

[0096] After receiving the high-level state vector, the deep Q-network performs a forward propagation computation and outputs a Q-value vector.

[0097] The Q-value vector is a numerical list, where each element corresponds to the expected cumulative reward of a pre-defined policy goal (e.g., merging calls into a conference, prioritizing new incoming calls, etc.), i.e., the Q-value.

[0098] Specifically, if the system has policy goals , the network outputs a Q-value vector .

[0099] The high-level decision module 122 then determines an optimal policy goal from the Q-value vector according to a pre-defined action selection policy.

[0100] For example, in the exploitation phase, a greedy policy can be directly adopted to select the action with the maximum Q-value: ; in the exploration phase, an ε-greedy policy can be adopted, i.e., with a probability of , the action corresponding to the maximum Q-value is selected, while with a small probability , a random action is selected to explore more possibilities.

[0101] S4、The policy goal is then sent to the low-level decision module 123.

[0102] Meanwhile, the data monitoring module 121 is triggered to perform a real-time resource probe.

[0103] This includes sending probe packets to specific network nodes to measure the round-trip delay and packet loss rate of the current network, and invoking operating system APIs to query the real-time load of the central processor and memory.

[0104] These real-time resource data and the policy goal are integrated into a low-level state vector, which serves as the input for low-level decision making.

[0105] S5、The low-level decision module 123 receives the low-level state After the vector, its embedded low-level resource agent first compares the network quality and device resource indicators in the state with a preset service quality threshold.

[0106] Only when all indicators meet the service quality threshold requirements, the decision-making process will continue.

[0107] After meeting the conditions, the low-level resource agent (also a DQN model) will forward-propagate the low-level state vector as input and output a Q-value vector corresponding to each atomic operation (such as direct mixing, mixing after audio stream transcoding, etc.).

[0108] The low-level decision-making module 123 finally selects the atomic operation with the maximum Q-value as output.

[0109] The final determined atomic operation is sent to the instruction execution module 124.

[0110] The function of the instruction execution module 124 is to translate this atomic operation as a logical instruction into one or more specific, executable control instructions.

[0111] For example, if the atomic operation is to place in a hold state, the instruction execution module 124 will generate a SIPre-INVITE request conforming to the RFC3264 standard, which contains an SDP attribute setting the media stream to inactive.

[0112] The generated instructions are then sent to the communication interface 130 or the related hardware audio processing unit for execution, thus completing the change of the call state.

[0113] The decision-making process ends here.

[0114] The internal implementation principle of the high-level decision-making module 122 and how it performs steps S2 and S3 in the method will be described in detail below. Referring to the accompanying Figure 3 , Figure 3 is the internal structure and data flow diagram of the high-level decision-making module according to an embodiment of the present application.

[0115] In step S2, the high-level decision-making module 122 is responsible for constructing a high-level state vector. The high-level state vector is integrated from the multi-source heterogeneous data provided by the data monitoring module 121.

[0116] In a specific embodiment, the high-level state vector contains the following parts:

[0117] First is the user workflow state vector​ .

[0118] The data monitoring module 121 collects the user's behavior sequence within a preset time window (e.g., the last 60 seconds), which includes the keyboard input frequency measured in the number of keystrokes per second, the mouse movement distance measured in pixels per second, the number of clicks per second, and a one-hot encoding representing the type of the current active application window.

[0119] The behavior sequence is input into a pre-trained long short-term memory (LSTM) network model.

[0120] The LSTM network model processes the time series data, and its output is a three-dimensional probability distribution vector, such as [0.8, 0.1, 0.1], which respectively corresponds to the probabilities of the user being in a focused state, a collaborative state, or an idle state.

[0121] This vector is the .

[0122] The pre-training process of the LSTM network model includes: first, a large number of user behavior sequence samples are collected, and at the same time, the samples are labeled with corresponding real workflow state labels through user manual labeling or according to specific application scenario rules (e.g., when the code editor is full screen and there is no instant messaging application running, it is labeled as a focused state).

[0123] Then, the labeled data set is divided into a training set and a validation set, and the LSTM model is iteratively trained through a supervised learning method, a back propagation algorithm, and a cross-entropy loss function until the classification accuracy of the LSTM network model on the validation set reaches a preset threshold (e.g., 95%).

[0124] Second, the communication session summary vector .

[0125] The summary vector describes the current global situation of communication, mainly including the types of current calls and new incoming calls (e.g., 0 represents VoIP, and 1 represents eLink) and the value level of the contact person.

[0126] In a specific embodiment, the value level of the contact person is quantified as an integer value from 1 to 5, which can be obtained by the data monitoring module 121 by querying a local or remote customer relationship management database, or calculated based on historical communication frequency, duration, etc., through a preset scoring rule.

[0127] In a specific implementation, the scoring rule based on historical data can be given by the following formula:

[0128] .

[0129] wherein, is the total number of historical calls, is the average call duration, is the time interval (e.g., in days) from the last call, , , are respective preset weight coefficients, represents the natural logarithm function, i.e., the logarithm with the natural constant as the base.

[0130] Here, is a common smoothing technique used to reduce the excessive impact of a very high number of calls and ensure that the function is still defined when the historical number of calls is 0.

[0131] represents the natural constant, an irrational number approximately equal to 2.71828.

[0132] In this formula, it serves as the base of the exponential function , building an exponentially decaying term. The value of this term will quickly approach 0 as the time interval from the last call increases, which allows the most recently contacted contacts to obtain a higher score, in line with the principle of the near-cause effect of communication importance.

[0133] The calculation result is then normalized and mapped to an integer level of 1 to 5.

[0134] For example, 5 represents the highest value contact, and 1 represents an unknown or low value contact.

[0135] In step S3, the high-level decision module 122 determines an optimal strategy target based on the constructed high-level state vector.

[0136] The strategy target is selected from a preset strategy target set , which includes merging calls into a conference, prioritizing new incoming calls, protecting the current call, or processing calls in order.

[0137] These targets correspond to macro operations such as mixing the audio streams of two calls, placing the current call on hold and connecting a new incoming call, rejecting or transferring a new incoming call, placing a new incoming call in a waiting queue and playing a prompt tone to the user, etc.

[0138] The high-level strategy agent embedded in the high-level decision module 122 can be implemented as a deep Q network.

[0139] The number of input layer neurons of the deep Q-network matches the number of high-level states The dimension of the vector matches the number of output layer neurons of the network.

[0140] The number of output layer neurons of the network matches the number of policy targets The number of intermediate targets matches the number of output neurons, each output neuron corresponds to the Q value of a policy target.

[0141] The high-level policy agent is trained by a high-level reward function to learn the optimal decision policy.

[0142] The calculation formula of the high-level reward function is:

[0143] ;

[0144] Among them, is the opportunity value reward, which can be calculated as the difference between the new incoming contact value level and the current call contact value level, i.e. .

[0145] is the user veto penalty, when the policy target executed by the system is vetoed by the user within a preset time through manual operation (e.g. manual hang up or switch), a fixed negative value is given, for example -10.

[0146] and are preset weight coefficients for adjusting the relative importance of the two rewards.

[0147] represents the value of the communication, represents the opportunity cost of interruption.

[0148] At the time of decision, the high-level policy agent inputs the current high-level state into the DQN network, and obtains the Q values of all available policy targets through a forward propagation calculation.

[0149] The high-level decision module 122 then selects the policy target with the maximum Q value as the result of this decision and passes it to the low-level decision module 123.

[0150] The internal implementation principle of the low-level decision module 123 and how it executes steps S4 and S5 in the method will be described in detail below. Referring to the accompanying Figure 4 , Figure 4 is the internal structure and data flow diagram of the low-level decision module according to an embodiment of the present application.

[0151] ​In step S4, the low-level decision module 123 is responsible for constructing the low-level state vector.

[0152] The low-level decision module 123 first receives the determined policy objective from the high-level decision module 122 .

[0153] Meanwhile, the data monitoring module 121 is triggered to obtain real-time data describing the current technical environment.

[0154] These real-time data collectively form the low-level state vector, which in one specific embodiment contains the following components:

[0155] Firstly, the network quality vector .

[0156] The data monitoring module 121 measures and quantifies the network performance in real time by sending ICMP probe packets to relevant network nodes (e.g. the SIP proxy server or media gateway of the VoIP service) or parsing RTCP reports.

[0157] The network quality vector may be specifically represented as [delay, jitter, packet loss rate], e.g. [80, 25, 0.5], representing a round-trip delay of 80 ms, a jitter of 25 ms, and a packet loss rate of 0.5%, respectively.

[0158] Secondly, the device resource vector .

[0159] The data monitoring module 121 obtains the current resource load of the terminal device 100 by invoking the application program interface provided by the operating system.

[0160] The device resource vector may be specifically represented as [central processor load, memory load], e.g. [0.65, 0.50], representing a central processor usage rate of 65% and a memory usage rate of 50%, respectively.

[0161] In step S5, the low-level decision module 123 determines an optimal atomic operation based on the constructed low-level state vector.

[0162] The execution process of step S5 is divided into two stages.

[0163] The first stage is the quality of service threshold evaluation.

[0164] Before making a model-based decision, the low-level decision module 123 first performs a deterministic pre-check.

[0165] It will be the lower-level state The network quality and device resource metrics in the vector are compared with a set of preset service quality thresholds.

[0166] The set of service quality thresholds includes: a minimum mean opinion score (MOS) estimate (e.g., 3.5), a maximum network latency (e.g., 150 milliseconds), and a maximum CPU load (e.g., 80%).

[0167] The decision-making process only proceeds to the next stage when all real-time metrics are within the allowable range of the service quality threshold.

[0168] If any metric fails to meet the requirements, the system will execute a preset, safe default action, such as rejecting new calls or playing a busy tone, to avoid a decrease in call quality due to insufficient resources.

[0169] The second stage is model-based selection of atomic operations.

[0170] Once the service quality threshold is met, the low-level resource agent embedded in the low-level decision module 123 begins to work.

[0171] The low-level resource agent can be implemented as a deep Q-network, with the low-level state as its input. Vector, output as a set of atomic operations Q-value evaluation for each operation.

[0172] Atomic operations are specific technical actions that control audio streams, such as direct mixing, audio stream transcoding followed by mixing, putting the stream in hold mode, or playing a cue tone.

[0173] Low-level resource agents use a low-level reward function Training is conducted to learn how to select operations that have the least impact on technical performance while meeting strategic objectives.

[0174] The training process for low-level resource agents can be carried out in an offline environment based on historical data or simulation.

[0175] During training, the system randomly samples a batch of transition samples from an experience replay pool, which includes (current low-level state, atomic operation performed, intrinsic reward obtained, next low-level state, and whether the policy objective has been achieved).

[0176] Based on these samples, the weights of the deep Q-network are updated using optimization algorithms such as Adam, with the goal of minimizing the loss function defined by the Bellman equation.

[0177] Through a large number of iterative training, the low-level resource agent can learn which atomic operation can maximize the cumulative Q value including the immediate intrinsic reward and the future long-term extrinsic reward when given the low-level state and the policy goal.

[0178] The calculation formula of the low-level reward function is:

[0179] ;

[0180] wherein, is the extrinsic reward, which is a fixed, large positive constant value (for example, +50) only when the high-level policy goal is successfully completed at the end, and is 0 in all other intermediate steps.

[0181] is the intrinsic reward, which is used to evaluate the technical performance of the single atomic operation itself.

[0182] It is calculated by a preset quality of service function, that is:

[0183] ;

[0184] wherein, is the additional processing delay introduced by the execution of the atomic operation, is the CPU and memory resources consumed by the execution of the operation, is the estimated average opinion score of the call after the execution of the operation.

[0185] In an embodiment, the function can be defined as a weighted sum: .

[0186] wherein , , is a preset weight coefficient.

[0187] The intrinsic reward mechanism guides the agent to select the atomic operation with lower delay, less resource consumption, and greater positive impact on the call quality (measured by MOS score).

[0188] In decision-making, the low-level resource agent inputs the current low-level state vector into the DQN network, and obtains the Q values of all selectable atomic operations through one forward propagation calculation.

[0189] The low-level decision-making module 123 finally selects the atomic operation with the maximum Q value as the result of this decision , and passes it to the instruction execution module 124.

[0190] ​​​Reference is made to the drawings Figure 5 , Figure 5 is a schematic diagram of the seamless handover decision-making process in a specific scenario according to an embodiment of the present application.

[0191] The complete workflow of the VoIP and eLink seamless handover decision-making method based on reinforcement learning provided by the present application will be described below through a specific implementation scenario.

[0192] In an initial state, it is assumed that a user is using a terminal device 100 to have a conversation with a contact via VoIP.

[0193] At the same time, the data monitoring module 121 calculates that the probability of the user being in a focused state is 0.9 through analysis of the user's behavior sequence (for example, continuous keyboard input and concentrated application window switching) via the LSTM model inside it.

[0194] The value level of the VoIP conversation contact is confirmed as level 2 (ordinary contact) by querying the CRM database.

[0195] S1, at this time, the communication interface 130 receives a new incoming call event of eLink.

[0196] The data monitoring module 121 analyzes the new incoming call event and obtains that the caller is a contact with a value level of 5 (high-value contact).

[0197] This event triggers the entire decision-making process.

[0198] S2, the data monitoring module 121 immediately constructs a high-level state vector.

[0199] The high-level state vector specifically includes: a user workflow state vector whose value is [0.9, 0.1, 0.0], representing the probabilities of being focused, collaborative, and idle, respectively; and a communication session summary vector which contains information about the current conversation value being 2 and the new incoming call value being 5.

[0200] The high-level state vector is transmitted to the high-level decision-making module 122.

[0201] S3, the high-level decision-making module 122 receives the high-level state vector.

[0202] The high-level strategy agent embedded therein performs forward calculation.

[0203] In the forward calculation, since the value of the new incoming call is much higher than that of the current conversation, the opportunity value reward a large positive value is generated.

[0204] Although the user is in a focused state, the weight of the high-value communication causes the Q-value of the strategy goal of prioritizing new incoming calls to be calculated as the highest.

[0205] Therefore, the high-level decision module 122 determines the strategy goal of prioritizing new incoming calls and outputs it.

[0206] S4, the low-level decision module 123 receives the strategy goal .

[0207] At the same time, the data monitoring module 121 performs a quick technical environment detection and obtains the network quality vector [45, 15, 0.1] (representing a delay of 45 ms, a jitter of 15 ms, and a packet loss rate of 0.1%) and the device resource vector [0.40, 0.35] (representing a CPU load of 40% and a memory load of 35%).

[0208] These data, together with the strategy goal , constitute the low-level state vector.

[0209] S5, the low-level decision module 123 first performs a quality of service threshold evaluation on the technical indicators in the low-level state vector.

[0210] Since the indicators such as the 45 ms delay and the 40% CPU load are within the preset good range, the evaluation passes. Subsequently, the low-level resource intelligent agent evaluates the available atomic operations for achieving the goal of prioritizing new incoming calls.

[0211] Among the optional atomic operations, the put-on-hold state introduces less additional resource consumption and less damage to the original call than the direct hang-up of the current call, and therefore has a higher intrinsic reward value.

[0212] Finally, the low-level resource intelligent agent outputs the atomic operation with the largest Q-value, i.e., the put-on-hold state.

[0213] The result of the low-level decision module 123 is the inevitable choice made by the low-level resource intelligent agent according to its training goal.

[0214] In its training process, the low-level resource intelligent agent has learned that, in the case of sufficient network and device resources, the put-on-hold state, as an atomic operation for achieving the goal of prioritizing new incoming calls, has a higher intrinsic reward (due to its lower resource consumption and less damage to the MOS score of the original call). The zero negative impact decision) is much higher than other operations (such as directly mixing, resulting in a lower MOS score estimate of the call confusion; or directly hang up, resulting in a very low intrinsic reward for completely interrupting the communication).

[0215] Therefore, its corresponding long-term cumulative Q value is the highest.

[0216] S6, the instruction execution module 124 receives the atomic operation to be placed in the hold state.

[0217] The instruction execution module 124 converts it into two specific control instruction sequences: instruction one, generate a SIP RE-INVITE request containing a=sendonly media attribute for the current VoIP call to place the VoIP call in the hold state; instruction two, generate a signaling to accept the eLink incoming call. The two instructions are sent to the communication interface 130 for execution.

[0218] The final execution result is that the user's current VoIP call is smoothly placed in the hold state, while the system automatically connects the eLink incoming call from the high-value contact, achieving the preservation of the user's original work context while ensuring the priority of high-value communication.

Claims

1. A method for seamless handover decision of VoIP and eLink call based on reinforcement learning, characterized in that, The method comprises the following steps: S1, real-time monitoring of a communication interface to receive a VoIP or eLink new incoming call event triggering decision; S2, obtaining a high-level state composed of a user workflow state and a communication session summary containing the new incoming call event; S3, determining a policy target by a high-level policy agent according to the high-level state; S4, obtaining a low-level state composed of the policy target, network quality and device resources; S5, determining an atomic operation for executing the policy target by a low-level resource agent according to the low-level state; S6, converting the atomic operation into a control instruction and sending the control instruction to a hardware audio processing unit or a software interface of a VoIP phone to change an audio stream path or a connection state of a current call to realize switching or management of VoIP and eLink calls. 2.The method of claim 1, wherein, The user workflow state comprises a probability distribution of the user currently being in a focused state, a collaborative state or an idle state, and the probability distribution is used for the high-level policy agent to evaluate a timing of executing the policy target. 3.The method of claim 2, wherein, The step of obtaining the user workflow state in step S2 specifically comprises: analyzing a behavior sequence of the user to determine the user workflow state; the behavior sequence comprises a keyboard input frequency, a mouse movement mode and a currently active application window. 4.The method of claim 1, wherein, The communication session summary comprises a call type of the current call and the new incoming call and a contact value level, and the contact value level is used for the high-level policy agent to evaluate priorities of different calls.

5. The method of claim 4, wherein the method further comprises: In step S3, the high-level policy agent determines the policy target based on a high-level reward function, and the high-level reward function comprises: an opportunity value reward calculated based on the contact value level and a preset negative reward value applied when a system decision is manually vetoed by the user.

6. The method of claim 1, wherein the method further comprises: determining whether the eLink call is in a voice activity state; and if the eLink call is in the voice activity state, determining whether the eLink call is in a voice activity state for a predetermined time period. The step of determining the atomic operation in step S5 specifically comprises: firstly, evaluating whether the network quality and the device resources meet a service quality threshold; then, only when the service quality threshold is met, the atomic operation is determined; the service quality threshold comprises a minimum mean opinion score (MOS) value, a maximum network delay and a maximum central processing unit (CPU) load.

7. The method of claim 1, wherein the method further comprises: determining whether the eLink call is in a voice activity detection (VAD) mode; and if the eLink call is in the VAD mode, determining whether the eLink call is in a silence descriptor (SID) mode. In step S5, the low-level resource agent determines the atomic operation based on a low-level reward function, and the low-level reward function comprises: an intrinsic reward for evaluating technical performance of the atomic operation and a preset extrinsic reward positive value obtained when the policy target is successfully completed. 8.The method of claim 7, wherein, The intrinsic reward is determined according to a delay, resource consumption and an estimated mean opinion score (MOS) value generated after execution of the atomic operation, and is used for selecting an atomic operation with the least impact on call quality. 9.The method of claim 1, wherein, In step S3, the policy target is selected from: merging calls into a conference, preferentially processing new incoming calls, protecting a current call or processing calls in sequence; the high-level policy agent selects the policy target to realize uninterrupted or low-interference transition among multiple call requests.

10. The method of claim 1, wherein the method further comprises: determining whether the eLink call is in a voice activity detection (VAD) state; and if the eLink call is in the VAD state, determining whether the eLink call is in a silence descriptor (SID) state. In step S5, the atomic operation is selected from: direct mixing, mixing after audio stream transcoding, being placed in a hold state or playing a prompt tone; The low-layer resource intelligent agent selects the atomic operation aiming to maintain audio stream continuity in the switching or management process, realizing seamless switching. The low-layer resource intelligent agent selects the atomic operation aiming to maintain audio stream continuity in the switching or management process, realizing seamless switching.

Images