Flow switching method and device and computer equipment

Abstract

The invention relates to the technical field of communication, and discloses a flow switching method and device and computer equipment. The method is executed by a network data analysis function (NWDAF) and comprises the following steps: acquiring cell information sent by target equipment, setting a first SIM card and a second SIM card on the target equipment, respectively calculating a first priority of the first SIM card and a second priority of the second SIM card according to the cell information, and if the first priority and the second priority do not meet a preset condition, determining the state of the target equipment based on the state of the target equipment. And determining a target action with the highest expected return in the Q table, and indicating the target equipment to perform cell switching based on the target action. According to the invention, by introducing the Q-learning algorithm, the intelligence and optimization of the flow selection of the multi-SIM card target equipment are realized, and the efficiency of flow switching and the connection stability are improved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a traffic switching method, device and computer equipment. Background Art

[0002] With the development of mobile communication technology, multi-SIM user equipment (MUSIM UE) is becoming more and more popular in daily life. Users can use multiple SIM cards in the same device to achieve seamless switching between different operators' networks, enjoy better communication quality and cost savings. However, in the actual application of multi-SIM cards, how to intelligently select the best traffic path to improve the stability and efficiency of network connection has become an important technical challenge. In a multi-SIM environment, user equipment usually faces the following problems: fluctuations in network signal strength, uneven network loads in different cells, differences in connection stability, and uncertain switching success rates. Traditional traffic switching methods mainly rely on predefined rules and thresholds. Although these methods are simple and easy to implement, they often show shortcomings such as poor adaptability and low efficiency in dynamic and complex network environments. Therefore, how to achieve intelligent, flexible and efficient traffic selection in a changing network environment has become the focus of the industry.

[0003] However, there are several main disadvantages in the current traffic selection of multi-SIM card user equipment. First, the rule-based traffic switching method relies on predefined thresholds and conditions, lacks flexibility and adaptability, and cannot cope with dynamic and complex network environments; second, although fuzzy logic control can handle uncertainty, the design and debugging of fuzzy rules and membership functions is complex and time-consuming, and it is difficult to achieve efficient real-time optimization in practical applications; finally, these methods are usually unable to self-learn and improve in switching strategies, resulting in difficulty in improving system performance in long-term use. Summary of the invention

[0004] In view of this, the present invention provides a traffic switching method, device and computer equipment to solve the problem of low traffic switching efficiency and accuracy of multi-SIM card user equipment in a dynamic network environment.

[0005] In a first aspect, the present invention provides a traffic switching method, which is performed by a network data analysis function NWDAF, and includes:

[0006] Acquire cell information sent by a target device; the target device is provided with a first SIM card and a second SIM card; the cell information is used to indicate the connection status of the first SIM card and the second SIM card with the cell;

[0007] Calculate the first priority of the first SIM card and the second priority of the second SIM card respectively according to the cell information;

[0008] If the first priority of the first SIM card and the second priority of the second SIM card do not meet the preset conditions, based on the state of the target device, determine the target action with the highest expected return in the Q table; the Q table is used to characterize the mapping relationship between the state and the expected return of the action; the Q table is obtained by reinforcement learning based on the Q-learning training model through the historical data of cell switching;

[0009] The target device is instructed to perform a cell handover based on the target action.

[0010] The traffic switching method provided in the embodiment of the present invention calculates the priority score of the SIM card, combines the current state of the target device, and combines the Q table obtained by Q-learning reinforcement learning training to select the target action with the highest expected return to perform cell switching. This method can intelligently make the best decision based on historical data and the current environment, optimize network performance and user experience, and can not only cope with dynamic changes in the network environment, but also reduce unnecessary switching operations by optimizing the cell switching strategy.

[0011] In an optional implementation manner, before respectively calculating the first priority of the first SIM card and the second priority of the second SIM card according to the cell information, the method further includes:

[0012] Set priority parameters and their corresponding weights;

[0013] Based on the priority parameters and the weights corresponding to the priority parameters, a first priority of the first SIM card and a second priority of the second SIM card are calculated respectively.

[0014] In the traffic switching method provided by the embodiment of the present invention, different priority parameters may have different effects on the decision of cell switching. By assigning appropriate weights to each parameter, it can ensure that the priority calculation is more accurate, so that the SIM card most suitable for the current situation can be selected for communication, and the performance and reliability of different SIM cards can be evaluated more accurately, which helps the system make more intelligent decisions when selecting to switch SIM cards, thereby optimizing network performance and user experience.

[0015] In an optional implementation manner, after acquiring the cell information sent by the target device, the method further includes:

[0016] Set the state and action of the Q table corresponding to the target device and the parameters of the Q-learning training model, and set the Q value corresponding to each state and action in the Q table to the preset value;

[0017] The Q-learning training model is trained through reinforcement learning based on the cell information, and the Q value in the Q table is updated.

[0018] The traffic switching method provided in the embodiment of the present invention sets the state and joint action of the Q table and sets parameters for the Q-learning training model, so that the decision-making process is more in line with the specific needs and characteristics of the device. At the same time, through continuous reinforcement learning training based on cell information, the Q value in the Q table can be continuously updated as the network environment and device status change. This dynamic adaptability enables the reinforcement learning model to flexibly respond to dynamic changes in the network environment and ensure that the decision-making strategy always remains in the optimal state.

[0019] In an optional implementation, the cell information includes at least one of signal strength, network load, connection stability, switching success rate, and whether it is a capacity enhancement cell.

[0020] The traffic switching method provided in the embodiment of the present invention can more comprehensively evaluate the connection quality between each SIM card and different cells by comprehensively considering multiple cell information indicators. This more accurate evaluation helps to more accurately calculate the priority score of the SIM card, thereby providing a more reliable basis for subsequent cell switching decisions.

[0021] In an optional implementation, if the first priority of the first SIM card and the second priority of the second SIM card do not meet the preset condition, before determining the target action with the highest expected return in the Q table based on the state of the target device, the method further includes:

[0022] If the first priority meets the preset conditions, the target action with the highest expected return is determined based on the Q table;

[0023] Perform simulation execution based on the target action to obtain simulation execution results;

[0024] Based on the simulation execution results, the Q value of the Q table is updated.

[0025] The traffic switching method provided by the embodiment of the present invention, when the priority of a certain SIM card meets the preset conditions, determines the target action with the highest expected return in the current state through the Q table and performs simulated execution to verify the effect of the decision. The simulated execution result can reflect the performance of different actions in the actual environment, thereby providing valuable feedback for the update of the Q table. By continuously updating the Q value of the Q table based on the simulated execution result, the accuracy and reliability of the Q table are gradually improved, thereby improving the quality of the decision and continuously optimizing its decision-making strategy.

[0026] In an optional embodiment, the method further includes:

[0027] If the first priority does not meet the preset requirements, the second priority of the second SIM card is calculated according to the cell information;

[0028] If the second priority meets the preset condition, the connection resources of the first SIM card are released.

[0029] The traffic switching method provided by the embodiment of the present invention can more reasonably allocate and utilize network resources by evaluating the priorities of two SIM cards. When the connection quality of the first SIM card is poor, the priority score of the second SIM card is evaluated, and the connection resources of the first SIM card are released so that these resources can be used more effectively by other devices or applications, which helps to promote network load balancing to alleviate network congestion, reduce delays and improve overall network performance.

[0030] In an optional embodiment, the method further includes:

[0031] Send a release confirmation message to the target device to obtain the switching result;

[0032] The switching record and Q table are updated based on the switching result.

[0033] The traffic switching method provided in the embodiment of the present invention can ensure that the connection resources of the first SIM card are correctly released and update the routing resources by sending a confirmation release message to the target device, and update the switching record and Q table based on the switching result, thereby ensuring that the data recorded by the system is consistent with the actual situation.

[0034] In an optional implementation, reinforcement learning training is performed on the Q-learning training model based on the cell information, and the Q value in the Q table is updated, including:

[0035] The cell information is divided into a training set and a test set according to a preset ratio;

[0036] After using the data of the training set to train the model, use the data of the test set to evaluate the model and obtain the evaluation results;

[0037] Based on the evaluation results, the parameters of the Q-learning training model are adjusted to obtain the target Q-learning training model;

[0038] Based on the target Q-learning training model, the Q value in the Q table is updated.

[0039] The traffic switching method provided by the embodiment of the present invention can further optimize the decision quality of the model by dividing the cell information into a training set and a test set, respectively used for model training and evaluation, and adjusting the model parameters based on the evaluation results, so as to ensure that the trained Q-learning model can better adapt to different network environments and cell conditions to improve the accuracy and efficiency of cell switching. At the same time, the Q-learning training model is updated based on the latest cell information and evaluation results, thereby enhancing the stability and reliability of the Q-learning training model.

[0040] In a second aspect, the present invention provides a flow switching device, the device comprising:

[0041] An acquisition module is used to acquire cell information sent by a target device; the target device is provided with a first SIM card and a second SIM card; the cell information is used to indicate the connection status of the first SIM card and the second SIM card with the cell;

[0042] A calculation module, used for calculating the first priority of the first SIM card and the second priority of the second SIM card respectively according to the cell information;

[0043] An action determination module is used to determine a target action with the highest expected return in a Q table based on the state of the target device when the first priority of the first SIM card and the second priority of the second SIM card do not meet the preset conditions; the Q table is used to characterize the mapping relationship between the state and the expected return of the action; the Q table is obtained by reinforcement learning based on the Q-learning training model through the historical data of cell switching;

[0044] The switching module is used to instruct the target device to perform cell switching based on the target action.

[0045] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the traffic switching method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0046] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the traffic switching method of the first aspect or any corresponding embodiment thereof.

[0047] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the traffic switching method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 is a system architecture diagram of a flow switching system according to an embodiment of the present invention;

[0050] Figure 2 is a flow chart of a flow switching method according to an embodiment of the present invention;

[0051] Figure 3 is an interactive schematic diagram of a traffic switching method according to an embodiment of the present invention;

[0052] Figure 4 is a structural block diagram of a flow switching device according to an embodiment of the present invention;

[0053] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0055] At present, there are several obvious defects in the traffic selection mechanism of multi-SIM card user equipment: first, the traditional traffic switching strategy based on fixed rules and thresholds lacks flexibility; second, although the fuzzy logic system processes uncertainty and fuzzy information to select and switch traffic, which shows potential in dealing with uncertainty, its complex rule formulation and membership function design process is not only time-consuming and labor-intensive, but also difficult to achieve efficient optimization in a real-time environment, affecting its actual application effect; finally, these mechanisms generally lack the ability of self-learning and continuous improvement, resulting in the difficulty of significantly improving system performance in long-term operation, limiting the continuous optimization of user experience. Therefore, exploring a traffic selection solution that can intelligently, flexibly and efficiently adapt to the dynamic network environment has become a hot spot and direction for the industry to explore.

[0056] According to an embodiment of the present invention, a traffic switching method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0057] This method is applied to the flow switching system, please refer to Figure 1 , Figure 1 The overall architecture of the system includes: Q-learning module, data collection module, data storage module, priority calculation module, execution module, network module, specifically:

[0058] Learning module: used to achieve intelligent traffic selection and optimization through reinforcement learning algorithm (Q-learning), specifically including Q-table initialization, parameter setting, action selection, Q-table update and model evaluation, that is, by initializing the Q-table, defining the state and action combination, and selecting an action with the highest expected return as the target action under the preset state, and finally continuously updating the Q-table through the learning process, and evaluating the model performance to optimize the strategy. This module includes the Q-table initialization submodule, the action selection submodule and the Q-table update submodule.

[0059] Data collection module: used to collect cell information of the dual SIM card connection of the target device in real time, such as signal strength, network load, connection stability and switching success rate, for subsequent analysis and decision-making. This module is divided into a cell information collection submodule and an information sending submodule. It is used to collect various network information related to the dual SIM card connection in real time to ensure that the decision is based on the latest data. The collected data is passed to the data storage module and the Q-learning module for further processing and analysis.

[0060] Data storage module: stores and manages the collected cell information, priority parameters and Q table data, and divides them into training sets and test sets to support the training and evaluation of the Q-learning model. This module is divided into a data storage submodule, a data partitioning submodule and a data provision submodule. It stores various data generated during the operation of the system to ensure data security and availability. It processes and divides the stored data to ensure the reasonable allocation of training and test data. It provides data support for the training and evaluation of the Q-learning model.

[0061] Priority calculation module: Calculates the priority score of each state based on the predefined priority parameters to guide traffic selection and switching decisions. This module is divided into priority parameter definition submodule, priority score calculation submodule, and score judgment submodule. Defines the various parameters and their weights used to calculate the priority score to ensure the accuracy and rationality of the calculation results. Calculates the priority score of each possible state based on the collected network information and the defined priority parameters. Determines subsequent operations based on the calculated priority score to ensure that the system makes the best decision.

[0062] Execution module: Executes traffic switching operations according to the actions selected by the Q-learning module, updates the network configuration and routing table, and records the switching results. This module is divided into an action execution submodule, a network configuration update submodule, and a result recording and reporting submodule. Executes corresponding traffic switching operations according to the actions selected by the Q-learning module to ensure rapid system response. Ensures the accuracy and timely update of network configuration to maintain the normal operation of the system. Records the switching results and reports to the core network to ensure the transparency and traceability of the system status.

[0063] Network communication module: responsible for data transmission and command communication between MUSIM UE and gNB, core network, and AMF, ensuring timely information transmission and coordinated operation of the system. It is divided into data transmission submodule, command sending submodule, and status transmission submodule. Ensure smooth data transmission between modules and maintain the overall coordination of the system. Send system commands to control the behavior of each submodule to ensure that the system runs as expected. Transmit system status and switching results to maintain information consistency among various parts of the system.

[0064] In this embodiment, a flow switching method is provided, which is applied to the above-mentioned flow switching system. Figure 2 is a flow chart of a flow switching method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0065] Step S101, obtaining cell information sent by a target device.

[0066] Specifically, the target device is provided with a first SIM card and a second SIM card, and the cell information is used to indicate the connection status of the first SIM card and the second SIM card with the cell.

[0067] Furthermore, the network data analysis function (NWDAF) needs to establish a stable communication connection with the target device. Through the base station gNB (Next Generation NodeB) in the existing 5G network, it is responsible for wireless communication with the target device MUSIM UE (Multi-SIM User Equipment). NWDAF sends a specific information request to the target device, which explicitly requires the target device to provide its current cell information. The target device MUSIM UE (Multi-SIM User Equipment) sends the cell information to which the current target device is connected to the core network through gNB.

[0068] Step S102: Calculate the first priority of the first SIM card and the second priority of the second SIM card according to the cell information.

[0069] Specifically, based on the cell information, a priority score evaluation table is established, and the NWDAF determines whether to perform traffic switching according to the priority score corresponding to the SIM card.

[0070] In some optional implementations, before the above step S102, the following steps are included:

[0071] Set priority parameters and their corresponding weights;

[0072] Based on the priority parameters and the weights corresponding to the priority parameters, a first priority of the first SIM card and a second priority of the second SIM card are calculated respectively.

[0073] The priority parameters include: current signal strength (RSSI), network load, connection stability, switching success rate and whether it is a capacity enhanced cell. Specifically, the current signal strength (RSSI) is used to characterize the received signal strength, indicating the connection quality between the device and the cell; the network load is used to characterize the network load of the cell, indicating the congestion level of the cell; the connection stability is used to characterize the historical connection stability between the device and the cell, indicating the reliability of the connection, connection stability = stable connection time / connection time; the switching success rate is used to characterize the success probability of the device switching to the cell, switching success rate = number of successful switching / number of switching; whether it is a capacity enhanced cell, the first SIM card tends to connect to the capacity enhanced cell and the second SIM card tends to connect to the non-capacity enhanced cell.

[0074] Preferably, as shown in Table 1, the weight values ​​of the first SIM card corresponding to the above priority parameters are 0.3, 0.2, 0.2, 0.1 and 0.2 respectively; the weight values ​​of the second SIM card corresponding to the above priority parameters are 0.4, 0.3, 0.2, 0.1 and 0 respectively. The priority calculation formula is:

[0075] Priority score = δ1 × current signal strength + δ2 × (1-network load) + δ3 × connection stability

[0076] +δ4×switching success rate+δ5×capacity enhancement cell

[0077] Table 1 Priority parameter table:

[0078]

[0079]

[0080] Step S103: if the first priority of the first SIM card and the second priority of the second SIM card do not meet the preset conditions, a target action with the highest expected reward is determined in the Q table based on the state of the target device.

[0081] Specifically, the Q table is used to characterize the mapping relationship between the state and the expected return of the action, and the Q table is obtained by reinforcement learning based on the Q-learning training model through the historical data of cell switching.

[0082] Furthermore, by introducing the Q-learning algorithm to form a Q table, the Q table stores the expected rewards of taking different actions under different states (i.e., the Q value in the Q table). Therefore, according to the current state of the target device, the action with the highest expected reward of all possible actions can be found in the Q table and determined as the target action. These actions correspond to different options for cell switching.

[0083] In some optional implementations, before step S103, the method further includes:

[0084] Set the state and action of the Q table corresponding to the target device and the parameters of the Q-learning training model, and set the Q value corresponding to each state and action in the Q table to the preset value;

[0085] The Q-learning training model is trained through reinforcement learning based on the cell information, and the Q value in the Q table is updated.

[0086] Set the state and action corresponding to the target device. Taking Table 2 as an example, the rows of the Q table represent the joint state of the two SIM cards. For example, (S1, S2) represents that the first SIM card is connected to cell A, the second SIM card is connected to cell B, and so on. The columns of the Q table represent the joint actions. For example, (A1, A2) represents that the first SIM card switches to cell A, the second SIM card switches to cell B, and so on. First, set the value of the Q table to zero, that is, the initial Q value of all state-action pairs is zero.

[0087] Table 2Q shows an example table:

[0088] (A1, A1) (A1, A2) (A2, A1) (A2, A2) (S1, S1) Q(S1,S1,A1,A1) Q(S1,S1,A1,A2) Q(S1,S1,A2,A1) Q(S1,S1,A2,A2) (S1, S2) Q(S1,S2,A1,A1) Q(S1,S2,A1,A2) Q(S1,S2,A2,A1) Q(S1,S2,A2,A2) (S2, S1) Q(S2,S1,A1,A1) Q(S2,S1,A1,A2) Q(S2,S1,A2,A1) Q(S2,S1,A2,A2) (S2, S2) Q(S2,S2,A1,A1) Q(S2,S2,A1,A2) Q(S2,S2,A2,A1) Q(S2,S2,A2,A2)

[0089] Then set the Q-learning training model parameters, including: setting the learning rate, discount factor, exploration probability, etc.

[0090] Preferably, the learning rate α is set with an initial value of 0.1, which indicates the weight of new information each time the Q value is updated; the discount factor γ is set with an initial value of 0.9, which indicates the impact of future rewards on the current decision; the exploration probability ε is initially set to 0.1 and gradually decreases. A high ε value indicates more random exploration, and a low ε value indicates more use of the best decision in the current Q table.

[0091] Finally, the Q-learning training model is trained through reinforcement learning based on the collected cell information, and the Q value in the Q table is updated.

[0092] Step S104: instructing the target device to perform cell switching based on the target action.

[0093] Specifically, according to the target action determined in step S103 (i.e., the cell switching option with the highest expected return), a corresponding switching instruction is prepared. This instruction needs to clearly indicate which cell the target device should switch to, as shown in Table 2 above. For example, if the current state is (S1, S2), it means that the first SIM card is connected to cell A and the second SIM card is connected to cell B. If the corresponding target action is (A2, A1), it means that the first SIM card will switch to cell B, the second SIM card will switch to cell B, and so on.

[0094] In summary, by obtaining cell information, setting priority parameters, including: current signal strength (RSSI), network load, connection stability, switching success rate, and whether it is a capacity enhancement cell, and setting priority weights corresponding to the above parameters, the priority scores of the first SIM card and the second SIM card are calculated based on the priority calculation formula:

[0095] Priority score = δ2 × current signal strength + δ1 × (1-network load) + δ3 × connection stability

[0096] +δ4×switching success rate+δ5×capacity enhancement cell

[0097] The cell information obtained at the same time is formed into a Q-table by introducing the Q-learning algorithm, which stores the expected rewards of taking different actions in different states, that is, the Q value, which is used to find the expected rewards of each possible action corresponding to the current state. First, the Q-table is initialized, that is, the initial Q value of all state-action pairs is 0. Then, the Q-learning parameters are set for model training. The parameters include setting the learning rate α, with an initial value of 0.1; the discount factor γ, with an initial value of 0.9, and the exploration probability ε, with an initial value of 0.1. The Q-learning training model is trained using the collected cell information. Finally, the Q value in the Q-table is updated based on the training results.

[0098] When the priority scores of the first SIM card and the second SIM card are both less than or equal to 0.5, it is considered that the first SIM card and the second SIM card are not satisfied with the preset conditions. At this time, based on the Q value in the Q table, the action with the highest expected return of all possible actions in the current state is found, and it is determined as the target action. The cell switching is performed according to the current state and the target action determined by the current state.

[0099] In some optional implementations, the cell information includes at least one of signal strength, network load, connection stability, handover success rate, and whether it is a capacity enhancement cell.

[0100] The target device MUSIM UE sends the cell information of the current dual SIM card connection to the core network through the gNB at preset time intervals, including: signal strength RSSI, network status (currently connected cell ID), cell congestion level, connection stability and handover success rate. The gNB also sends NWDAF whether the currently connected cell is a capacity enhancement cell. Based on these data, the Q-learning algorithm is trained, and the Q value in the Q table is updated through continuous state transition and action selection.

[0101] In some optional implementations, if the first priority of the first SIM card and the second priority of the second SIM card do not meet the preset conditions, before determining the target action with the highest expected return in the Q table based on the state of the target device, the method further includes:

[0102] If the first priority meets the preset conditions, the target action with the highest expected return is determined based on the Q table;

[0103] Perform simulation execution based on the target action to obtain simulation execution results;

[0104] Based on the simulation execution results, the Q value of the Q table is updated.

[0105] If the first priority score corresponding to the first SIM card is greater than 0.5 at this time, the Q-learning model training continues. Specifically, a current state S is selected from the data set. The action is selected based on the calculated priority score and the greedy strategy. The greedy strategy combines the balance of exploration and utilization for action selection: an action is randomly selected with a probability of ε to ensure that the algorithm can explore different actions; the joint action with the highest priority score is selected with a probability of (1-ε), and the best action is selected using known information. For example, the current state (S1, S2) means that the first SIM card is connected to cell A and the second SIM card is connected to cell B. The possible joint actions X are (A1, A1), (A1, A2), (A2, A1), (A2, A2).

[0106] Simulate the execution of actions and record the results. Simulate the execution of the selected joint action. For example, if the current state is (S1, S2) and the selected joint action is (A1, A2), the first SIM card remains connected to cell A, and the second card switches to cell B. If a better connection is detected, such as a higher score obtained through priority calculation, the corresponding joint action should be selected to perform the switching operation. Record the new state after the switch and calculate the immediate reward. If the switch is successful and the link is stable, a positive reward (+10) is given; if the switch fails or the connection is unstable, a negative reward (-10) is given, and the Q table is updated based on the reward score. The specific update formula is:

[0107]

[0108] Among them, Q(S1,S2,A1,A2) is the Q value of executing action X in the current state S;

[0109] α is the learning rate, which controls the update step size;

[0110] R is the immediate reward after performing the action;

[0111] γ is a discount factor that weighs the importance of future rewards;

[0112] is the Q value of choosing the best action in the new state.

[0113] In some optional embodiments, the method further comprises:

[0114] If the first priority does not meet the preset requirements, the second priority of the second SIM card is calculated according to the cell information;

[0115] If the second priority meets the preset condition, the connection resources of the first SIM card are released.

[0116] like Figure 3As shown, when the first priority corresponding to the first SIM card is less than or equal to 0.5, or when the energy-saving strategy decides to shut down the capacity enhancement cell, a shutdown instruction is sent to AMF. AMF sends the shutdown notification to NWDAF and MUSIM UE. MUSIM obtains the connection status of the current second SIM card, including the latest RSSI, network load, connection stability and switching success rate, and sends the collected information to NWDAF through AMF. NWDAF scores the second priority corresponding to the second SIM card. If the second priority score is greater than 0.5, it informs AMF, and AMF sends an N2 interface release request to gNB1 to request the release of the connection resources of the first SIM card.

[0117] In some optional embodiments, the method further comprises:

[0118] Send a release confirmation message to the target device to obtain the switching result;

[0119] The switching record and Q table are updated based on the switching result.

[0120] like Figure 3 As shown, when gNB1 receives the release request, it releases the connection resources of the first SIM card and sends a confirmation to the core network and MUSIM UE that the release is complete. After gNB1 releases the network resources of SIM card 1, MUSIM UE uses system commands to obtain the current network interface information (including the interface name, IP address, subnet mask and gateway of the first SIM card and the second SIM card). Before releasing the network resources, MUSIM UE records the current network configuration of the first SIM card for use when updating the routing table. MUSIM UE deletes all routing entries related to the first SIM card from the routing table, adds the network parameters of the second SIM card to the routing table, and sets it as the default route. Finally, the updated routing table is applied to the system to ensure that all data is transmitted through the second SIM card. Next, MUSIM verifies the connection stability, sends a test data packet through the second SIM card, verifies the stability and reliability of data transmission, and checks whether the network parameter configuration of the second SIM card is correct, including IP address, subnet mask, and gateway information. Finally, the handover result is reported to the core network, including the handover status and the current network status. After receiving it, AMF sends it to NWDAF. The NWDAF updates the switching record and the Q table. The NWDAF sends the cell to which the first SIM card is to be connected to the first SIM card, and the first SIM card is connected to the corresponding cell.

[0121] In some optional implementations, reinforcement learning training is performed on the Q-learning training model based on the cell information, and the Q value in the Q table is updated, including:

[0122] The cell information is divided into a training set and a test set according to a preset ratio;

[0123] After using the data of the training set to train the model, use the data of the test set to evaluate the model and obtain the evaluation results;

[0124] Based on the evaluation results, the parameters of the Q-learning training model are adjusted to obtain the target Q-learning training model;

[0125] Based on the target Q-learning training model, the Q value in the Q table is updated.

[0126] NWDAF first stores these data in the connected historical database. NWDAF divides the data into training sets (80%) and test sets (20%), with the first 19.2 hours of data per day as the training set and the last 4.8 hours of data as the test set. The training set is used to train the Q-learning algorithm, and the Q value in the Q table is updated through continuous state transitions and action selections. The model is then evaluated in the test set stage, and the performance of the Q-learning model is evaluated using the test data set. The Q-learning parameters and priority calculation weights are adjusted based on the evaluation results to optimize the model weights.

[0127] In this embodiment, a flow switching device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0128] This embodiment provides a flow switching device, such as Figure 4 As shown, including:

[0129] The acquisition module 401 is used to acquire the cell information sent by the target device; the target device is provided with a first SIM card and a second SIM card; the cell information is used to indicate the connection status between the first SIM card and the second SIM card and the cell.

[0130] The calculation module 402 is used to calculate the first priority of the first SIM card and the second priority of the second SIM card according to the cell information.

[0131] The action determination module 403 is used to determine the target action with the highest expected return in the Q table based on the state of the target device when the first priority of the first SIM card and the second priority of the second SIM card do not meet the preset conditions; the Q table is used to characterize the mapping relationship between the state and the expected return of the action; the Q table is obtained by reinforcement learning based on the Q-learning training model through the historical data of cell switching.

[0132] The switching module 404 is configured to instruct the target device to perform a cell switching based on the target action.

[0133] In some optional embodiments, the device further comprises:

[0134] The parameter setting module is used to set the priority parameters and the weights corresponding to the priority parameters.

[0135] The priority calculation module is used to calculate the first priority of the first SIM card and the second priority of the second SIM card based on the priority parameter and the weight corresponding to the priority parameter.

[0136] In some optional embodiments, the device further comprises:

[0137] The model parameter setting module is used to set the state and action of the Q table corresponding to the target device and the parameters of the Q-learning training model, and set the Q value corresponding to each state and action in the Q table to the preset value.

[0138] The first updating module is used to perform reinforcement learning training on the Q-learning training model based on the cell information and update the Q value in the Q table.

[0139] In some optional implementations, the action determination module 403 includes:

[0140] The target action submodule is used to determine the target action with the highest expected return based on the Q table when the first priority meets the preset conditions.

[0141] The simulation execution submodule is used to perform simulation execution based on the target action and obtain the simulation execution result.

[0142] The second updating submodule is used to update the Q value of the Q table based on the simulation execution result.

[0143] In some optional implementations, the action determination module 403 further includes:

[0144] The second priority calculation submodule is used to calculate the second priority of the second SIM card according to the cell information when the first priority does not meet the preset requirements.

[0145] The release submodule is used to release the connection resources of the first SIM card when the second priority meets the preset condition.

[0146] In an optional implementation, the action determination module 403 further includes:

[0147] The confirmation information submodule is used to send a confirmation release message to the target device to obtain the switching result.

[0148] The third updating submodule is used to update the switching record and the Q table based on the switching result.

[0149] In an optional implementation, the first update module includes:

[0150] The data grouping submodule is used to divide the cell information into a training set and a test set according to a preset ratio.

[0151] The evaluation result submodule is used to train the model using the data of the training set, and then use the data of the test set to evaluate the model to obtain the evaluation result.

[0152] The update model submodule is used to adjust the parameters of the Q-learning training model based on the evaluation results to obtain the target Q-learning training model.

[0153] The fourth updating submodule is used to update the Q value in the Q table based on the target Q-learning training model.

[0154] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0155] The flow switching device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0156] The embodiment of the present invention also provides a computer device having the above Figure 4 The flow switching device shown.

[0157] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0158] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0159] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0160] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0161] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0162] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0163] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0164] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0165] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0166] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A flow switching method, characterized in that: The method is performed by a network data analysis function NWDAF, and includes: Acquire cell information sent by a target device; the target device is provided with a first SIM card and a second SIM card; the cell information is used to indicate the connection status of the first SIM card and the second SIM card with the cell; calculating respectively a first priority of the first SIM card and a second priority of the second SIM card according to the cell information; If the first priority of the first SIM card and the second priority of the second SIM card do not meet the preset conditions, based on the state of the target device, determine the target action with the highest expected return in the Q table; the Q table is used to characterize the mapping relationship between the state and the expected return of the action; the Q table is obtained by reinforcement learning based on the Q-learning training model through the historical data of cell switching; The target device is instructed to perform cell switching based on the target action.

2. The method according to claim 1, characterized in that Before respectively calculating the first priority of the first SIM card and the second priority of the second SIM card according to the cell information, the method further includes: Setting priority parameters and weights corresponding to the priority parameters; Based on the priority parameters and the weights corresponding to the priority parameters, a first priority of the first SIM card and a second priority of the second SIM card are calculated respectively.

3. The method according to claim 2, characterized in that After obtaining the cell information sent by the target device, the method further includes: Setting the state and action of the Q table corresponding to the target device and the parameters of the Q-learning training model, and setting the Q value corresponding to each state and action in the Q table to a preset value; Reinforcement learning training is performed on the Q-learning training model based on the cell information, and the Q value in the Q table is updated.

4. The method according to any one of claims 1 to 3, characterized in that: The cell information includes at least one of signal strength, network load, connection stability, handover success rate, and whether it is a capacity enhancement cell.

5. The method according to claim 4, characterized in that If the first priority of the first SIM card and the second priority of the second SIM card do not meet the preset condition, before determining the target action with the highest expected return in the Q table based on the state of the target device, the method further includes: If the first priority meets the preset condition, determining the target action with the highest expected reward based on the Q table; Perform simulation execution based on the target action to obtain a simulation execution result; Based on the simulation execution result, the Q value of the Q table is updated.

6. The method according to claim 5, characterized in that The method further comprises: If the first priority does not meet the preset requirement, calculating the second priority of the second SIM card according to the cell information; If the second priority meets a preset condition, the connection resources of the first SIM card are released.

7. The method according to claim 4, characterized in that The method further comprises: Sending a release confirmation message to the target device to obtain a switching result; The switching record and the Q table are updated based on the switching result.

8. The method according to claim 3, characterized in that The performing reinforcement learning training on the Q-learning training model based on the cell information and updating the Q value in the Q table includes: Dividing the cell information into a training set and a test set according to a preset ratio; After using the data of the training set to train the model, the data of the test set is used to evaluate the model to obtain an evaluation result; Based on the evaluation result, adjusting the parameters of the Q-learning training model to obtain a target Q-learning training model; Based on the target Q-learning training model, the Q value in the Q table is updated.

9. A flow switching device, characterized in that: The device comprises: An acquisition module, used for acquiring cell information sent by a target device; the target device is provided with a first SIM card and a second SIM card; the cell information is used for indicating the connection status between the first SIM card and the second SIM card and the cell; a calculation module, configured to calculate the first priority of the first SIM card and the second priority of the second SIM card respectively according to the cell information; an action determination module, configured to determine, when the first priority of the first SIM card and the second priority of the second SIM card do not meet the preset conditions, a target action with the highest expected return in a Q table based on the state of the target device; the Q table is used to characterize the mapping relationship between the state and the expected return of the action; the Q table is obtained by reinforcement learning based on the Q-learning training model through the historical data of cell switching; A switching module is used to instruct the target device to perform cell switching based on the target action.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes a flow switching method according to any one of claims 1 to 8 by executing the computer instructions.

Images