A Reinforcement Learning-Based Intelligent Routing and Jump Recommendation Method

Abstract

This invention discloses an intelligent routing and jumping recommendation method based on reinforcement learning, relating to the field of computer network technology; it includes: Step 1: collecting multi-dimensional data, including user role information, page type, and user feedback data; Step 2: defining a state space S, and structuring the collected data, with each system state represented by a three-dimensional vector, each dimension corresponding to the user role code, page type code, and user feedback state code; Step 3: creating a Q-table, initializing the Q-value table according to the size of the state space S and action space A, with the Q-table dimension being |S|×|A|, and each element Q(s,a) initialized to a fixed value Q0; Step 4: performing action encoding, uniquely encoding the jumping target of each page, and establishing a mapping relationship table between action encoding and actual page jumping logic; Step 5: optimizing jumping actions based on a Q-Learning recommendation strategy; Step 6: generating and outputting recommendation results, and receiving feedback.

Description

Technical Field

[0001] This invention discloses an intelligent routing recommendation method based on reinforcement learning, which relates to the field of computer network technology. Background Technology

[0002] In existing business systems, users often face the problem of unclear next steps after submitting or completing page operations. Current systems mostly use fixed-rule redirection methods, such as pre-defined business process redirection logic. This approach has several drawbacks: First, because the rules are fixed, they cannot adapt to the personalized operating habits and needs of different users. For example, users with different roles have varying levels of familiarity with the business process, and the required redirection guidance should also differ. Second, in the face of business changes, such as adding new business functions or adjusting business processes, the cost of modifying and maintaining fixed rules is high, and it is difficult to respond quickly to changes. Third, rigid redirection methods cannot be dynamically adjusted based on real-time user feedback, resulting in a poor user experience, low redirection efficiency, and consequently affecting user satisfaction and system efficiency. Summary of the Invention

[0003] This invention aims to solve the problems of unclear jump paths, rigid fixed-rule jump methods, inability to adapt to different users or business changes, and poor user experience in existing business systems. By using an intelligent routing jump recommendation method based on reinforcement learning, dynamic self-learning and personalized page recommendations are achieved, thereby improving user jump efficiency and experience and reducing the configuration and maintenance costs of fixed rules.

[0004] The specific solution proposed in this invention is as follows:

[0005] This invention provides an intelligent route recommendation method based on reinforcement learning, comprising:

[0006] Step 1: Conduct multi-dimensional data collection, including user role information, page type, and user feedback data.

[0007] Step 2: Define the state space S, and perform structured encoding on the collected data. Each system state is represented by a three-dimensional vector, with each dimension corresponding to the user role encoding, page type encoding, and user feedback status encoding, respectively.

[0008] Step 3: Create the Q-table. Initialize the Q-value table according to the size of the state space S and the action space A. The Q-table has a dimension of |S|×|A|. Each element Q(s,a) is initialized to a fixed value Q0.

[0009] Step 4: Perform action coding, uniquely encode the redirection target for each page, and establish a mapping table between action codes and actual page redirection logic.

[0010] Step 5: Optimize the jump action based on the recommendation strategy of Q-Learning:

[0011] At each time step t, when the system is in state s t At that time, generate a random number r in the interval [0,1]. If r < ∈, and ∈ has an initial value of 0.2, from the action space A(s) t The user randomly selects an action from the given list; otherwise, they select the action with the highest Q-value in the current state. The user then executes action a. t Then enter the new state s t+1 Calculate instant rewards r based on user feedback and preset rules. t If a valid operation r is completed t =+5, if you click the cancel button r t =-6, if no action is taken within 3 seconds r t =-3; Update formula according to Q-Learning:

[0012]

[0013] Update the Q table, where the learning rate α = 0.1 and the discount factor γ = 0.9;

[0014] Step 6: Generate and output recommendation results, and receive feedback: When a user enters a certain page state, the system selects the action a with the maximum Q value from the action space A(s) corresponding to the current state s according to the Q table. * Calculate the difference ΔQ between the largest Q value and the second largest Q value; then apply action a. * The corresponding target page business name, redirect URL, and ΔQ are encapsulated and returned to the front end by the back end. The front end sets a confidence threshold θ and compares the received ΔQ with θ. If ΔQ ≥ θ, a recommended redirect pop-up is displayed in an appropriate position on the page, and the page redirection is triggered when the user clicks the recommended link. When ΔQ < θ, the front end does not display a redirect pop-up and executes the preset fallback rule.

[0015] Furthermore, step 1 of the aforementioned intelligent route jump recommendation method based on reinforcement learning includes:

[0016] Step 11: Collect user role information and obtain user identity tags, which include administrator role tags, ordinary employee role tags, and customer role tags;

[0017] Step 12: Collect page type. When the backend receives a request from the frontend, it determines the page type based on the URL path or interface identifier of the request, combined with the pre-established mapping relationship.

[0018] Step 13: Collect user feedback data. Through the front-end event listening mechanism, record the user's response to the recommended jump within 3 seconds of the recommendation prompt message being displayed. The response actions include clicking the jump link, clicking the cancel button, and no action.

[0019] Furthermore, in step 2 of the reinforcement learning-based intelligent routing and recommendation method, each system state is represented by a three-dimensional vector, denoted as: s = [role, page_type, feedback_status].

[0020] Each dimension corresponds to the user role code, page type code, and user feedback status code, respectively.

[0021] Furthermore, step 6 of the aforementioned intelligent route jump recommendation method based on reinforcement learning specifically includes:

[0022] S61: Optimal action selection. When a user enters a certain page state, the system selects the action a with the maximum Q value from the action space A(s) corresponding to the current state s, based on the Q table. * =argmax a∈A(s) Q(s,a), and simultaneously calculate the difference ΔQ = Q(s,a) between the largest and second largest Q values. max )-Q(s,a second ), and return ΔQ along with the recommendation results to the front end;

[0023] S62: Recommendation information assembly, including action a * The corresponding target page business name, redirect URL, and ΔQ are encapsulated into JSON format data and returned to the front end through the backend API interface;

[0024] S63: Front-end display and redirection. After receiving the recommendation data, the front-end sets a confidence threshold θ and compares the received ΔQ with θ:

[0025] If ΔQ≥θ, then a recommended redirect pop-up window will be displayed in an appropriate location on the page.

[0026] This invention also provides an intelligent route recommendation device based on reinforcement learning, comprising: a data acquisition module, a data encoding module, a Q-table management module, an action encoding module, a policy management module, and an output generation module.

[0027] The data collection module performs multi-dimensional data collection, including user role information, page type, and user feedback data.

[0028] The data encoding module defines a state space S, which performs structured encoding on the collected data. Each system state is represented by a three-dimensional vector, with each dimension corresponding to the user role encoding, page type encoding, and user feedback status encoding, respectively.

[0029] The Q-table management module creates the Q-table and initializes the Q-value table based on the size of the state space S and the action space A. The Q-table has a dimension of |S|×|A|, and each element Q(s,a) is initialized to a fixed value Q0.

[0030] The action coding module performs action coding, uniquely encoding the navigation target for each page, and establishing a mapping table between action codes and actual page navigation logic.

[0031] The strategy management module optimizes the navigation actions based on Q-Learning's recommendation strategy:

[0032] At each time step t, when the system is in state s t At that time, generate a random number r in the interval [0,1]. If r < ∈, and ∈ has an initial value of 0.2, from the action space A(s) t The user randomly selects an action from the given list; otherwise, they select the action with the highest Q-value in the current state. The user then executes action a. t Then enter the new state s t+1 Calculate instant rewards r based on user feedback and preset rules. t If a valid operation r is completed t =+5, if you click the cancel button r t =-6, if no action is taken within 3 seconds r t =-3; Update formula according to Q-Learning:

[0033]

[0034] Update the Q table, where the learning rate α = 0.1 and the discount factor γ = 0.9;

[0035] The output generation module generates and outputs recommendation results and receives feedback: When a user enters a certain page state, the system selects the action a with the maximum Q value from the action space A(s) corresponding to the current state s according to the Q table. * Calculate the difference ΔQ between the largest Q value and the second largest Q value; then apply action a. * The corresponding target page business name, redirect URL, and ΔQ are encapsulated and returned to the front end by the back end. The front end sets a confidence threshold θ and compares the received ΔQ with θ. If ΔQ ≥ θ, a recommended redirect pop-up is displayed in an appropriate position on the page, and the page redirection is triggered when the user clicks the recommended link. When ΔQ < θ, the front end does not display a redirect pop-up and executes the preset fallback rule.

[0036] Furthermore, the data acquisition module of the aforementioned reinforcement learning-based intelligent route redirection recommendation device performs the following:

[0037] Step 11: Collect user role information and obtain user identity tags, which include administrator role tags, ordinary employee role tags, and customer role tags;

[0038] Step 12: Collect page type. When the backend receives a request from the frontend, it determines the page type based on the URL path or interface identifier of the request, combined with the pre-established mapping relationship.

[0039] Step 13: Collect user feedback data. Through the front-end event listening mechanism, record the user's response to the recommended jump within 3 seconds of the recommendation prompt message being displayed. The response actions include clicking the jump link, clicking the cancel button, and no action.

[0040] Furthermore, the data encoding module of the reinforcement learning-based intelligent routing recommendation device uses a three-dimensional vector to represent each system state, denoted as: s = [role, page_type, feedback_status].

[0041] Each dimension corresponds to the user role code, page type code, and user feedback status code, respectively.

[0042] Furthermore, the generation and output module of the aforementioned reinforcement learning-based intelligent route jump recommendation device performs the following:

[0043] S61: Optimal action selection. When a user enters a certain page state, the system selects the action a with the maximum Q value from the action space A(s) corresponding to the current state s, based on the Q table. * =argmax a∈A(s) Q(s,a), and simultaneously calculate the difference ΔQ = Q(s,a) between the largest and second largest Q values. max )-Q(s,a second ), and return ΔQ along with the recommendation results to the front end;

[0044] S62: Recommendation information assembly, including action a * The corresponding target page business name, redirect URL, and ΔQ are encapsulated into JSON format data and returned to the front end through the backend API interface;

[0045] S63: Front-end display and redirection. After receiving the recommendation data, the front-end sets a confidence threshold θ and compares the received ΔQ with θ:

[0046] If ΔQ≥θ, then a recommended redirect pop-up window will be displayed in an appropriate location on the page.

[0047] The advantages of this invention are:

[0048] 1. Unlike traditional fixed-rule redirection methods, this invention eliminates the need for frequent rule modifications to adapt to business changes and user needs. The system can automatically optimize recommendation strategies through reinforcement learning, greatly reducing the manpower and time costs of rule configuration and maintenance.

[0049] 2. This invention can adapt to the personalized operating habits of different users and the dynamic changes in business. Whether it is a new user or an old user, whether it is an existing business process or a newly added business function, the system can continuously adjust the recommendation strategy through learning to maintain a good recommendation effect and improve the system's versatility and scalability.

[0050] 3. The introduction of the rule fallback mechanism in this invention avoids user operation interruption or recommendation interference caused by recommendation anomalies, thereby enhancing the reliability and usability of the system. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0052] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0053] Example 1

[0054] This invention provides an intelligent route recommendation method based on reinforcement learning, comprising:

[0055] Step 1: Conduct multi-dimensional data collection, which includes user role information, page type, and user feedback data.

[0056] For example, data collection modules can be deployed at key operation nodes in the expense reimbursement system, such as the reimbursement form filling page, the approval page, and the voucher upload page, to achieve real-time collection of user behavior data, including:

[0057] S11: User Role Information Collection: The system's backend permission management module interacts with the enterprise identity authentication center via the OAuth 2.0 protocol to obtain the role information of logged-in users. Upon identification, if the user is a regular employee, the "Regular Employee" role label is assigned and encoded as 0.

[0058] S12: Page Type Collection: When a user enters the expense report form filling page, the backend receives a request sent by the frontend based on the Axios library and parses its URL as " / reimbursement / apply". Based on the pre-defined mapping relationship {" / login":"login page"," / reimbursement / apply":"expense report form filling page"," / reimbursement / review":"review page"}, the backend uses regular expression matching rules to determine that this page is the expense report form filling page, encoded as 1.

[0059] S13: User Feedback Data Collection: After the system recommends a "View Reimbursement Standards" link to the user, the front-end uses the JavaScript `setTimeout` function to start a 3-second timer and records the user's feedback status through the `addEventListener` event listener mechanism. If the user clicks the link within 3 seconds, it is recorded as 1; if the user clicks the cancel button, it is recorded as -1; if no action is taken after the timer ends, it is recorded as 0.

[0060] Step 2: Define the state space S. The collected data is structured and encoded, with each system state represented by a three-dimensional vector. Each dimension corresponds to the user role code, page type code, and user feedback status code, respectively. Each state s∈S is represented by the three-dimensional vector s=[role,page_type,feedback_status]. To facilitate system storage and processing, binary encoding is used to compress the information in each dimension. For example, if a user clicks the "recommendation" button on the expense report form page, the corresponding state vector s=[0,1,1] is stored as "000101" in binary. The system establishes a mapping table between state codes and raw data for data reconstruction and analysis.

[0061] Step 3: Create the Q-table. Initialize the Q-value table based on the size of the state space S and the action space A. The Q-table has a dimension of |S|×|A|, and each element Q(s,a) is initialized to a fixed value Q0. Specifically, at system startup, based on the state space S, which contains 15 different states covering different roles, pages, and feedback state combinations, and the action space A, which includes 6 possible navigation actions on the expense report page (such as "View Expense Standards", "Add Expense Details", "Upload Attachments", "Submit Expense Report", "Return to Homepage", and "View Historical Expense Records"), a Q-table with a dimension of 15×6 is created. The initial value of each element Q(s,a) in the Q-table is set to 0.3.

[0062] Step 4: Perform action coding, uniquely encoding the redirection target for each page and establishing a mapping table between action codes and the actual page redirection logic. Specifically, for the expense report form filling page, the redirection target is coded as follows: "View Expense Standards" is coded as 0, "Add Expense Details" as 1, "Upload Attachment" as 2, "Submit Expense Report" as 3, "Return to Homepage" as 4, and "View Historical Expense Records" as 5. During system initialization, all action codes undergo uniqueness and conflict checks. Simultaneously, a mapping table between action codes and the actual page redirection logic is established for rapid transitions between recommendation result generation and page redirection.

[0063] Step 5: Optimize the jump action based on the recommendation strategy of Q-Learning:

[0064] At each time step t, when the system is in state s t At that time, generate a random number r in the interval [0,1]. If r < ∈, and ∈ has an initial value of 0.2, from the action space A(s) t The user randomly selects an action from the given list; otherwise, they select the action with the highest Q-value in the current state. The user then executes action a. t Then enter the new state s t+1 Calculate instant rewards r based on user feedback and preset rules. t If a valid operation r is completed t =+5, if you click the cancel button r t =-6, if no action is taken within 3 seconds r t =-3; Update formula according to Q-Learning:

[0065]

[0066] Update the Q table, where the learning rate α = 0.1 and the discount factor γ = 0.9.

[0067] The specific process can be as follows:

[0068] S51: Action Selection Strategy: At a certain time step t, the system is in state s. t , such as s t = [0,1,0], indicating that ordinary employees did not perform the previous recommended operation on the expense report page. The initial value of ∈ is set to 0.2. Generate a random number r in the interval [0,1]. If r < ∈, then from the action space A(s t A random action is selected from the given options; otherwise, the action with the largest Q value in the current state is selected. Here, ∈ decays with each training iteration according to a preset decay function ∈ = 0.2 - 0.0001 × n, where n is the iteration number, decreasing from 0.2 to 0.05.

[0069] S52: State Transition and Reward Calculation: User performs action a tAfter that, the system enters a new state s t+1 The user enters the process by performing the "Add Expense Details" action on the expense report form page. t+1 Taking the state [0,1,1] as an example, the reward r is calculated according to the rules. t If a valid operation r is completed t =+5.

[0070] S53: Q-value update, adjust the Q table according to the Q-Learning update formula:

[0071]

[0072] Given a learning rate α = 0.1 and a discount factor γ = 0.9, let s t Taking the "Add Fee Details" action code 1 as an example, in the current state, the original Q(s) t ,1)=0.3, in s t+1 In this state, Based on the formula above, substitute the data to calculate: Q(s) t ,1)←0.3+0.1[5+0.9*0.3-0.3]=0.3+0.1(5+0.27-0.3)=

[0073] 0.3 + 0.1 * 4.97 = 0.797, completing the Q-value update. The system employs a distributed transaction processing mechanism to ensure the accuracy and consistency of Q-value updates. It controls access to the Q-table during multi-user operations through distributed locks and logs each update, including information such as the Q-value before the update, the state-action pair, and the reward value.

[0074] Step 6: Generate and output recommendation results, and receive feedback: When a user enters a certain page state, the system selects the action a with the maximum Q value from the action space A(s) corresponding to the current state s according to the Q table. * Calculate the difference ΔQ between the largest Q value and the second largest Q value; then apply action a. * The corresponding target page business name, redirect URL, and ΔQ are encapsulated and returned to the front end by the back end. The front end sets a confidence threshold θ and compares the received ΔQ with θ. If ΔQ ≥ θ, a recommended redirect pop-up is displayed in an appropriate position on the page, and the page redirection is triggered when the user clicks the recommended link. When ΔQ < θ, the front end does not display a redirect pop-up and executes a preset fallback rule. The specific process is as follows:

[0075] Optimal action selection: When a user enters the expense report form page, the state s = [0, 1, 0]. The system traverses the Q-values ​​of each action in the action space A(s) and determines the action a corresponding to the maximum Q-value. *And record the second largest Q value, and calculate the difference ΔQ between the two. For example, suppose Q(s,0)=0.3, Q(s,1)=0.797, Q(s,2)=0.3, Q(s,3)=0.3, Q(s,4)=0.3, Q(s,5)=0.3, then a * For "Add Expense Details" (code 1), Q(s,a) max )=0.797,Q(s,a second =0.3, ΔQ = 0.797 - 0.3 = 0.497. The system uses parallel computing to improve the efficiency of Q-value calculation and sets a jitter threshold. When ΔQ is less than the jitter threshold but greater than the confidence threshold, the action with the largest or second largest Q-value is randomly selected as the recommendation.

[0076] S6.2, Recommendation Information Assembly: Assemble a * The business name of the corresponding page, the redirect URL, and ΔQ = 0.497 are encapsulated as JSON data: {"business_name":"Add Fee Details","url":" / reimbursement / add_detail","delta_Q":0.497}.

[0077] S6.3. The front-end sets a confidence threshold θ = 0.3. After receiving recommendation data from the back-end, it compares ΔQ with θ. When ΔQ = 0.497 > 0.3, a recommendation pop-up is displayed at an appropriate location on the page. When the user clicks the recommendation link, the window.location.href method is used to redirect the page, and the current page state information is recorded.

[0078] Rule fallback mechanism: When ΔQ < 0.3, the front end does not display a jump pop-up and executes the fallback rule. Since ΔQ = 0.497 > 0.3 in this operation, the pop-up is still displayed and the fallback is not triggered.

[0079] In summary, the method in this embodiment constructs a state space by collecting user roles, page types, and feedback behaviors, and dynamically optimizes page navigation strategies based on the Q-Learning algorithm. Personalized recommendations are achieved through Q-table updates and optimal action selection. Furthermore, a confidence assessment mechanism and rule-based fallback logic ensure accurate and reliable recommendation results, effectively improving user navigation efficiency and system intelligence.

[0080] Example 2

[0081] This invention also provides an intelligent route recommendation device based on reinforcement learning, comprising: a data acquisition module, a data encoding module, a Q-table management module, an action encoding module, a policy management module, and an output generation module.

[0082] The data collection module performs multi-dimensional data collection, including user role information, page type, and user feedback data.

[0083] The data encoding module defines a state space S, which performs structured encoding on the collected data. Each system state is represented by a three-dimensional vector, with each dimension corresponding to the user role encoding, page type encoding, and user feedback status encoding, respectively.

[0084] The Q-table management module creates the Q-table and initializes the Q-value table based on the size of the state space S and the action space A. The Q-table has a dimension of |S|×|A|, and each element Q(s,a) is initialized to a fixed value Q0.

[0085] The action coding module performs action coding, uniquely encoding the navigation target for each page, and establishing a mapping table between action codes and actual page navigation logic.

[0086] The strategy management module optimizes the navigation actions based on Q-Learning's recommendation strategy:

[0087] At each time step t, when the system is in state s t At that time, generate a random number r in the interval [0,1]. If r < ∈, and ∈ has an initial value of 0.2, from the action space A(s) t The user randomly selects an action from the given list; otherwise, they select the action with the highest Q-value in the current state. The user then executes action a. t Then enter the new state s t+1 Calculate instant rewards r based on user feedback and preset rules. t If a valid operation r is completed t =+5, if you click the cancel button r t =-6, if no action is taken within 3 seconds r t =-3; Update formula according to Q-Learning:

[0088]

[0089] Update the Q table, where the learning rate α = 0.1 and the discount factor γ = 0.9;

[0090] The output generation module generates and outputs recommendation results and receives feedback: When a user enters a certain page state, the system selects the action a with the maximum Q value from the action space A(s) corresponding to the current state s according to the Q table. * Calculate the difference ΔQ between the largest Q value and the second largest Q value; then apply action a. *The corresponding target page business name, redirect URL, and ΔQ are encapsulated and returned to the front end by the back end. The front end sets a confidence threshold θ and compares the received ΔQ with θ. If ΔQ ≥ θ, a recommended redirect pop-up is displayed in an appropriate position on the page, and the page redirection is triggered when the user clicks the recommended link. When ΔQ < θ, the front end does not display a redirect pop-up and executes the preset fallback rule.

[0091] The information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description in the method embodiment of the present invention, and will not be repeated here.

[0092] Similarly, the advantages of the device of the present invention are:

[0093] 1. Unlike traditional fixed-rule redirection methods, which require frequent rule modifications to adapt to business changes and user needs, the system can automatically optimize recommendation strategies through reinforcement learning, greatly reducing the manpower and time costs of rule configuration and maintenance.

[0094] 2. It can adapt to the personalized operating habits of different users and the dynamic changes in business. Whether it is a new user or an old user, whether it is an existing business process or a newly added business function, the system can continuously adjust the recommendation strategy through learning to maintain a good recommendation effect, thereby improving the system's versatility and scalability.

[0095] 3. A fallback mechanism is introduced to avoid user operation interruption or recommendation interference caused by abnormal recommendations, thereby enhancing the reliability and usability of the system.

[0096] It should be noted that not all steps and modules in the above processes and device structures are mandatory; some steps or modules can be omitted as needed. The execution order of each step is not fixed and can be adjusted as required. The system structure described in the above embodiments can be a physical structure or a logical structure. That is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0097] The embodiments described above are merely preferred embodiments for fully illustrating the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A smart routing and jump recommendation method based on reinforcement learning, characterized by: include: Step 1: Conduct multi-dimensional data collection, including user role information, page type, and user feedback data. Step 2: Define the state space S, and perform structured encoding on the collected data. Each system state is represented by a three-dimensional vector, with each dimension corresponding to the user role encoding, page type encoding, and user feedback status encoding, respectively. Step 3: Create the Q-table. Initialize the Q-value table according to the size of the state space S and the action space A. The Q-table has a dimension of |S|×|A|. Each element Q(s,a) is initialized to a fixed value Q0. Step 4: Perform action coding, uniquely encode the redirection target for each page, and establish a mapping table between action codes and actual page redirection logic. Step 5: Optimize the jump action based on the recommendation strategy of Q-Learning: At each time step t, when the system is in state s t At that time, generate a random number r in the interval [0,1]. If r < ∈, and ∈ has an initial value of 0.2, from the action space A(s) t The user randomly selects an action from the given list; otherwise, they select the action with the highest Q-value in the current state. The user executes action a. t Then enter the new state s t+1 Calculate instant rewards r based on user feedback and preset rules. t If a valid operation r is completed t =+5, if you click the cancel button r t =-6, if no action is taken within 3 seconds r t =-3; Update formula according to Q-Learning: Update the Q table, where the learning rate α = 0.1 and the discount factor γ = 0.9; Step 6: Generate and output recommendation results, and receive feedback: When a user enters a certain page state, the system selects the action a with the maximum Q value from the action space A(s) corresponding to the current state s according to the Q table. * Calculate the difference ΔQ between the largest Q value and the second largest Q value; then apply action a. * The corresponding target page business name, redirect URL, and ΔQ are encapsulated and returned to the front end by the back end. The front end sets a confidence threshold θ, compares the received ΔQ with θ, and if ΔQ≥θ, displays a recommended redirect pop-up in an appropriate position on the page. When the user clicks the recommended link, the page redirection operation is triggered. When ΔQ < θ, the front end does not display a redirect pop-up and executes the preset fallback rule.

2. The intelligent route jumping recommendation method based on reinforcement learning according to claim 1, characterized in that: Step 1 includes: Step 11: Collect user role information and obtain user identity tags, which include administrator role tags, ordinary employee role tags, and customer role tags; Step 12: Collect page type. When the backend receives a request from the frontend, it determines the page type based on the URL path or interface identifier of the request, combined with the pre-established mapping relationship. Step 13: Collect user feedback data. Through the front-end event listening mechanism, record the user's response to the recommended jump within 3 seconds of the recommendation prompt message being displayed. The response actions include clicking the jump link, clicking the cancel button, and no action.

3. The intelligent route jumping recommendation method based on reinforcement learning according to claim 1, characterized in that: In step 2, each system state is represented by a three-dimensional vector, denoted as: s = [role, page_type, feedback_status]. Each dimension corresponds to the user role code, page type code, and user feedback status code, respectively.

4. The intelligent route jumping recommendation method based on reinforcement learning according to claim 1, characterized in that: Step 6 specifically includes: S61: Optimal action selection. When a user enters a certain page state, the system selects the action a with the maximum Q value from the action space A(s) corresponding to the current state s, based on the Q table. * =argmax a∈A(s) Q(s,a), and simultaneously calculate the difference ΔQ = Q(s,a) between the largest and second largest Q values. max )-Q(s,a second ), and return ΔQ along with the recommendation results to the front end; S62: Recommendation information assembly, including action a * The corresponding target page business name, redirect URL, and ΔQ are encapsulated into JSON format data and returned to the front end through the backend API interface; S63: Front-end display and redirection. After receiving the recommendation data, the front-end sets a confidence threshold θ and compares the received ΔQ with θ: If ΔQ≥θ, then a recommended redirect pop-up window will be displayed in an appropriate location on the page.

5. A smart route recommendation device based on reinforcement learning, characterized in that: include: The module includes a data acquisition module, a data encoding module, a Q-table management module, an action encoding module, a strategy management module, and an output generation module. The data collection module performs multi-dimensional data collection, including user role information, page type, and user feedback data. The data encoding module defines a state space S, which performs structured encoding on the collected data. Each system state is represented by a three-dimensional vector, with each dimension corresponding to the user role encoding, page type encoding, and user feedback status encoding, respectively. The Q-table management module creates the Q-table and initializes the Q-value table based on the size of the state space S and the action space A. The Q-table has a dimension of |S|×|A|, and each element Q(s,a) is initialized to a fixed value Q0. The action coding module performs action coding, uniquely encoding the navigation target for each page, and establishing a mapping table between action codes and actual page navigation logic. The strategy management module optimizes the navigation actions based on Q-Learning's recommendation strategy: At each time step t, when the system is in state s t At that time, generate a random number r in the interval [0,1]. If r < ∈, and ∈ has an initial value of 0.2, from the action space A(s) t The user randomly selects an action from the given list; otherwise, they select the action with the highest Q-value in the current state. The user executes action a. t Then enter the new state s t+1 Calculate instant rewards r based on user feedback and preset rules. t If a valid operation r is completed t =+5, if you click the cancel button r t =-6, if no action is taken within 3 seconds r t =-3; Update formula according to Q-Learning: Update the Q table, where the learning rate α = 0.1 and the discount factor γ = 0.9; The output generation module generates and outputs recommendation results and receives feedback: When a user enters a certain page state, the system selects the action a with the maximum Q value from the action space A(s) corresponding to the current state s according to the Q table. * Calculate the difference ΔQ between the largest Q value and the second largest Q value; then apply action a. * The corresponding target page business name, redirect URL, and ΔQ are encapsulated and returned to the front end by the back end. The front end sets a confidence threshold θ, compares the received ΔQ with θ, and if ΔQ≥θ, displays a recommended redirect pop-up in an appropriate position on the page. When the user clicks the recommended link, the page redirection operation is triggered. When ΔQ < θ, the front end does not display a redirect pop-up and executes the preset fallback rule.

6. The intelligent route jump recommendation device based on reinforcement learning according to claim 5, characterized in that: The data acquisition module executes the following: Step 11: Collect user role information and obtain user identity tags, which include administrator role tags, ordinary employee role tags, and customer role tags; Step 12: Collect page type. When the backend receives a request from the frontend, it determines the page type based on the URL path or interface identifier of the request, combined with the pre-established mapping relationship. Step 13: Collect user feedback data. Through the front-end event listening mechanism, record the user's response to the recommended jump within 3 seconds of the recommendation prompt message being displayed. The response actions include clicking the jump link, clicking the cancel button, and no action.

7. The intelligent route jump recommendation device based on reinforcement learning according to claim 5, characterized in that: The data encoding module uses a three-dimensional vector to represent each system state, denoted as: s = [role, page_type, feedback_status]. Each dimension corresponds to the user role code, page type code, and user feedback status code, respectively.

8. The intelligent route jump recommendation device based on reinforcement learning according to claim 5, characterized in that: Execution of the output generation module: S61: Optimal action selection. When a user enters a certain page state, the system selects the action a with the maximum Q value from the action space A(s) corresponding to the current state s, based on the Q table. * =argmax a∈A(s) Q(s,a), and simultaneously calculate the difference ΔQ = Q(s,a) between the largest and second largest Q values. max )-Q(s,a second ), and return ΔQ along with the recommendation results to the front end; S62: Recommendation information assembly, including action a * The corresponding target page business name, redirect URL, and ΔQ are encapsulated into JSON format data and returned to the front end through the backend API interface; S63: Front-end display and redirection. After receiving the recommendation data, the front-end sets a confidence threshold θ and compares the received ΔQ with θ: If ΔQ≥θ, then a recommended redirect pop-up window will be displayed in an appropriate location on the page.

Images