Intelligent CRM management method based on reinforcement learning

Abstract

The invention discloses an intelligent CRM management method based on reinforcement learning, and the method comprises the following steps: S1, obtaining customer relationship data, and carrying out the preprocessing of the customer relationship data; s2, constructing a Markov decision process, defining a state and action mapping relation, and labeling behavior feedback to generate a triple set; s3, constructing a reinforcement learning framework, executing strategy sampling and experience playback on the triple set, and calculating a strategy updating direction; s4, performing convolution extraction and residual aggregation on the state sequence, and generating a context fusion vector in combination with cross-layer attention; s5, generating action distribution, estimating a dominant function and updating strategy parameters; s6, selecting a strategy action according to action distribution, mapping the strategy action into a structured CRM instruction, and constructing a behavior control flow diagram; and S7, recording execution feedback, and continuously optimizing the strategy and value network parameters by using a residual error correction mechanism. According to the method, closed-loop optimization of customer policy and CRM process execution is realized, and the response and conversion efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of customer relationship management technology, and in particular to an intelligent CRM management method based on reinforcement learning. Background Technology

[0002] In the field of Customer Relationship Management (CRM), enterprises generally rely on rule-driven process configuration, manually set customer segmentation criteria, and fixed-cycle marketing strategy execution. These traditional methods are no longer sufficient to meet the demands for efficient, accurate, and intelligent customer service and conversion in the face of increasingly complex customer behavior, rapidly shrinking decision-making windows, and diversified business chains.

[0003] Most existing CRM management methods employ decision-making logic based on static tags, dividing customers into limited levels or groups and configuring preset operation templates for each group. For example, a customer triggering a certain behavior is marked as a "potential churn," and the system then executes response actions such as pushing offers and sending reminders according to set rules. While this approach is simple to implement initially, it suffers from the following significant drawbacks in long-term operation: First, customer behavior changes frequently, making it difficult for fixed rules to adapt to dynamic scenarios; second, the response actions lack an optimality evaluation mechanism, failing to assess strategy quality from a long-term perspective; and third, strategy actions are usually tied to immediate conversion metrics, neglecting the balance between long-term value and customer lifetime value.

[0004] In recent years, some technical solutions have attempted to incorporate machine learning methods to classify, predict, or score customer behavior to support CRM strategy execution. However, these methods still primarily rely on supervised learning, depending on labeled data to train classifiers, and lack feedback mechanisms and adaptive learning capabilities. In complex customer management tasks involving multiple rounds of interaction, cross-stage marketing, and multi-channel integration, supervised models cannot dynamically adjust strategy direction based on real-time strategy execution results, leading to a disconnect between predictions and behavioral decisions, and lagging strategy updates.

[0005] Meanwhile, existing CRM process execution frameworks are mostly linear, lacking the ability to dynamically construct policy behavior structures. Especially in scenarios involving multi-task concurrency, multi-stage transitions, and context-driven conditions, process control exhibits problems such as path splitting, dependency conflicts, and execution delays. Furthermore, policy logic is mostly manually configured by business personnel, making it difficult to model, track, and optimize the long-term impact of customer behavior.

[0006] Therefore, how to provide an intelligent CRM management method based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] One objective of this invention is to propose an intelligent CRM management method based on reinforcement learning. This invention fully utilizes reinforcement learning strategy optimization mechanisms, deep state modeling technology, and multi-task behavior control flow graph generation structures. It describes in detail the complete algorithm path for achieving strategy sampling, dynamic decision-making, and automatic execution of CRM processes driven by customer behavior data. It has the advantages of strong adaptability, high optimality of response strategies, and significant improvement in customer conversion efficiency.

[0008] An intelligent CRM management method based on reinforcement learning according to an embodiment of the present invention includes the following steps: S1. Obtain customer relationship data, preprocess it, and generate a customer status sequence; S2. Construct a Markov decision process, defining the customer state as the environment state, the executable operations as the action space, and labeling each set of states and actions with behavioral feedback to form a set of triples. S3. Construct a reinforcement learning framework based on the policy gradient mechanism, perform policy sampling on the triplet set, construct training batches using the experience replay mechanism, and calculate the policy update direction based on the policy gradient method. S4. During the policy update process, channel-level convolution extraction and window-level residual aggregation operations are performed on the customer state sequence, and a context fusion vector is generated by combining the cross-layer attention mechanism. S5. Input the context fusion vector into the policy network to generate the action distribution, input each set of states and actions into the value network, estimate the state value and advantage function through the dual-path advantage reconstruction method, and complete the policy parameter update. S6. Select strategy actions based on action distribution, map strategy actions into structured CRM instruction sequences, and generate behavior control flow graphs by combining multi-task decoding structures to drive CRM execution flow. S7. Record feedback information during CRM execution and use the policy residual correction mechanism to update the policy network and value network parameters to continuously iterate the policy learning process.

[0009] Optionally, the customer relationship data includes user identifiers, interaction behaviors, timestamps, channel sources, and business tags. The preprocessing includes data cleaning, field alignment, anomaly removal, and behavior encoding operations. After preprocessing, the behavior sequence of each user is sliced ​​according to the time sequence to construct a state sequence with a time-series structure. A fixed-dimensional state representation set is generated based on the set window length and sliding step size, ultimately forming a customer state sequence.

[0010] Optionally, the policy gradient method adopts a joint policy calculation method based on GAE and cutoff reward period, and performs policy gradient direction update based on the current state distribution by constructing a weighted cumulative gradient function between short-term reward and long-term reward.

[0011] Optionally, the dual-path advantage reconstruction method simultaneously establishes two independent estimation paths based on the state value function and the action value function. The first path calculates the state baseline value through the state input, and the second path calculates the action expectation value through each set of states and actions. Finally, the advantage function is constructed through the difference between the two to evaluate the relative superiority of the strategy behavior in the current state and to serve as the basis for updating the value network.

[0012] Optionally, S2 specifically includes: S21. Obtain the preprocessed customer state sequence, and segment and label each customer's behavior segment according to the time sequence to ensure that each state has a clear temporal position and business semantics. S22. For each behavior segment, extract user behavior features, channel source and context information based on the user state content when the behavior is triggered, and combine them to construct a high-dimensional state vector as a representation of the environment state. S23. Clearly define the set of executable operations. The set of operations is set based on the actual CRM business process and covers all controllable behaviors including marketing push, human contact, service transfer, discount activation, interactive guidance, and content recommendation. A unique action code is assigned to each operation to form an action space. S24. For each combination of state and action, analyze customer behavior data, extract whether the customer responds, response delay, click depth, retention period and business conversion indicators, calculate the corresponding behavior feedback value, and mark the feedback result corresponding to the current state and action accordingly. S25. Combine the high-dimensional state vector, the corresponding action code, and the behavioral feedback value into a fixed-structure triplet to form decision sequence data that conforms to the Markov assumption.

[0013] Optionally, S3 specifically includes: S31. Construct a reinforcement learning framework and initialize a dual-network structure including a policy network and a value network. The policy network is used to generate action distributions, and the value network is used to evaluate the advantage score of state-action combinations. The policy network consists of a multi-layer fully connected structure, with each layer followed by an activation function. The last layer outputs a probability vector corresponding to the action dimension. The value network uses a parallel path structure to calculate the state value function and the action value function respectively. S32. Using the triplet set as initial experience data, select multiple state-action-feedback sequences from the triplet set according to a preset sampling interval to construct an experience replay buffer. The buffer is organized in the form of a double-ended queue and has a fixed capacity and a dynamic coverage mechanism based on the first-in-first-out rule to provide a dynamic sample source during the training process. S33. Extract multiple sets of state sequences in batches from the experience replay buffer, input them into the policy network in sequence, obtain the action probability distribution through forward propagation, determine the actual sampled action using a distributed random sampling mechanism, and form a state-action pair set with the state and the corresponding sampled action to calculate the behavioral tendency under the current policy. S34. Based on the set of state-action pairs, input the corresponding state into the value network, calculate the state value of the current state and the action value of the current state-action combination, calculate the advantage value based on the difference between the two, and perform a time discount accumulation operation on the advantage value in each trajectory to obtain a weighted cumulative advantage score. S35. Based on the cumulative advantage score and the action probability output by the policy network, calculate the policy gradient direction for each state-action pair, and use the backpropagation mechanism to backpropagate the gradient signal to the parameters of each layer of the policy network, and accumulate the gradient value of each layer. S36. Based on the gradient accumulation result, perform a parameter update operation on the policy network based on the learning rate adjustment factor, and optimize the objective function of the value network using the least squares error. Control the update rhythm of the two networks by sharing the number of training rounds to complete a complete policy iteration process. S37. Set the termination conditions for continuous strategy optimization. Determine whether to continue strategy training based on the cumulative reward improvement, strategy convergence magnitude, and sample diversity indicators. If the convergence criteria are met, retain the current parameter state as the deployment version of the CRM execution strategy.

[0014] Optionally, S6 specifically includes: S61. Receive the action distribution output by the policy network, use the maximum probability selection mechanism to determine the optimal policy action number in the current state, and perform a uniqueness check on the number to ensure that each policy action corresponds to the executable operation type in the CRM process. S62. Input the selected strategy action number into the instruction mapping unit, look up the mapping rule table, and construct a structured CRM instruction sequence containing execution target, operation parameters, trigger conditions and response type according to the CRM business behavior logic corresponding to the current action. The instruction sequence adopts a field-based hierarchical structure to clarify the dependency relationship between each operation field. S63. Input the structured CRM instruction sequence into the behavior decoding scheduling unit. Based on the field level of the instruction, start the parallel multi-task decoding path and construct execution nodes, condition nodes and jump nodes respectively to form a set of process decoding vectors with node identifiers and path indexes. S64. Based on the process decoding vector set, construct a behavior control flow graph. The control flow graph adopts a graph structure, where nodes represent independent operation behaviors in CRM execution, and edges represent scheduling relationships between behaviors. Generate an executable path set through a topology sorting algorithm. S65. Deploy the behavior control flow graph to the CRM task engine, schedule CRM actions node by node according to the current customer status and control flow path sequence, drive the CRM execution process, and complete a customer management task triggering process based on policy actions.

[0015] Optionally, the mapping rule table is a pre-built instruction mapping dictionary, which uses a key-value pair structure to store the correspondence between policy action numbers and CRM operation behaviors. Each mapping relationship includes an action number, the corresponding CRM business type, the execution target identifier, the parameter field template, and the operation priority index. All mapping relationships are indexed and classified according to the business process node number, and support calling different parameter template generation methods according to the context status label, which is used to convert policy action numbers into a structured CRM instruction sequence with business executability.

[0016] Optionally, the behavior decoding scheduling unit is a process building engine for structured CRM instruction sequences. It adopts a multi-path parallel scheduling architecture, maps the contents marked as execution fields, condition fields and jump fields in the instruction sequence to execution nodes, judgment nodes and branch nodes respectively, and constructs a graph structure topology according to the order relationship between fields. It generates path convergence nodes in the graph using process merging identifiers, and dynamically adjusts the decoding order according to the dependency relationship between tasks. Finally, it outputs a behavior control flow graph data structure containing node numbers, entry indexes and outgoing edge sets.

[0017] The beneficial effects of this invention are: First, by modeling customer behavior as a Markov decision process, this invention establishes a mapping relationship between state, action and reward, enabling real-time perception of changes in customer state and dynamic adjustment of response strategies based on behavioral feedback, thereby significantly improving the system's adaptability and the degree of personalized matching for customers.

[0018] Secondly, this invention designs a dual-network optimization structure consisting of state vector extraction, policy network reasoning, value network evaluation, and advantage function reconstruction. Combined with experience replay and policy gradient update mechanisms, it can accumulate long-term behavioral benefits in the process of continuous policy trial and error, effectively avoid short-sighted operational tendencies, and achieve a deep integration of policy optimality and customer lifetime value.

[0019] Finally, by mapping reinforcement learning policy actions into structured CRM instruction sequences and constructing a behavior control flow graph using multi-task decoding paths, this invention achieves automated execution and flexible configuration of CRM processes, effectively improving the accuracy and timeliness of customer outreach, reducing the frequency of human intervention, and providing practical technical support for enterprises to achieve intelligent, efficient, and closed-loop customer relationship management in complex business environments. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0021] Figure 1 This is a flowchart of an intelligent CRM management method based on reinforcement learning proposed in this invention; Figure 2 This is a CRM modeling flowchart for an intelligent CRM management method based on reinforcement learning proposed in this invention. Figure 3 This is a flowchart illustrating the behavior control generation process of an intelligent CRM management method based on reinforcement learning proposed in this invention. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0023] refer to Figure 1-3 A reinforcement learning-based intelligent CRM management method includes the following steps: S1. Obtain customer relationship data, preprocess it, and generate a customer status sequence; S2. Construct a Markov decision process, defining the customer state as the environment state, the executable operations as the action space, and labeling each set of states and actions with behavioral feedback to form a set of triples. S3. Construct a reinforcement learning framework based on the policy gradient mechanism, perform policy sampling on the triplet set, construct training batches using the experience replay mechanism, and calculate the policy update direction based on the policy gradient method. S4. During the policy update process, channel-level convolution extraction and window-level residual aggregation operations are performed on the customer state sequence, and a context fusion vector is generated by combining the cross-layer attention mechanism. S5. Input the context fusion vector into the policy network to generate the action distribution, input each set of states and actions into the value network, estimate the state value and advantage function through the dual-path advantage reconstruction method, and complete the policy parameter update. S6. Select strategy actions based on action distribution, map strategy actions into structured CRM instruction sequences, and generate behavior control flow graphs by combining multi-task decoding structures to drive CRM execution flow. S7. Record feedback information during CRM execution and use the policy residual correction mechanism to update the policy network and value network parameters to continuously iterate the policy learning process.

[0024] In this embodiment, the customer relationship data is obtained through accessing the CRM data interface, including user identifier, interaction time, behavior type, channel source, and business tag. Data cleaning is completed using field mapping and null value filling. The time format is unified and inconsistent codes are mapped and converted. A multi-dimensional behavior matrix is ​​constructed using behavior type and timestamp. Boundary constraints are applied to abnormal behavior points and a sliding window is used for sequence slicing. Fixed-length data segments are generated according to a set step size. One-hot encoding is used to vectorize the behavior type. The time position code and business tag vector are concatenated to form a state representation sequence, ultimately forming a customer state sequence.

[0025] In this embodiment, the policy gradient method adopts a joint policy calculation method based on GAE and cutoff reward period. By constructing a weighted cumulative gradient function between short-term reward and long-term reward, the policy gradient direction is updated based on the current state distribution.

[0026] In this embodiment, the dual-path advantage reconstruction method simultaneously establishes two independent estimation paths based on the state value function and the action value function. The first path calculates the state baseline value through the state input, and the second path calculates the action expectation value through each set of states and actions. Finally, the advantage function is constructed through the difference between the two to evaluate the relative superiority of the strategy behavior in the current state and to serve as the basis for updating the value network.

[0027] In this embodiment, S2 specifically includes: S21. Obtain the preprocessed customer state sequence, and segment and label each customer's behavior segment according to the time sequence to ensure that each state has a clear temporal position and business semantics. S22. For each behavior segment, extract user behavior features, channel source and context information based on the user state content when the behavior is triggered, and combine them to construct a high-dimensional state vector as a representation of the environment state. S23. Clearly define the set of executable operations. The set of operations is set based on the actual CRM business process and covers all controllable behaviors including marketing push, human contact, service transfer, discount activation, interactive guidance, and content recommendation. A unique action code is assigned to each operation to form an action space. S24. For each combination of state and action, analyze customer behavior data, extract customer response status, response delay, click depth, retention period, and business conversion metrics, and calculate the corresponding behavioral feedback value, which is expressed by the following formula: ; in, Indicates time step Behavioral feedback values, For weight parameters, This is the response identifier (1 for a response, 0 for no response). In response to the delay time, For maximum tolerable delay, For click depth, Maximum click depth For retention period, For the maximum retention period, For conversion indicators (converted to 1, not converted to 0), all behavioral data comes from CRM execution logs and interaction event logs; S25. The high-dimensional state vector, the corresponding action code, and the behavioral feedback value are combined into a fixed-structure triple to form decision sequence data that conforms to the Markov assumption. The triple is constructed as follows: ; in, Indicates time step The triplet, For a high-dimensional state vector, by the function Based on current behavioral characteristics Channel characteristics With context history Constructed Action encoding, by function Configure the operation object in CRM Obtained by mapping.

[0028] In this embodiment, S3 specifically includes: S31. Construct a reinforcement learning framework and initialize a dual-network structure including a policy network and a value network. The policy network is used to generate action distributions, and the value network is used to evaluate the advantage score of state-action combinations. The policy network consists of a multi-layer fully connected structure, with each layer followed by an activation function. The last layer outputs a probability vector corresponding to the action dimension. The value network uses a parallel path structure to calculate the state value function and the action value function respectively. S32. Using the triplet set as initial experience data, select multiple state-action-feedback sequences from the triplet set according to a preset sampling interval to construct an experience replay buffer. The buffer is organized in the form of a double-ended queue and has a fixed capacity and a dynamic coverage mechanism based on the first-in-first-out rule to provide a dynamic sample source during the training process. S33. Extract multiple sets of state sequences in batches from the experience replay buffer, input them into the policy network in sequence, obtain the action probability distribution through forward propagation, determine the actual sampled action using a distributed random sampling mechanism, and form a state-action pair set with the state and the corresponding sampled action to calculate the behavioral tendency under the current policy. S34. Based on the set of state-action pairs, input the corresponding states into the value network, calculate the state value of the current state and the action value of the current state-action combination, calculate the advantage value based on the difference between the two, and perform a time discount accumulation operation on the advantage value in each trajectory to obtain a weighted cumulative advantage score. The cumulative advantage score is calculated by the following formula: ; in, Indicates time step The generalized advantage estimate, Represents relative to the current time step Time offset index, This represents the index of the maximum time step in the current trajectory, used to determine the termination position of advantage accumulation. As a reward discount factor, As the dominant attenuation factor, Indicates time step Instant rewards For the current policy, the state The state value, For the current policy, the state The state value; S35. Based on the cumulative advantage score and the action probability output by the policy network, calculate the policy gradient direction for each state-action pair. Use a backpropagation mechanism to propagate this gradient signal back to the parameters of each layer of the policy network, accumulating the gradient value of each layer. The policy gradient direction is defined by the following formula: ; in, Indicates policy network parameters The gradient of the overall expected return, This represents the set of trainable parameters in the policy network, including all weight matrices, bias terms, and normalization parameters. Indicates according to the current strategy State-action pairs We perform expected sampling to form the statistical basis for policy gradient estimation. Indicates the current state Below, the policy network takes action. The logarithmic probability value, This represents the gradient direction of the logarithmic probability function of the policy output in the parameter space; S36. Based on the gradient accumulation result, perform a parameter update operation on the policy network based on the learning rate adjustment factor, and optimize the objective function of the value network using the least squares error. Control the update rhythm of the two networks by sharing the number of training rounds to complete a complete policy iteration process. S37. Set the termination conditions for continuous strategy optimization. Determine whether to continue strategy training based on the cumulative reward improvement, strategy convergence magnitude, and sample diversity indicators. If the convergence criteria are met, retain the current parameter state as the deployment version of the CRM execution strategy.

[0029] In this embodiment, S6 specifically includes: S61. Receive the action distribution output by the policy network, use the maximum probability selection mechanism to determine the optimal policy action number in the current state, and perform a uniqueness check on the number to ensure that each policy action corresponds to the executable operation type in the CRM process. S62. Input the selected strategy action number into the instruction mapping unit, look up the mapping rule table, and construct a structured CRM instruction sequence containing execution target, operation parameters, trigger conditions and response type according to the CRM business behavior logic corresponding to the current action. The instruction sequence adopts a field-based hierarchical structure to clarify the dependency relationship between each operation field. S63. Input the structured CRM instruction sequence into the behavior decoding scheduling unit. Based on the field level of the instruction, start the parallel multi-task decoding path and construct execution nodes, condition nodes and jump nodes respectively to form a set of process decoding vectors with node identifiers and path indexes. S64. Based on the process decoding vector set, construct a behavior control flow graph. The control flow graph adopts a graph structure, where nodes represent independent operation behaviors in CRM execution, and edges represent scheduling relationships between behaviors. Generate an executable path set through a topology sorting algorithm. S65. Deploy the behavior control flow graph to the CRM task engine, schedule CRM actions node by node according to the current customer status and control flow path sequence, drive the CRM execution process, and complete a customer management task triggering process based on policy actions.

[0030] In this embodiment, the mapping rule table is a pre-built instruction mapping dictionary. It uses a key-value pair structure to store the correspondence between policy action numbers and CRM operation behaviors. Each mapping relationship includes an action number, the corresponding CRM business type, the execution target identifier, the parameter field template, and the operation priority index. All mapping relationships are indexed and classified according to the business process node number, and support calling different parameter template generation methods according to the context status label, which is used to convert policy action numbers into a structured CRM instruction sequence with business executability.

[0031] In this embodiment, the behavior decoding scheduling unit is a process building engine for structured CRM instruction sequences. It adopts a multi-path parallel scheduling architecture, maps the contents marked as execution fields, condition fields, and jump fields in the instruction sequence to execution nodes, judgment nodes, and branch nodes, respectively, and constructs a graph structure topology based on the order relationship between fields. It generates path convergence nodes in the graph using process merging identifiers, and dynamically adjusts the decoding order according to the dependencies between tasks. Finally, it outputs a behavior control flow graph data structure containing node numbers, entry indexes, and outgoing edge sets.

[0032] Example 1: To verify the feasibility of this invention in practice, it was applied to a real-world business scenario of customer precision marketing and lifecycle management within an enterprise-level CRM platform. The platform primarily targets registered online users, who interact daily through multiple channels including web and mobile devices. The platform's existing CRM system used a rule-based configuration approach, categorizing customers into different levels based on fixed metrics such as registration time, recent active behavior, and purchase history, and sending SMS messages, emails, or app push notifications to specific customer groups using preset templates. However, with rapid user growth and increasingly complex behavioral patterns, this approach struggled to adapt to rapidly changing customer states, leading to a continuous decline in conversion rates and a significant increase in customer churn risk.

[0033] In this scenario, firstly, complete behavioral sequence data of active users on the platform over the past 90 days was collected, including events such as clicks, browsing, adding to cart, placing orders, cancellations, complaints, and service requests, totaling over 20 million behavioral records. These data underwent preprocessing operations, including field standardization, outlier removal, and time series slicing, ultimately constructing a standardized set of customer state sequences. Next, based on the proposed technical solution, the customer behavior data was modeled as a Markov decision process. The state vector consists of click frequency, conversion path depth, service response interval, historical purchase type, and current channel source within the user behavior window. The action space encompasses six CRM response behaviors: pushing coupons, issuing points, transferring to human assistance, recommending products, and sending reminder notifications.

[0034] A reinforcement learning policy network is trained using a constructed state-action-feedback triplet. Policy updates are implemented based on a merit-based reconstruction mechanism, with parameters updated daily. During CRM platform deployment, the action distribution generated by the policy network is mapped in real time to structured CRM execution instructions. The system generates a control flow graph based on these instructions and executes the process, achieving automated outreach and service delivery.

[0035] In practical application, the changes in key indicators such as user response rate, conversion rate, and reach accuracy of the platform before and after the implementation of this invention were compared. Before implementation, the CRM solution based on manual rules had an average customer response rate of 12.6%, a conversion rate of 3.8%, and a customer retention rate of 27.4% within 30 days. After implementing this invention, the customer response rate increased to 20.3%, the conversion rate increased to 7.9%, and the 30-day customer retention rate increased to 41.1%. Among high-value customer groups, the conversion increase brought about by strategic behavior was particularly significant. Platform data showed that the "push + recommendation + guidance" combined process triggered after the behavior control flow diagram was generated had a conversion rate 4.2 percentage points higher than that of a single push.

[0036] Furthermore, from an operational cost perspective, the automatic generation and control flow graph scheduling mechanism for reinforcement learning strategies proposed in this invention significantly reduces the frequency of manual rule maintenance and process configuration. The system automatically completed the training and deployment of 423 combined strategy paths within one month, reducing the manpower required for strategy design by approximately 78% compared to the original manual configuration.

[0037] The table below shows a comparison of key performance indicators before and after the implementation of this invention on the CRM platform: Table 1. Comparison of the implementation effects of intelligent CRM methods based on reinforcement learning

[0038] As can be seen from the above embodiments and the data analysis in Table 1, the present invention not only effectively improves the intelligence level and behavioral decision-making efficiency of CRM strategies, but also brings significant positive changes in core indicators such as customer conversion, response and retention, fully demonstrating the practical application value and promotion prospects of the present invention in complex customer management tasks.

[0039] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A smart CRM management method based on reinforcement learning, characterized in that, Includes the following steps: S1. Obtain customer relationship data, preprocess it, and generate a customer status sequence; S2. Construct a Markov decision process, define the customer state as the environment state, define the executable operations as the action space, and label each set of states and actions with behavioral feedback to form a set of triples. S3. Construct a reinforcement learning framework based on the policy gradient mechanism, perform policy sampling on the triplet set, construct training batches using the experience replay mechanism, and calculate the policy update direction based on the policy gradient method. S4. During the policy update process, channel-level convolution extraction and window-level residual aggregation operations are performed on the customer state sequence, and a context fusion vector is generated by combining the cross-layer attention mechanism. S5. Input the context fusion vector into the policy network to generate the action distribution, input each set of states and actions into the value network, estimate the state value and advantage function through the dual-path advantage reconstruction method, and complete the policy parameter update. S6. Select strategy actions based on action distribution, map strategy actions into structured CRM instruction sequences, and generate behavior control flow graphs by combining multi-task decoding structures to drive CRM execution flow. S7. Record feedback information during CRM execution and use the policy residual correction mechanism to update the policy network and value network parameters to continuously iterate the policy learning process.

2. The intelligent CRM management method based on reinforcement learning according to claim 1, characterized in that, The customer relationship data includes user identifiers, interaction behaviors, timestamps, channel sources, and business tags. The preprocessing includes data cleaning, field alignment, anomaly removal, and behavior encoding operations. After preprocessing, the behavior sequence of each user is sliced ​​according to the time sequence to construct a state sequence with a time-series structure. A fixed-dimensional state representation set is generated based on the set window length and sliding step size, ultimately forming a customer state sequence.

3. The intelligent CRM management method based on reinforcement learning according to claim 1, characterized in that, The policy gradient method adopts a joint policy calculation approach based on GAE and cut-off reward period. It performs policy gradient direction updates based on the current state distribution by constructing a weighted cumulative gradient function between short-term rewards and long-term returns.

4. The intelligent CRM management method based on reinforcement learning according to claim 1, characterized in that, The dual-path advantage reconstruction method simultaneously establishes two independent estimation paths based on state value function and action value function. The first path calculates the state baseline value through state input, and the second path calculates the action expectation value through each set of state and action. Finally, the advantage function is constructed through the difference between the two to evaluate the relative superiority of the strategy behavior in the current state and to serve as the basis for value network updates.

5. The intelligent CRM management method based on reinforcement learning according to claim 1, characterized in that, S2 specifically includes: S21. Obtain the preprocessed customer state sequence, and segment and label each customer's behavior segment according to the time sequence to ensure that each state has a clear temporal position and business semantics. S22. For each behavior segment, extract user behavior features, channel source and context information based on the user state content when the behavior is triggered, and combine them to construct a high-dimensional state vector as a representation of the environment state. S23. Clearly define the set of executable operations. The set of operations is set based on the actual CRM business process and covers all controllable behaviors including marketing push, human contact, service transfer, discount activation, interactive guidance, and content recommendation. A unique action code is assigned to each operation to form an action space. S24. For each combination of state and action, analyze customer behavior data, extract whether the customer responds, response delay, click depth, retention period and business conversion indicators, calculate the corresponding behavior feedback value, and mark the feedback result corresponding to the current state and action accordingly. S25. Combine the high-dimensional state vector, the corresponding action code, and the behavioral feedback value into a fixed-structure triplet to form decision sequence data that conforms to the Markov assumption.

6. The intelligent CRM management method based on reinforcement learning according to claim 1, characterized in that, S3 specifically includes: S31. Construct a reinforcement learning framework and initialize a dual-network structure including a policy network and a value network. The policy network is used to generate action distributions, and the value network is used to evaluate the advantage score of state-action combinations. The policy network consists of a multi-layer fully connected structure, with each layer followed by an activation function. The last layer outputs a probability vector corresponding to the action dimension. The value network uses a parallel path structure to calculate the state value function and the action value function respectively. S32. Using the triplet set as initial experience data, select multiple state-action-feedback sequences from the triplet set according to a preset sampling interval to construct an experience replay buffer. The buffer is organized in the form of a double-ended queue and has a fixed capacity and a dynamic coverage mechanism based on the first-in-first-out rule to provide a dynamic sample source during the training process. S33. Extract multiple sets of state sequences in batches from the experience replay buffer, input them into the policy network in sequence, obtain the action probability distribution through forward propagation, determine the actual sampled action using a distributed random sampling mechanism, and form a state-action pair set with the state and the corresponding sampled action to calculate the behavioral tendency under the current policy. S34. Based on the set of state-action pairs, input the corresponding state into the value network, calculate the state value of the current state and the action value of the current state-action combination, calculate the advantage value based on the difference between the two, and perform a time discount accumulation operation on the advantage value in each trajectory to obtain a weighted cumulative advantage score. S35. Based on the cumulative advantage score and the action probability output by the policy network, calculate the policy gradient direction for each state-action pair, and use the backpropagation mechanism to backpropagate the gradient signal to the parameters of each layer of the policy network, and accumulate the gradient value of each layer. S36. Based on the gradient accumulation result, perform a parameter update operation on the policy network based on the learning rate adjustment factor, and optimize the objective function of the value network using the least squares error. Control the update rhythm of the two networks by sharing the number of training rounds to complete a complete policy iteration process. S37. Set the termination conditions for continuous strategy optimization. Determine whether to continue strategy training based on the cumulative reward improvement, strategy convergence magnitude, and sample diversity indicators. If the convergence criteria are met, retain the current parameter state as the deployment version of the CRM execution strategy.

7. The intelligent CRM management method based on reinforcement learning according to claim 1, characterized in that, S6 specifically includes: S61. Receive the action distribution output by the policy network, use the maximum probability selection mechanism to determine the optimal policy action number in the current state, and perform a uniqueness check on the number to ensure that each policy action corresponds to the executable operation type in the CRM process. S62. Input the selected strategy action number into the instruction mapping unit, look up the mapping rule table, and construct a structured CRM instruction sequence containing execution target, operation parameters, trigger conditions and response type according to the CRM business behavior logic corresponding to the current action. The instruction sequence adopts a field-based hierarchical structure to clarify the dependency relationship between each operation field. S63. Input the structured CRM instruction sequence into the behavior decoding scheduling unit. Based on the field level of the instruction, start the parallel multi-task decoding path and construct execution nodes, condition nodes and jump nodes respectively to form a set of process decoding vectors with node identifiers and path indexes. S64. Based on the process decoding vector set, construct a behavior control flow graph. The control flow graph adopts a graph structure, where nodes represent independent operation behaviors in CRM execution, and edges represent scheduling relationships between behaviors. Generate an executable path set through a topology sorting algorithm. S65. Deploy the behavior control flow graph to the CRM task engine, schedule CRM actions node by node according to the current customer status and control flow path sequence, drive the CRM execution process, and complete a customer management task triggering process based on policy actions.

8. The intelligent CRM management method based on reinforcement learning according to claim 7, characterized in that, The mapping rule table is a pre-built instruction mapping dictionary. It uses a key-value pair structure to store the correspondence between policy action numbers and CRM operation behaviors. Each mapping relationship includes an action number, the corresponding CRM business type, the execution target identifier, the parameter field template, and the operation priority index. All mapping relationships are indexed and classified according to the business process node number, and support calling different parameter template generation methods based on the context status label. This is used to convert policy action numbers into a structured CRM instruction sequence with business executability.

9. The intelligent CRM management method based on reinforcement learning according to claim 7, characterized in that, The behavior decoding scheduling unit is a process building engine for structured CRM instruction sequences. It adopts a multi-path parallel scheduling architecture, maps the contents marked as execution fields, condition fields and jump fields in the instruction sequence to execution nodes, judgment nodes and branch nodes respectively, and constructs a graph structure topology according to the order relationship between fields. It generates path convergence nodes in the graph using process merging identifiers, and dynamically adjusts the decoding order according to the dependency relationship between tasks. Finally, it outputs a behavior control flow graph data structure containing node numbers, entry indexes and outgoing edge sets.

Images