User portrait dynamic updating method based on reinforcement learning

Abstract

The application discloses a user portrait dynamic updating method based on reinforcement learning, comprising the following steps: receiving original behavior data reported by a client, processing the original behavior data according to a predefined rule to obtain a portrait updating trigger event; defining a user state space and a user action space, and setting a reward function; training a deep reading event, a quick jump-out event, an exposure non-click event and a dialogue preference event according to the user state space, the user action space and the reward function, and extracting a user state vector at a current time from the deep reading event, the quick jump-out event, the exposure non-click event and the dialogue preference event; and the training process comprises an offline training stage, an online inference stage and a continuous learning stage. The application can adapt to different users and has an experience revisiting mechanism.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a method for dynamically updating user profiles based on reinforcement learning. Background Technology

[0002] Existing technologies include collaborative filtering recommendation based on user historical behavior and conventional user profile construction solutions. These solutions are widely used on mainstream platforms such as news reading, short videos, and content distribution. The overall technology revolves around data collection, offline statistics, profile generation, and business applications.

[0003] Existing technology (202511113544.3) discloses a method for real-time personalized content push based on user profiles. This method includes collecting user behavior datasets and historical interaction records, extracting real-time behavioral features and generating dynamic behavioral sequences; identifying behavioral delay features and converting them into interest feedback lag; determining dynamic influence weights by combining the interaction between user static attribute features and real-time behavioral features; predicting user interest tendencies in real time using lag and dynamic influence weights, generating fluctuation deviation values; and triggering dynamic updates to the user profile and adjusting the push strategy when the deviation value exceeds a preset threshold. In this scheme, the profile update strategy is a "rule-driven threshold trigger" rather than a "data-driven self-learning" approach, which cannot adapt to different users and lacks an experience review mechanism.

[0004] Therefore, the inability of existing technologies to adapt to different users and the lack of an experience follow-up mechanism are technical problems that urgently need to be solved in this field. Summary of the Invention

[0005] In view of this, the present invention provides a method for dynamically updating user profiles based on reinforcement learning, in order to solve the problems of existing technologies being unable to adapt to different users and lacking an experience follow-up mechanism.

[0006] This application provides a method for dynamically updating user profiles based on reinforcement learning, including the following steps:

[0007] The system receives raw behavioral data reported by the client, which includes at least: reading progress data, reading duration data, dialogue content data, and dialogue sentiment data. It processes the reading progress data, reading duration data, dialogue content data, and dialogue sentiment data according to predefined rules to obtain profile update trigger events. The predefined rules include deep reading judgment rules, quick exit judgment rules, exposure without click judgment rules, and dialogue preference judgment rules. The profile update trigger events include: deep reading events, quick exit events, exposure without click events, and dialogue preference events.

[0008] The deep reading determination rule is defined as follows: if the reading progress data reaches a first preset threshold and the reading duration data reaches a second preset threshold, then the deep reading event is generated; the quick exit determination rule is defined as follows: if the reading duration data does not reach a third preset threshold and the reading progress data does not reach a fourth preset threshold, then the quick exit event is generated, wherein the first preset threshold is greater than the fourth preset threshold; the exposure without click determination rule is defined as follows: if the target content is exposed to the visible area of ​​the user terminal's display interface and there is no corresponding click behavior within a fifth preset threshold, then the exposure without click event is generated, wherein the fifth preset threshold is greater than the second preset threshold; the dialogue preference determination rule is defined as follows: parsing multi-turn dialogue semantics, identifying user preferences, and generating the dialogue preference event;

[0009] Define a user state space and a user action space, and set a reward function. The user state space is defined as: embedding the current user profile vector, the feature vector of the short-term behavior sequence, and the recent dialogue sentiment vector. The short-term behavior sequence is an event sequence composed of deep reading events, quick exit events, exposure without click events, and dialogue preference events within a preset time period.

[0010] The user action space is defined as follows: the action corresponding to each moment is the incremental adjustment value for the weight of the i-th label; the continuous adjustment amount is discretized into a finite number of values, and the user action space is limited to a discrete set: ,

[0011] The reward function is set up to calculate the reward based on the user behavior generated after the user profile is updated.

[0012] The training process utilizes the user state space, user action space, and reward function to train on deep reading events, quick exit events, exposure-without-click events, and dialogue preference events. It extracts the current user state vector from these events. The training process includes an offline training phase, an online inference phase, and a continuous learning phase.

[0013] During the offline training phase, based on historically accumulated deep reading events, quick exit events, exposure-without-click events, and dialogue preference events, the state transition process is simulated and the corresponding instant rewards are calculated to pre-train the Q network. The experience samples generated during pre-training are stored in the experience replay buffer.

[0014] During the online reasoning phase, the user state vector extracted from deep reading events, quick exit events, exposure without click events, and dialogue preference events at the current moment is received in real time. The Q network outputs the optimal action and performs profile updates. At the same time, the real-time experience samples generated by online interaction are stored in the experience replay buffer.

[0015] During the continuous learning phase, while the online inference phase is running, mixed data, including historical and real-time experience samples, is sampled from the experience replay buffer at preset intervals to fine-tune the Q-network. The fine-tuned parameters are then synchronized in real-time to the Q-network used in the online inference phase.

[0016] The instant reward is calculated based on changes in deep reading events, quick exit events, exposure without clicking events, and conversation preference events. The action includes adding, deleting, or modifying at least one interest tag in the reading user profile.

[0017] During the training of the Q-network, the loss function is:

[0018] ,

[0019] In the formula, Represents the loss function. Represents the mathematical expectation. Represents the user state space, Indicates the state at the next moment. Indicates the action to be taken at the next moment. Indicates an immediate reward. Indicates the discount factor. Indicates the target network parameters. Represents the user action space. This indicates a maximize operation. Indicates the target Q value. This represents the current Q value.

[0020] Optionally, the reward function is expressed as: ,

[0021] In the formula, Represents the reward function, This indicates the weight of each corresponding indicator. This indicates the change in click-through rate of recommended content after the update. This indicates the percentage change in reading progress data greater than 0.9; The proportional change indicating a positive emotional tone in the dialogue; This represents the cumulative number of updates.

[0022] Optionally, .

[0023] Optionally, the expression for the target Q value is:

[0024] ,

[0025] In the formula, This represents the target Q value.

[0026] Compared with existing technologies, the reinforcement learning-based user profile dynamic update method provided by this invention achieves at least the following beneficial effects:

[0027] This invention achieves adaptability to different users and includes an experience-based feedback mechanism. The system requires no manual rule intervention, automatically learning the optimal profile update strategy through a reinforcement learning trial-and-error mechanism. Through continuous learning, it quickly captures user interest shifts, preventing profile lag. The interest tags in the user profile can be adjusted in real-time according to changes in the user's actual reading behavior, rather than being statically fixed. By uniformly converting predefined rules into profile update trigger events, it overcomes the limitations of traditional, singular behavior collection methods.

[0028] Of course, any product implementing the present invention does not need to achieve all of the above additional technical effects while solving the background technical problem. It is sufficient for the product to solve the background technical problem first. The additional technical effects are effects that are beyond the understanding of those skilled in the art when the specific structure of the present invention is combined with a specific environment.

[0029] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0031] Figure 1 This is a schematic diagram of the structure of the reinforcement learning-based user profile dynamic update method provided by the present invention. Detailed Implementation

[0032] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0033] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0034] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0035] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0036] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0037] See Figure 1 As shown, Figure 1 This is a schematic diagram of the user profile dynamic update method based on reinforcement learning provided by the present invention. This embodiment provides a user profile dynamic update method based on reinforcement learning, including the following steps:

[0038] Step S1: Receive raw behavioral data reported by the client; the raw behavioral data includes at least: reading progress data, reading duration data, dialogue content data, and dialogue sentiment data; process the reading progress data, reading duration data, dialogue content data, and dialogue sentiment data according to predefined rules to obtain profile update trigger events; the predefined rules include deep reading judgment rules, quick exit judgment rules, exposure without click judgment rules, and dialogue preference judgment rules; the profile update trigger events include: deep reading events, quick exit events, exposure without click events, and dialogue preference events; among them,

[0039] The deep reading determination rule is defined as follows: a deep reading event is generated when the reading progress data reaches a first preset threshold and the reading duration data reaches a second preset threshold; a quick exit event is generated when the reading duration data does not reach a third preset threshold and the reading progress data does not reach a fourth preset threshold, wherein the first preset threshold is greater than the fourth preset threshold; the exposure-without-click determination rule is defined as follows: an exposure-without-click event is generated when the target content is exposed to the visible area of ​​the user terminal's display interface and there is no corresponding click behavior within a fifth preset threshold, wherein the fifth preset threshold is greater than the second preset threshold; the dialogue preference determination rule is defined as follows: parsing multi-turn dialogue semantics, identifying user preferences, and generating the dialogue preference event;

[0040] Specifically, in step S1, the server receives raw behavioral data reported by the client in real time.

[0041] When a user engages in any interactive activity on the client, such as reading, clicking, chatting, or swiping, the client collects all relevant fields in real time. In addition to regular interaction data, it simultaneously captures content attribute data, reading status data (such as reading progress data, reading duration data, reading completion rate, and bounce depth data), and chat content data. For example, all relevant fields include the user's unique identifier, operation time data, article title data, content tag data, article length data, content style data, number of images data, reading duration data, reading progress data, chat content data, and chat sentiment data.

[0042] Subsequently, the following standardized operations are performed according to the predefined data dictionary:

[0043] Field mapping: Mapping raw field names to standard internal field names (e.g., article_title) title);

[0044] Type conversion: Convert reading progress data to single-precision floating-point type float, with a valid value range of 0.0~1.0; convert reading time data to integer type int, with the unit of measurement being seconds; convert image count data to integer type int.

[0045] Unit normalization: Timestamps are uniformly converted to Unix millisecond-level integers;

[0046] Unit normalization: All time fields within the system uniformly use Unix timestamps with millisecond precision;

[0047] Standardization of enumeration values: Various content styles correspond to fixed enumeration values ​​(such as rational analysis and technical analysis), and the dialogue sentiment tendency field is uniformly mapped to three fixed labels: positive (positive behavior) and negative (negative behavior).

[0048] Missing value filling: Fill missing but required fields with default values ​​(such as empty strings or 0);

[0049] Format encapsulation: The standardized fields are assembled into a structured standard JSON format, and each behavioral data is generated into an independent log line.

[0050] This ultimately generates standardized behavior logs. For example:

[0051]

[0052] It should be noted that this standardized behavior log is the raw behavior data received by the server in real time.

[0053] The server filters and transforms each piece of reading progress data, reading duration data, dialogue content data, and dialogue sentiment data according to the following steps: Specifically, the server reads the original behavior logs (such as reading progress data, reading duration data, dialogue content data, and dialogue sentiment data) one by one and checks whether they match the predefined rules.

[0054] Filter out invalid behavioral data, such as synchronously removing data with missing fields, abnormal timestamps, and conflicting behavioral logic (e.g., reading progress data is greater than 0 but page dwell time is 0), to reduce the data processing load of the subsequent large language model.

[0055] The predefined rules include rules for determining deep reading, quick exit, exposure without click, and dialogue preference. If any one of these rules is met, the data is considered valid behavioral data. If none of these rules is met, the original behavioral log is discarded and the user profile is not updated.

[0056] The deep reading determination rule is defined as follows: when the reading progress data reaches a first preset threshold and the reading time data reaches a second preset threshold, a deep reading event is generated; the first preset threshold can be 90% and the second preset threshold can be 10 seconds.

[0057] The quick exit judgment rule is defined as follows: if the reading time data does not reach the third preset threshold and the reading progress data does not reach the fourth preset threshold, a quick exit event is generated; the third preset threshold can be 5 seconds and the fourth preset threshold can be 10%.

[0058] The rule for determining if content is exposed but not clicked is defined as follows: if no corresponding click behavior occurs within a fifth preset threshold after the target content is exposed to the visible area of ​​the user terminal's display interface, an exposure-not-clicked event is generated; the fifth preset threshold can be 5 minutes.

[0059] The rule for determining if content was not clicked is implemented by writing user content exposure records to a Redis cache.

[0060] Cache key naming rules: For example, unique identifier for the user, unique identifier for the content;

[0061] The cached value stores the timestamp of this exposure.

[0062] Cache expiration time: The cache expires automatically after 300 seconds, and the exposure record is automatically cleared after expiration; if a click log is received, the cache key is deleted; if the cache key expires naturally, an exposure unclicked event is triggered.

[0063] The dialogue preference determination rule is defined as follows: parsing the semantics of multi-turn dialogues, identifying user preferences and generating dialogue preference events. The aforementioned user preferences can be the user's explicit likes or dislikes.

[0064] The specific handling method for this dialogue preference event:

[0065] Use lightweight pre-trained models (such as DistilBERT fine-tuned intent classifiers) or rule-based keyword and sentiment dictionaries;

[0066] Input the current dialogue round and the context data of the last 3 rounds, and output the topic (such as the collaboration between Retrieval Enhancement Generation (RAG) and the Agent) and the sentiment polarity (such as positive for liking and negative for rejecting).

[0067] A conversational preference event is generated only when the confidence level of the sentiment polarity is greater than 0.7 and contains explicit preference keywords (such as "like" or "don't recommend"). The event details include the topic, sentiment polarity, and original query.

[0068] It should be noted that the DistilBERT mentioned above is a lightweight distillation pre-trained model of BERT proposed in 2019. Downstream fine-tuning for intent classification is a common standard baseline solution in the fields of dialogue systems and intelligent customer service natural language processing, and belongs to existing technology.

[0069] The original paper on the DistilBERT base model is published in: Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter[J]. arXiv preprint, arXiv:1910.01108, October 2, 2019.

[0070] Transform into profile update trigger events: For each piece of valid behavioral data, extract key fields based on deep reading judgment rules, quick exit judgment rules, exposure without click judgment rules, and dialogue preference judgment rules, and output them in a fixed event format. The fixed event format includes:

[0071] Unique event identifier: Globally unique string UUID, with no auto-increment conflicts; database auto-incrementing numeric primary key;

[0072] Event types: corresponding to in-depth reading judgment rules, quick exit judgment rules, exposure without click judgment rules, and dialogue preference judgment rules;

[0073] Unique user identifier, event timestamp;

[0074] Event details: Includes core data for events that trigger profile updates, such as in-depth reading events which include user topic, tags, reading progress data, and reading duration data; and exposure without clicks events which include tags and exposure counts.

[0075] Rule Identifier: Records the matching rule identifier, which is the unique number of the predefined rules that this event matched, including the deep reading judgment rule, the quick exit judgment rule, the exposure without click judgment rule, and the dialogue preference judgment rule.

[0076] It should be noted that for the rule of determining if an exposure is not clicked, an exposure cache is used with the user's unique identifier and the content's unique identifier as keys. The exposure timestamp is recorded. When click behavior data is received, the corresponding cache is cleared. The cache that has not been cleared after a timeout (e.g., more than 5 minutes) is periodically scanned, and an exposure not clicked event is generated for that exposure. Deep reading events, quick exit events, and dialogue preference events do not involve cross-log aggregation; a single valid behavior data point is used for direct determination.

[0077] The deep reading event, quick exit event, exposure unclicked event, and dialogue preference event output in step S1 will be directly input into step S3 to reduce invalid calls.

[0078] Step S2: Define the user state space and user action space, and set the reward function. The user state space is defined as: embedding the current user profile vector, the feature vector of the short-term behavior sequence, and the recent dialogue sentiment vector. The short-term behavior sequence is an event sequence composed of deep reading events, quick exit events, exposure without click events, and dialogue preference events within a preset time period.

[0079] The user action space is defined as follows: the action corresponding to each moment is the incremental adjustment value for the weight of the i-th label; the continuous adjustment amount is discretized into a finite number of values, and the user action space is limited to a discrete set: ,

[0080] The reward function is set up to calculate the reward based on the user behavior generated after the user profile is updated.

[0081] Specifically, the user's current profile vector (d is the number of labels, and each component is normalized to) ); Indicates a point in time User profile vectors. This indicates that the user profile vector is dynamically updated and will change as new user behavior data (clicks, browsing, purchases, etc.) is collected.

[0082] This means that the vector is a A 1-dimensional real vector, that is, it has There are 10 components, each of which is a real number.

[0083] Here It is the predefined total number of tags.

[0084] (3) The number of tags refers to the characteristic dimensions used to characterize a user, such as interest tags: sports, technology, food, fashion;

[0085] Attribute tags: Age group (18 to 24, 25 to 35), gender preference;

[0086] Behavioral tags: highly active, nighttime user, price-sensitive;

[0087] The number of tags, d, is determined by the business scenario (it could be several hundred or even tens of thousands).

[0088] (4) Normalize each component to The values ​​for each dimension are compressed to between 0 and 1, including 0 and 1.

[0089] 0 indicates that the user has no inclination or match for this tag. 1 indicates that the user is a perfect match or has a very strong inclination for this tag.

[0090] The purpose of normalization is to eliminate the influence of different units of measurement between different tags (such as click count data). (Duration of stay data).

[0091] It facilitates unified calculations (such as cosine similarity, dot product, etc.).

[0092] This is often achieved through softmax, sigmoid, or min-max scaling.

[0093] The short-term behavioral sequence is an event sequence composed of deep reading events, quick bounce events, exposure without click events, and dialogue preference events within 24 hours; the feature vectors of deep reading events, quick bounce events, exposure without click events, and dialogue preference events within the past 24 hours are also included. (m represents the behavioral type encoding dimension, such as the count / intensity of profile update trigger events like deep reading events, quick bounce events, and exposure without click events);

[0094] Recent conversation sentiment embedding vector (k is the emotion embedding dimension, such as a 32-dimensional vector). That is... .

[0095] User Action Space definition:

[0096] Each movement Indicates a specific tag The weight adjustment amount is discretized into a finite number of options: If multiple labels need to be adjusted, then multiple actions will be output sequentially.

[0097] The reward function is expressed as:

[0098] ,

[0099] In the formula, Represents the reward function, This indicates the weight of each corresponding indicator. This indicates the change in click-through rate of recommended content after the update. This indicates the percentage change in reading progress data greater than 0.9; The proportional change indicating a positive emotional tone in the dialogue; This represents the cumulative number of updates.

[0100] The step count penalty is designed to prevent excessively frequent updates. Each update increments the step count by 1, and the cumulative penalty increases linearly.

[0101] Weight For hyperparameters, optionally, For example, take It is 0.4. It is 0.3. It is 0.2. It is 0.1.

[0102] Step S3: Train the deep reading event, quick exit event, exposure without click event, and dialogue preference event based on the user state space, user action space, and reward function. Extract the current user state vector from the deep reading event, quick exit event, exposure without click event, and dialogue preference event. The training process includes an offline training stage, an online inference stage, and a continuous learning stage.

[0103] During the offline training phase, based on historically accumulated deep reading events, quick exit events, exposure-without-click events, and dialogue preference events, the state transition process is simulated and the corresponding instant rewards are calculated to pre-train the Q network. The experience samples generated during pre-training are stored in the experience replay buffer.

[0104] Specifically, this offline training phase already possesses basic "profile update decision-making capabilities" before entering the online environment, thus avoiding the harm to user experience caused by random exploration during the cold start phase.

[0105] Historical logs are inexpensive data available offline, making full use of them to train the initial model.

[0106] During the online reasoning phase, the user state vector extracted from deep reading events, quick exit events, exposure without click events, and dialogue preference events at the current moment is received in real time. The Q network outputs the optimal action and performs profile updates. At the same time, the real-time experience samples generated by online interaction are stored in the experience replay buffer.

[0107] Specifically, the operational steps in the online inference phase are as follows: extracting the user state vector from the deep reading event, the quick exit event, the exposure without clicking event, and the dialogue preference event at the current moment. Q-network forward computation For all actions; select the next action; perform a more accurate portrait and observe the state at the next moment. and ,Will Store in the experience revisit buffer.

[0108] During the inference phase, the loss function is not calculated and the network parameters are not updated; only forward inference is performed. This reflects the separation of "user behavior strategy" and "target strategy" in reinforcement learning algorithms. The Q-network plays the role of "user behavior strategy" at this time, only responsible for decision-making and not for learning (learning is completed asynchronously during the continuous learning phase).

[0109] The inference phase operates in real-time, with millisecond-level inference capabilities, meeting the low-latency requirements of online business. Online data is continuously added to the experience pool, preparing "fresh samples" for subsequent continuous learning.

[0110] During the continuous learning phase, while the online inference phase is running, mixed data, including historical and real-time experience samples, is sampled from the experience replay buffer at preset intervals to fine-tune the Q-network. The fine-tuned parameters are then synchronized in real-time to the Q-network used in the online inference phase.

[0111] The instant reward is calculated based on changes in deep reading events, quick exit events, exposure without click events, and dialogue preference events. The action includes adding, deleting, or modifying at least one interest tag in the reading user profile. During the training of the Q-network, the loss function is:

[0112] ,

[0113] In the formula, Represents the loss function. Represents the mathematical expectation. Represents the user state space, Indicates the state at the next moment. Indicates the action to be taken at the next moment. Indicates an immediate reward. Indicates the discount factor. Indicates the target network parameters. Represents the user action space. This indicates a maximize operation. Indicates the target Q value. This represents the current Q value.

[0114] The steps for the continuous learning phase are as follows: A fine-tuning task is triggered every morning at midnight, sampling a batch from the buffer, which mixes offline historical samples (40%) and online samples from the most recent 24 hours (60%); the same loss function is then calculated. Backpropagation; the fine-tuned 𝜃 is synchronized to the Q network used in the online inference stage; the target network is synchronously soft-updated.

[0115] It should be noted that the continuous learning phase and the offline training phase use the exact same mathematical framework for reinforcement learning (the same loss function, the same update formula). The only difference is:

[0116] The data source changes from "pure historical logs" to "a mixture of historical and real-time data", and the learning rate φ can be reduced (e.g., from 0.001 to 0.0001) to avoid new data drastically changing the learned stable strategy (preventing catastrophic forgetting).

[0117] Optionally, the expression for the target Q value is:

[0118] ,

[0119] In the formula, This represents the target Q value.

[0120] Compared with existing technologies, the reinforcement learning-based user profile dynamic update method provided in this embodiment achieves at least the following beneficial effects:

[0121] This embodiment can adapt to different users and has an experience feedback mechanism; the system does not require manual rule intervention, and automatically learns the optimal profile update strategy through a reinforcement learning trial-and-error mechanism; through a continuous learning mechanism, it quickly captures user interest drift and avoids profile lag; it allows the interest tags in the user profile to be adjusted in real time according to changes in the user's actual reading behavior, rather than being statically fixed. By uniformly converting predefined rules into profile update trigger events, it breaks through the limitations of traditional single-faceted behavior collection.

[0122] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0123] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for dynamically updating user profiles based on reinforcement learning, characterized in that, Includes the following steps: The system receives raw behavioral data reported by the client, which includes at least: reading progress data, reading duration data, dialogue content data, and dialogue sentiment data. It processes the reading progress data, reading duration data, dialogue content data, and dialogue sentiment data according to predefined rules to obtain profile update trigger events. The predefined rules include deep reading judgment rules, quick exit judgment rules, exposure without click judgment rules, and dialogue preference judgment rules. The profile update trigger events include: deep reading events, quick exit events, exposure without click events, and dialogue preference events. The deep reading determination rule is defined as follows: if the reading progress data reaches a first preset threshold and the reading duration data reaches a second preset threshold, then the deep reading event is generated; the quick exit determination rule is defined as follows: if the reading duration data does not reach a third preset threshold and the reading progress data does not reach a fourth preset threshold, then the quick exit event is generated, wherein the first preset threshold is greater than the fourth preset threshold; the exposure without click determination rule is defined as follows: if the target content is exposed to the visible area of ​​the user terminal's display interface and there is no corresponding click behavior within a fifth preset threshold, then the exposure without click event is generated, wherein the fifth preset threshold is greater than the second preset threshold; the dialogue preference determination rule is defined as follows: parsing multi-turn dialogue semantics, identifying user preferences, and generating the dialogue preference event; Define a user state space and a user action space, and set a reward function. The user state space is defined as: embedding the current user profile vector, the feature vector of the short-term behavior sequence, and the recent dialogue sentiment vector. The short-term behavior sequence is an event sequence composed of deep reading events, quick exit events, exposure without click events, and dialogue preference events within a preset time period. The user action space is defined as follows: the action corresponding to each moment is the incremental adjustment value for the weight of the i-th label; the continuous adjustment amount is discretized into a finite number of values, and the user action space is limited to a discrete set: , The reward function is set up to calculate the reward based on the user behavior generated after the user profile is updated. The training process utilizes the user state space, user action space, and reward function to train on deep reading events, quick exit events, exposure-without-click events, and dialogue preference events. It extracts the current user state vector from these events. The training process includes an offline training phase, an online inference phase, and a continuous learning phase. During the offline training phase, based on historically accumulated deep reading events, quick exit events, exposure-without-click events, and dialogue preference events, the state transition process is simulated and the corresponding instant rewards are calculated to pre-train the Q network. The experience samples generated during pre-training are stored in the experience replay buffer. During the online reasoning phase, the user state vector extracted from deep reading events, quick exit events, exposure without click events, and dialogue preference events at the current moment is received in real time. The Q network outputs the optimal action and performs profile updates. At the same time, the real-time experience samples generated by online interaction are stored in the experience replay buffer. During the continuous learning phase, while the online inference phase is running, mixed data, including historical and real-time experience samples, is sampled from the experience replay buffer at preset intervals to fine-tune the Q-network. The fine-tuned parameters are then synchronized in real-time to the Q-network used in the online inference phase. The instant reward is calculated based on changes in deep reading events, quick exit events, exposure without clicking events, and conversation preference events. The action includes adding, deleting, or modifying at least one interest tag in the reading user profile. During the training of the Q-network, the loss function is: ， In the formula, Represents the loss function. Represents the mathematical expectation. Represents the user state space, Indicates the state at the next moment. Indicates the action to be taken at the next moment. Indicates an immediate reward. Indicates the discount factor. Indicates the target network parameters. Represents the user action space. This indicates a maximize operation. Indicates the target Q value. This represents the current Q value.

2. The method for dynamically updating user profiles based on reinforcement learning according to claim 1, characterized in that, The reward function is expressed as follows: , In the formula, Represents the reward function, This indicates the weight of each corresponding indicator. This indicates the change in click-through rate of recommended content after the update. This indicates the percentage change in reading progress data greater than 0.9; The proportional change indicating a positive emotional tone in the dialogue; This represents the cumulative number of updates.

3. The method for dynamically updating user profiles based on reinforcement learning according to claim 2, characterized in that, 。 4. The method for dynamically updating user profiles based on reinforcement learning according to claim 1, characterized in that, The expression for the target Q value is: ， In the formula, This represents the target Q value.

Images