Precise information delivery system and method based on multi-modal perception and reinforcement learning

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an accurate information delivery system and method based on multi-modal perception and reinforcement learning, and the method comprises the steps: when an evaluation window ends, the system calls a content category identifier in a user behavior sequence to calculate an information entropy value, and carries out the statistics of a negative feedback rate; the system generates a long-term reward value which is in negative correlation with the information entropy value and the negative feedback rate, and the reinforcement learning model is updated in a backtracking manner in combination with the instant reward, so that the technical problem of reward index dislocation in reinforcement learning is solved, the optimization target of the model is converted into a low-entropy concentration state of cultivating a user from an induced click state, and the user experience is improved. Therefore, the degradation of the model strategy is avoided.

Description

Technical Field

[0001] This invention relates to a system and method for precise information delivery based on multimodal perception and reinforcement learning, belonging to the field of artificial intelligence technology. Background Technology

[0002] With the development of mobile internet technology, information delivery systems are increasingly incorporating machine learning techniques, particularly reinforcement learning, to replace traditional static rules or shallow statistical models and achieve dynamic and adaptive personalized content delivery. To further improve the accuracy of perception and decision-making, existing technologies are beginning to integrate multimodal data, user behavior, and environmental information. This approach of integrating multimodal perception and reinforcement learning has also been explored in other complex decision-making fields. Chinese invention patent CN118514081A discloses a robot satellite assembly method and system based on multimodal reinforcement learning. This solution attempts to integrate visual perception, QR code target information, and force and torque information to guide the robotic arm to complete assembly tasks in the physical world. However, the physical manipulation and environmental perception problems solved by such technologies are completely different from the fundamental challenges in the field of information delivery. In information delivery scenarios, even with the introduction of multimodal data, the core technical challenge remains how to design an effective reward mechanism to cope with human complexity and subjective intent.

[0003] However, these systems that attempt to incorporate reinforcement learning generally adopt a design approach stemming from engineering practice. To facilitate quantification and rapid iteration, these systems tend to treat the user's immediate click feedback as the sole or highest-weighted reward signal for the reinforcement learning agent. This design causes the machine learning model's training objective to deviate from the actual application objective of improving long-term user retention. The model's optimization direction shifts to maximizing short-term click probabilities. The direct consequence of this deviation is that the model will autonomously learn and converge to an inducing strategy, pushing highly stimulating or misleading information in exchange for short-term clicks, even if this behavior interferes with the user's browsing focus, leading to frequent topic switching or quick content closure. This convergence direction of the model's strategy ultimately results in a decline in long-term user retention.

[0004] Therefore, the technical problem to be solved by this invention is how to design a composite reward mechanism that can objectively measure the degree of drift in a user's browsing state based on changes in the content category of the user's behavior sequence within an evaluation window, and combine the measurement result with real-time feedback to generate retrospective rewards, thereby guiding the model strategy toward convergence towards the direction of long-term optimal value. Summary of the Invention

[0005] This invention provides a system and method for precise information delivery based on multimodal perception and reinforcement learning. Its main purpose is to solve the problem that existing reinforcement learning systems rely too much on instant click feedback as a reward signal, which causes the model training objective to deviate from the long-term value of the user and leads to policy degradation.

[0006] To achieve the above objectives, this invention provides an information delivery system based on multimodal perception and reinforcement learning, the system comprising: The multimodal data fusion and feature extraction module is used to receive multi-source heterogeneous data and generate multimodal context feature vectors; The dynamic user intent modeling and profile update module is used to process user interaction behavior sequences, generate user state vectors, and record user behavior sequences in the user interaction behavior sequences. A reinforcement learning-based precise matching and decision engine is used to concatenate multimodal context feature vectors with user state vectors as state inputs to the reinforcement learning model, and select and execute delivery actions based on the state inputs. The online learning and feedback evolution closed-loop module is used to respond to real-time user feedback on the delivery action, determine the first reward value, and initiate a retrospective reward calculation at the end of the evaluation window with a preset time length. This calculation retrieves the content category identifiers from the user behavior sequence recorded by the dynamic user intent modeling and profile update module, calculates the information entropy value based on the content category identifiers, and counts the negative feedback rate within the evaluation window to generate a second reward value. The second reward value is a function value that is negatively correlated with the information entropy value and the negative feedback rate. The online learning and feedback evolution closed-loop module is also used to accumulate the first reward value and the second reward value to obtain the final reward value, and retrospectively distribute the final reward value to all state input and delivery action pairs within the evaluation window to update the parameters of the reinforcement learning model.

[0007] Preferably, the user behavior sequence recorded by the dynamic user intent modeling and profile update module contains a series of content items; when calculating the information entropy value, the online learning and feedback evolution closed-loop module specifically extracts the content category identifiers corresponding to all content items within the evaluation window and forms a category identifier sequence, and then calculates its Shannon entropy based on the category identifier sequence. Second reward value The calculation rules include: ,in For Shannon entropy, For negative feedback rate, and Preset weighting coefficients to characterize the relative importance of Shannon entropy and negative feedback rate.

[0008] Preferably, the multimodal data fusion and feature extraction module is used to call a pre-trained convolutional neural network to process sensor signals and a pre-trained language model to process text logs, so as to automatically extract high-dimensional features. The multimodal data fusion and feature extraction module is also used to determine whether the user is in a moving state or a stationary browsing state based on the sensor data, and adjust the weight ratio of each modality feature in real time through a dynamic weight allocation algorithm according to the determined state. When the user is in a moving state, the weight of sensor data and GPS data is increased, and when the user is in a stationary browsing state, the weight of text log data is increased.

[0009] Preferably, the dynamic user intent modeling and profile update module is used to take the user interaction behavior sequence as input and output an intent embedding vector representing the user's current instantaneous intent through a sequence deep learning model. The dynamic user intent modeling and profile update module is also used to call an online learning algorithm to fuse the intent embedding vector with the historical user state vector and generate an updated user state vector through incremental fine-tuning. The update of the user state vector adopts a dual mechanism of timed and triggered updates, which are performed at a preset frequency and triggered instantly when the user generates a preset high-value behavior.

[0010] Preferably, the reinforcement learning-based precise matching and decision engine is used to employ a deep Q-network as the reinforcement learning model; the reinforcement learning-based precise matching and decision engine is also used to enable the exploration-utilization balance mechanism when selecting the delivery action, to set a first exploration probability for new users or new content, and to reduce the first exploration probability according to a preset decay rule based on data accumulation.

[0011] Preferably, the reinforcement learning-based precise matching and decision engine is used to filter candidate content from the content library that meets the preset threshold for matching degree with the intent embedding vector generated by the dynamic user intent modeling and profile update module, and combine it with the currently available channels and time periods to form a set of combined actions as the selection space for delivery actions; when generating the set of combined actions, the reinforcement learning-based precise matching and decision engine is also used to set content diversity constraints, which are used to limit the upper limit of the repetition rate of the same content category identifier within a preset time window.

[0012] Preferably, the online learning and feedback evolution closed-loop module is used to achieve hot model updates using online gradient descent when updating the parameters of the reinforcement learning model. This module also incorporates a champion challenger framework to continuously conduct multi-strategy A / B testing, and automatically switches the newly optimized model to the new online master model when the core metrics of the new optimized model consistently outperform the online master model. The switching employs a canary release strategy, first covering a small proportion of traffic with the new optimized model, and then gradually increasing the coverage proportion after observing no anomalies. The multimodal data fusion and feature extraction module includes: HTTP and HTTPS protocol interfaces for connecting to user terminal APPs, sensor and GPS data from smart devices; an MQTT protocol interface for connecting to third-party data sources; and a streaming computing platform with a built-in window calculation mechanism and backpressure mechanism to automatically trigger traffic shaping strategies when data traffic exceeds the platform's carrying capacity threshold.

[0013] Preferably, the dynamic user intent modeling and profile update module has built-in invalid behavior filtering rules to remove interference data such as accidental clicks and duplicate searches before processing user interaction behavior sequences; the dynamic user intent modeling and profile update module also has a zero-padding strategy to pad to a fixed length when the length of the user interaction behavior sequence is insufficient before inputting it into the sequence deep learning model.

[0014] Preferably, the online learning and feedback evolution closed-loop module includes: a feedback data collection unit for collecting real-time user feedback through user terminal tracking, the real-time user feedback including feedback type, feedback duration and feedback scenario; a feedback database for storing real-time user feedback; and a data verification unit for setting up a data verification mechanism to eliminate fake or abnormal real-time user feedback through timestamp comparison and device identification verification.

[0015] A method for precise information delivery based on multimodal perception and reinforcement learning, the method comprising: Step a: Receive multi-source heterogeneous data and generate multimodal context feature vectors; process user interaction behavior sequences, generate user state vectors, and record user behavior sequences in the user interaction behavior sequences; Step b: Concatenate the multimodal context feature vector with the user state vector as the state input of the reinforcement learning model, and select and execute the delivery action based on the state input; Step c: Respond to real-time user feedback on the delivery action and determine the first reward value; Step d: At the end of an evaluation window with a preset time length, a retrospective reward calculation is initiated. This calculation retrieves the content category identifier in the user behavior sequence, calculates the information entropy value based on the content category identifier, and counts the negative feedback rate within the evaluation window, thereby generating a second reward value. The second reward value is a function value that is negatively correlated with the information entropy value and the negative feedback rate. Step e: Add the first reward value and the second reward value together to obtain the final reward value; Step f involves retrospectively allocating the final reward value to all state input-action pairs within the evaluation window to update the parameters of the reinforcement learning model.

[0016] Compared with the prior art, the beneficial effects of the present invention are: 1. By constructing a composite reward calculation mechanism in the reinforcement learning decision engine, the optimization objective of the model is aligned with the user's long-term focus state. At the end of an evaluation window, this mechanism retrieves the behavior sequence generated by the dynamic user intent modeling module and objectively measures the degree of topic drift for the content items that have been interacted with in the sequence. The system retrospectively assigns a long-term reward value that is negatively correlated with both the degree of drift and the negative feedback rate to all decision sequences within the window, thereby transforming the training objective of the model from optimizing immediate clicks to optimizing the user's long-term retention value.

[0017] 2. The aforementioned composite reward mechanism substantially changes the convergence direction of the decision-making model's strategy. For a delivery action that, while receiving immediate positive feedback, leads to highly dispersed browsing topics for the user, the system will add a negative long-term reward value at the end of the evaluation window. The online learning module uses this reshaped total reward value to update the model parameters. This allows the model's optimal strategy to converge to actively guiding users to engage in more focused browsing with greater thematic continuity during iterative evolution, thus avoiding strategy degradation caused by the model's pursuit of short-term metrics.

[0018] 3. A technical closed loop with highly coupled data flow and functional logic was constructed. The dynamic user intent modeling module and the reinforcement learning decision engine achieved dual functional association. On the one hand, the user state vector generated by the user intent module was used to construct the state input of the decision engine to achieve accurate matching of the user's current state. On the other hand, the user behavior sequence on which the intent module depends was retrieved by the reward calculation unit of the decision engine as the key data source for calculating the aforementioned long-term reward. This mechanism ensures that the model's intent recognition and policy optimization use the same source data, ensuring that the user state on which the decision model is based is coherent and consistent when optimizing long-term goals. Attached Figure Description

[0019] Figure 1This is a system functional architecture diagram based on the composite reward mechanism of the present invention; Figure 2 This is a graph showing the training convergence relationship between information entropy and reward value in this invention. Figure 3 This is a system engineering deployment architecture diagram for the present invention, which combines model operation and maintenance. Detailed Implementation

[0020] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention is further described below with reference to specific illustrations. However, the following embodiments are for explanation only and are not intended to limit the scope of protection of this invention.

[0021] This invention discloses a system and method for precise information delivery based on multimodal perception and reinforcement learning. The system architecture is a closed-loop system consisting of data perception, intent cognition, intelligent decision-making, and online evolution. Physically, the system can be deployed on a cloud server or distributed computing cluster. It mainly includes: a multimodal data fusion and feature extraction module, a dynamic user intent modeling and profile update module, a reinforcement learning-based precise matching and decision engine, and an online learning and feedback evolution closed-loop module. The multimodal data fusion and feature extraction module serves as the data entry point, responsible for real-time aggregation of user and environmental data to generate unified contextual features. The dynamic user intent modeling and profile update module, as the core of user cognition, is responsible for tracking user behavior and outputting dynamic user states. The reinforcement learning-based precise matching and decision engine calculates and executes the optimal delivery action based on the state inputs provided by the first two modules. Finally, the online learning and feedback evolution closed-loop module collects feedback after action execution, not only for calculating immediate rewards but, more importantly, for triggering a retrospective reward reshaping based on an evaluation window. This reshaping reward signal is then used to continuously optimize the decision engine and intent modeling module, ensuring that the system strategy aligns with maximizing long-term benefits. The direction of convergence is achieved. In a specific implementation scenario, the operation flow of the system of this invention is detailed as follows: The multimodal data fusion and feature extraction module starts working. The objective challenge faced by this module lies in the variability of user states and the heterogeneity of data sources. To address this challenge, this module receives interaction logs and GPS data reported by user terminal APP through its built-in streaming computing platform via standard HTTP and HTTPS protocol interfaces, and simultaneously connects to third-party data sources or smart device sensor data via MQTT protocol interface. The platform's built-in window calculation mechanism and backpressure mechanism ensure that the system can operate under high-concurrency data flow. The system ensures stability; upon receiving data, it doesn't simply combine the data, but instead uses a dedicated pre-trained deep learning model for automated feature extraction. Specifically, sensor signals and accelerometer data are input into a pre-trained convolutional neural network (CNN) to extract time and frequency domain features, while text logs, such as search terms and page titles, are input into a pre-trained language model to extract deep semantic features. Furthermore, this module also performs a crucial dynamic weight allocation; the system determines the user's state in real time based on sensor data, and when it determines the user is in motion, the accelerometer variance exceeds a threshold. And the GPS speed is greater than the threshold. When the user is in a static browsing state, the weight of sensor data and GPS data features is automatically increased during fusion. Conversely, when the user is in a static browsing state, the weight of text log features is increased. All heterogeneous features are aligned to the same high-dimensional space through feature embedding technology, and after the above dynamic weighted fusion, a multi-modal context feature vector with unified dimension is finally generated. This vector constitutes the first part of the environmental state required by the reinforcement learning model.

[0022] Meanwhile, the dynamic user intent modeling and profile update module works in parallel, aiming to solve the technical problems of slow user profile updates and inability to capture instantaneous intent in traditional user profile updates. This module obtains user interaction behavior sequences from the user terminal. Before inputting the sequences into the model, the system first activates invalid behavior filtering rules, eliminating interference data such as accidental clicks and repeated searches based on click duration (less than 0.5 seconds) or high-frequency repetition. Subsequently, to meet the input requirements of the sequence model, the insufficiently long sequences are padded to a fixed length using a zero-padding strategy, based on the most recent 50 interactions. The processed behavior sequences are then input into a sequence deep learning model, which, in a preferred embodiment, is coded as a Transformer. The model uses a Long Short-Term Memory (LSTM) network or an internal attention mechanism to capture the temporal dependencies between behaviors, outputting an intent embedding vector representing the user's current instantaneous intent. This intent embedding vector is used for content filtering in the subsequent decision engine, and is also input along with the user's historical user state vector into an online learning algorithm and an incremental fine-tuning algorithm to generate an updated user state vector through fusion calculation. This update process employs a dual mechanism of timed and triggered updates: it updates every 30 minutes at a preset frequency, and is triggered instantly when the user performs a preset high-value behavior, such as sharing or saving. In this way, the module continuously outputs two core results: intent embedding... The system first generates an input vector and a dynamically updated user state vector. Then, a reinforcement learning-based precise matching and decision-making engine is activated, serving as the core of the system's decision-making process. This engine constructs the state input required for decision-making, defined as a combined vector formed by concatenating the multimodal context feature vector output by the multimodal data fusion and feature extraction module from the preceding steps with the user state vector output by the dynamic user intent modeling and profile update module. Next, the engine constructs an action space, filtering from the content library those with intent embedding vectors just generated by the dynamic user intent modeling module. Through cosine similarity calculation, candidate content with a matching degree greater than 0.8 that meets a preset threshold is selected for delivery, and this is combined with currently available channels. The system combines content, channel, and time period to form a set of combined actions. When generating this action set, the system also enforces content diversity constraints, actively checking and limiting the repetition rate of the same content category identifier to no more than 30% within a preset time window over the past 24 hours to avoid user fatigue. After defining the state and action space, the system calls a reinforcement learning model, preferably a Deep Q-Network (DQN), inputting the current state vector. The DQN calculates the expected long-term cumulative reward (Q-value) for each action in the action set. When selecting the final action, the system uses an exploration-utilization balancing mechanism, setting a higher initial exploration probability for new users or new content. This gives it a 20% chance to randomly select an action. As data accumulates, this probability of exploration decreases gradually by 0.01 after every 100 interactions, according to a preset decay rule, and the system increasingly selects the action with the highest Q value. The final selected action is then executed by the delivery system.

[0023] Finally, the online learning and feedback evolution closed-loop module is responsible for realizing the system's self-evolution. This is the core mechanism of the invention, aiming to solve the problem of misaligned reward metrics caused by the over-reliance on instant click feedback in existing reinforcement learning systems. This module collects real-time user feedback after the execution of the delivery action through user terminal tracking, such as clicks, pauses, and closes. The feedback data includes feedback type, feedback duration, and feedback scenario. Before being uploaded to the feedback database, it undergoes a data verification mechanism, including timestamp comparison and device identification verification, to eliminate forged or abnormal data. For these instant feedbacks, the system determines a first reward value (…). This is a sparse scalar, with a +0.1 bonus for each click. However, the system does not use this as the final training signal. Instead, it waits for an evaluation window of a preset length, either at the end of a session or after a fixed 30-minute period. Upon the window's end, the system immediately initiates a retrospective reward calculation: this calculation retrieves the complete user behavior sequence recorded in the dynamic user intent modeling and profile update module within the window, extracts all interacted content items and their corresponding content category identifiers, and forms a category identifier sequence. Then, based on this category identifier sequence, its Shannon entropy is calculated. A higher The value objectively characterizes the highly dispersed nature of the topics browsed by users during this period, indicating a high-entropy drift state; simultaneously, the system statistically analyzes the negative feedback rate within this window. The percentage of actions that are turned off or ignored; subsequently, the system generates a second reward value based on a preset negative correlation function. In a preferred embodiment, the calculation rule is as follows: ,in and To predetermine the weighting coefficients, this formula ensures that both high entropy and high negative feedback rates result in a punitive negative reward value; the system then sums the first reward value of all (state-action) pairs within the window with this uniformly calculated second reward value to obtain the final reward value. Finally, the module retrospectively assigns this final reward value to all state input and action pairs within the evaluation window, and stores these reshaped reward (state, action, final reward, next state) tuples in the experience replay pool. This pool is used to update the network parameters of the DQN model using an online gradient descent method, i.e., hot model updates, which also updates the user intent model. In addition, the module has a built-in champion-challenger framework to continuously conduct multi-strategy A / B testing. When the core metrics of the newly optimized model, such as long-term retention rate, are consistently better than the online main model, the system adopts a canary release strategy, first switching 1% of the traffic, observing no abnormalities, and then gradually increasing the coverage ratio, automatically switching the new model to the new online main model.

[0024] Example 1: This example demonstrates the specific application and operation of the aforementioned technical solution in a high-concurrency information flow delivery platform. The reinforcement learning model previously deployed on this platform used only the user's immediate click feedback as the sole positive reward signal. In the initial operation phase of the platform, the reinforcement learning model iterated based on the single objective of maximizing immediate click rewards. The system strategy converged to pushing highly stimulating and timely information with low content value, leading to a short-term increase in click-through rate. However, backend user behavior analysis showed that the median session duration of users was decreasing, while the frequency of content topic switching and application exit rate were increasing. This indicates that the optimization objective of the model has deviated from the business objective of improving long-term user retention, and the system has fallen into a dilemma of reward indicator misalignment. The user's browsing behavior exhibits a typical high-entropy drift state.

[0025] To address this issue, the system switches to the composite reward mechanism of this invention, which operates as follows: an evaluation window is set to 30 minutes; at a certain time point... The system is in a state where the input is a concatenation of a multimodal context feature vector and a user state vector. Based on a reinforcement learning-based precise matching and decision engine, a delivery action is executed. This action pushes content with a highly enticing title to the user; the user is attracted by the title and clicks, and the system responds with this immediate user feedback. Determine a first reward value at each moment. After opening the content, the user found it did not match the title and closed it after a brief 3-second pause. This action was recorded as negative feedback by the system. Subsequently, the user switched between three unrelated content categories over the next 5 minutes, from finance to entertainment, and then to society. At the specified time, the evaluation window ends, and the online learning and feedback evolution closed-loop module initiates retrospective reward calculation. This calculation retrieves the complete user behavior sequence within this window and, by analyzing the distribution of content category identifiers, calculates the Shannon entropy of that sequence. For a relatively high value, For example; at the same time, the negative feedback rate in the system statistics window. Due to behaviors such as rapid shutdown, this value is also at a relatively high level. For example; the system then... The rules calculate the second reward value and set the weight. , ,but The system then calculates the final reward value. The final reward value of -1.05 was retroactively allocated. Moment (status input, action delivery) Yes, and is used to update the parameters of the reinforcement learning model.

[0026] Meanwhile, for another delivery action within this window The content it pushes highly matches the user's current intent embedding vector, resulting in a prolonged user dwell time after clicking. Furthermore, subsequent browsing activities all revolved around this topic, causing the window to... Low value and Its final reward value is 0. The calculated value is positive. Through this reward reshaping based on information entropy and negative feedback rate, the model's Q-value function or policy network will autonomously learn, after multiple iterations, to execute the delivery action. While induced clicks offer immediate positive rewards, their long-term cumulative expected value (Q-value) is significantly lower due to high costs. Punishment is negative; conversely, performing the delivery action is positive. Cultivating focus leads to higher long-term cumulative expectations; ultimately, the system's decision model converges to a strategy that prioritizes guiding users into a low-entropy focused state, proactively avoiding short-term inducements that could cause users to drift towards high entropy; the user behavior sequence data recorded by the dynamic user intent modeling and profile update module not only provides user cognition for the construction of state inputs, but also its derived information entropy value serves as a core data source, correcting the optimization objective of the online learning and feedback evolution closed-loop module. The two modules achieve functional coupling through the same source data, enabling the reinforcement learning model's strategy evolution direction to shift from optimizing immediate clicks to optimizing long-term user value.

[0027] Example 2: This example compares the composite reward mechanism of this invention with the standard instant reward mechanism in an offline simulation environment based on A / B testing. The environment uses a publicly available news recommendation dataset as the content library and constructs a simulated user agent. The agent's response rules are set as follows: when receiving content highly relevant to its current intent, it generates click and long-stay feedback and tends to maintain topic consistency in subsequent interactions; when receiving highly persuasive but low-relevance content, it has a certain probability of generating a click, but this is followed by a rapid close. This behavior is counted as a negative feedback, triggering frequent topic switching, which in turn leads to an increase in information entropy. The experimental setup included a control group and an experimental group based on the present invention. Both groups used a Deep Q-Network (DQN) as a reinforcement learning model and were trained with 1 million interactions in the same simulation environment. The control group model used the standard reward function in the prior art, i.e., only the first reward value. As the sole training signal, a click is counted as +1.0, and no click is counted as 0. The sample model of this invention adopts the composite reward mechanism of this invention, and its evaluation window is set to 10 consecutive delivery actions, with its final reward value... Depend on With the second reward value The summation is obtained; among which, The calculation follows In this experiment, the weighting coefficient was set to as well as After training, the two models were evaluated on 100,000 independent test sessions, and their average click-through rate (CTR) and average session information entropy were calculated. Average negative feedback rate in conversations And the average session length, which is measured in the number of interactions.

[0028] The test evaluation results are shown in Table 1.

[0029] Table 1: Comparison of Model Performance under Different Reward Mechanisms

[0030] The data in Table 1 show that the control group model, which relies solely on instant clicks for rewards, achieved an average click-through rate of 12.3%, an average session information entropy of 1.48, and an average session negative feedback rate of 21.5%. The sample group model of this invention, which adopts a composite reward mechanism, achieved an average click-through rate of 10.9%, reduced the average session information entropy to 0.62, reduced the average session negative feedback rate to 5.3%, and increased the average session length from 4.2 interactions to 7.9 interactions.

[0031] Example 3: To further separate and verify the information entropy value in the composite reward mechanism of the present invention The key technological contribution for backtracking computation is that, based on the experimental environment and evaluation indicators of Example 2, this embodiment adds a comparative sample group and compares it with the control group in Example 2 and the sample group of this invention. This embodiment adopts the experimental environment, user agent model and evaluation indicators of Example 2; the reinforcement learning model of the comparative sample group has its reward function set in a conventional and predictable improvement method in the art, that is, only at the first reward value... Based on (click count +1.0), a penalty for immediate negative feedback is added, and its reward function is: ,in, The weighting coefficient is set to 1.0, consistent with the setting of the sample group in Example 2 of the present invention; it should be noted that the reward function of this comparative sample group does not include the information entropy value. The calculations and considerations, its The results represent only the immediate negative feedback penalty, not the backtracking calculation results based on the evaluation window. All samples (control group, comparative sample group, and sample group of this invention) were trained under the same conditions for 1 million interactions and their performance was evaluated on 100,000 independent test sessions. The key metrics comparison is shown in Table 2.

[0032] Table 2: Performance Comparison of Models under Different Reward Mechanisms (Including Comparative Models)

[0033] The data in Table 2 confirms that the conventional technical improvement approach of penalizing only immediate negative feedback (comparative sample group), while reducing the negative feedback rate, fails to guide the model to learn the user's attention state, and its information entropy value... No improvement was observed compared to the control group; in contrast, the sample group of this invention introduced an information entropy-based approach. Retrospective second reward value This shifts the model strategy from optimizing immediate actions to optimizing session states, resulting in an increase in average session length while reducing information entropy and negative feedback rate.

[0034] Example 4: This example combines Figures 1 to 3 This section describes the information delivery system and methods based on multimodal perception and reinforcement learning, such as... Figure 1As shown, the system starts with unlabeled data sources, including sensors, GPS, text logs, and APP interactions. The multimodal data fusion and feature extraction module receives this data, performs dynamic weight allocation, and outputs a multimodal context feature vector. Simultaneously, the dynamic user intent modeling and profile update module processes user interaction sequences and uses a sequence deep learning model to output a user state vector, while also recording the user behavior sequence for backtracking. A reinforcement learning-based precise matching and decision engine receives the multimodal context feature vector and the user state vector and concatenates them into a state input. The engine employs a Deep Q-Network (DQN) and enables an exploratory balancing mechanism. The final output is: the delivery action is sent to the user terminal, and the user terminal generates real-time user feedback such as clicks and closes, which flows to an online learning and feedback evolution closed-loop module. This module determines the first reward value immediately and initiates a retrospective reward calculation to generate a second reward value in the future. The final output is: the reward value is used to update the model parameters of the decision engine. This retrospective reward calculation, as the core mechanism of this invention, is initiated at the end of the evaluation window. It retrieves the user behavior sequence provided by the dynamic user intent modeling and profile update module for calculation, and calculates based on the information entropy H and the negative feedback rate. A second reward value that is negatively correlated with the two is generated. The goal is to resolve the misalignment of reward indicators, cultivate a low-entropy state of focus, and use this as the basis for calculation to provide to the online learning and feedback evolution closed-loop module.

[0035] like Figure 2 As shown in the figure, the horizontal axis represents the number of training epochs multiplied by 100,000, the left vertical axis represents the average final reward value, and the right vertical axis represents the average information entropy H. The data in the figure shows that as the number of training epochs increases from 1 to 10, the average final reward value, represented by the solid line, exhibits a monotonically increasing trend, growing from -0.8 to nearly 0.8, while the average information entropy H, represented by the dashed line, exhibits a monotonically decreasing trend, decreasing from 1.4 to slightly above 0.6. This indicates that the model's optimization objective successfully converges to cultivating a low-entropy focused state. Figure 3As shown, in this architecture, multimodal data sources, including raw data generated by user terminals and third-party data, as well as real-time user feedback, are sent to the data access and streaming computing platform. This platform has built-in protocol interfaces HTTP and MQTT, as well as window computing and backpressure mechanisms. After processing, it generates a cleaned data stream and sends it to the core service and decision engine. The core service and decision engine integrates a multimodal feature extraction module, a dynamic user intent modeling module, a reinforcement learning decision engine, and an online learning evolution module. On the one hand, it reads content from the content library of the data storage area to perform information delivery. On the other hand, it writes feedback / experience to the feedback database and experience replay pool. The core service also updates itself through hot model updates and outputs the newly optimized model to the model operation and release module. The release module achieves stable iteration of the model through A / B testing and canary release strategies using the champion challenger framework.

[0036] Example 5: This example illustrates a standardized engineering procedure for offline calibration and determination of key operating parameters in the aforementioned technical solution, specifically addressing the evaluation window duration, etc. Weight coefficients in the function and Key operating parameters are defined, and a reproducible calibration procedure is provided. The calibration is performed in an offline simulation environment using a historical dataset containing 1 million real user session logs, and a simulated user agent with characteristics consistent with that in Example 2 is enabled. The optimization objective of this procedure is set as a non-real-time business metric, namely, the simulated 7-day user retention rate. The calibration process determines the duration of the evaluation window, which is fixed by the system. , As a temporary coefficient, the impact of evaluation window lengths of 5 minutes, 10 minutes, 20 minutes, 30 minutes and 60 minutes on the simulated 7-day user retention rate was tested. The test results are shown in Table 3.

[0037] Table 3: Calibration Table of the Impact of Assessment Window Duration on Long-Term Retention Rate

[0038] Table 3 shows that a window that is too short (5 minutes) cannot effectively capture the user's complete drift behavior, while a window that is too long (60 minutes) results in sparse reward signals and excessive latency. In the dataset and environment of this experiment, a 20-minute evaluation window achieved the highest retention rate of 24.8%. Therefore, 20 minutes was used as the preset duration for the evaluation window in subsequent calibrations. After determining the evaluation window to be 20 minutes, the system started... Medium weighting coefficient Information entropy weight and Calibration of negative feedback rate weights; the system employs a grid search method, traversing and testing within a pre-defined two-dimensional parameter space, where... The test range is [0.0, 2.0], with a step size of 0.5. The test range is [0.0, 2.0], with a step size of 0.5, and the simulated 7-day user retention rate is used as the final evaluation metric. The calibration results are shown in Table 4.

[0039] Table 4: Weighting Coefficients and Grid search calibration table for the impact of simulated 7-day retention rate

[0040] The data analysis in Table 4 shows that when and At that time, corresponding to the control group setting in Example 2, the retention rate was the lowest, at 15.2%; when and At that time, corresponding to the reward method of the comparative sample group in Example 3, the retention rate improved. It reached 18.5%, but the improvement was limited; when and When the technology range employing the composite reward mechanism of this invention is reached, the retention rate is higher than in the previous two scenarios; in this calibration, when and At that time, the system achieved a peak retention rate of 24.8%; if or The value of continues to increase, so that or For example, the retention rate actually decreased, indicating that excessively high penalty weights make the model strategy too conservative, excessively suppressing necessary browsing exploration, which is also detrimental to long-term retention; through this calibration procedure, the system determined and This is the optimal parameter combination for the current deployment environment; furthermore, this embodiment also discloses a specific implementation procedure for the dynamic weight allocation algorithm in the multimodal data fusion and feature extraction module: the system analyzes the accelerometer signal in the sensor data and calculates its signal variance within a 1-second time window. And set a variance threshold, such as ;when When the system determines that the user is in a mobile state, the fusion weight assigned to the sensor data and GPS data features is 0.7, and the weight assigned to the text log data features is 0.3; when When the system determines that the user is in a static browsing state, the weights are reversed, with a weight of 0.3 assigned to the sensor data and GPS data features, and a weight of 0.7 assigned to the text log data features.

[0041] Example 6: This example further discloses the specific engineering implementation procedures of the invalid behavior filtering rules and zero-filling strategy executed by the aforementioned dynamic user intent modeling and profile update module when processing user interaction behavior sequences. These procedures aim to ensure the data quality and format standardization of the data input to the sequence deep learning model. After receiving the original user interaction behavior sequence, the system activates the invalid behavior filtering rules, which include at least two sub-procedures: the first is accidental click filtering. The system traverses the interaction records in the sequence. When the behavior type of a record is click and its feedback duration is less than a preset effective click duration threshold, the system will filter out the accidental clicks. At the specified time, the record is marked as invalid and removed from the sequence; the second item is duplicate search filtering. The system checks adjacent search behaviors in the sequence. When two records both have the same search behavior, their search terms are identical, and the timestamp interval is less than a preset duplicate search interval threshold, the system will remove them. At 10:00, the next search record is marked as invalid and removed.

[0042] After performing the invalid behavior filtering, the system obtains a length of The system executes a clean sequence of actions, then initiates a zero-padding strategy to meet the fixed-length input specification requirements of the Transformer deep learning model; the system sets a fixed input sequence length. If the system detects the current sequence length If the condition is met, a padding operation is performed; this operation does not fill in random values ​​or 0, but rather fills in the beginning or end of the sequence. A dedicated padding token ([PAD]Token) is a specific token with the number 0 in the model vocabulary. Subsequent sequence deep learning models will ignore the positions of these padding tokens during the self-attention calculation process through the AttentionMask mechanism, thereby achieving equal-length processing of the input sequence without introducing noise information.

[0043] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A system for precise information delivery based on multimodal perception and reinforcement learning, characterized in that the system... include: The multimodal data fusion and feature extraction module is used to receive multi-source heterogeneous data and generate multimodal context feature vectors; The dynamic user intent modeling and profile update module is used to process user interaction behavior sequences, generate user state vectors, and record user behavior sequences in the user interaction behavior sequences. A reinforcement learning-based precise matching and decision engine is used to concatenate multimodal context feature vectors with user state vectors as state inputs to the reinforcement learning model, and select and execute delivery actions based on the state inputs. The online learning and feedback evolution closed-loop module is used to respond to real-time user feedback on the delivery action, determine the first reward value, and initiate a retrospective reward calculation at the end of the evaluation window with a preset time length. This calculation retrieves the content category identifiers in the user behavior sequence recorded by the dynamic user intent modeling and profile update module, calculates the information entropy value based on the content category identifiers, and counts the negative feedback rate within the evaluation window, thereby generating the second reward value. The second reward value is a function value that is negatively correlated with the information entropy value and the negative feedback rate. The online learning and feedback evolution closed-loop module is also used to accumulate the first reward value and the second reward value to obtain the final reward value, and to retrospectively distribute the final reward value to all state input and action pairs within the evaluation window to update the parameters of the reinforcement learning model.

2. The information delivery system based on multimodal perception and reinforcement learning according to claim 1, characterized in that, The dynamic user intent modeling and profile update module records a series of content items in the user behavior sequence. The online learning and feedback evolution closed-loop module, when calculating information entropy, specifically extracts the content category identifiers corresponding to all content items within the evaluation window and forms a category identifier sequence, then calculates its Shannon entropy based on the category identifier sequence. Second reward value The calculation rules include: ,in For Shannon entropy, For negative feedback rate, and Preset weighting coefficients to characterize the relative importance of Shannon entropy and negative feedback rate.

3. The information delivery system based on multimodal perception and reinforcement learning according to claim 1, characterized in that, The multimodal data fusion and feature extraction module is used to call a pre-trained convolutional neural network to process sensor signals and a pre-trained language model to process text logs, so as to automatically extract high-dimensional features. The multimodal data fusion and feature extraction module is also used to determine whether the user is in a moving state or a stationary browsing state based on sensor data, and adjusts the weight ratio of each modality feature in real time through a dynamic weight allocation algorithm according to the determined state. When the user is in a moving state, the weight of sensor data and GPS data is increased, and when the user is in a stationary browsing state, the weight of text log data is increased.

4. The information delivery system based on multimodal perception and reinforcement learning according to claim 1, characterized in that, The dynamic user intent modeling and profile update module takes user interaction behavior sequences as input and outputs an intent embedding vector representing the user's current instantaneous intent through a sequence deep learning model. The module also calls online learning algorithms to fuse the intent embedding vector with historical user state vectors and generate updated user state vectors through incremental fine-tuning. The update of user state vectors adopts a dual mechanism of timed and triggered updates, updating at a preset frequency and triggering an immediate update when the user generates a preset high-value behavior.

5. The information delivery system based on multimodal perception and reinforcement learning according to claim 1, characterized in that, The reinforcement learning-based precise matching and decision engine is used to employ a deep Q-network as the reinforcement learning model. The reinforcement learning-based precise matching and decision engine is also used to enable an exploration-utilization balance mechanism when selecting delivery actions, setting a first exploration probability for new users or new content, and reducing the first exploration probability according to a preset decay rule based on data accumulation.

6. The information delivery system based on multimodal perception and reinforcement learning according to claim 4, characterized in that, The precision matching and decision engine based on reinforcement learning is used to filter candidate content from the content library that meets the preset threshold for matching degree with the intent embedding vector generated by the dynamic user intent modeling and profile update module. Combined with the currently available channels and time periods, a set of combined actions is formed as the selection space for the delivery actions. When generating a set of combined actions, the reinforcement learning-based precise matching and decision engine is also used to set content diversity constraints. These constraints limit the maximum repetition rate of the same content category identifier within a preset time window.

7. The information delivery system based on multimodal perception and reinforcement learning according to claim 1, characterized in that, The online learning and feedback evolution closed-loop module is used to achieve hot model updates using online gradient descent when updating the parameters of the reinforcement learning model. This module also incorporates a champion challenger framework to continuously conduct multi-strategy A / B testing. When the core metrics of the newly optimized model consistently outperform the online master model, it automatically switches the new optimized model to the new online master model. The switching employs a canary release strategy, initially covering a small percentage of traffic with the new optimized model, and gradually increasing the coverage percentage after observing no anomalies. The multimodal data fusion and feature extraction module includes: HTTP and HTTPS protocol interfaces for connecting to user terminal apps, sensor and GPS data from smart devices; an MQTT protocol interface for connecting to third-party data sources; and a streaming computing platform. The streaming computing platform incorporates a window calculation mechanism and backpressure mechanism to automatically trigger traffic shaping strategies when data traffic exceeds the platform's capacity threshold.

8. The information delivery system based on multimodal perception and reinforcement learning according to claim 4, characterized in that, The dynamic user intent modeling and profile update module has built-in invalid behavior filtering rules to remove interference data such as accidental clicks and duplicate searches before processing user interaction behavior sequences. The dynamic user intent modeling and profile update module also has a zero-padding strategy to pad the length of user interaction behavior sequences to a fixed length before inputting them into the sequence deep learning model.

9. The information delivery system based on multimodal perception and reinforcement learning according to claim 1, characterized in that, The online learning and feedback evolution closed-loop module includes: a feedback data collection unit for collecting real-time user feedback through user terminal tracking, which includes feedback type, feedback duration, and feedback scenario; a feedback database for storing real-time user feedback; and a data verification unit for setting up a data verification mechanism to eliminate fake or abnormal real-time user feedback through timestamp comparison and device identification verification.

10. A method for precise information delivery based on multimodal perception and reinforcement learning, characterized in that, The method includes: Step a: Receive multi-source heterogeneous data and generate multimodal context feature vectors; process user interaction behavior sequences, generate user state vectors, and record user behavior sequences in the user interaction behavior sequences; Step b: Concatenate the multimodal context feature vector with the user state vector as the state input of the reinforcement learning model, and select and execute the delivery action based on the state input; Step c: Respond to real-time user feedback on the delivery action and determine the first reward value; Step d: At the end of an evaluation window with a preset time length, a retrospective reward calculation is initiated. This calculation retrieves the content category identifier in the user behavior sequence, calculates the information entropy value based on the content category identifier, and counts the negative feedback rate within the evaluation window, thereby generating a second reward value. The second reward value is a function value that is negatively correlated with the information entropy value and the negative feedback rate. Step e: Add the first reward value and the second reward value together to obtain the final reward value; Step f involves retrospectively allocating the final reward value to all state input-action pairs within the evaluation window to update the parameters of the reinforcement learning model.

Images