Intelligence collection and editing method and device based on reinforcement learning

Abstract

The present application relates to the technical field related to intelligent recommendation, and specifically to a method and device for intelligence collection and editing based on reinforcement learning. The method includes: obtaining intelligence collection and editing logs; constructing a selection button click rate estimation model and a push button click rate estimation model based on the intelligence collection and editing logs; wherein the selection button click rate estimation model is used to estimate the selection button click rate of intelligence, and the push button click rate estimation model is used to estimate the push button click rate of intelligence; constructing a fusion model of the output results of the selection button click rate estimation model and the output results of the push button click rate estimation model; wherein the fusion model is a deep learning model, and is also used to adjust internal parameters based on the latest operations of intelligence collection and editing personnel, so that the fusion value output by the fusion model is more in line with the expectations of intelligence collection and editing personnel; candidate intelligence recommendations are made based on the selection button click rate estimation model, the push button click rate estimation model and the fusion model.

Description

Technical Field

[0001] The present application relates to the technical field related to intelligent recommendation, and specifically to an intelligence collection and editing method and device based on reinforcement learning. Background Art

[0002] With the development of information technology and open source big data, many companies have established competitive intelligence departments, built their own intelligence collection and editing systems, and deployed intelligence collection and editing personnel. Intelligence collection and editing personnel use intelligence collection and editing systems to select intelligence reports, and then compile reports and provide them to corporate decision makers for reference, so as to reduce the company's own risks and enhance competitiveness. Companies in highly competitive industries regard intelligence collection and editing as a necessary task.

[0003] Because the most important features of intelligence services are accuracy and timeliness. Intelligence services serve corporate decision-making. If intelligence is provided after an important decision is made, then no matter how critical and good the intelligence is, it will not play its due role. Therefore, companies need to improve the efficiency of intelligence collection and editing as much as possible under the condition of limited manpower of intelligence collection and editing personnel. Summary of the invention

[0004] In view of this, the embodiments of the present application are committed to providing an intelligence collection and editing method and device based on reinforcement learning.

[0005] This application provides an intelligence collection and editing method based on reinforcement learning, including:

[0006] Obtain intelligence collection and editing logs;

[0007] Based on the intelligence collection and editing log, a selection button click rate prediction model and a push button click rate prediction model are respectively constructed;

[0008] The report button click rate prediction model is used to estimate the report button click rate of intelligence, and the push button click rate prediction model is used to estimate the push button click rate of intelligence;

[0009] Construct a fusion model of the output results of the selection button click rate estimation model and the push button click rate estimation model;

[0010] The fusion model is a reinforcement learning model, and is also used to adjust internal parameters based on the latest operations of intelligence editors, so that the fusion value output by the fusion model better meets the expectations of intelligence editors;

[0011] Among them, the click rate of the report button is positively correlated with the integration value, and the click rate of the push button is positively correlated with the integration value;

[0012] Candidate intelligence recommendation is performed based on the selection button click rate prediction model, the push button click rate prediction model and the fusion model.

[0013] In some embodiments, the model for estimating the click rate of the report button uses a dual-tower deep neural network model to estimate the click rate of the report button.

[0014] One side of the model for predicting the click rate of the reporting button is input with the features of user operations in the intelligence collection and editing log, and the other side is input with the candidate intelligence in the intelligence collection and editing log. The intelligence collection and editing log based on the actual collection and editing operations of intelligence editors is trained offline in advance.

[0015] In some embodiments, the push button click rate prediction model uses a dual deep neural network to estimate the push button click rate. One side inputs the features of user operations in the intelligence collection and editing log, and the other side inputs the candidate intelligence in the intelligence collection and editing log. The intelligence collection and editing log based on the actual collection and editing operations of intelligence editors is trained offline in advance.

[0016] In some embodiments, the selection button click rate prediction model and the push button click rate prediction model are retrained based on the latest data at every preset time interval.

[0017] In some embodiments, the fusion model uses T-power form to fuse the click rate and duration to obtain the fused value q_final;

[0018] The fusion formula is: q_final=select_ctr T ∗ push_ctr (1-T)

[0019] The value range of T is from 0.40 to 1.0, with a step size of 0.01. select_ctr is the click rate of the select button; push_ctr is the click rate of the push button.

[0020] In some embodiments, the fusion model also includes: using the DQN network model for online application to implement the Markov decision process, and using --greed method to search for the best result.

[0021] In some embodiments, based on the editor's request, candidate intelligence is displayed;

[0022] If the request is to sort by release time, then the candidate intelligence sorted by release time is displayed;

[0023] If the request is to sort by content similarity, then the candidate intelligence sorted by content similarity is displayed.

[0024] If the request is to sort by source importance, then candidate intelligence sorted by source importance is displayed;

[0025] If the request is to sort by intelligence, candidate intelligence sorted by integration value is displayed.

[0026] This application also provides an intelligence collection and editing device based on reinforcement learning, including:

[0027] The acquisition module is used to obtain intelligence collection and editing logs;

[0028] A construction module is used to construct a selection button click rate prediction model and a push button click rate prediction model based on the intelligence collection and editing log; wherein the selection button click rate prediction model is used to estimate the selection button click rate of intelligence, and the push button click rate prediction model is used to estimate the push button click rate of intelligence; a fusion model of the output results of the selection button click rate prediction model and the output results of the push button click rate prediction model is constructed; wherein the fusion model is a reinforcement learning model, and is also used to adjust internal parameters based on the latest operations of intelligence collection and editing personnel, so that the fusion value output by the fusion model is more in line with the expectations of intelligence collection and editing personnel; wherein the selection button click rate is positively correlated with the fusion value, and the push button click rate is positively correlated with the fusion value;

[0029] The recommendation module is used to recommend candidate intelligence based on the selection button click rate prediction model, the push button click rate prediction model and the fusion model.

[0030] The present application provides an electronic device, including:

[0031] A processor, and a memory for storing a program executable by the processor;

[0032] The processor is used to implement the above-mentioned intelligence collection and editing method based on reinforcement learning by running the program in the memory.

[0033] The present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the above-mentioned intelligence collection and editing method based on reinforcement learning.

[0034] In the scheme provided by the present application, firstly, the intelligence collection and editing log is obtained; based on the intelligence collection and editing log, a selection button click rate prediction model and a push button click rate prediction model are respectively constructed; wherein, the selection button click rate prediction model is used to estimate the selection button click rate of intelligence, and the push button click rate prediction model is used to estimate the push button click rate of intelligence; a fusion model of the output results of the selection button click rate prediction model and the output results of the push button click rate prediction model is constructed; wherein, the fusion model is a reinforcement learning model, and is also used to adjust internal parameters based on the latest operations of intelligence collection and editing personnel, so that the fusion value output by the fusion model is more in line with the expectations of intelligence collection and editing personnel; wherein, the selection button click rate is positively correlated with the fusion value, and the push button click rate is positively correlated with the fusion value; candidate intelligence recommendations are made based on the selection button click rate prediction model, the push button click rate prediction model and the fusion model. With such a setting, the present application reduces the need for manual operation and speeds up intelligence processing through automated log acquisition and model construction. The model for estimating the click rate of the report and push buttons can predict user behavior based on historical data, thereby improving the relevance and accuracy of intelligence recommendations. The fusion model can dynamically adjust internal parameters according to the latest operations of intelligence editors, so that the recommendation results are more in line with the actual needs and expectations of users. Through the positive correlation between the fusion value output by the fusion model and the click rate of the report and push buttons, the intelligence that users may be interested in can be recommended more accurately, thereby improving user satisfaction. The reinforcement learning mechanism enables the system to learn and adapt to changes in market and user needs in real time, quickly adjust the recommendation strategy, and maintain the timeliness and relevance of intelligence push. Through the intelligent recommendation system, limited intelligence collection and editing resources can be more effectively allocated to the intelligence that users are most interested in, thereby improving overall work efficiency. Through accurate click rate estimation and fusion models, the push of invalid or irrelevant intelligence is reduced, and the risk of wrong recommendations is reduced. In summary, this intelligence collection and editing method based on reinforcement learning not only improves the efficiency and accuracy of intelligence collection and editing through intelligent data processing and model prediction, but also enhances the adaptability of the system and user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0036] Figure 1 This is a schematic diagram of an intelligence collection and editing operation interface provided by an embodiment of the present application;

[0037] Figure 2It is a schematic diagram of the intelligence collection and editing operation interface provided by an embodiment of the present application;

[0038] Figure 3 It is a flowchart of an intelligence collection and editing method based on reinforcement learning provided by an embodiment of the present application.

[0039] Figure 4 This is a schematic diagram of reinforcement learning provided by an embodiment of the present application.

[0040] Figure 5 This is a system implementation process provided by an embodiment of the present application.

[0041] Figure 6 This is a structural diagram of a model for estimating click rate of a report button provided by an embodiment of the present application.

[0042] Figure 7 This is a structural diagram of a push button click rate estimation model provided by an embodiment of the present application.

[0043] Figure 8 This is a DQN network model structure diagram provided by an embodiment of the present application.

[0044] Fig. 9 It is a structural diagram of an intelligence collection and editing device based on reinforcement learning provided by an embodiment of the present application.

[0045] Fig.10 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Application Overview

[0048] With the development of information technology and open source big data, many companies have established competitive intelligence departments, built their own intelligence collection and editing systems, and deployed intelligence collection and editing personnel. Intelligence collection and editing personnel use intelligence collection and editing systems to select intelligence reports, and then compile reports and provide them to corporate decision makers for reference, so as to reduce the company's own risks and enhance competitiveness. Companies in highly competitive industries regard intelligence collection and editing as a necessary task.

[0049] Because the most important features of intelligence services are accuracy and timeliness. Intelligence services serve corporate decision-making. If intelligence is provided after an important decision is made, then no matter how critical and good the intelligence is, it will not play its due role. Therefore, companies need to improve the efficiency of intelligence collection and editing as much as possible under the condition of limited manpower of intelligence collection and editing personnel.

[0050] In order to improve the efficiency of intelligence collection and editing, it is necessary to use a more effective intelligence collection and editing system. The intelligence collection and editing system usually consists of five processing links, including: 1) information collection, 2) information processing, 3) intelligence analysis, 4) intelligence push, and 5) intelligence collection and editing. The first four links are mainly completed automatically or semi-automatically by the system, and the fifth link is mainly completed manually by intelligence collection and editing personnel.

[0051] Among them, the "intelligence push" involved in link 4) needs to push appropriate potential intelligence in a timely and accurate manner, and needs to reduce the interference of pushing invalid content, which has a great impact on the work efficiency of intelligence editors. The "intelligence collection and editing" involved in link 5) needs to provide the most suitable operation interface for the work of editors, which also has a great impact on the work efficiency of intelligence editors. Optimizing these two links is an urgent need and effective direction to improve the efficiency of intelligence collection and editing.

[0052] In the intelligence collection and editing system, the "intelligence push" involved in link 4) is usually done by combining the intelligence subject demand characteristics preset by intelligence editors with the semantic information of intelligence content. In addition, how to better match intelligence subject demand characteristics with intelligence content semantic information has become a hot issue in the current knowledge push field, and a large number of technical achievements have been accumulated and made public, involving commonly used content recommendation technologies, such as recommendations based on content similarity, recommendations based on collaborative filtering, and recommendations based on embedded vector matching, etc.

[0053] Regarding the "intelligence collection and editing" involved in step 5), the usual method is that the system provides an intelligence collection and editing operation interface, and provides multiple sorting methods for the intelligence content pushed in step 4), such as: sorting by release time, sorting by content similarity, sorting by source importance, etc. For each piece of pushed intelligence, two types of operation buttons are usually provided: "select" and "push". Selecting means adding the candidate intelligence to the formal intelligence collection, and pushing means pushing the candidate intelligence to the person in need of intelligence. The intelligence collection and editing operation interface provided in step 5) is shown below Figure 1 shown.

[0054] The above methods are all intelligence collection and editing systems that push intelligence and sort interfaces based on the characteristics of the historical environment or the characteristics of historical preset intelligence theme requirements. They cannot adjust the system strategy in a timely and dynamic manner according to the editing and editing operations of intelligence editors. Changes in the external environment such as market competition are often changing rapidly. Intelligence editors will change the selection or push basis of intelligence editing operations in a timely manner based on their accumulated intelligence work experience and the latest external environment.

[0055] For example, Figure 1 In the operation interface shown, the intelligence editors selected or pushed the relevant content of Title 2 based on the enterprise's intelligence needs; however, when the external environment suddenly changed, the intelligence editors promptly changed the intelligence basis and turned to selecting or pushing the relevant content of Title 5, and the relevant content of Title 2 became invalid content.

[0056] Ideally, the intelligence collection and editing system should be sensitive to the change in the basis of intelligence, and sort the intelligence content related to Title 5 before the intelligence content related to Title 2, so as to attract the attention and attention of intelligence collection and editing personnel. However, in the previous solution, the change in the basis of intelligence could not be effectively captured and learned by the intelligence collection and editing system in a timely manner, and the previous sorting effect was still maintained, resulting in invalid content drowning out the truly valuable intelligence content, which not only affected the efficiency of intelligence collection and editing, but also brought the risk of missing out on truly valuable intelligence to the enterprise.

[0057] This application improves the existing technical solutions.

[0058] The intelligence collection and editing operation interface involved in stage 5) "Intelligence collection and editing" adds "smart sorting" and "refresh" buttons on the basis of the existing interface, such as Figure 2 As shown. In the intelligence push subsystem involved in link 4) "intelligence push", an intelligent sorting module based on reinforcement learning is added on the basis of the commonly used content recommendation technology module. The present invention adopts reinforcement learning, so that every time the intelligence editor refreshes, the intelligence push subsystem can obtain the best candidate intelligence push result in the current state by requesting the online service of the DQN network model. It solves the problem that the traditional method learns according to the historical environment or historical samples of the push system, and cannot make dynamic adjustments in time according to the current latest state of the push system, thereby causing intelligence value deviation.

[0059] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0060] Exemplary Methods

[0061] Figure 3FIG. 1 is a flowchart of an intelligence collection and editing method based on reinforcement learning provided by an embodiment of the present application. Figure 3 As shown, the method includes the following contents.

[0062] Step S310, obtaining intelligence collection and editing log;

[0063] This step involves collecting and organizing the operation logs of intelligence editors in the editing system. These logs include various operations performed by intelligence editors on intelligence content, such as selection, push, refresh, etc. Log data is the basis for subsequent model training, and it is necessary to record in detail the user's behavior pattern, operation time, selected intelligence content, and other information.

[0064] Step S320, constructing a selection button click rate prediction model and a push button click rate prediction model based on the intelligence collection and editing log;

[0065] The report button click rate prediction model is used to estimate the report button click rate of intelligence, and the push button click rate prediction model is used to estimate the push button click rate of intelligence;

[0066] In this step, two models are constructed based on the collected intelligence collection and editing logs:

[0067] Model for estimating the click rate of the report button: The goal of this model is to predict the likelihood of intelligence editors clicking the "report" button. The model analyzes the characteristics of intelligence content and the behavior patterns of intelligence editors to estimate the probability of each piece of intelligence being reported.

[0068] Push button click rate prediction model: Similar to the report selection model, this model predicts the probability of intelligence editors clicking the "push" button. It also analyzes intelligence content and user behavior to estimate the likelihood of intelligence being pushed.

[0069] These two models are usually constructed using machine learning or deep learning techniques, which train the models through historical data so that they can predict user behavior based on input features.

[0070] Step S330, constructing a fusion model of the output results of the selection button click rate prediction model and the output results of the push button click rate prediction model;

[0071] The fusion model is a reinforcement learning model, and is also used to adjust internal parameters based on the latest operations of intelligence editors, so that the fusion value output by the fusion model better meets the expectations of intelligence editors;

[0072] Among them, the click rate of the report button is positively correlated with the integration value, and the click rate of the push button is positively correlated with the integration value;

[0073] The fusion model combines the outputs of the report button click rate prediction model and the push button click rate prediction model to generate a comprehensive recommendation score. This model uses deep learning technology and can dynamically adjust internal parameters based on the latest operations of intelligence editors. The design of the fusion model makes the report button click rate and the push button click rate positively correlated with the final fusion value, that is, if an intelligence is predicted to be more likely to be reported or pushed, its fusion value will also increase accordingly.

[0074] Step S340: Recommend candidate intelligence based on the selection button click rate prediction model, the push button click rate prediction model and the fusion model.

[0075] In this step, the three models built earlier (the model for estimating the click rate of the report button, the model for estimating the click rate of the push button, and the fusion model) are used to recommend candidate intelligence. The system will sort all candidate intelligence according to the fusion value output by the fusion model, and give priority to recommending intelligence with high fusion values ​​to intelligence editors. In this way, intelligence editors can more efficiently see the most relevant and most likely intelligence of interest, thereby improving overall work efficiency.

[0076] In some embodiments, the report button click rate prediction model uses a dual-tower deep neural network model to predict the report button click rate; one side of the report button click rate prediction model inputs the characteristics of user operations in the intelligence collection and editing log, and the other side inputs the candidate intelligence in the intelligence collection and editing log, and the intelligence collection and editing log based on the actual collection and editing operations of the intelligence editors is trained offline in advance.

[0077] In some embodiments, the push button click rate prediction model uses a dual-tower deep neural network to estimate the push button click rate. One side inputs the features of user operations in the intelligence collection and editing log, and the other side inputs the candidate intelligence in the intelligence collection and editing log. The intelligence collection and editing log based on the actual collection and editing operations of intelligence editors is trained offline in advance.

[0078] The dual-tower deep neural network model (DNN) consists of two independent sub-networks (or "towers"), each of which processes different input data and merges the data in some way (such as dot product, concatenation, etc.) at the top layer of the network to make the final prediction. This structure is particularly suitable for processing data with two different types of features, such as user behavior features and item features.

[0079] The specific contents of the prediction model for the click rate of the report button are as follows:

[0080] Model structure: A dual-tower DNN model is used, with one side processing user operation features and the other side processing candidate intelligence features.

[0081] Input features: User operation features: including behavioral data of intelligence editors in the system, such as operation activity, intelligence style preferences, intelligence interest points, etc.

[0082] Candidate intelligence features: include various attributes of intelligence content, such as keywords, sources, release time, etc.

[0083] Training process: Using the real operation logs of intelligence editors as training data, the model learns the relationship between user behavior patterns and intelligence content through offline training, thereby predicting the probability of users clicking the "report" button.

[0084] The push button click rate estimation model is as follows:

[0085] Model structure: The dual-tower DNN model is also used, and its structure is similar to the prediction model of the click rate of the selected report button.

[0086] Input features: User operation features: Same as the selection model, including user behavior and preference information.

[0087] Candidate intelligence characteristics: Same as the selection model, including various attributes of intelligence.

[0088] Training process: Similar to the report selection model, offline training is performed using the real operation logs of intelligence editors to learn and predict the probability of users clicking the "push" button.

[0089] In some embodiments, the selection button click rate prediction model and the push button click rate prediction model are retrained based on the latest data at every preset time interval.

[0090] Regularly retraining the models for predicting the click rate of the opt-in button and the push button is a key step in ensuring that the intelligence collection and editing system remains highly accurate and relevant. The following is a detailed description of this process:

[0091] The Importance of Regular Retraining

[0092] Adaptability: The interests and behavior patterns of intelligence editors may change over time. Regularly updating the model ensures that the system adapts to these changes and maintains the accuracy of recommendations.

[0093] Data freshness: As new intelligence is constantly added and user behavior data accumulates, training models with the latest data can improve the real-time and relevance of predictions.

[0094] Performance optimization: Regular retraining can help identify and correct deviations or deficiencies that may occur in the model in real-world applications, thereby optimizing overall performance.

[0095] Steps for retraining

[0096] Data Collection:

[0097] Collect the latest intelligence collection and editing logs, including users' click behaviors, characteristics of intelligence content, etc.

[0098] Ensure the quality and integrity of data and perform necessary data cleaning and preprocessing.

[0099] Model selection:

[0100] Select appropriate machine learning algorithms and model structures, such as the dual-tower DNN model.

[0101] Adjust model parameters based on the model's performance and business needs.

[0102] Offline training:

[0103] Train the model using the latest data collected.

[0104] Perform cross-validation to evaluate the performance of the model on different data sets and ensure the generalization ability of the model.

[0105] Model Evaluation:

[0106] The performance of the model is evaluated through various metrics such as precision, recall, F1 score, etc.

[0107] Compare the performance of the old and new models to ensure that the new model improves on key metrics.

[0108] Model deployment:

[0109] Deploy the trained new model to the production environment to replace the old model.

[0110] Monitor the performance of the model in actual applications to ensure its stability and effectiveness.

[0111] Feedback loop:

[0112] Collect feedback data after model deployment, including user satisfaction with recommendation results.

[0113] Use this feedback data for the next model training to form a continuous optimization cycle.

[0114] Technical Implementation

[0115] Automated processes: The process of data collection, model training, and deployment can be performed regularly through automated tools and scripts.

[0116] Resource management: Ensure that there are sufficient computing resources to support large-scale data training and model updates.

[0117] Security and Privacy: Comply with relevant data protection regulations and privacy policies when handling user data and deploying models.

[0118] Through this regular retraining mechanism, the intelligence collection and editing system can continuously provide high-quality intelligence recommendations to meet the ever-changing needs of users, thereby improving overall work efficiency and user satisfaction.

[0119] Furthermore, in the fusion model, the T-power form is used to fuse the click rate and the duration to obtain the fused value q_final;

[0120] The fusion formula is: q_final=select_ctr T ∗ push_ctr (1-T)

[0121] The value range of T is from 0.40 to 1.0, with a step size of 0.01. select_ctr is the click rate of the select button; push_ctr is the click rate of the push button.

[0122] Fusion formula

[0123] q_final=select_ctr T ∗ push_ctr (1-T)

[0124] in:

[0125] q_final is the recommended value after fusion, which is used to determine the ranking of intelligence.

[0126] T is a parameter between 0.40 and 1.0, which is used to adjust the weight of the report button click rate and the push button click rate in the fusion value.

[0127] select_ctr is the click rate of the selection button, which indicates the possibility that the user will select a piece of information for further processing.

[0128] push_ctr is the push button click rate, which indicates the possibility that a user will push a piece of information to other users or systems.

[0129] The role of parameter T

[0130] The value of T determines the relative importance of the two click rates in the final fusion value. When T is close to 1, the click rate of the report button has a greater impact on the fusion value; when T is close to 0.4, the click rate of the push button has a greater impact.

[0131] By adjusting the value of T, the system can optimize the recommendation effect of intelligence according to different scenarios or user behavior patterns.

[0132] Value step

[0133] The value of T is set in steps of 0.01, which means that there are 60 possible values ​​of T in the range of 0.40 to 1.0. This fine-grained adjustment provides a high degree of flexibility, allowing the system to be precisely adjusted according to specific user behavior and business needs.

[0134] The implementation process is as follows:

[0135] Data collection: Collect user operation data on intelligence, including data logs of operation behaviors such as selection and push.

[0136] Model training: Use the collected data to train the prediction model for the click rate of the opt-in and push button.

[0137] Parameter adjustment: Dynamically adjust the value of T based on statistical analysis of user behavior or real-time feedback.

[0138] Fusion calculation: Calculate the q_final value of each piece of intelligence based on the adjusted T value.

[0139] Intelligence sorting: Sort candidate intelligence according to the calculated q_final value, and give priority to intelligence with high q_final value.

[0140] The solution provided by this application has the following advantages:

[0141] Flexibility: By dynamically adjusting the value of T, the system can adapt to different user behaviors and business needs.

[0142] Real-time: The recommendation strategy can be adjusted in real time based on the user’s latest operations to improve the accuracy and relevance of recommendations.

[0143] Optimization effect: Optimize the sorting and recommendation effects of intelligence by precisely controlling the weight of the selection and push button click rates.

[0144] This fusion method provides an effective tool for intelligence collection and editing systems to improve the relevance of intelligence recommendations and user satisfaction.

[0145] In some embodiments, the fusion model uses the DQN network model for online application to implement the Markov decision process and adopts --greed method to search for the best result.

[0146] The fusion model uses the Deep Q-Network (DQN) model for online application, which is a method of applying reinforcement learning to intelligence collection and editing systems. The following is a detailed explanation of this application method:

[0147] DQN is an algorithm that combines deep learning with Q learning (a simple reinforcement learning algorithm) to solve reinforcement learning problems with high-dimensional observation spaces. DQN learns through the following steps:

[0148] 1. State: In the intelligence collection and editing system, the state can be expressed as the current operating environment of the intelligence editor, including the user's basic attributes, current refresh status, and operation sequence characteristics.

[0149] 2. Action: An action is a decision that the intelligence push subsystem can take, such as choosing different \(T \) parameter values ​​to combine the selection and push button click rates.

[0150] 3. Reward: Reward is defined based on the feedback of intelligence editors on the push results, such as the number of times the "select report" or "push" button is clicked.

[0151] 4. Q-value: Q-value represents the expected utility of taking a specific action in a given state. The goal of DQN is to learn a Q function that predicts the long-term reward for each action.

[0152] Markov Decision Process (MDP)

[0153] DQN implements a Markov decision process, which is a mathematical framework for modeling the decisions of decision makers in an incompletely certain environment. In MDP, the state transition of the system has the Markov property, that is, the next state depends only on the current state and action, but not on the previous state and action sequence.

[0154] ϵ-greedy strategy

[0155] In order to strike a balance between exploration and exploitation, DQN adopts the ϵ-greedy strategy:

[0156] 1. Exploration: Randomly select an action with a small probability ϵ, which helps discover new action values.

[0157] 2. Exploitation: Select the currently estimated best action with probability 1-ϵ, which helps to exploit the known best policy.

[0158] As time goes by, the value of ϵ decreases, which means that the system will rely more on the learned policy to make decisions.

[0159] Online Application

[0160] In the intelligence collection and editing system, the online application of the DQN model usually includes the following steps:

[0161] 1. Online service request: Whenever the intelligence editor performs an operation (such as clicking the "Refresh" button), the system will request the online service of the DQN model.

[0162] 2. State input: The current state of the intelligence editor is passed as input to the DQN model.

[0163] 3. Action selection: The DQN model predicts the Q value of each possible action based on the current state and selects the action according to the ϵ-greedy strategy.

[0164] 4. Reward feedback: The feedback from intelligence editors on the push results is used as a reward signal to update the Q-value estimate of the DQN model.

[0165] 5. Model Update: The DQN model is regularly updated using the collected reward feedback to improve the accuracy of its predictions.

[0166] Through this online application, the DQN model can learn and optimize the decision-making process of the intelligence push subsystem in real time, thereby improving the efficiency and quality of intelligence collection and editing.

[0167] Furthermore, in some embodiments, the intelligence collection and editing method based on reinforcement learning further includes:

[0168] Display candidate intelligence based on editorial requests;

[0169] If the request is to sort by release time, then the candidate intelligence sorted by release time is displayed;

[0170] If the request is to sort by content similarity, then the candidate intelligence sorted by content similarity is displayed.

[0171] If the request is to sort by source importance, then candidate intelligence sorted by source importance is displayed;

[0172] If the request is to sort by intelligence, candidate intelligence sorted by integration value is displayed.

[0173] The solution provided by this application is further described below in conjunction with specific embodiments:

[0174] The solution provided by this application is described below in conjunction with specific embodiments:

[0175] Reinforcement learning is used to describe the process of how an intelligent agent learns during the interaction with the environment. For the intelligence push subsystem, the intelligence editors are the environment, and the intelligence push subsystem is the intelligent agent with strategy. Figure 4As shown, the intelligence push subsystem decides action a1 (referring to the subsystem displaying the sorted push results on the interface) to the intelligence editor in the push result refresh 1 according to the intelligence editor's status s1 (operation status, intelligence style preference, intelligence interest points, etc.). After browsing the results of this action, the intelligence editor gives a reward r1 (such as clicking the report and push buttons on the interface). The intelligence push subsystem then decides the next action a2 according to the current intelligence editor's status s2, and so on, until the intelligence editor leaves, thus completing the operation sequence of an intelligence editor. This sequence forms a Markov decision process. ; The goal of learning can be simply abstracted into finding the maximum The strategy π.

[0176] Further, see Figure 5 When the intelligence editor clicks the "Smart Sorting" button on the intelligence editing interface, the intelligence push subsystem will perform the following steps:

[0177] Step 1: Collecting feedback from intelligence editors

[0178] The characteristics of the intelligence editor's status are collected, including: basic attributes of intelligence editors, such as operational activity, intelligence style preferences, intelligence interests, etc.; refresh status of intelligence editors, such as the number of refreshes, refresh intervals, etc.; operation sequence information of intelligence editors, such as how many candidate intelligences are displayed recently, which intelligences are selected, and which intelligences are pushed.

[0179] Step 2: Operation feedback log stitching

[0180] The samples are organized at the granularity of each refresh action when the intelligence editor clicks the "Refresh" button on the interface, the characteristics of the intelligence editor's status are recorded in the system log, and the log files are processed and spliced ​​to form samples that can be used by the machine learning model.

[0181] Step 3: Estimated click-through rate of the selected button

[0182] The industry-wide dual-tower DNN (deep neural network) model is used to estimate the click rate of the report button. The name "dual towers" vividly explains the structural composition of the model. The model structures on both sides are like two tall towers, and the top interoperability layer is like an aerial corridor built by two tower spires, responsible for the communication of information on both sides. In the dual-tower model for estimating the click rate of the report button, one side inputs the operational feedback of the intelligence editors clicking the "report" button and the features obtained by splicing the logs, and the other side inputs the content features of the candidate intelligence. This model is trained offline in advance with the click logs of the actual online push results, and then applied to the online for estimation. The output is the estimated value select_ctr of the click rate of the report button of the result to be pushed. The dual-tower model is updated daily. The structure of the dual-tower model for estimating the click rate of the report button is as follows: Figure 6 shown.

[0183] Step 4: Push button click rate estimation:

[0184] The industry-wide dual-tower DNN (deep neural network) model structure is used to estimate the click rate of the push button. One side inputs the feedback of the intelligence editor clicking the "push" button and the features obtained by splicing the logs, and the other side inputs the content features of the candidate intelligence. This model is trained offline in advance with the click logs of the real online push results, and then applied to the online for estimation. The output is the estimated push button click rate value push_ctr of the push result to be pushed. The dual-tower model is updated daily. The structure of the dual-tower model for estimating the click rate of the push button is as follows: Figure 7 shown.

[0185] Step 5: Select and push the fusion value prediction

[0186] Use the T-power form to fuse the click rate of the select button and the click rate of the push button to get the fused prediction value q_final. The fusion formula is: q_final=select_ctr T ∗ push_ctr (1-T) The value range of T is from 0.40 to 1.0, and the step size is 0.01, so a total of 60 possible actions are selected as candidates.

[0187] Use the Deep Q-learning method commonly used in the industry in reinforcement learning to learn, and train a DQN network model (Deep Q-learning Network) offline in advance. The model structure is as follows Figure 8 As shown. The action reward function value It is the model's prediction of the action to be used in the current state s. The reward value that will be obtained; 𝜃 is the weight.

[0188] For each set of push results of the intelligence push subsystem, the intelligence editor will give a set of real feedback. By recording the different states of the intelligence editor, abstracting the feature representation, and then forming an operation sequence training sample with refresh as the granularity, a DQN model can be trained. The input of the model is the current state of the intelligence editor, including the intelligence style preference, intelligence interest points, refresh times, recent operation sequence information, etc. of the intelligence editor. The output of the model is the estimated reward value of the result pushed to the intelligence editor by each candidate action in this state. In this way, by traversing the output of all candidate actions of the DQN model, taking the candidate action corresponding to the maximum reward value as the T fusion parameter of this push, and acting on the line, a closed loop of continuous learning and continuous optimization is formed. Of course, in order to ensure the exploration of unknown possibilities, a 1% probability will be used for random selection.

[0189] The DQN network model uses a three-layer network structure. The last layer is a fully connected layer with 60 output nodes, corresponding to the estimated reward value of each action. The model organizes samples at a refresh granularity, first randomly selects T samples, does not use a simulator, uses the industry's commonly used Off-policy algorithm for training, and updates the model every day.

[0190] Reward design:

[0191] Current reward value Immediate Reward(t) = ln(1 + total number of clicks on the refresh selection button at time t * total number of clicks on the refresh push button at time t)

[0192] Total Reward(t) = Immediate Reward(t) + r * ImmediateReward(t+1) + r 2 * Immediate Reward(t+2) + r 3 * Immediate Reward(t+3) + …

[0193] The attenuation coefficient is r = 0.5

[0194] The online working mode of DQN network model:

[0195] Every time the intelligence editor clicks the "Refresh" button, the intelligence push subsystem requests the online service of the DQN network model. The online service calculates the reward value (i.e., the reward value for each action) corresponding to all candidate actions (60 candidates) in this state based on the basic attributes of the intelligence editor, the current refresh status, and the operation sequence characteristics through the DQN network model. The estimated value of the corresponding reward , and then use the industry's commonly used method (i.e., 1% probability of randomly selecting T, 99% probability of selecting the maximum estimated reward value) Corresponding T) returns the T value to the intelligence push subsystem for fusion q_final=select_ctr T ∗ push_ctr (1-T) .

[0196] Step 6: Overall ranking of candidate intelligence

[0197] The intelligence push subsystem performs overall sorting of the intelligence to be pushed based on the predicted value q_final which is a fusion of the estimated click rate of the report button and the estimated click rate of the push button, and then inserts the overall sorted results into the cache queue of the intelligence push subsystem's results to be pushed.

[0198] Step 7: Candidate intelligence refresh push

[0199] The intelligence editor clicks the refresh button on the intelligence editing interface to perform a refresh operation, and the intelligence push subsystem takes the result from the cache queue of the results to be pushed and displays it to the intelligence editor. Then repeat step 1.

[0200] Exemplary Devices

[0201] The device embodiments of the present application can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0202] Fig. 9 FIG. 1 is a block diagram of an intelligence collection and editing device based on reinforcement learning provided by an embodiment of the present application. Fig. 9 As shown, the device comprises:

[0203] An acquisition module 91 is used to acquire intelligence collection and editing logs;

[0204] A construction module 92 is used to construct a selection button click rate prediction model and a push button click rate prediction model based on the intelligence collection and editing log; wherein the selection button click rate prediction model is used to estimate the selection button click rate of intelligence, and the push button click rate prediction model is used to estimate the push button click rate of intelligence; a fusion model of the output results of the selection button click rate prediction model and the output results of the push button click rate prediction model is constructed; wherein the fusion model is a reinforcement learning model, and is also used to adjust internal parameters based on the latest operations of intelligence collection and editing personnel, so that the fusion value output by the fusion model is more in line with the expectations of intelligence collection and editing personnel; wherein the selection button click rate is positively correlated with the fusion value, and the push button click rate is positively correlated with the fusion value;

[0205] The recommendation module 93 is used to recommend candidate intelligence based on the selection button click rate prediction model, the push button click rate prediction model and the fusion model.

[0206] Exemplary Electronic Devices

[0207] Below, reference Fig.10 To describe an electronic device according to an embodiment of the present application. Fig.10 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0208] like Fig.10 As shown, electronic device 1000 includes one or more processors 1010 and memory 1020 .

[0209] The processor 1010 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1000 to perform desired functions.

[0210] The memory 1020 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1010 may run the program instructions to implement the intelligence collection and editing method based on reinforcement learning and / or other desired functions of the various embodiments of the present application described above. Various contents such as category correspondences may also be stored in the computer-readable storage medium.

[0211] In one example, the electronic device 1000 may further include: an input device 1030 and an output device 1040 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0212] In addition, the input device 1030 may also include, for example, a keyboard, a mouse, an interface, etc. The output device 1040 may output various information to the outside, including analysis results, etc. The output device 1040 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0213] Of course, to simplify, Fig.10Only some of the components in the electronic device related to the present application are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application conditions.

[0214] Exemplary computer program products and computer-readable storage media

[0215] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the intelligence collection method based on reinforcement learning according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0216] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0217] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the intelligence collection and editing method based on reinforcement learning according to various embodiments of the present application described in the above “Exemplary Method” section of this specification.

[0218] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0219] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for collecting and compiling intelligence based on reinforcement learning, characterized in that: include: Obtain intelligence collection and editing logs; Based on the intelligence collection and editing log, a selection button click rate prediction model and a push button click rate prediction model are respectively constructed; The report button click rate prediction model is used to estimate the report button click rate of intelligence, and the push button click rate prediction model is used to estimate the push button click rate of intelligence; Construct a fusion model of the output results of the selection button click rate estimation model and the push button click rate estimation model; The fusion model is a reinforcement learning model, and is also used to adjust internal parameters based on the latest operations of intelligence editors, so that the fusion value output by the fusion model better meets the expectations of intelligence editors; Among them, the click rate of the report button is positively correlated with the integration value, and the click rate of the push button is positively correlated with the integration value; Recommend candidate intelligence based on the selection button click rate prediction model, the push button click rate prediction model and the fusion model; In the fusion model, the click rate and duration are fused in the form of T power to obtain the fused value q_final; The fusion formula is: q_final=select_ctr T push_ctr (1-T) The value range of T is from 0.40 to 1.0, with a step size of 0.

01. select_ctr is the click rate of the select button; push_ctr is the click rate of the push button.

2. The intelligence collection and editing method based on reinforcement learning according to claim 1 is characterized in that: The model for estimating the click rate of the report button uses a dual-tower deep neural network model to estimate the click rate of the report button. One side of the model for predicting the click rate of the reporting button is input with the features of user operations in the intelligence collection and editing log, and the other side is input with the candidate intelligence in the intelligence collection and editing log. The intelligence collection and editing log based on the actual collection and editing operations of intelligence editors is trained offline in advance.

3. The intelligence collection and editing method based on reinforcement learning according to claim 1 is characterized in that: The push button click rate prediction model adopts a dual-tower deep neural network to estimate the push button click rate. One side inputs the features of user operations in the intelligence collection and editing log, and the other side inputs the candidate intelligence in the intelligence collection and editing log. The intelligence collection and editing log based on the actual collection and editing operations of intelligence editors is trained offline in advance.

4. The intelligence collection and editing method based on reinforcement learning according to claim 1 is characterized in that: At preset intervals, the click-through rate prediction model for the opt-in button and the click-through rate prediction model for the push button are retrained based on the latest data.

5. The intelligence collection and editing method based on reinforcement learning according to claim 1 is characterized in that: Also includes: The fusion model adopts the DQN network model for online application, realizes the Markov decision process, and adopts --greed method to search for the best result.

6. The intelligence collection and editing method based on reinforcement learning according to claim 1 is characterized in that: Get requests and display candidate intelligence based on the requests; If the request is to sort by release time, then the candidate intelligence sorted by release time is displayed; If the request is to sort by content similarity, then the candidate intelligence sorted by content similarity is displayed. If the request is to sort by source importance, then candidate intelligence sorted by source importance is displayed; If the request is to sort by intelligence, candidate intelligence sorted by integration value is displayed.

7. An intelligence collection and editing device based on reinforcement learning, characterized in that: include: The acquisition module is used to obtain intelligence collection and editing logs; A construction module is used to construct a selection button click rate prediction model and a push button click rate prediction model based on the intelligence collection and editing log; wherein the selection button click rate prediction model is used to estimate the selection button click rate of intelligence, and the push button click rate prediction model is used to estimate the push button click rate of intelligence; a fusion model of the output results of the selection button click rate prediction model and the output results of the push button click rate prediction model is constructed; wherein the fusion model is a reinforcement learning model, and is also used to adjust internal parameters based on the latest operations of intelligence collection and editing personnel, so that the fusion value output by the fusion model is more in line with the expectations of intelligence collection and editing personnel; wherein the selection button click rate is positively correlated with the fusion value, and the push button click rate is positively correlated with the fusion value; A recommendation module, used for recommending candidate intelligence based on the click rate prediction model of the report button, the click rate prediction model of the push button and the fusion model; In the fusion model, the click rate and duration are fused in the form of T power to obtain the fused value q_final; The fusion formula is: q_final=select_ctr T push_ctr (1-T) The value range of T is from 0.40 to 1.0, with a step size of 0.

01. select_ctr is the click rate of the select button; push_ctr is the click rate of the push button.

8. An electronic device, characterized in that: include: A processor, and a memory for storing a program executable by the processor; The processor is used to implement the intelligence collection and editing method based on reinforcement learning as described in any one of claims 1 to 6 by running the program in the memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to execute the intelligence collection and editing method based on reinforcement learning as described in any one of claims 1 to 6.

Images