Product recommendation method and device based on burying point data, electronic equipment and medium

Abstract

The invention provides a product recommendation method and device based on burying point data, electronic equipment and a medium, and can be applied to the technical field of big data. The method comprises the following steps: acquiring interface burying point data authorized by a user; first data processing is conducted on the interface burying point data, preference marks are obtained, the preference marks comprise N user preference area marks, M user preference time marks and L user preference product marks, and N, M and L are all positive integers; based on the preference mark, a Q-learning algorithm is used to acquire a maximum Q value combination; and recommending products to the user according to the mark information in the maximum Q value combination. According to the method provided by the invention, the customer product suitable for the user can be autonomously found through the behavior of the user, the unique interface of the user is obtained, and the satisfaction degree of the user can be greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and more specifically to a product recommendation method, device, electronic device and medium based on buried data. Background Art

[0002] Nowadays, national-level banking apps have hundreds of millions of customers, with comprehensive financial services and rich product types. However, the information carrying space of the interface is limited. In order to highlight key products, the design trend has gradually changed from the traditional full-bar list display to a lightweight card shelf that displays a small number of selected products. However, due to the complexity of financial scenarios, the types of customers and conditions for handling different products are different. Only by placing the most suitable financial products for customers on the top of the shelf can we more actively promote conversion and provide customers with more intimate and intelligent services.

[0003] At present, through the analysis of conventional financial product recommendation methods in the market, it is found that product recommendations are mostly based on customer segmentation, and the data sources of customer segmentation are mostly basic customer information, user assets, etc. However, the recommendation effect based on customer segmentation is not ideal, and it often fails to accurately capture user preferences and needs, resulting in inaccurate recommendation results and poor user experience. Summary of the invention

[0004] In view of the above problems, according to a first aspect of the present invention, a product recommendation method based on buried point data is provided, characterized in that the method includes: obtaining interface buried point data authorized by the user; performing a first data processing on the interface buried point data to obtain preference tags, wherein the preference tags include N user preference area tags, M user preference time tags and L user preference product tags, and N, M, and L are all positive integers; based on the preference tags, using a Q-learning algorithm to obtain a maximum Q value combination; and making product recommendations to the user according to the tag information in the maximum Q value combination.

[0005] According to some exemplary embodiments, before obtaining the interface tracking data authorized by the user, the method also includes: designing tracking on the front-end interface, including tracking clicks on the navigation bar, buttons and partition modules; and tracking clicks, browsing and application operations for each product and / or service.

[0006] According to some exemplary embodiments, the first data processing of the interface buried data specifically includes: obtaining the area click ratio according to the number of clicks on the navigation bar, buttons and partition modules, and selecting N user preference area tags based on the area click ratio; obtaining the user product views according to the clicks, browsing and application operations on each product and / or service, and selecting L user preference product tags based on the user product views; and obtaining the user activity according to the number of clicks on the navigation bar, buttons and partition modules and the clicks, browsing and application operations on each product and / or service, and selecting M user preference time tags based on the user activity.

[0007] According to some exemplary embodiments, the method of obtaining the maximum Q value combination based on the preference mark using the Q-leaming algorithm specifically includes: establishing a state space and a behavior space based on the preference mark; defining a reward function as a reward value corresponding to executing a behavior in the behavior space, wherein the reward value is determined based on the number of user clicks or user satisfaction; randomly selecting an initial state s and the next behavior a based on the state space and the behavior space to obtain an initial Q value; using the Q-learning algorithm to update the rules to gradually update the Q value until the maximum Q value is obtained; and taking the state corresponding to the maximum Q value as the maximum Q value combination.

[0008] According to some exemplary embodiments, the Q-leaming algorithm update rule is used to gradually update the Q value until the maximum Q value is obtained, specifically including: an environment interaction step: the agent performs the selected behavior a, interacts with the environment, and observes the obtained reward value and the next state s'; a target Q value calculation step: calculates the target Q value of the agent selecting behavior a' in the next state s'; a Q value update step: based on the target Q value, updates the Q value; and repeats the environment interaction step, the target Q value calculation step and the Q value update step until the maximum Q value is obtained.

[0009] According to some exemplary embodiments, before gradually updating the Q value using the Q-learning algorithm update rule, it also includes setting an initial ε value, wherein the selecting behavior a' includes: selecting behavior a' according to an ε-greedy strategy.

[0010] According to some exemplary embodiments, selecting behavior a' according to the ε-greedy strategy specifically includes: generating a random number in the range of [0, 1); if the random number is less than the ε value, selecting a random behavior in the behavior space as behavior a'; and if the random number is greater than the ε value, selecting the behavior with the largest current Q value as behavior a'.

[0011] According to some exemplary embodiments, after the Q value updating step, the method further comprises: updating the ε value by using exponential decay or linear decay.

[0012] According to a second aspect of the present invention, a product recommendation device based on buried point data is proposed, and the device includes: an interface buried point data acquisition module, used to: acquire interface buried point data authorized by the user; a tag acquisition module, used to: perform a first data processing on the interface buried point data to obtain preference tags, wherein the preference tags include N user preference area tags, M user preference time tags and L user preference product tags, and N, M, and L are all positive integers; a maximum Q value combination acquisition module, used to: based on the preference tags, use the Q-leaming algorithm to obtain the maximum Q value combination; and a recommendation module, used to: recommend products to the user according to the tag information in the maximum Q value combination.

[0013] According to some exemplary embodiments, the product recommendation device based on embedded point data may also include a embedded point design unit.

[0014] According to some exemplary embodiments, the tracking point design unit can be used to design tracking points on the front-end interface, including tracking points for clicks on navigation bars, buttons, and partition modules; and tracking points for clicks, browsing, and application operations for each product and / or service.

[0015] According to some exemplary embodiments, the tag acquisition module may include a user preference area tag selection unit, a user preference product tag selection unit, and a user preference time tag selection unit.

[0016] According to some exemplary embodiments, the user preference area mark selection unit may be used to obtain an area click ratio according to the number of clicks on the navigation bar, button and partition module, and select N user preference area marks based on the area click ratio.

[0017] According to some exemplary embodiments, the user preference product tag selection unit can be used to obtain user product views based on the click, browse and application operations on each product and / or service, and select L user preference product tags based on the user product views.

[0018] According to some exemplary embodiments, the user preference time stamp selection unit can be used to obtain user activity based on the number of clicks on the navigation bar, buttons and partition modules, and the clicks, browsing and application operations on each product and / or service, and select M user preference time stamps based on the user activity.

[0019] According to some exemplary embodiments, the maximum Q-value combination acquisition module may include a space establishment unit, a reward function definition unit, an initial Q-value acquisition unit, a maximum Q-value acquisition module and a maximum Q-value combination acquisition unit.

[0020] According to some exemplary embodiments, the space establishing unit may be configured to establish a state space and a behavior space based on the preference tags.

[0021] According to some exemplary embodiments, the reward function definition unit may be used to define a reward function as a reward value corresponding to the execution action, wherein the reward value is determined according to the number of user clicks or user satisfaction.

[0022] According to some exemplary embodiments, the initial Q value acquisition unit may be used to randomly select an initial state s and a next behavior a based on the state space and the behavior space to obtain an initial Q value.

[0023] According to some exemplary embodiments, the maximum Q value acquisition module may be used to gradually update the Q value using a Q-learning algorithm update rule until the maximum Q value is acquired.

[0024] According to some exemplary embodiments, the maximum Q value combination acquisition unit may be configured to take the state corresponding to the maximum Q value as the maximum Q value combination.

[0025] According to some exemplary embodiments, the maximum Q value acquisition module may include an environment interaction unit, a target Q value calculation module, a Q value updating unit and a maximum Q value acquisition unit.

[0026] According to some exemplary embodiments, the environment interaction unit may be used in an environment interaction step: the agent performs a selected behavior a, interacts with the environment, and observes the obtained reward value and the next state s'.

[0027] According to some exemplary embodiments, the target Q value calculation module may be used in a target Q value calculation step: calculating the target Q value of the agent selecting behavior a' in the next state s'.

[0028] According to some exemplary embodiments, the Q value updating unit may be used in a Q value updating step: updating the Q value based on the target Q value.

[0029] According to some exemplary embodiments, the maximum Q value acquisition unit may be used to repeat the environment interaction step, the target Q value calculation step and the Q value update step until the maximum Q value is acquired.

[0030] According to some exemplary embodiments, the target Q value calculation module may include an initial ε value setting module.

[0031] According to some exemplary embodiments, the initial ε value setting module may be used to further set an initial ε value before gradually updating the Q value using the Q-learning algorithm update rule, wherein the selecting behavior a' includes: selecting behavior a' according to an ε-greedy strategy.

[0032] According to some exemplary embodiments, the initial ε value setting module may include a random number generating unit, a first selecting unit and a second selecting unit.

[0033] According to some exemplary embodiments, the random number generation unit may be configured to generate a random number in the range of [0, 1).

[0034] According to some exemplary embodiments, the first selection unit may be configured to select a random behavior in the behavior space as the behavior a′ if the random number is smaller than the ε value.

[0035] According to some exemplary embodiments, the second selection unit may be configured to select a behavior with the largest current Q value as behavior a' if the random number is greater than the ε value.

[0036] According to a third aspect of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method as described above.

[0037] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method as described above.

[0038] According to a fifth aspect of the present invention, there is provided a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method as described above is implemented.

[0039] One or more of the above embodiments have the following advantages or beneficial effects: by analyzing the user's embedded data in the application interface, it is possible to gain an in-depth understanding of the user's behavior and interests, thereby achieving personalized recommendations; at the same time, the Q-leaming algorithm can continuously optimize and recommend strategies based on the user's preferences and behavior patterns, so that the recommendation results are more in line with the user's needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0041] Figure 1The application scenario diagram of the product recommendation method, device, equipment, and medium based on buried point data according to an embodiment of the present invention is schematically shown.

[0042] Figure 2 The flowchart of the product recommendation method based on buried point data according to an embodiment of the present invention is schematically shown.

[0043] Figure 3 A flowchart of a method for designing tracking points in a front-end interface according to an embodiment of the present invention is schematically shown.

[0044] Figure 4 The flowchart of the first data processing method according to the embodiment of the present invention is schematically shown.

[0045] Figure 5 The flowchart of the method for obtaining the maximum Q value combination by using the Q-leaming algorithm according to an embodiment of the present invention is schematically shown.

[0046] Figure 6 The flowchart of the method for gradually updating the Q value by using the Q-leaming algorithm update rule according to an embodiment of the present invention is schematically shown.

[0047] Figure 7 The flowchart of the method for introducing the ε-greedy strategy according to an embodiment of the present invention is schematically shown.

[0048] Figure 8 The flowchart of the method for selecting behavior a' according to an embodiment of the present invention is schematically shown.

[0049] Fig. 9 The structural block diagram of a product recommendation device based on buried point data according to an embodiment of the present invention is schematically shown.

[0050] Fig.10 A block diagram of an electronic device suitable for a product recommendation method based on buried point data according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0051] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0052] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0053] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0054] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0055] In the technical solution of the present invention, the acquisition, storage and application of user personal information involved all comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0056] First, the technical terms described in this article are explained and described as follows.

[0057] Q-learning: A reinforcement learning algorithm used to train an agent to make decisions in an unknown environment to maximize rewards. The goal of Q-learning is to learn a strategy that enables the agent to choose the best action in a given environment to maximize the cumulative reward. In Q-leaning, the agent achieves this goal by maintaining a Q-value function (or Q-table).

[0058] Q-value function: represents the expected reward for taking an action in a given state. By continuously interacting with the environment and trying different actions, the agent updates the Q-value function so that the Q-value gradually approaches the optimal value.

[0059] Tracking: Inserting code or tags into software or applications to collect specific behavior data of users on the interface. This behavior data may include various operations such as user clicks, browsing, searching, and purchasing.

[0060] ε-greedy strategy: A probability-based action selection method to balance the trade-off between exploring unknown actions and exploiting known actions.

[0061] At present, for the intelligent layout and product placement of front-end interface shelves, the widely used recommendation algorithms can be roughly divided into the following three directions.

[0062] The first is the content-based recommendation algorithm, which recommends relevant content based on the content that the user is interested in.

[0063] The second is the recommendation algorithm based on collaborative filtering. This method is based on the assumption that users with the same preferences will have the same interest bias, and uses scores to characterize users' preferences for different products. Collaborative filtering algorithms are also divided into two types: one is user-based and the other is item-based. The principle of user-based collaborative filtering is to find neighboring user groups with similar preferences to the current user based on the user's preference for items. The knn algorithm is generally used to make recommendations for the current user based on the historical preference information of these k neighbors. The advantage of this recommendation is that there is no need to establish a connection between the recommended items. The items may be completely unrelated, but the user's potential interests can be discovered, and personalized recommendation results can be generated for each user; its disadvantage is that the growth rate of users will be much faster than the growth rate of products, and when it reaches a certain level, it will become a bottleneck for product recommendations. The recommendation based on items is very similar to the user-based recommendation. It is based on the user's preference for items, and then finds the similarity between items, and then recommends similar items to users. This method has a high recommendation accuracy and improved coverage. Its performance will not fluctuate too much as the number of users increases, but it cannot provide users with personalized recommendation results.

[0064] Finally, there is the recommendation algorithm based on association rules. This method is common in e-commerce systems. The primary goal of the recommendation system based on association rules is to mine association rules, that is, the collection of items that are purchased by many users at the same time. The items in these collections can recommend each other. This method has a large amount of calculation, and there is also the possibility that popular products will be over-recommended.

[0065] However, after analyzing the conventional financial product recommendation methods on the market, it was found that product recommendations are mostly based on customer segmentation, and the data sources for customer segmentation are mostly basic customer information, user assets, etc., and the recommendation effect is not ideal and cannot meet user needs.

[0066] Based on this, an embodiment of the present invention provides a product recommendation method based on buried point data, characterized in that the method includes: obtaining interface buried point data authorized by the user; performing a first data processing on the interface buried point data to obtain preference tags, wherein the preference tags include N user preference area tags, M user preference time tags and L user preference product tags, and N, M, and L are all positive integers; based on the preference tags, using the Q-leaming algorithm to obtain the maximum Q value combination; and recommending products to the user according to the tag information in the maximum Q value combination. According to the product recommendation method based on buried point data provided by the present invention, by analyzing the buried point data of the user in the application interface, the user's behavior and interests can be deeply understood, thereby realizing personalized recommendations; at the same time, the Q-learning algorithm can continuously optimize and recommend strategies according to the user's preferences and behavior patterns, so that the recommendation results are more in line with the user's needs.

[0067] It should be noted that the product recommendation method, device, equipment and medium based on buried data determined by the present invention can be used in the field of big data technology, can also be used in the financial field, and can also be used in various fields other than the field of big data technology and the financial field. The application field of the product recommendation method, device, equipment and medium based on buried data provided by the embodiments of the present invention is not limited.

[0068] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0069] Figure 1 The application scenario diagram of the product recommendation method, device, equipment, and medium based on buried point data according to an embodiment of the present invention is schematically shown.

[0070] like Figure 1 As shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0071] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0072] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0073] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0074] It should be noted that the product recommendation method based on buried data provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the product recommendation device based on buried data provided in the embodiment of the present invention can generally be set in the server 105. The product recommendation method based on buried data provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Correspondingly, the product recommendation device based on buried data provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0075] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0076] Figure 2 The flowchart of the product recommendation method based on buried point data according to an embodiment of the present invention is schematically shown.

[0077] like Figure 2 As shown, the product recommendation method 200 based on buried point data of this embodiment may include operations S210 to S240.

[0078] In operation S210, the interface tracking data authorized by the user is obtained.

[0079] In an embodiment of the present invention, the interface tracking data is represented as data of specific behaviors in an application or website, and may include user activities such as clicks, browsing, searches, purchases, and comments.

[0080] In an embodiment of the present invention, before collecting user interface embedding data, the purpose of collecting data should be clarified to ensure that the data collection process is legal. The user should be clearly aware of the scope and purpose of data collection and have the right to choose whether to authorize.

[0081] Figure 3 A flowchart of a method for designing tracking points in a front-end interface according to an embodiment of the present invention is schematically shown.

[0082] like Figure 3 As shown, the method for designing a tracking point in the front-end interface of this embodiment may include operation S310.

[0083] In operation S310, tracking points are designed on the front-end interface, including tracking points for clicks on the navigation bar, buttons, and partition modules; and tracking points for clicks, browsing, and application operations for each product and / or service.

[0084] In an embodiment of the present invention, the design of tracking points can identify key interactive elements, including navigation bars, buttons, and partition modules. These elements are parts that users often interact with applications. Recording these interactions can provide important user behavior data.

[0085] Furthermore, for the identified key interactive elements, embedding codes can be embedded in their corresponding code positions. The embedding codes are usually JavaScript code segments used to capture user clicks and interactive behaviors.

[0086] In an embodiment of the present invention, corresponding tracking events can be defined for each product and / or service. Tracking events can be user behaviors such as clicks, browsing, and applications. Each tracking event can be associated with a specific product or service. For tracking events, corresponding parameters can also be set to record relevant information. For example, the product ID, click timestamp, product category, and other information clicked by the user are recorded, and this information will be used for subsequent data analysis.

[0087] In an embodiment of the present invention, a tracking point is designed in the front-end interface, which can accurately record data when the user performs specific operations. Compared with recording data through the back-end, the front-end tracking point can obtain user behavior information more accurately and avoid data loss or inaccuracy due to network delays and other reasons.

[0088] Return to reference Figure 2In operation S220, the interface buried point data is subjected to a first data processing to obtain preference tags, wherein the preference tags include N user preference area tags, M user preference time tags and L user preference product tags, and N, M and L are all positive integers.

[0089] In an embodiment of the present invention, in order to provide a more comprehensive and detailed user information portrait, a first data processing operation may be performed on the interface buried data to obtain the user's preferences in three dimensions: different interface areas, different time periods, and different products.

[0090] Figure 4 The flowchart of the first data processing method according to the embodiment of the present invention is schematically shown.

[0091] like Figure 4 As shown, the first data processing method of this embodiment may include operations S410 to S430.

[0092] In operation S410, according to the number of clicks on the navigation bar, the button, and the partition module, the area click ratio is obtained, and based on the area click ratio, N user preference area marks are selected.

[0093] In an embodiment of the present invention, the number of clicks of users on each area can be collected. By statistics and calculation, the ratio of the number of clicks of each area to the total number of clicks can be obtained as the area click ratio. Further, the area click ratio can be sorted and the top N user preference area labels can be selected. For example, the top three areas in the area click ratio statistics are selected as customer favorite areas and are marked as A1, A2, and A3.

[0094] In operation S420, the user product views are obtained according to the click, browse and apply operations on each product and / or service, and L user preferred product tags are selected based on the user product views.

[0095] In an embodiment of the present invention, the number of views of users on each product and / or service can be collected. The number of views can refer to the sum of the number of clicks, views and applications of each product and / or service by the user. Further, the views can be sorted according to the number of views, and the top L user preference product labels can be selected. For example, the top three products and / or services with the highest number of views are selected as the three categories of products that customers pay the most attention to, and are marked as P1, P2, and P3.

[0096] In operation S430, the user activity is obtained according to the number of clicks on the navigation bar, buttons and partition modules, and the clicks, browsing and application operations on each product and / or service, and M user preference time tags are selected based on the user activity.

[0097] In an embodiment of the present invention, the number of clicks of users on each area and the number of views of users on each product and / or service can be collected. These data can be expressed as user activity, that is, the frequency of interaction between the user and the application or website within a certain time range.

[0098] In an embodiment of the present invention, the user's activity at 24 time points in a day can be counted from 0:00 to 24:00, with 1 hour as the time unit. Further, the activity can be sorted and the top M user preference time tags can be selected. For example, the two time points with the highest user activity in a day can be selected as the customer time preference, marked as T1 and T2.

[0099] Return to reference Figure 2 In operation S230, based on the preference labels, a maximum Q value combination is obtained using a Q-learning algorithm.

[0100] According to the embodiment of the present invention, in order to gradually learn the strategy of taking the best action in different states, so as to accurately match the user's preference, the Q-learning algorithm is further adopted. The optimization process by the Q-learning algorithm will be specifically analyzed below.

[0101] Figure 5 The flowchart of the method for obtaining the maximum Q value combination by using the Q-learning algorithm according to an embodiment of the present invention is schematically shown.

[0102] like Figure 5 As shown, the method of using the Q-learning algorithm to obtain the maximum Q value combination in this embodiment may include operations S510 to S550.

[0103] In operation S510, a state space and a behavior space are established based on the preference tags.

[0104] In an embodiment of the present invention, the state space is a set consisting of user preference tags, wherein each state includes one of three preference tags, such as (A1, T1, P1); the behavior space is a set consisting of all operations or behaviors that can be executed by the system, wherein each selection of a preference tag is regarded as a behavior a.

[0105] In operation S520, a reward function is defined as a reward value corresponding to executing the behavior in the behavior space, wherein the reward value is determined according to the number of user clicks or user satisfaction.

[0106] In an embodiment of the present invention, the reward function can determine the reward value based on the user's behavioral feedback. Specifically, it can be determined based on the number of user clicks or satisfaction, where the user's satisfaction can be defined by behaviors such as purchases and applications. This can encourage the system to learn to make more accurate recommendations for products that the user is interested in.

[0107] In the embodiment of the present invention, the reward value given by the reward function should have a clear range. Take the reward value of the preferred area determined according to the number of user clicks as an example. If the number of clicks on areas A1, A2, and A3 is 3, 2, and 1 respectively, then A1 can be determined to be 3 points, A2 to be 2 points, and A3 to be 1 point.

[0108] In operation S530, based on the state space and the behavior space, an initial state s is randomly selected, and a next behavior a is selected to obtain an initial Q value.

[0109] In an embodiment of the present invention, in order to be able to explore in state space and behavior space, an initial state s can be randomly selected, such as (A1, T1, P1), and the next behavior a is selected. In this state, there is an initial Q value, denoted as Q(s, a).

[0110] In operation S540, the Q value is gradually updated using a Q-learning algorithm update rule until a maximum Q value is obtained.

[0111] Figure 6 The flowchart of the method for gradually updating the Q value by using the Q-learning algorithm update rule according to an embodiment of the present invention is schematically shown.

[0112] like Figure 6 As shown, the method of gradually updating the Q value by using the Q-learning algorithm update rule in this embodiment may include operations S610 to S640.

[0113] In operation S610, the environment interaction step: the agent performs the selected behavior a, interacts with the environment, and observes the obtained reward value and the next state s'.

[0114] In an embodiment of the present invention, the environmental interaction step may include: the intelligent agent makes recommendations to the user based on the selected behavior a, and the user performs corresponding feedback on the recommended content based on the recommendation result, such as clicking on the recommended area; the environment (application or web page) receives the user's feedback behavior, calculates and returns the reward value and the next state s'.

[0115] In operation S620, the target Q value calculation step is to calculate the target Q value of the agent selecting behavior a' in the next state s'.

[0116] In the embodiment of the present invention, it is necessary to select the behavior a' in the next state s'. In order to further find the optimal recommendation method, an ε-greedy strategy can be introduced.

[0117] Figure 7 The flowchart of the method for introducing the ε-greedy strategy according to an embodiment of the present invention is schematically shown.

[0118] like Figure 7 As shown, the method for introducing the ε-greedy strategy of this embodiment may include operation S710.

[0119] In operation S710, before the Q value is gradually updated by using the Q-learning algorithm update rule, an initial ε value is set, wherein the selecting behavior a' includes: selecting behavior a' according to an ε-greedy strategy.

[0120] In the embodiment of the present invention, ε is the exploration rate, which is used to control the probability of exploring new behaviors in the recommendation process. Before the Q-learning algorithm starts to update, the initial ε value needs to be set. Usually, the initial ε value is a large value, such as 0.8 or 0.9, indicating that the system is more inclined to explore new behaviors at the beginning; as the algorithm executes, the ε value will gradually decrease, and the system is more inclined to use the learned optimal strategy.

[0121] Figure 8 The flowchart of the method for selecting behavior a' according to an embodiment of the present invention is schematically shown.

[0122] like Figure 8 As shown, the method for selecting behavior a' in this embodiment may include operations S810 to S830.

[0123] In operation S810, a random number is generated within the range of [0, 1).

[0124] In operation S820, if the random number is smaller than the ε value, a random behavior is selected in the behavior space as behavior a'.

[0125] In an embodiment of the present invention, if the random number is less than the ε value, a behavior a' is randomly selected with a probability of ε, which means that the system has a probability of ε to select a random behavior. This exploration helps to discover potential high-quality recommendations, especially in the initial stage of the algorithm or when encountering unknown situations.

[0126] In operation S830, if the random number is greater than the ε value, the behavior with the largest current Q value is selected as behavior a'.

[0127] In an embodiment of the present invention, if the random number is greater than the ε value, the behavior a' with the maximum Q value is selected with a probability of 1-ε, so as to utilize the optimal strategy learned in the current state. This means that the system has a probability of 1-ε to select the behavior that can bring the greatest benefit in the current state. Such a utilization process helps the system to maintain a relatively stable recommendation strategy and avoid excessive exploration that may lead to system instability.

[0128] According to the embodiment of the present invention, the ε-greedy strategy is used to select the behavior a', which can maintain stability and accuracy in the recommendation process and help improve the adaptability of the system so that it can cope with the ever-changing user needs.

[0129] Return to reference Figure 6 , in operation S630, Q value updating step: based on the target Q value, updating the Q value.

[0130] In an embodiment of the present invention, the Q value may be updated according to the following formula:

[0131] Q(s,a)←Q(s,a)+α[r+γmaxQ(s,a 1 )-Q(s,a)] (1)

[0132] Among them, α is the learning rate, which determines how much of the error is to be learned this time; γ is the attenuation of the expected result in the future (ie, Q value); r is the reward value.

[0133] In an embodiment of the present invention, after the Q value updating step, the method further comprises: updating the ε value by using exponential decay or linear decay.

[0134] In an embodiment of the present invention, the process of updating the ε value is intended to gradually reduce the degree of exploration, so that in the process of interaction between the agent and the environment, the chance of random exploration is gradually reduced, and the agent is more inclined to utilize the learned optimal strategy.

[0135] In the embodiment of the present invention, exponential decay is a commonly used method to reduce the ε value in the form of an exponential function. Generally, the ε value decreases in the form of an exponential function after each Q value update, for example, by the following formula:

[0136] ε=ε*deacy_factor (2)

[0137] Where deacy_factor is the attenuation factor, which is usually less than 1.

[0138] In the embodiment of the present invention, linear decay is another commonly used method, which uses a linear function to reduce the ε value. Unlike exponential decay, the linear decay process is relatively stable. Generally, the ε value decreases linearly after each Q value update, for example, by the following formula:

[0139] ε=ε-deacy_step (3)

[0140] Among them, deacy_step is the attenuation step size.

[0141] In operation S640, steps S610-S630 are repeated until a maximum Q value is obtained.

[0142] In an embodiment of the present invention, the environment interaction step, the target Q value calculation step and the Q value update step are performed in a cycle. Through continuous learning and optimization, the Q value table will gradually converge to the optimal value, representing the expected benefit of performing the best behavior in each state.

[0143] Return to reference Figure 5 , in operation S550, the state corresponding to the maximum Q value is used as the maximum Q value combination.

[0144] According to an embodiment of the present invention, the Q-learning algorithm can find the preference tag combination with the most clicks or the highest customer satisfaction based on the actual behavior data of the customer, thereby achieving more personalized recommendations. This means that recommendations can be made based on the specific preferences and needs of each customer, thereby improving the accuracy of the recommendations and user satisfaction.

[0145] Return to reference Figure 2 In operation S240, product recommendations are made to the user according to the tag information in the maximum Q value combination.

[0146] In an embodiment of the present invention, the system can update the recommended content displayed on the UI interface based on the tag information in the maximum Q value combination, including displaying products that may be of interest in the user preference area on the UI interface during the corresponding user active period for user viewing and selection.

[0147] The product recommendation method based on buried data provided by the present invention overcomes the shortcomings that current product recommendations are mostly based on customer segmentation, and the data sources for customer segmentation are mostly basic customer information, user assets, etc., and the recommendation effect is not ideal. By focusing on the in-depth analysis of user buried data, learning user behavior through buried data, and finding the preference tag combination with the most clicks or the highest customer satisfaction through the Q-learning algorithm, it is possible to recommend the most suitable products to users and update the user's UI interface at the same time. Specifically, it brings the following beneficial effects:

[0148] 1. Through in-depth analysis of user tracking data, user preferences are counted. This not only takes into account user preferences for products, but also sets exclusive recommendation areas and recommended time periods for users, thereby achieving flexible control of the user interface.

[0149] 2. By using the Q-learning algorithm, it can automatically learn user preference patterns and optimize product recommendation strategies. This automated processing reduces the burden of manual intervention;

[0150] 3. Through the ε-greedy strategy, the system can explore and utilize to a certain extent during the recommendation process. The exploration process allows the system to have the opportunity to discover new recommendation strategies, thereby continuously optimizing the recommendation effect, while the utilization process allows the system to maintain the learned optimal strategy to a certain extent, avoiding recommendation instability caused by excessive exploration;

[0151] 4. It greatly reduces the manpower required for product recommendations, leverages user behavior to make intelligent recommendations for customer products, and combines market data changes to update product status in real time. This is not only smarter than human management, but also greatly improves efficiency.

[0152] Based on the above-mentioned product recommendation method based on buried data, the present invention also provides a product recommendation device based on buried data. Fig. 9 The device is described in detail.

[0153] Fig. 9 The structural block diagram of a product recommendation device based on buried point data according to an embodiment of the present invention is schematically shown.

[0154] like Fig. 9 As shown, the product recommendation device 900 based on buried point data according to this embodiment includes an interface buried point data acquisition module 910, a mark acquisition module 920, a maximum Q value combination acquisition module 930 and a recommendation module 940.

[0155] The interface buried point data acquisition module 910 can be used to acquire the interface buried point data authorized by the user. In one embodiment, the interface buried point data acquisition module 910 can be used to perform the operation S210 described above, which will not be repeated here.

[0156] The tag acquisition module 920 can be used to perform a first data processing on the interface buried point data to obtain a preference tag, wherein the preference tag includes N user preference area tags, M user preference time tags and L user preference product tags, and N, M, and L are all positive integers. In one embodiment, the tag acquisition module 920 can be used to perform the operation S220 described above, which will not be repeated here.

[0157] The maximum Q value combination acquisition module 930 may be used to acquire the maximum Q value combination based on the preference tag using a Q-learning algorithm. In one embodiment, the maximum Q value combination acquisition module 930 may be used to perform the operation S230 described above, which will not be described in detail herein.

[0158] The recommendation module 940 may be used to recommend products to the user according to the tag information in the maximum Q value combination. In one embodiment, the recommendation module 940 may be used to perform the operation S240 described above, which will not be described in detail here.

[0159] According to an embodiment of the present invention, the product recommendation device based on embedded point data 900 may further include a embedded point design unit.

[0160] The design tracking unit can be used to design tracking in the front-end interface, including tracking clicks on navigation bars, buttons, and partition modules; and tracking clicks, browsing, and application operations for each product and / or service. In one embodiment, the design tracking unit can be used to perform the operation S310 described above, which will not be repeated here.

[0161] According to an embodiment of the present invention, the tag acquisition module 920 may include a user preference area tag selection unit, a user preference product tag selection unit, and a user preference time tag selection unit.

[0162] The user preference area mark selection unit can be used to obtain an area click ratio according to the number of clicks on the navigation bar, button and partition module, and select N user preference area marks based on the area click ratio. In one embodiment, the user preference area mark selection unit can be used to perform the operation S410 described above, which will not be repeated here.

[0163] The user preferred product mark selection unit may be used to obtain the user product views according to the click, browse and apply operations for each product and / or service, and select L user preferred product marks based on the user product views. In one embodiment, the user preferred product mark selection unit may be used to perform the operation S420 described above, which will not be described in detail here.

[0164] The user preference time mark selection unit can be used to obtain user activity according to the number of clicks on the navigation bar, button and partition module and the click, browse and apply operations on each product and / or service, and select M user preference time marks based on the user activity. In one embodiment, the user preference time mark selection unit can be used to perform the operation S430 described above, which will not be repeated here.

[0165] According to an embodiment of the present invention, the maximum Q value combination acquisition module 930 may include a space establishment unit, a reward function definition unit, an initial Q value acquisition unit, a maximum Q value acquisition module and a maximum Q value combination acquisition unit.

[0166] The space establishment unit may be used to establish a state space and a behavior space based on the preference tag. In one embodiment, the space establishment unit may be used to perform the operation S510 described above, which will not be described in detail here.

[0167] The reward function definition unit may be used to define a reward function as a reward value corresponding to the execution action, wherein the reward value is determined according to the number of user clicks or user satisfaction. In one embodiment, the reward function definition unit may be used to perform the operation S520 described above, which will not be described in detail herein.

[0168] The initial Q value acquisition unit may be used to randomly select an initial state s based on the state space and the behavior space, and select a next behavior a to obtain an initial Q value. In one embodiment, the initial Q value acquisition unit may be used to perform the operation S530 described above, which will not be described in detail here.

[0169] The maximum Q value acquisition module may be used to gradually update the Q value using the Q-learning algorithm update rule until the maximum Q value is obtained. In one embodiment, the maximum Q value acquisition module may be used to perform the operation S540 described above, which will not be described in detail herein.

[0170] The maximum Q value combination acquisition unit may be used to take the state corresponding to the maximum Q value as the maximum Q value combination. In one embodiment, the maximum Q value combination acquisition unit may be used to perform the operation S550 described above, which will not be described in detail here.

[0171] The maximum Q value acquisition module may include an environment interaction unit, a target Q value calculation module, a Q value update unit and a maximum Q value acquisition unit.

[0172] The environment interaction unit can be used for the environment interaction step: the agent performs the selected behavior a, interacts with the environment, and observes the obtained reward value and the next state s'. In one embodiment, the environment interaction unit can be used to perform the operation S610 described above, which will not be repeated here.

[0173] The target Q value calculation module can be used in the target Q value calculation step: calculating the target Q value of the agent selecting behavior a' in the next state s'. In one embodiment, the target Q value calculation module can be used to perform the operation S620 described above, which will not be repeated here.

[0174] The Q value updating unit may be used in the Q value updating step: updating the Q value based on the target Q value. In one embodiment, the Q value updating unit may be used to perform the operation S630 described above, which will not be described in detail herein.

[0175] The maximum Q value acquisition unit may be used to repeat the environment interaction step, the target Q value calculation step and the Q value update step until the maximum Q value is obtained. In one embodiment, the maximum Q value acquisition unit may be used to perform the operation S640 described above, which will not be described in detail here.

[0176] The target Q value calculation module may include an initial ε value setting module.

[0177] The initial ε value setting module can be used to set an initial ε value before gradually updating the Q value using the Q-learning algorithm update rule, wherein the selecting behavior a' includes: selecting behavior a' according to the ε-greedy strategy. In one embodiment, the initial ε value setting module can be used to perform the operation S710 described above, which will not be repeated here.

[0178] According to an embodiment of the present invention, the initial ε value setting module may include a random number generating unit, a first selecting unit and a second selecting unit.

[0179] The random number generation unit may be used to generate a random number in the range of [0, 1). In one embodiment, the random number generation unit may be used to perform the operation S810 described above, which will not be described in detail herein.

[0180] The first selection unit may be used to select a random behavior in the behavior space as behavior a' if the random number is less than the ε value. In one embodiment, the first selection unit may be used to perform the operation S820 described above, which will not be described in detail herein.

[0181] The second selection unit may be used to select the behavior with the largest current Q value as behavior a' if the random number is greater than the ε value. In one embodiment, the second selection unit may be used to perform the operation S830 described above, which will not be described in detail here.

[0182] According to an embodiment of the present invention, any multiple modules of the interface buried point data acquisition module 910, the mark acquisition module 920, the maximum Q value combination acquisition module 930 and the recommendation module 940 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the interface buried point data acquisition module 910, the mark acquisition module 920, the maximum Q value combination acquisition module 930 and the recommendation module 940 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the interface buried point data acquisition module 910, the mark acquisition module 920, the maximum Q value combination acquisition module 930 and the recommendation module 940 can be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function can be executed.

[0183] Fig.10 A block diagram of an electronic device suitable for a product recommendation method based on buried point data according to an embodiment of the present invention is schematically shown.

[0184] like Fig.10 As shown, the electronic device 1000 according to an embodiment of the present invention includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0185] In RAM 1003, various programs and data required for the operation of electronic device 1000 are stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Processor 1001 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 1002 and / or RAM 1003. It should be noted that the program can also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0186] According to an embodiment of the present invention, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The electronic device 1000 may further include one or more of the following components connected to the I / O interface 1005: an input portion 1006 including a keyboard, a mouse, etc.; an output portion 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1008 including a hard disk, etc.; and a communication portion 1009 including a network interface card such as a LAN card, a modem, etc. The communication portion 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage portion 1008 as needed.

[0187] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0188] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 1002 and / or RAM 1003 described above and / or one or more memories other than ROM 1002 and RAM 1003.

[0189] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present invention.

[0190] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 1001. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0191] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1009, and / or installed from the removable medium 1011. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0192] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0193] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages, specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed completely on the user computing device, partially on the user device, partially on the remote computing device, or completely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0194] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0195] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A product recommendation method based on embedded point data, characterized in that: The method comprises: Obtain user-authorized interface tracking data; Performing a first data processing on the interface buried data to obtain preference tags, wherein the preference tags include N user preference area tags, M user preference time tags, and L user preference product tags, where N, M, and L are all positive integers; Based on the preference labels, using a Q-learning algorithm to obtain a maximum Q value combination; and Products are recommended to the user according to the tag information in the maximum Q value combination.

2. The method according to claim 1, characterized in that Before obtaining the interface tracking data authorized by the user, the method further includes: Design tracking points on the front-end interface, including tracking clicks on the navigation bar, buttons, and partition modules; as well as tracking clicks, browsing, and application operations for each product and / or service.

3. The method according to claim 2, characterized in that The performing first data processing on the interface buried point data specifically includes: According to the number of clicks on the navigation bar, the button and the partition module, obtain the area click ratio, and select N user preference area marks based on the area click ratio; According to the click, browse and apply operations for each product and / or service, obtain the user product browsing volume, and select L user preferred product tags based on the user product browsing volume; and The user activity is obtained according to the number of clicks on the navigation bar, buttons and partition modules, as well as the clicks, browsing and application operations on each product and / or service, and M user preference time tags are selected based on the user activity.

4. The method according to any one of claims 1 to 3, characterized in that: The method of obtaining the maximum Q value combination based on the preference mark by using the Q-learning algorithm specifically includes: Based on the preference tags, establishing a state space and a behavior space; A reward function is defined as a reward value corresponding to executing a behavior in the behavior space, wherein the reward value is determined according to the number of user clicks or user satisfaction; Based on the state space and behavior space, randomly select an initial state s and select the next behavior a to obtain an initial Q value; Using the Q-learning algorithm update rule to gradually update the Q value until the maximum Q value is obtained; and The state corresponding to the maximum Q value is used as the maximum Q value combination.

5. The method according to claim 4, characterized in that The Q-value is gradually updated by using the Q-learning algorithm update rule until the maximum Q value is obtained, specifically including: Environment interaction step: The agent performs the selected behavior a, interacts with the environment, and observes the reward value and the next state s'; Target Q value calculation step: Calculate the target Q value of the agent's selected behavior a' in the next state s'; Q value updating step: updating the Q value based on the target Q value; and The environment interaction step, the target Q value calculation step and the Q value update step are repeated until the maximum Q value is obtained.

6. The method according to claim 5, characterized in that Before gradually updating the Q value by using the Q-learning algorithm update rule, it also includes setting an initial ε value, wherein the selecting behavior a' includes: selecting behavior a' according to an ε-greedy strategy.

7. The method according to claim 6, wherein selecting behavior a' according to the ε-greedy strategy specifically comprises: Generate a random number in the range [0, 1); If the random number is less than the ε value, a random behavior is selected in the behavior space as behavior a'; as well as If the random number is greater than the ε value, the behavior with the largest current Q value is selected as behavior a'.

8. The method according to claim 6 or 7, characterized in that: After the Q value updating step, the method further comprises: updating the ε value by using exponential decay or linear decay.

9. A product recommendation device based on embedded point data, characterized in that: The device comprises: The interface buried point data acquisition module is used to: obtain the interface buried point data authorized by the user; A tag acquisition module is used to: perform a first data processing on the interface buried point data to obtain a preference tag, wherein the preference tag includes N user preference area tags, M user preference time tags and L user preference product tags, and N, M and L are all positive integers; A maximum Q value combination acquisition module is used to: acquire the maximum Q value combination based on the preference mark using a Q-learning algorithm; and The recommendation module is used to recommend products to users according to the tag information in the maximum Q value combination.

10. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images