Interactive simulation method and device for recommendation system based on reinforcement learning

Abstract

The present invention at least relates to an interactive simulation method and device for a recommendation system based on reinforcement learning. Specifically disclosed is an interactive simulation method for a recommendation system based on reinforcement learning, comprising: normalizing vectorized representations of multiple historical data items of user behavior to obtain multiple unit vector data; constructing a tree index structure based on the multiple unit vector data using a spatial index method; retrieving the most similar historical data item to a given data item in the tree index structure; and simulating user feedback on the given data item based on the most similar historical data item.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning, and more specifically to an interactive simulation method and device for a recommendation system based on reinforcement learning. Background Art

[0002] The list-wise recommendation system is an intelligent model built based on the historical data of existing users, which recommends multiple items, packages, articles, movies, etc. that users may be interested in. The list-wise recommendation system based on reinforcement learning regards the recommendation process as a serialized interaction process between users and the recommendation system, and implements the operation of updating the online recommendation system model in real time based on the behavioral data fed back by online users in real time.

[0003] like Figure 1 As shown in Figure 1, the recommendation process can be viewed as a sequential interaction between the user and the search engine. If the search engine is viewed as an agent and the user as an environment, the product search problem can be viewed as a typical sequential decision problem:

[0004] 1) Every time a user requests a page view (PV), the agent makes a corresponding sorting decision and displays the products to the user;

[0005] 2) Users give feedback signals such as clicks and page turns based on the search engine's ranking results;

[0006] 3) The search engine receives feedback signals and makes new ranking decisions when new PV requests are made;

[0007] 4) This process will continue until the user purchases the product or exits the search.

[0008] The five elements of reinforcement learning are:

[0009] 1) State space

[0010] The state is defined as the user's historical behavior, that is, the latest N items that the user has clicked before the recommendation. Clicking allows the user to browse, collect, or purchase.

[0011] 2) Action Space (action)

[0012] An action is defined as a list of items to be recommended to a user.

[0013] 3) Reward

[0014] The search engine takes corresponding actions according to the current state. That is, after recommending K product lists to the user, the search engine obtains the immediate reward of the current state-action according to the user's feedback on the recommended list (ignore, browse, collect or purchase).

[0015] 4) Transition probability

[0016] In the present invention, the state transition is defined as follows: the current state is the N items that the user has recently clicked (i.e., browsed, collected or purchased, etc.), and the action is the K items newly recommended to the user. If the user ignores all of these items, then the state at the next moment is the same as the current state. If the user clicks on two of the items, then the state at the next moment is based on the current state, removing two items from the front and placing the two clicked items at the end.

[0017] 5) Discount factor Γ

[0018] The value range of Γ is [0,1], which defines the reward coefficient when calculating the pending reward. In simple terms, it means how much you care about future rewards. The larger the Γ, the more you care about future rewards, and the smaller the Γ, the more you care about current benefits.

[0019] Before the list-wise recommendation system based on reinforcement learning goes online, offline training and evaluation are required. How to train the parameters of the recommendation system and evaluate its performance offline is a difficult task. The ground-truth data and corresponding feedback in the user history records are limited, and the data is very sparse for the entire action space (that is, all possible combinations of items, packages, articles, and movies). This will cause two problems. First, only part of the state-action data pairs can be obtained for training, and it is impossible to model all situations (which may cause overfitting). Second, it will cause inconsistency between the online and offline environments.

[0020] To this end, an online environment simulator is proposed in the prior art. The simulator outputs simulated online rewards when the state and action selected by the recommendation system are input, so that the recommendation system can train parameters offline according to the simulated rewards. The online environment simulator is mainly built based on the user's historical behavior data. Its basic idea is that given a similar state and action, different users will also give similar feedback.

[0021] However, the problem with this online environment simulator is that it is necessary to calculate the similarity between a given state-action and each piece of historical data one by one, which requires a huge amount of calculation. In order to improve the calculation efficiency, the prior art also proposes to group and stratify the historical state-action data before calculating the similarity, and use part of the data to characterize the characteristics of the entire data. However, although this improves the calculation efficiency, it brings uncertainty to the simulation accuracy of the simulator. Summary of the invention

[0022] In view of one or more of the above technical problems, there is currently a need for a simulation method and device that can both improve the calculation speed and ensure the simulation accuracy of the simulator. The present invention proposes an interactive simulator for an optimization algorithm for a recommendation system based on reinforcement learning. The present invention optimizes the method for solving the similarity between historical state-action data and given state-action data based on the User-Agent algorithm. By using spatial indexing technology in a high-dimensional space to create a dynamic index for historical state and action features, it is possible to quickly retrieve the historical state-action pairs with the greatest similarity to the given state-action pair. While improving the solution speed, it does not sacrifice accuracy. The processing accuracy and processing time efficiency of the reinforcement learning user simulator are greatly improved.

[0023] According to one embodiment of the present invention, an interactive simulation method for a recommendation system based on reinforcement learning is proposed, comprising: normalizing vectorized representations of multiple historical data items of user behavior to obtain multiple unit vector data; constructing a tree index structure based on the multiple unit vector data using a spatial indexing method; retrieving historical data items that are most similar to a given data item in the tree index structure; and simulating user feedback on the given data item based on the most similar historical data items.

[0024] According to one embodiment, an interactive simulation device for a recommendation system based on reinforcement learning is provided, comprising: a memory storing computer executable instructions; and a processor configured to implement the method described above when executing the computer executable instructions stored in the memory.

[0025] According to one embodiment, an interactive simulation device for a recommendation system based on reinforcement learning is provided, comprising components configured to execute the steps of the method described above.

[0026] According to one embodiment, a non-transitory computer-readable storage medium is provided, storing instructions, which implement the method described above when executed by a processor.

[0027] According to one embodiment, a recommendation system based on reinforcement learning is provided, which is trained using the interactive simulation device as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0029] Figure 1 shows the interaction between the search engine and the user during reinforcement learning;

[0030] Figure 2 A flow chart of an interactive simulation method according to an embodiment of the present invention is shown;

[0031] Figure 3 A flow chart of an interactive simulation method according to another embodiment of the present invention is shown;

[0032] Figure 4 The comparison between the simulation method in the prior art and the simulation method of the present invention is shown. DETAILED DESCRIPTION

[0033] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. Unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0034] In order to solve the above problems of the existing user-agent interaction simulator, the present invention innovatively proposes a method and device for constructing a tree index structure for the spatial data of historical behavior using a dynamic spatial indexing method, and obtaining the historical data most similar to the given state and behavior data by searching in the tree index structure. Compared with the method of calculating the cosine distance between the given data and each historical data (or sampled historical data) one by one in the prior art, the method of the present invention significantly improves the calculation efficiency and accuracy.

[0035] Refer to the following Figure 2 An interactive simulation method 200 according to an embodiment of the present invention is described.

[0036] First, in step 201, the vectorized representations of multiple historical data items of user behavior are normalized to obtain multiple unit vector data.

[0037] The vectorized representation of multiple historical data items of user behavior is obtained by vectorizing feature texts of the historical data items of user behavior, for example, by using Natural Language Processing (NLP).

[0038] According to one embodiment, each historical data item includes at least:

[0039] 1. A list of recommended products (i.e., actions) provided by a search engine at a certain historical moment. The product list may include M products.

[0040] 2. The multiple products that the user has recently clicked before the recommendation is about to be made (i.e., status). For example, the latest N products that the user has browsed, collected, or purchased before the recommendation list is about to be provided.

[0041] 3. User feedback (i.e., reward) on the recommended product list. That is, whether the user ignored, browsed, collected, or purchased one or more products in the recommended product list.

[0042] Preferably, a product can be represented by multiple features such as product ID, product category, product attributes, etc. A user can be represented by multiple features such as user ID, user category, user preference, etc. Therefore, the obtained data item is usually a high-dimensional vector.

[0043] The purpose of normalizing the text vector and converting it into a unit vector is to enable the spatial indexing technology to be applied thereto in the next step to construct a tree index structure.

[0044] Spatial indexing technology is conventionally used in the field of geographic information systems, and the similarity measurement indicator in spatial indexing technology is Euclidean distance. This is completely different from content-based recommendation systems, in which the similarity measurement indicator is cosine distance. Therefore, there is no precedent in the prior art for applying spatial indexing technology to content-based recommendation systems.

[0045] Here is a brief introduction to the difference between Euclidean distance and cosine distance. Euclidean distance can reflect the absolute difference of individual numerical features, so it is more used in analysis that needs to reflect the difference from the numerical size of the dimension, such as using user behavior indicators to analyze the similarity or difference of user values. Cosine distance distinguishes differences more from the direction, but is not sensitive to absolute values. It is more used to use users' ratings of content to distinguish the similarity and difference of interests, and at the same time corrects the problem of inconsistent metrics between users. Therefore, content-based recommendation systems usually use cosine distance to judge similarity.

[0046] Taking this into consideration, in order to enable the spatial indexing technology to be applied to the field of content-based recommendation systems such as the present invention, the present invention further normalizes the vectorized representation of the text. The Euclidean distance and cosine distance of the normalized vector are equivalent. Therefore, after the data is normalized, the spatial indexing technology based on the Euclidean distance can be applied to the application scenario of the present invention, that is, the construction of the user simulator in the content-based recommendation system.

[0047] The historical data items can preferably be obtained from a database that specifically stores the user's historical behavior trajectory, where the data of each recommendation-feedback interaction can be stored as a historical data item. When calling historical data, historical data within a specific period of time (e.g., 1 year) can be selected; or the latest specific number of historical data items can be selected.

[0048] In step 202, a tree index structure is constructed based on the plurality of unit vectors using a spatial indexing technique.

[0049] After obtaining the unit vectors of multiple historical data items, the spatial indexing technology is used to construct a tree index structure.

[0050] Different from common indexing techniques (e.g., B-tree) for low-dimensional data, spatial indexing techniques can be used for indexing high-dimensional data. The state data and action data in the present invention are both high-dimensional data, so spatial indexing techniques can be used to construct a tree index structure for them.

[0051] Common spatial indexing technologies include, for example, R* tree, R tree, and R+ tree, etc. As long as the spatial indexing technology is suitable for processing high-dimensional data structures, it can be applied to the present invention.

[0052] Among these spatial indexing technologies, the R* tree is particularly preferred. The creation, insertion, deletion and query algorithms of the R* tree are basically similar to those of the R tree. The main difference is that the R* tree introduces a unique insertion path selection mechanism when inserting objects, which reduces the overlap between the intermediate node directory rectangles to improve its indexing performance. In addition, the R* tree uses forced reinsertion technology in its tree creation and insertion process to effectively optimize the structure of the tree and better solve the problem of indexing performance degradation caused by unreasonable insertion order of spatial objects. Therefore, the R* tree is particularly suitable for scenarios in recommendation systems where the tree structure needs to be updated after iteration according to the transition probability, that is, inserting and deleting nodes.

[0053] The R* tree index can be constructed by the insertion algorithm and forced reinsertion algorithm of the R* tree. The insertion algorithm of the R* tree is the basis for constructing the R* tree. The construction of the R* tree starts from an empty tree and continuously calls the insertion algorithm to insert the spatial objects in the spatial data set into the R* tree one by one until a complete R* tree is generated. Therefore, the insertion algorithm of the R* tree is crucial in its various index algorithms. The forced reinsertion algorithm of the R* tree can reinsert the data that is not in the right position during the insertion process, thereby ensuring that each data is in the right position.

[0054] The insertion algorithm and forced reinsertion algorithm of the R* tree will be introduced below and will not be described in detail here.

[0055] In step 203, the historical data item most similar to the given data item is retrieved in the constructed tree index structure.

[0056] The data items given by the recommendation model are input into the tree index structure for retrieval. According to one embodiment, the given data items at least include: a plurality of commodities (status) recently clicked by a given user and a given recommended commodity list (action).

[0057] Similar to historical data items, commodities can be characterized by commodity ID, commodity category, commodity attributes, etc. Users can be characterized by user ID, user category, user preference, etc. In order to input a given data item into the tree index structure constructed using dynamic spatial indexing technology, the given data item must also be preprocessed similar to historical data, that is, the text is converted into a vector and then normalized to obtain the unit vector of the given data item.

[0058] Similarly, the retrieval process is described by taking the R* tree as an example. As those skilled in the art can understand, although the R* tree is preferred, other tree structures can also be used.

[0059] The query of the R* tree is a recursive process. Starting from the root node, determine whether the minimum bounding rectangle (MBR) of the root node intersects with the query area. If not, return an empty set directly, that is, the spatial data set does not contain the object to be queried. If it intersects, match it with the directory rectangle of each lower-level child node pointed to by the root node. If the directory rectangle of a child node intersects with the query area, continue to search and match the nodes of the lower layer of the node. And so on, until the leaf node that intersects with it is found. Judge each data item in the found leaf node to see whether its MBR intersects with the query area. If it intersects, it is the query result.

[0060] In step 204, the user's feedback on the given data item is simulated based on the most similar historical data item.

[0061] Preferably, the user's feedback on the recommended product list in the most similar historical data item may be used as the user's feedback on the given data item.

[0062] As described above, the user's feedback on a given data item may include at least one or more of the following: ignoring, browsing, collecting, or purchasing one or more items in the recommended item list.

[0063] In the vector space model, feedback can be vectorized into a reward sequence. For example, if two products are recommended, the user's feedback on the two products can be ignored, browsed, collected or purchased, and the corresponding rewards are 0, 1, 3, and 5. Then there are 16 possible reward sequences for the user's feedback on the two recommended products: (0,0), (0,1), (0,3), (0,5), (1,0), (1,1), (1,3), (1,5), (3,0), (3,1), (3,3), (3,5), (5,0), (5,1), (5,3), (5,5), defined as:

[0064] u={u1,…,u 16}

[0065] However, since the reward to be input into the recommendation system for parameter training must be a value rather than a vector, it is necessary to multiply the reward sequence by the discount factor to convert the reward sequence into a specific reward value (r t ).

[0066]

[0067] where k is the order in the recommendation list, K is the length of the recommendation list, and Γ∈(0,1].

[0068] In the above described embodiment, the reward sequence is stored in each node of the tree structure together with the state and the action. In an optional embodiment, the reward sequence can also be stored separately in another database, and after the most similar historical data item is retrieved, the reward sequence corresponding to the historical data item is retrieved separately in this database.

[0069] The simulation method described in the above embodiment creates a dynamic spatial index for high-dimensional data of historical behaviors. Without sampling, the historical data item with the greatest similarity to a given data item can be quickly retrieved, thereby greatly improving the processing accuracy and processing time of the reinforcement learning user simulator.

[0070] According to a preferred embodiment, the obtained vector can also be subjected to dimensionality reduction processing before normalization. The vector obtained by natural language processing is a high-dimensional vector. Dimensionality reduction processing of high-dimensional vectors can extract effective information, discard useless information, ensure the mutual independence of features, enhance the spatial expression of objects, and improve the calculation processing speed and calculation result accuracy of similarity. Preferably, principal component analysis technology (PCA, Principal Components Analysis) can be used to reduce the dimension of feature data. This is just an example, and other dimensionality reduction techniques known to those skilled in the art can also be used for dimensionality reduction. Although dimensionality reduction processing is preferred, it is not necessary. Dimensionality reduction is a common technical means in this field and will not be described in detail here.

[0071] After obtaining the unit vector, a vector space model (VSM) is constructed to facilitate subsequent processing.

[0072] According to one embodiment of the present invention, the tree index structure constructed using the spatial index method can be dynamically updated according to a single search. That is, the search in the spatial index structure can be synchronized with the update of the tree structure (insertion or deletion of nodes) without the need to reorganize the tree structure regularly. Figure 3 An example flowchart of the interactive simulation method according to this embodiment is provided. Figure 2 The method shown differs from the method shown in that it includes a step of updating the tree index structure.

[0073] Steps 301-305 and Figure 2 Steps 202-204 in are similar and will not be further described here.

[0074] In step 306, it is determined whether the training process has met the termination condition. The termination condition may be set to, for example, reaching a certain number of iterations, or other termination conditions commonly used in the art.

[0075] If the end condition is met, the iteration ends. This means that the training process is complete and the recommendation system is ready for launch.

[0076] If the end condition is not met, the process proceeds to step 307 to determine whether the current state has changed compared to the time before the given data item was retrieved based on the transition probability.

[0077] If the state has not changed, that is, the user has ignored all the items in the recommendation list and has not clicked (i.e., browsed, collected or purchased, etc.) any item in the recommendation list, then the process can directly return to step 303 to retrieve the next given data item in the tree index structure.

[0078] If the state has changed, that is, the user has clicked on one or more items in the recommendation list, the process proceeds to step 308 to update the tree index structure.

[0079] In one embodiment, the tree index structure is updated based on the given data item and the simulated feedback. Preferably, updating the tree index structure includes: inserting the given data item and the simulated feedback that have been retrieved into the tree index structure according to the transition probability, and deleting the earliest historical data item from the tree index structure. In this way, the tree index structure can keep using the latest data, thereby improving the timeliness of the simulation of user feedback.

[0080] The transition probability means that if the user ignores all the recommended items, the state at the next moment is the same as the current state. If the user clicks on one or more of the items, the state at the next moment will change relative to the current state. For example, if the user clicks on two of the items, the state at the next moment is based on the current state, removing two items from the front and placing the two items clicked at the end.

[0081] The update of R* tree is realized by deleting and inserting nodes. The insertion algorithm and deletion algorithm of R* tree will be introduced below and will not be described in detail here.

[0082] Then, the process returns to step 303, and in the next training process of the recommendation system, the next given data item given by the recommendation system is input into the updated tree index structure to retrieve the reward sequence of the next given data item. According to the transition probability, the state in the next given data item is also updated. For the interactive simulator, if the state changes according to the transition probability, the state in the given data item input into the interactive simulator in the next iteration needs to be updated.

[0083] The following describes the insertion algorithm and deletion algorithm used when constructing and updating the tree index structure by taking the R* tree as an example.

[0084] 1. Insertion algorithm of R* tree

[0085] Starting from the root node, select the insertion path according to the following requirements.

[0086] a) If the child node of the current node points to a leaf node, then select the index item whose directory rectangle overlaps with the new data rectangle with the smallest increase in area; if the overlap area increase is the same, then select the index item with the smallest increase in directory rectangle area; if the area increase is also the same, then select the index item with the smallest directory rectangle area.

[0087] b) If the child node of the current node points to an intermediate node, then select the index item whose directory rectangle has the smallest area increase after the newly added data rectangle is inserted; if the area increase is the same, then select the index item with the smallest area.

[0088] After inserting all the data in the vector space model into the R* tree structure, a dynamically indexable tree structure of historical data items is obtained.

[0089] In addition, the R* tree can use its unique forced reinsertion technology during its tree construction process. When the position where the data is inserted is not optimal, the data can be deleted and forced to be reinserted to ensure that each data is in the best position. Compared with other spatial indexing technologies, the tree structure constructed in this way is more optimized and can achieve better query performance.

[0090] 2. Forced reinsertion algorithm for R* tree.

[0091] The forced reinsertion of the R* tree is for the same set of spatial objects. Due to the different insertion orders, the constructed R* trees are different. The spatial objects inserted first may have introduced an "inappropriate" directory matrix into the tree, which makes the tree structure unfavorable for query. Therefore, the R* tree performs "forced reinsertion" during the insertion process to obtain dynamic structural reorganization. The advantage of this algorithm of the R* tree is that it ensures that the data is in the best position when the tree is built and updated, thereby improving retrieval performance.

[0092] First, calculate the distance from the center of the minimum bounding rectangle (MBR) of the M+1 index items or data items contained in the node to be split to the center of the MBR of the node, and sort the M+1 items in node from large to small according to the distance. Second, remove the first T items from node and adjust the MBR of node accordingly. Finally, starting from the maximum distance or minimum distance, call the insertion algorithm in sequence to reinsert these T items.

[0093] 3. Deletion algorithm of R* tree.

[0094] To delete a specified spatial object from the R* tree, first use the query algorithm to find the leaf node containing the spatial object to be deleted, then delete the data item information of the spatial object to be deleted in the leaf node, and adjust the directory rectangle of the index item of the parent node from the leaf node upward until the root node. If the deletion of the spatial object causes the corresponding leaf node to underflow, that is, the number of data items contained in the leaf node is less than the lower limit of the node capacity m specified by the R* tree, then all the data items of the leaf node need to be reinserted into the leaf node layer, and the index item information of its parent node containing the leaf node needs to be deleted. This process extends to the upper nodes, and the structure of R* needs to be adjusted accordingly to ensure that the number of index items of all intermediate nodes is not less than m. If only one child node is left in the root node, the root node needs to be deleted, and its child node is used as the new root node of R*, and the depth of R* is reduced by one layer.

[0095] Below, refer to Figure 4 Describe the differences between the present invention and the prior art.

[0096] Figure 4 Two flow charts are shown, the left one is a flow chart of an interactive simulation method in the prior art, and the right one is a flow chart of an interactive simulation method according to an embodiment of the present invention.

[0097] As can be seen from the figure, the method in the prior art is to determine the similarity by calculating the cosine value of the given data item and the historical data item one by one. This requires a huge amount of calculation and has low calculation efficiency. In order to reduce the amount of calculation, the prior art also groups and stratifies the historical data items. However, although such a method improves the calculation efficiency, it loses the calculation accuracy. Therefore, it is impossible to obtain satisfactory simulation results, which affects the training effect.

[0098] The present invention creatively proposes to construct a tree index structure for high-dimensional vector data using a dynamic spatial indexing technology conventionally used in the field of geographic information systems after normalizing the text vector, thereby achieving rapid retrieval. Compared with the prior art, the present invention eliminates the process of calculating the cosine value, greatly reducing the amount of calculation. In addition, the method of the present invention does not require data sampling, has high simulation accuracy, and has good effects.

[0099] According to another embodiment of the present invention, an interactive simulation device for a recommendation system based on reinforcement learning is provided, comprising: a memory storing computer executable instructions; and a processor configured to implement the method described above when executing the computer executable instructions stored in the memory.

[0100] According to another embodiment of the present invention, an interactive simulation device for a recommendation system based on reinforcement learning is provided, comprising components configured to execute the steps of the method described above.

[0101] It should be understood that the components here are only logical modules divided according to the specific functions they implement, and are not used to limit the specific implementation method. In actual implementation, the device can be implemented as an independent physical entity, or it can also be implemented by a single entity (for example, a processor (CPU or DSP, etc.), an integrated circuit, etc.).

[0102] According to another embodiment of the present invention, a non-transitory computer-readable storage medium is provided, which stores instructions, and when the instructions are executed by a processor, the method described above is implemented. The computer-readable storage medium can be any type of storage medium, such as an optical storage device (e.g., a CD, a digital versatile disc, a Blu-ray disc, etc.) or a tape storage device (e.g., a hard disk drive).

[0103] According to another embodiment of the present invention, a recommendation system based on reinforcement learning is provided, which is trained using the interactive simulation device as described above.

[0104] From the detailed description, many features and advantages of the present invention are apparent, therefore, the appended claims are intended to cover all such features and advantages of the present invention that fall within the true spirit and scope of the present invention. In addition, since many modifications and variations will readily occur to those skilled in the art, it is not desired to limit the present invention to the exact construction and operation illustrated and described, and accordingly, all suitable modifications and equivalents that fall within the scope of the present invention may be adopted.

Claims

1. An interactive simulation method for a recommendation system based on reinforcement learning, comprising: Normalizing the vectorized representations of multiple historical data items of user behavior to obtain multiple unit vector data; Using a spatial index method to construct a tree index structure based on the plurality of unit vector data; Retrieve the historical data item most similar to the given data item in the tree index structure; simulating user feedback on a given data item based on the most similar historical data item; and After obtaining the simulated user feedback on the given data item, determine whether the tree index structure needs to be updated based on the transition probability. If the tree index structure needs to be updated, the given data item that has been retrieved and its simulated feedback are inserted into the tree index structure, and the earliest historical data item is deleted from the tree index structure. Then, the historical data item that is most similar to the next given data item is retrieved in the updated tree index structure until the predetermined end condition is met.

2. The method of claim 1, wherein: The tree-like index structure constructed using the spatial index method can be dynamically updated according to a single retrieval.

3. The method of claim 1, wherein: Retrieving the historical data items most similar to a given data item in a tree index structure involves: Performing text vectorization on a given data item and normalizing the obtained vectorized representation to obtain a unit vector of the given data item; and The unit vector of a given data item is input into the tree index structure for retrieval to obtain the historical data item most similar to the given data item.

4. The method of claim 1, further comprising: Perform feature data dimensionality reduction on historical data text vectors before normalization; as well as A vector space model is constructed based on a plurality of unit vector data after normalization.

5. The method of claim 1, wherein: Simulating user feedback on a given data item according to the most similar historical data item includes: obtaining user feedback on a recommended product list in the most similar historical data item as user feedback on the given data item.

6. The method according to claim 1, wherein each of the historical data items comprises at least: A list of recommended products at a certain historical moment, multiple products that the user has recently clicked before the recommendation is made, and user feedback on the recommended product list.

7. The method of claim 1, wherein: The given data items include at least: a plurality of items most recently clicked by the user given by the recommendation model and a list of recommended items, and wherein the simulated feedback includes at least one or more of the following: ignoring, browsing, collecting or purchasing one or more items in the recommended item list.

8. The method of claim 1, wherein: The tree index structure is an R* tree.

9. An interactive simulation device for a recommendation system based on reinforcement learning, comprising: a memory storing computer executable instructions; as well as A processor, configured to implement the method according to any one of claims 1 to 8 when executing computer executable instructions stored in the memory.

10. An interactive simulation device for a recommendation system based on reinforcement learning, comprising components configured to execute the steps of the method according to any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 8.

12. A recommendation system based on reinforcement learning, wherein the recommendation system is trained using the interactive simulation device as claimed in claim 10.

Images