Method, computer device, and computer program for providing optimal routes using expert knowledge

Abstract

A method, computer device, and computer program product are provided that use expert knowledge to provide an optimal route. [Solution] A method for providing an optimal route using expert knowledge includes predicting the next action of a target user through an optimal route prediction model learned using a graph-shaped path that represents a user behavior trajectory on a session-by-session basis, and recommending the predicted action path as an optimal route, wherein the recommending step includes grasping a user intent based on a previous behavior trajectory in the target user's current session, and determining a path corresponding to the user intent as one of the optimal routes.

Description

[Technical field]

[0001] The following description relates to a technique for finding patterns from user behavior and providing optimal routes. [Background technology]

[0002] In the current situation where a huge amount and variety of content is provided, services that recommend content to users based on various criteria have appeared. A typical example of such services is the personalized recommendation service.

[0003] A personal recommendation service is a service that recommends content that is suitable for a user based on the user's activities (eg, content usage patterns, content purchase patterns, etc.), that is, personalized content.

[0004] Existing methods for such personalized recommendation services analyze a user's activity patterns based on their past activities, search for other users who have similar patterns to those, and recommend to the user content that the other users have used or purchased.

[0005] Korean Patent Publication No. 10-1647364 (registered on August 4, 2016) discloses a technology that targets multiple user groups and recommends suitable content from among multiple pieces of content based on a score for each user group based on the interactions of multiple users with each piece of content, and a priority based on the user group to which the user belongs, thereby recommending content that is more suitable to the user. Summary of the Invention [Problem to be solved by the invention]

[0006] A method and apparatus are provided for modeling a user experience and recommending an optimal route to achieve a goal.

[0007] A method and apparatus are provided for recommending a route that reflects expert knowledge as an optimal route for achieving a user goal. [Means for solving the problem]

[0008] A method executed on a computer device, the computer device including at least one processor configured to execute computer-readable instructions contained in a memory, the method including a step of predicting, by the at least one processor, a next action of a target user using an optimal path prediction model learned by a graph-form path representing a user behavior trajectory on a per-session basis, and recommending the predicted action path as an optimal path, the step of recommending including a step of understanding a user intent based on a previous behavior trajectory in the target user's current session, and a step of determining a path corresponding to the user intent as one of the optimal routes.

[0009] According to one aspect, the determining step may determine a path based on expert knowledge by calling an expert AI model corresponding to the user intent from among expert AI models trained on paths determined as correct answer data while taking into account rules defined in expert knowledge.

[0010] According to another aspect, the optimal route prediction model may include an expert AI model for each expert category that has learned a path based on expert knowledge of the corresponding category, and the determining step may determine a path based on expert knowledge using an expert AI model of a category that is appropriate for the user intent.

[0011] According to another aspect, the path in the form of a graph is composed of states at each time step, actions in the states, and rewards for the actions, where the states are defined by the content of the service screen consumed by the user and may further include at least one of the type of service, environmental information associated with the user, personal information of the user, and a session category, the actions are defined by user activity in the states, and the rewards are defined by user satisfaction for the actions.

[0012] According to another aspect, paths based on expert knowledge may be rewarded more highly than other paths.

[0013] According to yet another aspect, the determining step may include displaying an interface on a service screen of the current session to guide the expert knowledge-based path.

[0014] According to another aspect, the determining step may further include soliciting user input associated with the user intent through an interface for guiding a path based on expert knowledge.

[0015] According to yet another aspect, the determining step may further include providing additional information corresponding to the user input as an expert knowledge-based path.

[0016] According to yet another aspect, the determining step may include defining a current situation of the target user and guiding a personalized path based on the current situation.

[0017] According to yet another aspect, the current situation of the target user may be defined by at least one of the target user's user behavior history within a current session or a recent period of time, the user's environmental information, and the user's personal information.

[0018] A computer program recorded on a computer-readable recording medium is provided for executing the above method on a computer device.

[0019] The present invention provides a computer device comprising at least one processor configured to execute computer-readable instructions contained in a memory, wherein the at least one processor processes a process of predicting a next action of a target user using an optimal path prediction model learned from a graph-form path representing a user behavior trajectory on a per-session basis, and recommending the predicted action path as an optimal path, and the at least one processor grasps a user intent based on a previous behavior trajectory in the target user's current session, and determines a path corresponding to the user intent as one of the optimal routes. Effect of the Invention

[0020] Embodiments of the present invention utilize artificial intelligence models to find patterns in user behavior and recommend optimal routes to user goals while eliminating unnecessary steps.

[0021] According to an embodiment of the present invention, by expressing and modeling user experience as a path, it is possible to build a hyper-personalized model for predicting the optimal route, and to build a platform that is independent of a specific domain or service.

[0022] According to an embodiment of the present invention, by recommending a route represented by expert knowledge as a route suitable for a user intent, an optimal user experience can be guided to reach a user goal more quickly. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 illustrates an example of a network environment in accordance with an embodiment of the present invention. [Diagram 2]FIG. 2 is a block diagram illustrating an example of a computing device in accordance with an embodiment of the present invention. [Diagram 3] 4 is a flow chart illustrating an example method that may be performed by a computing device in accordance with an embodiment of the present invention. [Figure 4] FIG. 2 illustrates an example of a user experience scenario in accordance with an embodiment of the present invention. [Diagram 5] FIG. 2 illustrates an example of a user experience scenario in accordance with an embodiment of the present invention. [Figure 6] FIG. 2 is a diagram illustrating an example of services connected in a network form in an embodiment of the present invention. [Figure 7] FIG. 4 illustrates an example of a graph path representing user recorded session data in accordance with one embodiment of the present invention. [Figure 8] FIG. 2 is a diagram showing an example of a path configuration representing a user experience in one embodiment of the present invention. [Figure 9] FIG. 11 is an exemplary diagram illustrating a process of finding an optimal route using a model that is trained by expressing a user experience as a path in an embodiment of the present invention; [Figure 10] FIG. 11 is an exemplary diagram illustrating a process of finding an optimal route using a model that is trained by expressing a user experience as a path in an embodiment of the present invention; [Figure 11] 1 is a flowchart illustrating an example of a process for providing an optimal route using expert knowledge in an embodiment of the present invention. [Figure 12] FIG. 13 illustrates an example of a path configuration including expert experience in one embodiment of the present invention. [Figure 13] FIG. 2 illustrates an example of an expert-embedded AI model in accordance with one embodiment of the present invention. [Figure 14] FIG. 2 illustrates an example UA service scenario in one embodiment of the present invention. [Figure 15] FIG. 2 illustrates an example UA service scenario in one embodiment of the present invention. [Figure 16]FIG. 2 illustrates an example UA service scenario in one embodiment of the present invention. [Figure 17] FIG. 2 illustrates an example UA service scenario in one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0025] SUMMARY OF THE DISCLOSURE The present invention relates to a technique for finding patterns in user behavior and providing optimal routes.

[0026] An embodiment including the subject matter specifically disclosed in this specification can recommend an optimal route for achieving a user's goal on a platform that can link multiple services.

[0027] The optimum route recommendation system according to the embodiment of the present invention may be realized by at least one computer device, and the optimum route recommendation method according to the embodiment of the present invention may be executed by at least one computer device included in the optimum route recommendation system. In this case, a computer program according to the embodiment of the present invention may be installed and executed in the computer device, and the computer device may execute the optimum route recommendation method according to the embodiment of the present invention according to the control of the executed computer program. The above-mentioned computer program may be combined with the computer device and recorded in a computer-readable recording medium to execute the optimum route recommendation method on a computer.

[0028] Fig. 1 is a diagram showing an example of a network environment in one embodiment of the present invention. The network environment in Fig. 1 shows an example including a plurality of electronic devices 110, 120, 130, 140, a plurality of servers 150, 160, and a network 170. Such Fig. 1 is merely an example for explaining the invention, and the number of electronic devices and the number of servers are not limited to those shown in Fig. 1. Furthermore, the network environment in Fig. 1 is merely an example for explaining an environment applicable to this embodiment, and the environment applicable to this embodiment should not be limited to the network environment in Fig. 1.

[0029] The electronic devices 110, 120, 130, 140 may be fixed terminals or mobile terminals realized by computer devices. Examples of the electronic devices 110, 120, 130, 140 include smartphones, mobile phones, navigation systems, personal computers (PCs), notebook PCs, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), tablets, etc. As an example, FIG. 1 shows a smartphone as an example of the electronic device 110, but in the embodiment of the present invention, the electronic device 110 may mean one of various physical computer systems that can communicate with other electronic devices 120, 130, 140 and / or servers 150, 160 via a network 170 using a substantially wireless or wired communication method.

[0030] The communication method is not limited, and may include not only a communication method utilizing a communication network (for example, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network 170 can include, but also short-range wireless communication between devices. For example, the network 170 may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, and the like. Furthermore, the network 170 may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or hierarchical network, and the like.

[0031] Each of the servers 150, 160 may be implemented by one or more computing devices that communicate with the multiple electronic devices 110, 120, 130, 140 via the network 170 to provide instructions, code, files, content, services, etc. For example, the server 150 may be a system that provides a service (such as a personalized recommendation service, for example) to the multiple electronic devices 110, 120, 130, 140 connected via the network 170.

[0032] 2 is a block diagram showing an example of a computer device according to an embodiment of the present invention. Each of the electronic devices 110, 120, 130, and 140 and the servers 150 and 160 described above may be realized by a computer device 200 shown in FIG.

[0033] As shown in FIG. 2, the computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input / output interface 240. The memory 210 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, the permanent mass storage device such as a ROM or a disk drive may be included in the computer device 200 as a separate permanent storage device separate from the memory 210. The memory 210 may also store an operating system and at least one program code. Such software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory 210 through the communication interface 230, which is not a computer-readable recording medium. For example, the software components may be loaded into the memory 210 of the computing device 200 based on a computer program installed by a file received over the network 170 .

[0034] The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the processor 220 by the memory 210 or the communication interface 230. For example, the processor 220 may be configured to execute instructions received according to program code stored in a storage device such as the memory 210.

[0035] The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (for example, the above-mentioned recording device) via the network 170. For example, requests, instructions, data, files, etc. generated by the processor 220 of the computer device 200 according to program codes recorded in a recording device such as the memory 210 may be transmitted to other devices via the network 170 under the control of the communication interface 230. Conversely, signals, instructions, data, files, etc. from other devices may be received by the computer device 200 via the communication interface 230 of the computer device 200 via the network 170. The signals, instructions, data, etc. received via the communication interface 230 may be transmitted to the processor 220 or the memory 210, and the files, etc. may be recorded on a recording medium (the above-mentioned permanent recording device) that the computer device 200 may further include.

[0036] The input / output interface 240 may be a means for interfacing with the input / output device 250. For example, the input device may include a device such as a microphone, a keyboard, or a mouse, and the output device may include a device such as a display and a speaker. As another example, the input / output interface 240 may be a means for interfacing with a device in which functions for input and output are integrated into one, such as a touch screen. The input / output device 250 may be configured as one device together with the computer device 200.

[0037] Also, in other embodiments, computing device 200 may include fewer or more components than those of Figure 2. However, most prior art components need not be explicitly illustrated. For example, computing device 200 may be implemented to include at least some of the input / output devices 250 described above, and may further include other components such as a transceiver, a database, etc.

[0038] Specific embodiments of a method and apparatus for providing an optimal route to achieve a user goal are described below.

[0039] In this specification, the term "path" refers to all user experiences on the Internet that are undertaken to achieve a user goal, and includes all search activities on the Internet (search, exploration, navigation, etc.). In other words, the path does not simply refer to search activities, but may include all user experiences consisting of various interlocking services (search service, map service, shopping service, easy payment service, blog service, AD service, etc.).

[0040] This embodiment can provide a model (Optimal Composable Experience Action Network, hereinafter referred to as the "OCEAN model") that trains an artificial intelligence model based on user behavior patterns to provide the optimal experience for the user and easily generate and expand user experiences between services.

[0041] In this embodiment, a platform independent of a specific domain or service can be constructed through the OCEAN model. In this specification, the platform refers to an integrated platform including a multi-domain and cross platform for providing a route on a network where multiple services are linked.

[0042] The computer device 200 according to the present embodiment may provide a personalized recommendation service to a client by connecting to a dedicated application installed on the client or a web / mobile site related to the computer device 200. An optimal route recommendation system implemented by a computer may be configured in the computer device 200. For example, the optimal route recommendation system may be implemented in the form of a program that operates independently, or may be implemented in the form of an in-app of a specific application so as to be operable on the specific application.

[0043] The processor 220 of the computer device 200 may be implemented with components for performing the following optimal route recommendation method. Depending on the embodiment, the components of the processor 220 may be selectively included in the processor 220 or may be excluded from the processor 220. Also, depending on the embodiment, the components of the processor 220 may be separated or combined to represent the functions of the processor 220.

[0044] Such processor 220 and its components may control the computer device 200 to execute steps included in the following optimal route recommendation method. For example, the processor 220 and its components may be implemented to execute instructions from the operating system code and at least one program code contained in the memory 210.

[0045] Here, the components of the processor 220 may represent different functions that are performed by the processor 220 according to instructions provided by program code stored in the computing device 200 .

[0046] The processor 220 may read the necessary instructions from the memory 210, which is loaded with instructions related to the control of the computing device 200. In this case, the read instructions may include instructions for controlling the processor 220 to perform the following steps:

[0047] The steps included in the optimal route recommendation method described below may be performed in an order different from that shown in the figures, some of the steps may be omitted, or additional processes may be included.

[0048] FIG. 3 is a flow chart illustrating an example method that may be performed by a computing device in accordance with an embodiment of the present invention.

[0049] 3, in step 310, the processor 220 may collect user historical session data including all behavior trajectories in each session for each user on a session-by-session basis. The processor 220 may track user behavior in each session and collect a series of all experiences made in the session as one sample data to be used as learning data for model learning.

[0050] In step 320, the processor 220 may represent the user recorded session data of each session as a graph path. The processor 220 may represent each of the user recorded session data of each session as at least one or more paths. In this case, each path may be composed of a state, an action, and a reward. The state may mean a current situation of the user at each time step (contents consumed by the user on a screen provided by the service), the action may mean a user activity in the current situation such as clicking, scrolling, going back, searching again, etc., and the reward may mean a state obtained by an action, i.e., a user's satisfaction with a state-action pair.

[0051] In step 330, the processor 220 may generate an OCEAN model, which is a model for optimal route prediction, by learning user experience expressed as a graph path. The processor 220 may connect services that can be interoperated based on multi-domain or cross-platform in a network form. User experience collected by user logs for services may be expressed as a graph path of the network. In this case, the processor 220 may train the model using a path history, an action for each state, and a reward for each path.

[0052] In step 340, the processor 220 may predict the next user behavior by the OCEAN model for the user record session data collected from the current session of the target user, and recommend an optimal route on the network. The processor 220 may predict the next action by using the previous path history according to the user log in the current session for the target user who is using at least one of the services connected in the network, and recommend the predicted behavior path as the optimal route.

[0053] The OCEAN model according to the present embodiment directly models user experience and may include user experience for at least one or more services. The OCEAN model is built with a composable graph path structure, so it is highly deterministic and expert knowledge can be expressed as paths on the network. Furthermore, the OCEAN model is also a hyper-personalized model, and can guide users through continuous experiences to optimally achieve their goals.

[0054] This embodiment learns user recorded session data on a per-session basis in which a purpose is defined by the user, and can automatically find the purpose of a target user from among multiple learned purposes based on the target user's behavior within the current session.

[0055] 4 and 5 are diagrams illustrating example user experience scenarios in accordance with one embodiment of the present invention.

[0056] Figure 4 illustrates a series of scenarios in which a user searches for dining tables in the search window to achieve a specific goal of "buying a dining table," clicks on a blog post from the search results screen, searches for the product name identified in the blog post, navigates to the page of the searched product, and places an order for the product.

[0057] On the other hand, Figure 5 shows an example scenario that goes through more processes to achieve the same goal as Figure 4.

[0058] A user cycles through states and actions to achieve a goal, where a state may be defined as a current user state representing the contents of a service screen, and an action may be defined as a user activity such as clicking, scrolling, going back, searching again, etc.

[0059] In this embodiment, services that can be interconnected may be connected in the form of a network. In the network, a user experience may be represented as a path on the network, and a model may be trained based on the history of the path, the action for each state, and the reward for each path. The model trained in this way may predict the next user action and link the predicted actions to guide the user to achieve a goal via an optimal route.

[0060] For example, referring to FIG. 6, various services such as a search service, a map service, a shopping service, a pay service, a blog service, an advertisement service, etc. may be connected to one network as interoperable services.

[0061] Paths for user experience in various services may be generated, and an OCEAN model 600 may be constructed to predict an optimal route through comprehensive learning of paths of services connected in a network form.

[0062] FIG. 7 illustrates an example of a graph path representing user recorded session data in one embodiment of the present invention.

[0063] Users A, B, and C are all trying to purchase the same product, and all have the same goal to achieve, but the process (user experience scenario) they take to reach that goal is different. Figure 7 shows the network representation of the three users' experiences in achieving their goal, and as each user goes through a different process, a wide variety of paths are drawn. We can see that User A was unable to purchase the product he desired, while User C, who had the most specific understanding of the product, succeeded in purchasing the product via the shortest route out of the three.

[0064] In this embodiment, an OCEAN model 600 is generated that can predict the optimal route from among multiple routes by learning user experience composed of graph paths, and this makes it possible to provide guidance for achieving goals by recommending the same route as user C to other users.

[0065] FIG. 8 shows an example of a configuration of paths representing user experiences in one embodiment of the present invention.

[0066] Referring to FIG. 8, the network of the OCEAN model 600 may be composed of one or more paths 800, each of which may consist of a state 801 at each time step, an action 802 for the state 801, and a reward 803 for the action 802.

[0067] States 801 may be defined for all screens provided to the user in all services that can be integrated, and all activities that the user can take for each service (e.g., clicking, searching, browsing, etc.) may be defined as actions 802, and rewards 803 may be designed based on the level of satisfaction with the actions 802.

[0068] The state 801 may be defined to include all elements that can determine the user state. It may include not only the screen that the user is consuming at the current time stage, but also the type of service (e.g., search, shopping, map, blog, etc.), environmental information (e.g., time, location, weather, season, etc.), personal information of the user (e.g., age, gender, preferences, etc.), and session category (e.g., interior, fashion, car, etc.). This makes it possible to provide a service that recommends a personalized optimal route.

[0069] The state 801 may be classified into a service state, an environment state, a user state, a session state, etc. depending on the element, and may be composed of various variables that represent the current situation, such as a current query, an age, a current screen, a location, a time, etc.

[0070] Actions 802 refer to user actions and may include queries (eg, text, image, voice, etc.) and gestures (eg, view, click, scroll, back, favorite, etc.).

[0071] Reward 803 may represent a user's satisfaction with action 802. As an example, reward 803 may be determined by a direct method that receives direct feedback from the user to measure the satisfaction with action 802. As another example, reward 803 may be determined by an indirect method that predicts the satisfaction with action 802 based on additional actions such as dwell time on a page, purchasing a product, clicking on a page, etc.

[0072] All user logs in one session may be graphed and the user experience may be expressed as various paths 800. The user experience in all services that can be linked, without depending on a specific service, may be expressed as a path 800 consisting of a state 801, an action 802, and a reward 803. In this way, by modeling the user experience itself, a platform that is not dependent on a specific domain or service can be constructed.

[0073] All sets of data passed through through trial and error to achieve a goal (i.e., including all successful and unsuccessful experiences) may be used as learning data.

[0074] For example, an example of a user experience connected to a corrective knowledge panel service is shown in Table 1. In this case, the corrective knowledge panel service is a service that provides basic detailed information desired by the user about, for example, a region or a famous place so that the user can immediately view it. In other words, corrective search refers to a search type in which documents displayed on a search result screen for a given query are pre-matched for each query, and the corrective knowledge panel service refers to a service screen that includes corrective search results. User experiences related to such searches include correcting typos and inputting queries that are more suitable for preferred information. [Table 1]

[0075] States 801, actions 802, and rewards 803 may be defined so that the model can be trained by collecting samples of user experiences as described above. [Table 2]

[0076] A reward 803 may be defined as a user's satisfaction with an action 802 .

[0077] As an example, reward 803 may be determined based on impression-to-click rate (CTR), and may be applied as a positive indicator if there are many user clicks on an impression.

[0078] As another example, the reward 803 may be determined based on the probability of clicking on a particular collection, text, point, etc., and may be applied as a positive indicator if the user clicks on similar locations many times, or as a negative indicator if the click locations are scattered.

[0079] As another example, the reward 803 may be determined based on the duration of stay, and may be applied as a positive indicator if the user stays in a certain state for a certain amount of time or more, and as a negative indicator if the user stays in a certain state for less than the certain amount of time.

[0080] Additionally, the reward 803 may be determined based on the difference in dwell time, and may be applied as a positive indicator if the dwell time in the current situation is longer than the previous state, and may be applied as a negative indicator if the dwell time in the current situation is shorter than the previous state.

[0081] As another example, the reward 803 may be determined based on scrolling, with a positive indicator being applied if there is little scrolling and a long stay on a particular part, and a negative indicator being applied if there is a lot of meaningless scrolling.

[0082] As another example, the reward 803 may be determined based on an action pair count, where a positive indicator may be applied if the number of times the previous action and the current action occur together is above a certain level, and a negative indicator may be applied if the number of times the previous action and the current action occur together is below a certain level.

[0083] As another example, the reward 803 may be determined based on an action transition probability, and may be applied as a positive indicator if the probability of transitioning from the previous action to the current action is above a certain level, and may be applied as a negative indicator if the probability is below the certain level.

[0084] As yet another example, the reward 803 may be determined based on the probability that the current action is the final action of the session, and may be applied as a positive indicator if many users end the session in the current situation, and as a negative indicator if many users take other actions in the current situation. [Table 3]

[0085] If the reward for the final state of the session is higher than a certain level, it may be used as a successful experience, and if the reward for the final state of the session is lower than the certain level, it may be used as a failed experience.

[0086] In other words, for learning the OCEAN model 600, user record session data including all behavioral trajectories in the corresponding session can be used as learning data on a session-by-session basis, and the learning data can be appropriately processed and used depending on the service status, model learning method, etc. For example, in the case of a search service, a query can be defined as an action 802, and in the case of a shopping service, a click can be defined as an action 802. When language model learning is applied, the state 801 and the action 802 can all be expressed in language and utilized.

[0087] The processor 220 may construct the OCEAN model 600 by learning a path 800 that represents the user's current situation at each time stage in a session as a state 801, the user's activities as actions 802, and the user's satisfaction as a reward 803.

[0088] As an example, the processor 220 may generate the OCEAN model 600 through reinforcement learning based on the state 801, the action 802, and the reward 803. As another example, the processor 220 may generate the OCEAN model 600 through learning a language model that understands the state 801, the action 802, and the reward 803 themselves as a language. As another example, the processor 220 may generate the OCEAN model 600 through neural network learning for a path 800 of a graph structure composed of the state 801, the action 802, and the reward 803. Depending on the embodiment, the OCEAN model 600 may also be generated by ensemble learning using two or more models.

[0089] The process of model training based on reinforcement learning is as follows:

[0090] As one definition for reinforcement learning, a Markov decision process (MDP) may be defined as a tuple (S,A,τ,r,γ,μ), where S and A are sets of states and actions. τ maps every s,a∈S×A to a conditional distribution τ(·|s,a) over states (S), and r maps any s,a∈S×A to r(s,a)∈[0,1]. γ∈[0,1] is a discount factor, and μ is a distribution over states (S).

[0091] Also, θ∈R d A policy parameterized by θ is the conditional distribution π for action (A) θ Define (·|s,h), where s∈S denotes the current situation and h denotes additional information for the condition (e.g., past state-action-reward history).

[0092] Furthermore, given MDP(S, A, τ, r, γ, μ) and policy π θ and the associated value function:

number

number

[0093] The goal of reinforcement learning is to find a policy that maximizes a value function V(θ).

[0094] The processor 220 may indirectly perform policy learning by predicting the maximum reward (Q-function) for each state-action pair to extract the optimal policy. In this case, the learning algorithm may be based on temporal difference (TD) learning, and may utilize a model-based method (i.e., knowledge of the reward function and state transition distribution is required) such as value iteration or a model-free method (no corresponding knowledge is required) such as Q-learning.

[0095] If the policy is differentiable, a gradient-based method may be used to optimize the value function V(θ). In this embodiment, the flat gradient problem caused by sparse signals can be solved mainly by pre-conditioning / normalizing the ascending direction (natural gradient, TRPO and PPO).

[0096] The processor 220 may directly model the sequence of states, actions, and rewards while considering offline reinforcement learning with sequence modeling. The user's behavior trajectory in a session may be composed of a static data set that can learn a sequence model, and the decision transformer for sequence modeling is as shown in Equation (2).

number

[0097] Here, R t represents the returns-to-go (sum of future rewards) that can be specified at test time.

[0098] On the other hand, the trajectory transformer is expressed as equation (3).

number

[0099] Here, state, action, reward, R t If x is discretized, a modified beam search may be used at test time to approximate the trajectory that maximizes the reward.

[0100] The OCEAN model 600 is trained by user experiences including achievement goals, and therefore contains useful resources for all tasks related to the service without additional fine-tuning of the model, so that all services can be personalized based on the user experience. However, if necessary, a method of optimizing the model service by adjusting the model such as fine-tuning is also possible.

[0101] Referring to FIG. 9 , the processor 220 may represent each user experience that achieves the same goal as a path 800 composed of a state 801, an action 802, and a reward 803, and construct an OCEAN model 600 through reinforcement learning on the path 800.

[0102] There are various paths to achieve the goal, and the processor 220 can find the optimal path while predicting the next action using the OCEAN model 600. Referring to Fig. 10, the processor 220 may use the OCEAN model 600 to find and recommend the optimal path to achieve the goal.

[0103] The user's past actions and state history sequence (state[0], state[1], ..., state[n-1], action[0], action[1], ..., action[n-1]) that lead to the current situation within a session are the input of OCEAN model 600, and the final state-action pair (state[n] based on action[n]) is the output of OCEAN model 600. The n in action[n] refers to the number of actions the user has attempted to achieve a goal.

[0104] In the OCEAN model 600, not only the previous log but all user behavior trajectories in the same session, i.e., all paths (states, actions, rewards) in the previous history, affect the prediction of the next action. For example, action[2] can be predicted using state[1], action[1], reward[1], action[3] can be predicted using state[1], state[2], action[1], action[2], reward[1], reward[2], and action[4] can be predicted using state[1], state[2], state[3], action[1], action[2], action[3], reward[1], reward[2], reward[3].

[0105] In the following, an embodiment for providing an optimal route using expert knowledge will be specifically described.

[0106] In this embodiment, one of the services realized by the OCEAN model 600 is a universal advisor (hereinafter, referred to as "UA") service.

[0107] UA services may be implemented in a bottom-up approach, where a collection of paths becomes an experience, a collection of experiences becomes an advisor, and a collection of advisors completes the universal advisor.

[0108] UA services can provide personalized information, appearing at the moments of need throughout the search flow to guide users to achieve their goals.

[0109] FIG. 11 is a flow diagram illustrating an example process for using expert knowledge to provide an optimal route in accordance with an embodiment of the present invention.

[0110] Referring to FIG. 11, in step 1110, the processor 220 may generate an expert AI model by learning centered on a path based on expert knowledge. The processor 220 may collect user experiences that achieve a goal based on expert knowledge of the corresponding category for each expert category (e.g., interior design, fashion, cooking, law, etc.). As an example, the processor 220 may track all behavioral trajectories in an expert's session as a user experience by an expert, and collect all the series of experiences made in the session as one sample data, and use them as learning data for model learning. For example, for the goal of purchasing a table, the processor 220 may extract user record session data on the route by which a user corresponding to an interior design expert purchased the table, and collect a path based on expert knowledge. As another example, the processor 220 may collect a path defined as correct answer data while considering the rules defined by the expert knowledge. For example, to develop expert knowledge for an interior design expert, interior elements (materials, textures, colors, shapes), space styles (natural, modern, vintage, Scandinavian, etc.), and space types (living room, kitchen, bedroom, bathroom, hallway, etc.) may be defined, and rules that are criteria for judging excellent interiors, such as what criteria the expert uses to change the space while considering each item, may be defined. Paths based on expert knowledge may be collected based on an image set that is judged to be excellent interiors according to the interior design expert's rules. The processor 220 may generate an expert AI model for each expert category by collecting paths for each expert category, classifying the collected paths into the corresponding categories, and learning the classified paths. Thus, the processor 220 may generate an expert AI model for a UA service by modeling a user experience that reflects the expert knowledge. The processor 220 includes expert experiences when expressing user experiences collected in a user log for a service as paths in the form of a graph, and therefore the expert experiences may also be expressed as paths on the same network.

[0111] In step 1120, the processor 220 may understand the search intent of the target user based on the user record session data collected from the current session of the target user. Among the states and actions extracted from the user log data in the session, the final state of the session may be defined as the user's potential goal. The next action of the target user may be predicted using a model that learns the action trajectory of the session in which the goal was successfully achieved, and the target user's potential goal may be understood from other past user behavior patterns based on the prediction result. The processor 220 may determine the moment when a UA service is needed according to the search intent of the target user. When the target user's potential goal is understood, the processor 220 may check whether an expert path is included in multiple paths to achieve the goal.

[0112] In step 1130, when the processor 220 recognizes that the moment when the UA service is needed is based on the search intent of the target user, the processor 220 may call an expert AI model suitable for the search intent. For example, when the processor 220 finds an interior expert path among multiple past paths that have achieved a potential goal recognized as the user's search intent, the processor 220 may determine that the moment when the UA service is needed is, and call an interior expert AI model that has learned the path of the interior expert.

[0113] In step 1140, the processor 220 may recommend the path according to the expert AI model as one of the optimal routes for the target user. The processor 220 provides a UA service based on the search intent of the target user, where the UA service may use the expert AI model to guide an expert path corresponding to the user's current situation.

[0114] The processor 220 can predict the next action of the target user based on the history of the current session, and can grasp the potential goal indicating the user's search intent. The processor 220 can request the user for information required to provide the next state based on the path to achieve the user's potential goal, and can provide the next state as the optimal path according to the user's input.

[0115] FIG. 12 is a diagram showing an example of the configuration of a path including expert experience in one embodiment of the present invention.

[0116] To provide UA services, not only general user experiences but also expert experiences are modeled as paths in a graph structure.

[0117] Referring to FIG. 12, the processor 220 may model both a general user experience 1201 of purchasing a light via a general product search path and an expert experience 1202 of purchasing a light via a path defined as expert knowledge as user experiences that achieve a common goal of purchasing a light.

[0118] When the processor 220 identifies “light purchase” as a potential purpose of the target user, it may recommend to the target user a path based on general user experience 1201, or a path based on expert experience 1202 via the UA service.

[0119] The processor 220 may model the expert experience 1202 for the UA service to build an expert AI model. Referring to FIG. 13, the expert AI model 1300 may include AI models in various fields such as interior (space), fashion, cooking, law, etc., and the processor 220 may grasp the potential purpose of the target user and provide the UA service using an AI model suitable for the purpose.

[0120] A new path that cannot be created by the general user experience 1201 can be created through the expert AI model 1300. In this case, the expert experience 1202 may be reflected in the OCEAN model 600 in a form in which a higher reward is applied than the general user experience 1201 in the modeling process.

[0121] The expert AI model 1300 may be constructed as an independent model for each category, and may operate in a manner of selecting and calling an expert AI model 1300 suitable for a user's search intent.

[0122] 14 to 17 are diagrams showing examples of UA service scenarios in one embodiment of the present invention.

[0123] FIG. 14 shows an example of an interior related UA service scenario.

[0124] When a user uses a search service with the search term "Scandinavian interior", the processor 220 understands the user intent to change his / her space into a Scandinavian-style interior based on the user's current situation (search term, history in the current session, etc.). If the user's search intent belongs to the expert category, the processor 220 may provide a UA service of the corresponding category. As an example, the processor 220 may determine that a UA service is needed at a moment when the user's current situation falls into the expert category.

[0125] The interior expert AI model is a model of experience based on the knowledge of interior experts, and a UA service based on the interior expert AI model may provide recommendations based on expert knowledge by requesting an image of an interior space from a user, extracting information about the interior space from the input image and detecting objects within the space (understanding), analyzing the interior space information of the input image and the object detection results by applying rules defined by the knowledge of interior experts (reasoning), and determining a recommendation solution appropriate for the analysis results (decision making).

[0126] Referring to FIG. 14, when the processor 220 determines that the user's search intent indicates that UA services from an interior design expert are required, the processor 220 may display a UA interface 1401 on a search result screen 1400 corresponding to the search term "Scandinavian interior."

[0127] The processor 220 may request an image input related to an interior space from a user in order to find and execute a path optimized for the user's search intent through the OCEAN model 600. When the UA interface 1401 is selected on the search result screen 1400, the processor 220 may provide an image input interface 1410 to request an image input. The processor 220 may add the image input to the image input interface 1410 as the user's current situation, and may recommend interior results 1420 suitable for the space of the input image on the search result screen 1400 as a path corresponding to the user's state.

[0128] FIG. 15 shows an example of a UA service scenario related to shopping.

[0129] If a user has the intent to purchase a particular product but does not know any specific information about the product and enters the search term "mushroom shaped light," the user's intent to purchase the light may be understood based on the search term and the history of the current session.

[0130] Referring to FIG. 15, when the processor 220 determines that the user's search intent indicates that UA services from a shopping expert are required at this time, the processor 220 may display a UA interface 1501 on a search result screen 1500 corresponding to the search term "mushroom-shaped light."

[0131] The user experience of achieving the purchase of the light may be modeled to include not only general user experience but also expert experience. The processor 220 may request image input related to the product to be purchased in order to find and execute an expert path through the OCEAN model 600. When the UA interface 1501 is selected on the search result screen 1500, the processor 220 may provide an image input interface 1510 to request image input. The processor 220 may add the image input to the image input interface 1510 as the user's current situation, and may recommend shopping information (e.g., product details, product homepage, product purchase page, retailer information, etc.) 1520 related to the product on the input image on the search result screen 1500 as a path corresponding to the user state.

[0132] The processor 220 can eliminate unnecessary steps through the UA service and easily and quickly suggest the optimal route to achieve a goal, without the user having to go through multiple processes to search for product information.

[0133] The processor 220 can understand the user intent based on the user's current situation, such as search terms and session history, and can suggest cross-platform results according to the user intent. For example, if a user inputs the search term "how to remove coffee stains" in a situation where the user has spilled coffee on his / her clothes, the processor 220 can grasp the user's potential purpose of removing the coffee stain and link a search service that provides text including know-how on how to remove the coffee stain, a shopping service for purchasing detergent for removing the coffee stain, and a location information service that provides a list of dry cleaners near the user's current location. In other words, the UA service can actively guide the user to achieve his / her purpose while moving between multiple platforms such as a search service, a shopping service, and a location information service.

[0134] As a UA service, the processor 220 is also capable of common sense reasoning: if a user uploads an image of a coffee stained shirt instead of a search term, common sense reasoning from image analysis may identify the underlying goal of removing the stain and suggest a path to achieve this goal.

[0135] In the OCEAN model 600, the user experience is modeled as a path in the form of a graph. For example, it is possible to learn from search history data what final result a user reached for the search term "when you have a headache," and the final result selected by many users ultimately becomes common sense. In order to learn such user experience, when the search term "when you have a headache" is entered, the final result can be predicted based on the user's common sense. Many common sense examples such as "take a headache medicine when you have a headache," "eat rice when you are hungry," and "use an umbrella when it rains" can be reflected in the OCEAN model 600, and a UA service using the OCEAN model 600 can provide an experience optimized by common sense inference.

[0136] The processor 220 can also proactively provide notifications to users in a multi-domain or cross-platform environment when they are unable to complete a goal in a previous session. For example, if a user is unable to purchase a desired product in a shopping service because the product is out of stock, the UA service can monitor the status of the product and notify the user when the product is restocked.

[0137] There are several criteria for determining whether a user has achieved their goal. For example, a user may be deemed to have achieved their goal if they purchase a product after searching, or if they move to a different page after searching and stay on the page for a certain period of time.

[0138] In the process of constructing the OCEAN model 600, the final state in the user behavior history data (e.g., whether many users ended the session on the product search page, ended the session after purchasing a product, etc.) may be learned probabilistically. In other words, for each state, the probability that it is the final state in which the goal is achieved may be learned.

[0139] If the user ends the session in a state that is not likely to be the final state, the goal may be considered not to have been achieved. The user action sequence and the current state may be continuously monitored to see if they have achieved the goal.

[0140] One of the key elements in guiding users along the optimal path to achieve their goals is personalization. How well a service can understand the user's state plays a key role in determining recommendations.

[0141] 16 and 17 show an example of a UA service scenario that takes into account the user state.

[0142] The processor 220 may recognize the user's current situation through the UA service and provide additional information. Fig. 16 shows a product details screen 1600. When a user of Korean nationality currently residing in Korea is viewing a product whose shoe size is displayed in American standard, it may be determined that the UA service is required at that time, and a UA interface 1601 may be displayed on the product details screen 1600.

[0143] If the user resides in the United States or has American nationality, the display of the UA interface 1601 on the product details screen 1600 is omitted.

[0144] When the UA interface 1601 is selected on the product details screen 1600, the processor 220 may provide Korean size standards or recommend sizes suitable for the user's personal information as additional information 1610 corresponding to the user's current situation on the product details screen 1600.

[0145] The more personal information, such as the user's shoe size, favorite color, favorite brand, etc., that is stored in the OCEAN model 600 as user status, the more optimal the path the processor 220 can provide the UA service.

[0146] 17 shows a shopping service screen 1700. If a user has recently purchased a sofa and searches for interior-related keywords such as interior, lighting, rugs, etc. are increasing, a user state may be defined based on such history. In the process of the user using the shopping service screen 1700, a UA interface 1701 including additional information related to the user state may be provided on the shopping service screen 1700 using an interior expert AI model.

[0147] On the other hand, for a user who needs to lose weight and is searching for training wear for the purpose of exercising, the user state may be defined based on recent search terms, preferred brands, seasonal information, etc., and for such a user, related information may be recommended using at least one of a fashion expert AI model and a food expert AI model.

[0148] In other words, the processor 220 may define the user's current situation based on the user's behavioral history during the current session or a recent period of time, the user's environmental information (e.g., time, location, weather, season, etc.), and the user's personal information (e.g., age, gender, preferences, etc.), and may guide a personalized path.

[0149] Therefore, the processor 220 can utilize expert AI models from various domains to provide more accurate and useful recommendations, and can recommend different paths for each user by understanding the user state and providing necessary information for the current situation.

[0150] Conventionally, recommendation information has been provided based on rules defined by a service provider, but the UA service using the OCEAN model 600 of this embodiment can provide personalized recommendation information according to a user state defined based on the user's history through an AI model.

[0151] In other words, UA services using the OCEAN model 600 can predict the purpose based on the user experience and guide the optimal route, providing results based on more diverse variables. By directly modeling the experience based on expert knowledge as well as the general user experience, it is possible to respond to all situations, and by expanding the path, it is possible to connect with new services as much as you like.

[0152] In this way, according to the embodiment of the present invention, it is possible to find patterns from user behavior using an artificial intelligence model, eliminate unnecessary steps, and recommend an optimal route to the user's goal. In particular, according to the embodiment of the present invention, by expressing and modeling the user experience as a graph path, it is possible to build a hyper-personalized model to predict the optimal route, and to build a platform that is independent of a specific domain or service. Furthermore, according to the embodiment of the present invention, it is possible to guide the user to an optimal user experience to reach the goal more quickly by recommending a route expressed by expert knowledge as a route suitable for the user intent.

[0153] The above-mentioned devices may be realized by hardware components, software components, and / or a combination of hardware and software components. For example, the devices and components described in the embodiments may be realized using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or various devices capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications that run on the OS. The processing device may also access, record, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, one processing device may be described as being used, but one skilled in the art will understand that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0154] The software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to operate as desired or instruct the processing device, either individually or collectively. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed and stored or executed in a distributed manner on computer systems connected by a network. The software and data may be stored on one or more computer-readable storage media.

[0155] The method according to the embodiment may be realized in the form of program instructions executable by various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously records a program executable by a computer, or one that temporarily records the program for execution or download. The medium may be one of various recording means or storage means in the form of a single or multiple hardware devices combined, and may be a medium directly connected to a certain computer system, or may be one that is distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and media configured to record program instructions such as ROMs, RAMs, flash memories, etc. Examples of other media include recording media and storage media managed by app stores that distribute applications, sites that supply or distribute various other software, and servers.

[0156] Although the embodiments have been described based on limited embodiments and drawings as above, those skilled in the art will be able to make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from that described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a manner different from that described, or may be counterbalanced or replaced by other components or equivalents, and still achieve suitable results.

[0157] Therefore, different embodiments are within the scope of the appended claims provided that they are equivalent to the claims.

Claims

1. 1. A method implemented on a computer device, comprising: The computing device includes at least one processor configured to execute computer-readable instructions contained in a memory; The method comprises: predicting a next action of the target user through an optimal path prediction model learned according to a path in a graph form representing a user behavior trajectory in a session unit by the at least one processor, and recommending the predicted action path as an optimal path. Including, The recommending step comprises: understanding a user intent based on a previous behavioral trajectory of the target user in a current session; and determining a path corresponding to the user intent as one of the optimal paths; A method comprising:

2. The determining step comprises: Among a plurality of expert AI models trained by paths determined as correct answer data in consideration of rules defined in expert knowledge, an expert AI model corresponding to the user intent is called to determine a path based on expert knowledge. The method according to claim 1 , characterized in that

3. The optimal route prediction model is For each expert category, an expert AI model that learns a path based on the expert knowledge of the corresponding category is included, The determining step comprises: determining a path based on said expert knowledge through an expert AI model of a category appropriate for said user intent; The method according to claim 1 , characterized in that

4. The graph-shaped path is composed of a state at each time step, an action at the state, and a reward for the action, The state is defined by the content of a service screen consumed by the user, and further includes at least one of a type of service, environmental information associated with the user, personal information of the user, and a session category; The action is defined by user activity in the state; The reward is defined as user satisfaction with the action. The method according to claim 1 , characterized in that

5. Expertise-based paths offer higher rewards than other paths. The method according to claim 4, characterized in that

6. The determining step comprises: displaying an interface for guiding a path based on expert knowledge on a service screen of the current session; The method of claim 1 , comprising:

7. The determining step comprises: and requesting user input associated with the user intent via an interface for guiding the path based on the expert knowledge. The method of claim 6 further comprising:

8. The determining step comprises: providing additional information corresponding to the user input as the expert knowledge-based path; The method of claim 7 further comprising:

9. The determining step comprises: Defining a current situation of the target user and guiding a personalized path based on the current situation. The method of claim 1 , comprising:

10. The current situation of the target user is defined by at least one of the following: a user behavior history within a current session or a recent certain period of time of the target user, user environment information, and user personal information. The method according to claim 9, characterized in that

11. A computer program recorded on a computer-readable recording medium for causing a computer device to execute the method according to any one of claims 1 to 10.

12. 1. A computing device comprising: At least one processor configured to execute computer-readable instructions contained in the memory. Including, The at least one processor A process of predicting the next action of a target user through an optimal path prediction model learned from a graph-shaped path that represents the user behavior trajectory for each session, and recommending the predicted action path as the optimal path. Process The at least one processor Understanding a user intent based on a previous behavioral trajectory of the target user in a current session; determining a path corresponding to the user intent as one of the optimal paths; A computer device comprising:

13. The at least one processor Among a plurality of expert AI models trained by paths determined as correct answer data in consideration of rules defined in expert knowledge, an expert AI model corresponding to the user intent is called to determine a path based on expert knowledge.

13. The computer device of claim 12.

14. The optimal route prediction model is For each expert category, an expert AI model that has learned a path based on the expert knowledge of the corresponding category is included; The at least one processor determining a path based on said expert knowledge through an expert AI model of a category appropriate for said user intent; 14. The computer device of claim 13.

15. The graph-shaped path is composed of a state at each time step, an action at the state, and a reward for the action, The state is defined by the content of a service screen consumed by the user, and further includes at least one of a type of service, environmental information associated with the user, personal information of the user, and a session category; the action is defined by a user activity in the state; The reward is defined as a user's satisfaction with the action.

13. The computer device of claim 12.

16. Expertise-based paths offer higher rewards than other paths.

16. The computer device of claim 15.

17. The at least one processor Displaying an interface for expert knowledge-based path guidance on the service screen of the current session.

13. The computer device of claim 12.

18. The at least one processor soliciting user input associated with the user intent via an interface to guide the expert knowledge-based path; 20. The computer device of claim 17,

19. The at least one processor providing additional information corresponding to the user input as a path based on the expert knowledge; 20. The computer device of claim 18,

20. The at least one processor Defining a current situation of the target user based on at least one of a user behavior history within a current session or a recent period of time of the target user, user environmental information, and user personal information, and guiding a personalized path based on the current situation.

13. The computer device of claim 12.

Images