Method for recommending contents and system thereof

Abstract

A content recommendation method and system are provided. According to some embodiments of this disclosure, the content recommendation method may include the following steps: obtaining user feedback information regarding recommended content; updating a reinforcement learning model by reflecting rewards for the feedback information; and re-performing content recommendations for the user using the updated reinforcement learning model. According to the above method, customized content that the user likes can be accurately recommended.

Description

Content recommendation method and system The present disclosure relates to a content recommendation method and system, and more particularly, to a method for recommending customized content to a user and a system for executing the method. With the widespread adoption of OTT (Over-The-Top) services, interest in technologies that recommend customized content to users is rapidly increasing. Furthermore, these recommendation technologies are widely used not only in OTT services but also in areas such as product recommendations, music, and news. Meanwhile, recommendation technologies used in the OTT sector are largely based on analyzing users' viewing history. For example, existing recommendation technologies analyze a user's viewing history to identify characteristics of the user's favorite content (e.g., genre, actors, themes, etc.) and use these characteristics to provide recommendations (e.g., recommending content with similar characteristics). However, the above recommendation technology has obvious limitations, because the above recommendation content depends on the user's past viewing patterns and cannot quickly reflect changes in user preferences in the recommendation results. Patent Literature Korean Patent No. 10-2022-0080563 (published on June 14, 2022) Technical problem to be solved by the invention The technical problem to be solved by some embodiments of the present disclosure is to provide a method for accurately recommending customized content preferred by a user and a system for executing the method. Another technical problem to be solved by some embodiments of the present disclosure is to provide a content recommendation method that can quickly respond to changes in user preferences and a system for executing the method. Another technical problem to be solved by some embodiments of the present disclosure is to provide a method for accurately calculating user feedback rewards when recommending content using a reinforcement learning model. Another technical problem to be solved by some embodiments of the present disclosure is to provide a method for accurately identifying false feedback when recommending content using a reinforcement learning model. Another technical problem to be solved by some embodiments of the present disclosure is to provide a method for solving the problem of some content being continuously recommended or excluded from recommendation targets when recommending content using a reinforcement learning model. Another technical problem to be solved by some embodiments of the present disclosure is to provide various feedback information that can be used for model training when using a reinforcement learning model to recommend content. It should be understood that the technical problems disclosed herein are not limited to the above-mentioned technical problems, and a person having ordinary knowledge in the industry can clearly understand other technical problems not mentioned through the following description. Technical solutions to the problem To address the aforementioned technical issues, a content recommendation method according to one embodiment of the present disclosure is performed by at least one computing device and may include the following steps: using a reinforcement learning model to determine recommended content to be provided to a user; obtaining feedback information from the user regarding the recommended content; updating the reinforcement learning model by reflecting a reward for the feedback information; and re-executing content recommendations for the user using the updated reinforcement learning model. In some embodiments, the recommended content may be provided to the user via a main area of ​​a content service page. In some embodiments, the step of determining the recommended content may include the following steps: screening multiple candidate contents from pre-registered content based on at least one of advertising costs, registration time and the user's viewing history analysis results; and determining the recommended content from the multiple candidate contents using the reinforcement learning model. In some embodiments, the step of determining the recommended content may include the following steps: training the reinforcement learning model using the user's feedback information collected through random recommendations; and determining the recommended content using the trained reinforcement learning model. In some embodiments, the step of determining the recommended content may include the following steps: performing initialization or discount processing for the accumulated rewards of at least a portion of the pre-registered content; and determining the recommended content using the reinforcement learning model that reflects the processed accumulated rewards. In some embodiments, the reinforcement learning model is based on Thompson sampling, and the step of determining the recommended content may include the following steps: determining the current context of the user based on the user's terminal type and current time information; selecting a probability distribution corresponding to the current context from multiple probability distributions corresponding to each pre-registered content; and determining the recommended content from the pre-registered content based on the selected probability distribution. In some embodiments, the reinforcement learning model is a Thompson sampling-based model, the feedback information includes at least one positive feedback information and at least one negative feedback information, and the step of updating the reinforcement learning model may include the following steps: aggregating the rewards of the at least one positive feedback information to calculate a first reward; aggregating the rewards of the at least one negative feedback information to calculate a second reward; and updating the probability distribution corresponding to the recommended content based on the first reward and the second reward. In some embodiments, the feedback information is multiple, and the step of updating the reinforcement learning model may include the following steps: predicting the total reward of the multiple feedback information by a machine-learning model; and updating the reinforcement learning model based on the predicted total reward. In some embodiments, the feedback information may include information related to the selection and playback of the recommended content. To solve the above technical problems, according to some embodiments of the present disclosure, a content recommendation system includes one or more processors; and a memory that stores instructions, and the one or more processors execute the stored instructions to perform the following operations: using a reinforcement learning model to determine recommended content to be provided to a user; obtaining feedback information from the user regarding the recommended content; updating the reinforcement learning model by reflecting a reward for the feedback information; and using the updated reinforcement learning model to re-execute content recommendations for the user. To address the aforementioned technical issues, a computer program according to some embodiments of the present disclosure is stored on a computer-readable recording medium and, in conjunction with a computing device, performs the following steps: using a reinforcement learning model to determine recommended content to provide to a user; obtaining user feedback information regarding the recommended content; updating the reinforcement learning model by reflecting a reward corresponding to the feedback information; and re-executing content recommendations for the user using the updated reinforcement learning model. Effects of the Invention According to some embodiments of the present disclosure, a reinforcement learning model can be gradually updated based on user feedback information, and the updated reinforcement learning model can be used to determine recommended content for the corresponding user. In this way, customized content that the user likes can be accurately recommended. Furthermore, the accumulated rewards for pre-registered content can be initialized or discounted periodically or irregularly, allowing the reinforcement learning model (or recommendation results) to quickly reflect changes in user preferences. In addition, when registering new content, the accumulated rewards of existing content can be initialized or discounted. In this case, the problem of new content being excluded from recommendation targets due to existing content can be easily solved. Furthermore, machine learning models can be used to predict rewards for user feedback or the parameter values ​​of a probability distribution (e.g., a beta distribution) corresponding to recommended content. In this case, rewards for feedback can be accurately calculated and reflected in the reinforcement learning model. Furthermore, by utilizing monitoring information about the user's subsequent actions, false feedback in recommended content can be accurately identified. This can prevent performance degradation of reinforcement learning models caused by false feedback. Furthermore, various feedback information related to the selection or playback of recommended content can be used to update (train) the reinforcement learning model, thereby improving the recommendation accuracy of the reinforcement learning model. The effects of the technical idea of ​​the present disclosure are not limited to the above-mentioned effects, and those skilled in the art will clearly understand other effects not mentioned from the following description. Next, multiple embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The advantages and features of the present disclosure, as well as the methods for achieving these advantages and features, can be clearly understood by referring to the detailed embodiments and drawings described later. However, the technical concepts of the present disclosure are not limited to the following embodiments and can be implemented in various forms. The purpose of the following embodiments is only to make the technical concepts of the present disclosure more complete and to fully inform those with ordinary knowledge in the technical field of the present disclosure of the scope of the present disclosure. The technical concepts of the present disclosure are defined only by the scope of the patent application. Furthermore, when describing the various embodiments of the present disclosure, if it is determined that detailed descriptions of related well-known structures or functions would obscure the gist of the embodiments of the present disclosure, their detailed descriptions will be omitted. Unless otherwise defined, the terms (including technical and scientific terms) used in the following embodiments may be used with the meanings that are generally understood by a person of ordinary skill in the art to which the present disclosure belongs, but this may vary depending on the intentions of a person of ordinary skill working in the relevant art, precedents, the emergence of new technologies, etc. The terms used in the present disclosure are for the purpose of describing the embodiments, not for the purpose of limiting the scope of the present disclosure. In the following embodiments, expressions in the singular include the plural concept unless the context clearly indicates the singular. Also, expressions in the plural include the singular concept unless the context clearly indicates the plural. In addition, the terms such as first, second, A, B, (a), (b) etc. used in the following embodiments are only used to distinguish certain technical features from other technical features, and the nature, order or sequence of the technical features are not limited by the above terms. Various embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. FIG1 is an exemplary structural diagram illustrating a content service providing system according to some embodiments of the present disclosure. In FIG1 and other figures, a content recommendation system 10 is shown as “recommendation system 10” and will be referred to as “recommendation system 10” hereinafter for ease of description. As shown in FIG1 , a content service providing system according to an embodiment may be a system that provides content services to a plurality of user terminals 12 - 1 to 12 - n. For example, the content service providing system may be a system (platform) that provides an OTT (Over-The-Top) service or a content streaming service. However, the scope of the present disclosure is not limited thereto. Hereinafter, when referring to any user terminal (e.g., 12 - 1 ) or collectively referring to all user terminals 12 - 1 to 12 - n, reference numeral 12 is used. As shown in the figure, the content service providing system can be configured to include a recommendation system 10 and a content providing system 11. FIG1 shows that the content providing system 11 and the recommendation system 10 are implemented in separate physical computing devices / systems, but this is only for the purpose of functionally distinguishing the structural features 10 and 11 of the content service providing system. In actual environments, the two structural features 10 and 11 can be implemented in an integrated form within the same physical computing device / system, which is self-evident. The content provision system 11 may be a computing device / system for providing content to users. For example, the content provision system 11 may be a system (platform) that streams requested content from pre-registered content to the user terminal 12. In this case, the content provision system 11 may also be referred to as a "content streaming system (platform)" or "content streaming server." The content provision system 11 may be implemented in any form. For example, the content provided by the service can be in various formats, such as video (images), images, audio (voice), text, and multimedia, and can take any form. Furthermore, the service content can encompass various types of content, such as movies, television programs, serials, webtoons, music, and news. However, for ease of understanding, the following description assumes that the service content is in a playable format, such as multimedia and video. Furthermore, the recommendation system 10 may be a computing device / system for recommending content to users. In other words, the recommendation system 10 may determine recommended content for users and provide the recommended content to the users directly or through the content provision system 11. For example, the recommended content may be presented (provided) to the user by displaying it in the hero area of ​​a content service page (screen). For this purpose, reference will be made to Figures 3 through 5 . As shown in Figure 2 , the recommendation system 10 can use a reinforcement learning model to determine recommended content. Specifically, the recommendation system 10 can repeatedly perform the following process: determining recommended content using the reinforcement learning model 20, obtaining user feedback on the recommended content, and updating the reinforcement learning model 20 using the obtained feedback. In this way, the reinforcement learning model 20 can be trained to accurately recommend user-customized content. This will be explained in more detail with reference to the figures following Figure 6 . Feedback (e.g., user actions related to recommended content) information includes all types of feedback-related information that can be obtained from users, such as direct (explicit) feedback, indirect (implicit) feedback, positive feedback, negative feedback, etc. For example, such feedback information may include: content selection-related information (e.g., whether a content page link was clicked, the number of clicks, the number of other content searches / played / clicked before selection, the time required to make a selection, etc.); content playback / viewing / consumption-related information (e.g., whether the play button was clicked, the number of pauses during playback, the duration of playback, the total playback time, whether the content was played to completion, the time required to play, the number of other content searches / played / clicked before playback, whether to continue watching when switching devices, the time required to complete playback, etc.); bookmarking-related information (e.g., whether the bookmark button was clicked, whether the unbookmark button was clicked, etc.); sharing-related information; and comment-related information (e.g., whether a comment was written and the content of the comment, etc.). However, the scope of this disclosure is not limited to this. For further information on feedback information, please refer to the description of Figures 3 to 5. For reference, the term "viewing" can be interpreted as encompassing meanings such as "listening" and "appreciating," etc. For example, when the content is in the form of audio, viewing the content can be interpreted as listening to or appreciating the content. The reward (or reward score) used to update (train) the reinforcement learning model 20 can be calculated based on the feedback information of the example. However, the specific calculation method can be designed in many ways. For example, the shorter the time required until the recommended content is selected or played, the higher the reward value can be calculated. As another example, the fewer contents the user has selected or played before selecting or playing the recommended content, the higher the reward value may be calculated. As another example, a higher reward value may be calculated for a longer playback time or continuous playback time of the recommended content. As another example, the fewer the number of pauses, the higher the value of the reward can be calculated. As another example, the shorter the time required to fully view the recommended content, the higher the value of the reward can be calculated. As another example, when the user continues to watch the recommended content through other terminals, a higher reward value can be calculated. As another example, when a recommended content is played or saved, a higher reward may be given than when the recommended content is simply selected. As another example, rewards for feedback information may be calculated based on various combinations of the above examples. For more information on the reward calculation method, please refer to the description of Figures 10 to 12. For reference, the recommendation system 10 can obtain user feedback information through the content providing system 11 (eg, a server providing content service pages) or directly from the user terminal 12. The feedback information can be obtained in any manner. The reinforcement learning model 20 can be based on, for example, a multi-armed bandit (MAB) algorithm. More specifically, for example, the reinforcement learning model 20 can be based on Thompson sampling, upper confidence bound (UCB), an ε-greedy algorithm, or the like. However, for ease of understanding, the following description assumes that the reinforcement learning model is based on Thompson sampling. However, the scope of this disclosure is not limited to this. The technical concepts of this disclosure are not directly related to the Beta distribution and can be applied to other types of MAB algorithms without substantial modification. The recommendation system 10 described above can be implemented on at least one computing device. For example, all functions of the recommendation system 10 can be implemented on a single computing device, or a first function of the recommendation system 10 can be implemented on a first computing device and a second function on a second computing device. Furthermore, specific functions of the recommendation system 10 can be implemented on multiple computing devices. A computing device may encompass any device with computing functionality, and examples of such devices will be seen with reference to FIG. 20 . For reference, because a computing device is a collection of interacting technical features (e.g., memory, processor, etc.), it can in some cases be referred to as a "computing system." Of course, the term "computing system" can also encompass the concept of a collection of multiple interacting computing devices. The user terminal 12 may refer to a terminal used by a user to receive content services. In addition, the user terminal 12 may be implemented by various types of computing devices such as smart phones, notebook computers, and desktop computers, and may be implemented in any device. Users can access various content through the terminal 12. For example, users can access and log in to the content service page through the terminal 12 to watch desired content, or receive recommended content. Furthermore, users can also provide various feedback regarding the recommended content through the terminal 12. As shown in FIG1 , the content service providing system 100 and the user terminal 12 can communicate via a network. The network can be implemented by any type of wired or wireless network, such as a local area network (LAN), a wide area network (WAN), a mobile radio communication network, or a wireless broadband Internet (Wibro). Thus far, a content service provision system according to some embodiments of the present disclosure has been briefly described with reference to Figures 1 and 2 . Below, schematic diagrams illustrating exemplary methods for providing (displaying) recommended content and user feedback information according to some embodiments of the present disclosure will be briefly described with reference to Figures 3 through 5 . However, for clarity of the present disclosure, the reference numbers of the reinforcement learning model 20 may be omitted when not directly referencing the accompanying figures, and the reference numbers of the reinforcement learning model 20 may be changed as necessary. FIG3 illustrates providing recommended content via a profile selection page 30 within a content service page according to some embodiments of the present disclosure. The profile selection page 30 may be, for example, a page (screen) displayed on the user terminal 12 upon successful login (e.g., when the content service supports multiple profiles), but the scope of the present disclosure is not limited thereto. 3 , the recommended content determined by the recommendation system 10 may be displayed in a main area 34 of the profile selection page 30. For example, information such as static images, short videos (clips), and text of the recommended content may be displayed in the main area 34. However, the scope of the present disclosure is not limited thereto. For reference, the main area 34 may refer to an area occupying a relatively large proportion or a visually prominent area in the corresponding page 30. Those generally skilled in the art are already familiar with the concept of the main area, and thus an explanation thereof will be omitted. In this embodiment, the recommendation system 10 uses the user's login information to select a reinforcement learning model (i.e., a customized / personalized reinforcement learning model) corresponding to the corresponding user (or user type), and can determine recommended content using the selected reinforcement recommendation model. The plurality of profiles 31 to 33 may or may not correspond to the user in a 1:1 manner. However, in the following, for ease of understanding, it is assumed that one profile (eg, 31) represents one user. FIG4 illustrates a display of recommended content on a main service page 40 of a content service page according to some embodiments of the present disclosure. The main service page 40 may be, for example, a page (screen) displayed on the user terminal 12 when a profile (e.g., 31) is selected or a user successfully logs in, but the scope of the present disclosure is not limited thereto. As shown in FIG4 , the main service page 40 may include a category menu area 41, a main area 42, and other content areas (e.g., 43). Furthermore, the main service page 40 may include a representative information area 46 for recommended content, a play button 47, and a content indicator 48. The main service page 40 may provide a content search function based on categories, themes, etc. (see 45), and may also provide a profile change function (see 44), etc. Similar to the above-mentioned content, the recommended content may be displayed in the main area 42 of the main service page 40. In addition, representative information 46 of the recommended content (eg, title, genre, actors, etc.) may be displayed near the main area 42. For reference, the content indicator 48 may be understood as having a search function for recommended contents and a function of notifying the total number of recommended contents (eg, 6). On the page 40 , for example, when the user selects the play button 47 , feedback information related to the content playing may be generated. Furthermore, for example, when a user selects (clicks) the representative information area 46 or the main area 42 of recommended content, feedback information related to the content selection may be generated (for example, when a link to a recommended content page is embedded in the main area 42). In this case, the screen of the user terminal 12 moves from the main service page 40 to the recommended content page (see 50 in FIG. 5 ), and feedback information related to the content selection may be generated. 5 shows a content page 50 according to some embodiments of the present disclosure. The content page 50 may be, for example, a page (screen) displayed on the user terminal 12 when the user selects corresponding content (eg, selects the main area 42), but the scope of the present disclosure is not limited thereto. As shown in FIG5 , the content page 50 may include a category menu area 41 , a main area 51 , and a content details area 52 . Furthermore, the content page 50 may also include a play button 53 , a favorites button 54 , a share button 55 , and the like. The content of the corresponding page 50 may be displayed in the main area 51 . On the above page 40, for example, when the user selects the play button 53, feedback information related to the content playback can be generated. Of course, even when the user is watching or finishes watching the content, feedback information such as the number of pauses and the total playback (viewing) time can also be generated. In addition, feedback information related to collection and sharing can also be generated through the collection button 54 and the share button 55. 3 to 5 illustrate examples of providing recommended content and user feedback information according to some embodiments of the present disclosure. As described above, providing recommended content in the primary area (e.g., 42) of the content service page maximizes the effectiveness of the recommended content and easily attracts user interest and attention. The following describes various methods that can be performed in the recommendation system 10 described above, with reference to FIG6 and subsequent figures. However, for ease of understanding, the description will proceed assuming that all steps / operations of the methods described below are performed in the recommendation system 10 described above. Therefore, if the subject matter of a particular step / operation is omitted, it can be understood that it is performed in the recommendation system 10. However, in actual environments, the specific steps / operations of the methods described below can also be performed in other computing devices. FIG6 is an exemplary flow chart schematically illustrating a content recommendation method according to some embodiments of the present disclosure. However, this is only a preferred embodiment for achieving the purpose of the present disclosure, and it is self-evident that some steps may be added or deleted as needed. As shown in FIG6 , the content recommendation method according to an embodiment may begin with step S61 of initializing a reinforcement learning model. A reinforcement learning model may refer to a model that is customized (specialized) for a specific user (or a specific type of user) (e.g., a model that learns feedback information from a specific user to recommend content to that user). For reference, user types may be classified based on user demographic characteristics (eg, gender, age group, etc.), viewing history, etc. (eg, using clustering techniques, etc.), but the scope of the present disclosure is not limited thereto. Meanwhile, the specific method of initializing the reinforcement learning model may vary according to the embodiment. In some embodiments, initialization can be performed (completed) by assigning initial values ​​to the parameters of the reinforcement learning model (e.g., cumulative rewards, parameters (element numbers) of the probability distribution, etc.) by the recommendation system 10. For example, if the reinforcement learning model is based on Thompson sampling and Beta distribution, the recommendation system 10 can assign appropriate initial values ​​(e.g., "1") to the parameters of the Beta distribution corresponding to each content. In this case, the initial value can be a preset fixed value, or it can be a value dynamically determined based on the characteristics of the content, the type of user, the user's viewing history, or feedback information. For example, the recommendation system 10 can collect user feedback information for specific content (e.g., new content) and determine the initial value therefrom. For related information, please refer to the description of Figure 17. In other embodiments, as shown in FIG7 , the recommendation system 10 can predict the probability distribution of specific content (i.e., the probability distribution of recommendation success or reward) using a trained machine-learning model 71, and can use the predicted probability distribution as the initial probability distribution for the corresponding content. For example, the recommendation system 10 can use the machine-learning model 71 to predict the probability distribution corresponding to the corresponding content from user feedback information 72 for the specific content (e.g., collected through random recommendations, etc.). The machine-learning model 71 can be configured to receive the feedback information 72 and output confidence scores 73 for various probability distributions. Depending on the situation, it can also be configured to further receive user profile information (e.g., gender, age group, etc.) and content feature information (e.g., genre, actor, etc.). The machine-learning model 71 can be trained using a training set consisting of feedback information (e.g., reference 72) and correct answer probability distributions (i.e., correct answer category labels). Such a training set can be generated, for example, by analyzing the recommendation history of the corresponding user or users (e.g., extracting past feedback information and the probability distribution used for recommendations from a history of good recommendation results to generate the training set). The machine learning model 71 may be implemented based on a neural network (eg, an RNN-based model that receives a sequence of feedback, a self-attention-based model), but the scope of the present disclosure is not limited thereto. In some cases, the recommendation system 10 may use the machine learning model 71 of the aforementioned embodiment to determine a probability distribution to be applied to new content or to change the probability distribution of specific content that meets specified conditions (e.g., content that consistently receives negative feedback information). In some other embodiments, the reinforcement learning model may be initialized based on various combinations of the above embodiments. In step S62, initial training of the reinforcement learning model can be performed. For example, the recommendation system 10 can collect user feedback by randomly selecting recommended content within a certain time period. Furthermore, the recommendation system 10 can use the collected feedback information to perform initial training of the reinforcement learning model. In some cases, step S62 can be omitted. For example, before the reinforcement learning model is fully trained, the recommendation results will be similar to those of random recommendations (i.e., the proportion of exploration will be much greater than that of exploitation). Therefore, the initialized reinforcement learning model can be directly used for actual recommendations. In step S63, the content recommendation process and the feedback-based update process can be repeatedly performed using the reinforcement learning model. FIG8 shows the detailed process of this step. In step S81, a reinforcement learning model may be used to determine recommended content. For example, the recommendation system 10 may use Thompson sampling to determine at least one recommended content from the probability distribution corresponding to each content. The number of recommended content may vary depending on the situation. The operation of Thompson sampling is well known to those skilled in the art, so its explanation will be omitted. In some embodiments, as shown in FIG9 , the recommendation system 10 can screen multiple candidate content from pre-registered content and use a reinforcement learning model to determine recommended content from the multiple candidate content ( S91 , S92 ). For example, the recommendation system 10 can screen candidate content based on registration time, advertising costs, production costs, content production entity, and user viewing history analysis results. For example, the recommendation system 10 can screen new (recently registered) content, content with advertising costs / production costs exceeding a standard value, content produced directly by content service providers, content produced by specific content producers, and preferred content based on the user's viewing history (e.g., content of similar types to previously viewed content or content with the same actors). In some other embodiments, the recommendation system 10 can determine the user's current context and use a reinforcement learning model corresponding to the determined context to determine recommended content. This can further improve the accuracy of recommendations, as will be described with reference to Figures 18 and 19. In some other embodiments, recommended content may be determined based on various combinations of the above embodiments. Depending on the situation, the recommendation system 10 may determine additional recommended content based on the rules. For example, the recommendation system 10 may also recommend previously sampled content (eg, new content, content with high advertising costs, etc.) to the user based on the rules. In step S82 , user feedback information regarding the recommended content may be obtained. In step S83, the reinforcement learning model can be updated (trained) by reflecting the rewards for the feedback information. Updating the reinforcement learning model may mean updating the parameters of the reinforcement learning model used for content recommendation (e.g., the cumulative rewards for each content, the UCB for each content, the number of recommendations for each content, the probability distribution parameters for each content, etc.). However, the specific methods (e.g., the reward calculation method, the update method, etc.) may vary depending on the embodiment. In some embodiments, the recommendation system 10 can calculate a first reward by aggregating (e.g., summing) the rewards for at least one piece of positive feedback information, and aggregating the rewards for at least one piece of negative feedback information (e.g., not selected) to calculate a second reward. In this case, the rewards for each piece of feedback information can be pre-set. Furthermore, the recommendation system 10 can update the reinforcement learning model based on the first and second rewards. For example, the recommendation system 10 can update the first parameter "α" of the Beta distribution corresponding to the recommended content based on the first reward (e.g., summing), and update the second parameter "β" of the Beta distribution based on the second reward (using Thompson sampling). In other embodiments, as shown in FIG10 , the recommendation system 10 can use a trained machine learning model 101 to predict rewards 103 (i.e., total rewards) for multiple pieces of feedback 102. Furthermore, the recommendation system 10 can update a reinforcement learning model based on the rewards 103. For example, the recommendation system 10 can use a first machine learning model to predict a first reward for multiple pieces of positive feedback, and a second machine learning model to predict a second reward for multiple pieces of negative feedback. Furthermore, the recommendation system 10 can update the parameters of a beta distribution corresponding to the recommended content based on the first and second rewards. For example, the training set for the machine learning model 101 can consist of different types of feedback and rewards for correct answers. Alternatively, for example, the training set can be generated using feedback information included in the recommendation history of a corresponding user or multiple users, and the reward for correct answers can be calculated based on, for example, the total play time of the recommended content and the time required to fully watch the content. However, the scope of this disclosure is not limited to this. As shown in the figure, the machine learning model 101 can be implemented based on a neural network, but the scope of this disclosure is not limited to this. In some other embodiments, as shown in FIG11 , the recommendation system 10 can determine (predict) the reward weight assigned to each of a plurality of feedback information (e.g., 112) (e.g., refer to W1 to W3) through a trained machine learning model 111. Furthermore, the recommendation system 10 can calculate (e.g., by summing the weights) the reward (i.e., the total reward) for the plurality of feedback information (e.g., 112) based on the determined reward weights. The machine learning model 111 can be configured to receive different types of feedback information 112 and output a predicted reward weight for each feedback information, and to perform training based on the difference between the reward 113 calculated based on the plurality of predicted reward weights (e.g., summed) and the correct answer reward 114. In other embodiments, as shown in FIG12 , the recommendation system 10 can use a trained machine learning model 121 to predict a parameter value 123 of a probability distribution (e.g., a Beta distribution) corresponding to recommended content 124 from at least one piece of feedback information 122. Furthermore, the recommendation system 10 can update the corresponding probability distribution based on the predicted parameter value 123. The machine learning model 121 can be configured to receive feedback information (e.g., 122) and output a parameter value (e.g., 123) that reflects the probability distribution of rewards. The training set for such a model 121 can consist of feedback information and correct answer parameter values. In some other embodiments, the reinforcement learning model may be trained based on various combinations of the above embodiments. Figure 13 illustrates the process of updating a Thompson sampling-based reinforcement learning model. As shown in Figure 13 , the probability distribution corresponding to each of the content items 131 to 133 can be gradually updated based on the user's feedback information (i.e., reward information), thereby accurately recommending customized content (e.g., item 131) that the user prefers. In addition, the recommendation system 10 can update the reinforcement learning model immediately each time feedback information is obtained, or it can perform updates at a time delayed from the acquisition time (for example, periodic updates, updates when the number of feedback exceeds a standard value, etc.). For example, the recommendation system 10 can perform immediate updates only for some content (for example, new content, content with accumulated rewards below a standard value, content with high advertising costs / production costs, content produced directly by the content service operator, etc.), and perform immediate / delayed updates for all content. Alternatively, considering that the impact of feedback information on the model decreases as the reinforcement learning model is trained, the recommendation system 10 can increase the update delay time (for example, the update period) over time. Delayed updates will be further explained later with reference to Figures 14 and 15. This will be described again with reference to FIG. 8 . In step S84, the updated reinforcement learning model may be used to re-perform content recommendation for the user. For example, the recommendation system 10 may determine one or more recommended content in the same manner as in step S81 above. Thus far, a content recommendation method according to some embodiments of the present disclosure has been briefly described with reference to Figures 6 and 13. Based on the above, a reinforcement learning model can be gradually updated based on user feedback information, and the updated reinforcement learning model can be used to determine recommended content for the corresponding user. Thus, customized content that the user likes can be accurately recommended. Furthermore, a machine-trained model can be used to predict the reward for feedback information or to predict the parameter values ​​of a probability distribution (e.g., a beta distribution) corresponding to the recommended content. In this case, the reward for the feedback information can be accurately calculated and accurately reflected in the reinforcement learning model. Hereinafter, a delayed update method according to some embodiments of the present disclosure will be described with reference to FIG. 14 . As shown in FIG14 , the recommendation system 10 can collect and accumulate feedback information 142, 143 for recommended content over a certain period of time, starting from the time the content is recommended (see “C”). Furthermore, the recommendation system 10 can calculate rewards for the accumulated feedback information 142, 143 and update the reinforcement learning model (e.g., updating the probability distribution of content C) by reflecting the calculated rewards. In other words, the recommendation system 10 can perform the update of the reinforcement learning model at a delayed time (see “T4”) rather than immediately. The above-mentioned certain period may be a pre-set value or a value determined based on the length of the content (ie, learning time), the update cycle of the reinforcement learning model, etc. Delayed updates not only alleviate the increased load caused by frequent model updates, but also allow for the idle time between updates to identify false feedback. False feedback refers to feedback that doesn't align with the user's original intent, such as incorrect clicks. This type of feedback negatively impacts the performance of reinforcement learning models, so it's best to remove it before updating the model whenever possible. The following describes methods for addressing this false feedback issue. FIG. 15 is a schematic diagram illustrating a false feedback determination method according to some embodiments of the present disclosure. As shown in Figure 15 , assume that after recommending content (see "C"), recommendation system 10 receives feedback 152 from user 151. Recommendation system 10 can then collect monitoring information about user 151's subsequent actions to determine whether feedback 152 is fraudulent. Figure 15 illustrates monitoring performed directly by recommendation system 10, but other devices can also perform the monitoring. Then, the recommendation system 10 can determine whether the obtained feedback 152 is false feedback based on the collected monitoring information (see 153). For example, if the subsequent action of the user 151 meets a predefined action pattern (for example, the action pattern of returning to the previous page in a very short time after clicking on the content, the action pattern of randomly searching the content area, etc.), the recommendation system 10 can determine that the feedback 152 is false feedback. In response to such a judgment result, the recommendation system 10 can delete the feedback 152. In other words, the recommendation system 10 can delete the feedback 152 before updating the reinforcement learning model. In this way, the recommendation performance of the reinforcement learning model can be improved. In some cases, the recommendation system 10 may also determine false feedback using a machine learning model. For example, the recommendation system 10 may perform the determination using a machine learning model that receives subsequent actions (e.g., a sequence of actions) following a specific feedback and outputs a confidence score for the false feedback. So far, embodiments related to delayed update have been described with reference to Figures 14 and 15. Next, referring to Figure 16, a method for initializing and discounting accumulated rewards according to some embodiments of the present disclosure will be described. Even when using reinforcement learning models, in some cases, some content may be continuously recommended or excluded from the recommendation list. For example, when users repeatedly receive positive feedback about a particular piece of content, the cumulative reward increases rapidly (for example, the value of the first parameter 'α' of the beta distribution increases rapidly), leading to the phenomenon of persistent recommendation of that particular piece of content (i.e., the proportion of exploitation becomes excessive). In this case, the constant appearance of the same content in the recommendation list may cause users to lose interest in the recommended content. Furthermore, when users' preferences change, their interest in recommended content may also decrease. As another example, when existing content with high cumulative rewards is repeatedly recommended, a problem may arise in which new content is continuously excluded from the recommendation list. In other words, a problem may arise in which new content does not have a suitable opportunity to secure rewards. To address the above issues, the recommendation system 10 can initialize or discount the accumulated rewards for pre-registered content. Initializing or discounting the accumulated rewards can include initializing or discounting the parameter values ​​of a probability distribution (e.g., a Beta distribution) determined by the accumulated rewards. Depending on the circumstances, the recommendation system 10 can also simultaneously initialize or discount other parameters, such as the number of content recommendations. Specifically, as shown in FIG16 , the recommendation system 10 can initialize or discount the accumulated rewards for existing content 161 and 162 in response to the registration of new content 163. This can solve the problem of new content 163 being continuously excluded from recommendation targets. Of course, it can also solve the problem of some existing content (e.g., 161) being continuously recommended. As another example, to solve the above problem, the recommendation system 10 may periodically initialize or discount the accumulated rewards of pre-registered content. As another example, the recommendation system 10 may select content that meets preset conditions and initialize or discount the accumulated rewards of only the selected content. For example, the recommendation system 10 may select content whose number of recommendations or accumulated rewards exceeds a standard value from pre-registered content. As yet another example, initialization or discount processing of accumulated rewards may be performed based on various combinations of the above examples. The above method can also be used to quickly reflect changes in user preferences in a reinforcement learning model. For example, the recommendation system 10 can periodically or irregularly initialize or discount the accumulated rewards of the reinforcement learning model or a portion of the content. This can increase the influence of recent user feedback on the reinforcement learning model, allowing changes in user preferences to be quickly reflected in the model. Furthermore, according to some other embodiments of the present disclosure, as shown in FIG17 , the recommendation system 10 can address the aforementioned issue by appropriately setting the initial reward for new content 173, rather than discounting the accumulated rewards for existing content 171 and 172. Specifically, when registering new content 173, the recommendation system 10 can recommend the new content 173 to a large number of users to quickly collect a large amount of feedback information. The recommendation system 10 can then calculate the initial reward for the new content 173 based on the collected feedback information. This can address the issue of new content 173, which is of general interest to a large number of users, being excluded from the recommendation targets for a specific user. So far, embodiments related to the method for initializing the accumulated reward, etc. have been described with reference to Figures 17 and 18. Next, a context-based content recommendation method according to some embodiments of the present disclosure will be described with reference to Figures 18 and 19. This embodiment relates to a method for further improving recommendation accuracy by considering the user's current context. The user context can be determined (differentiated) based on the type of user terminal (i.e., access terminal) 12 (e.g., mobile phone, desktop computer, etc.), the layout of the content service page (e.g., the size and location of the main area, etc.), and time information (e.g., weekday / weekend, time period, day of the week, etc.), but the scope of this disclosure is not limited thereto. Specifically, as shown in Figure 18, assume that three contexts 181 to 183 are defined, and a reinforcement learning model (see 184 to 186) is constructed for each context. These reinforcement learning models 184 to 186 can be constructed using feedback information collected from the corresponding contexts 181 to 183. In this case, the recommendation system 10 can determine the user's current context based on information such as the type of the user's terminal and the current time. Furthermore, the recommendation system 10 can use the model (e.g., probability distribution) from among the multiple reinforcement learning models 184 to 186 that corresponds to the user's current context to determine recommended content. This allows the recommendation system 10 to consider context-dependent user preferences (e.g., content viewed varies depending on time of day, weekends, etc.) when determining recommended content. Ultimately, this can significantly improve user satisfaction with recommended content. Furthermore, in some embodiments, as shown in FIG19 , the model parameters for only a portion of the pre-registered content (referenced as 191 ) can be subdivided by context. For example, a specific content 191 may have multiple probability distributions corresponding to different contexts 192 through 194 , while other content may not. Thus, when determining whether to recommend the corresponding content 191 , the recommendation system 10 can use the probability distribution corresponding to the user's current context among the multiple probability distributions. 18 and 19 , the context-based content recommendation method according to some embodiments of the present disclosure has been described. As described above, by considering the user's current context to determine recommended content, the accuracy of recommendations can be further improved. An exemplary computing device 200 capable of implementing the recommendation system 10 of some embodiments of the present disclosure is described below with reference to FIG. 20 . FIG. 20 is a diagram illustrating an exemplary hardware structure of a computing device 200 . As shown in Figure 20, the computing device 200 may include one or more processors 201, a bus 203, a communication interface 204, a memory 202 for loading a computer program executed by the processor 201, and a storage 205 for storing a computer program 206. However, Figure 20 only shows the technical features related to the embodiments of the present disclosure. Therefore, a person with ordinary knowledge in the field to which the present disclosure belongs can know that in addition to the technical features shown in Figure 20, other general technical features may also be included. In other words, the computing device 200 may also include various technical features in addition to the technical features shown in Figure 20. In addition, in some cases, the computing device 200 may be configured in a form that omits some of the technical features shown in Figure 20. In the following, each technical feature of the computing device 200 will be explained. The processor 201 can control the overall operation of the various components of the computing device 200. The processor 201 can include at least one of a central processing unit (CPU), a microprocessor unit (MPU), a microcontroller unit (MCU), a graphics processing unit (GPU), a neural processing unit (NPU), or any other processor known in the industry. Furthermore, the processor 201 can execute calculations for at least one application or program to perform operations / methods according to the embodiments of the present disclosure. The computing device 200 can have one or more processors. Furthermore, memory 202 can store various data commands and / or information. Memory 202 can load computer program 206 from storage 205 to execute the methods and / or operations of the embodiments of the present disclosure. Memory 202 can be implemented as volatile memory such as random access memory (RAM), but the technical scope of the present disclosure is not limited thereto. Furthermore, the bus 203 can provide a communication function between the technical features of the computing device 200. The bus 203 can be implemented by various types of buses, such as an address bus, a data bus, and a control bus. In addition, the communication interface 204 can support wired and wireless communication of the computing device 200. In addition, the communication interface 204 can also support various communication methods other than network communication. To this end, the communication interface 204 can include a communication module known in the art to which this disclosure pertains. In addition, the storage 205 can non-temporarily store one or more computer programs 206. The storage 205 can include non-volatile memory such as read-only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, a hard drive, a removable hard drive, or any other computer-readable recording medium known in the art to which the present disclosure pertains. Furthermore, the computer program 206 may include one or more instructions that, when loaded into the memory 202, cause the processor 201 to perform operations / methods according to various embodiments of the present disclosure. In other words, the processor 201 may perform operations / methods according to various embodiments of the present disclosure by executing the one or more loaded instructions. For example, the computer program 206 may include instructions for performing the following operations: using a reinforcement learning model to determine recommended content to provide to a user; obtaining user feedback information regarding the recommended content; updating the reinforcement learning model by reflecting rewards corresponding to the feedback information; and re-performing content recommendations for the user using the updated reinforcement learning model. In this case, the recommendation system 10 according to some embodiments of the present disclosure may be implemented by the computing device 200. Furthermore, in some embodiments, computing device 200 shown in FIG. 20 may represent a virtual machine implemented using cloud technology. For example, computing device 200 may be a virtual machine running on one or more physical servers included in a server farm. In this case, at least a portion of processor 201, memory 202, and storage 205 shown in FIG. 20 may be virtual hardware, and communication interface 204 may also be implemented using virtualized network elements such as a virtual switch. Thus far, an exemplary computing system 200 capable of implementing the recommendation system 10 of some embodiments of the present disclosure has been described with reference to FIG. 20 . The various embodiments of the present disclosure and their effects have been described above with reference to Figures 1 to 20. The effects of the technical concept of the present disclosure are not limited to the effects described above, and those skilled in the art will clearly understand other effects not described below. Furthermore, although the above embodiments describe multiple technical features combined into one or in combination, the technical concepts of the present disclosure are not necessarily limited to these embodiments. In other words, within the scope of the purpose of the technical concepts of the present disclosure, all technical features can be selectively combined into one or more for operation. The technical concepts of the present disclosure described so far can be implemented by computer-readable code on a computer-readable medium. The computer program recorded on the computer-readable recording medium can be transmitted to another computing device via a network such as the Internet and installed in the other computing device, thereby being used in the other computing device. Although various operations are shown in a specific order in the drawings, it should not be understood that the operations must be performed in the specific order shown or in sequence, or that all the shown operations must be performed to obtain the desired results. In some cases, multi-processing and parallel processing may also be advantageous. The above describes various embodiments of the present disclosure with reference to the drawings, but a person of ordinary skill in the art to which the present disclosure belongs will understand that the technical ideas of the present disclosure can be implemented in other specific forms without changing the technical ideas or essential features of the present disclosure. Therefore, it should be understood that the embodiments described above are exemplary and non-restrictive in all aspects. The scope of protection of the present disclosure should be interpreted by the attached patent application scope, and all technical ideas within its equivalent scope should be understood to be included in the scope of rights of the technical ideas defined by the present disclosure. 10: Recommendation system 11: Content provision system 12-1, 12-2, 12-n: User terminal 20: Reinforcement learning model 30: Profile selection page 31, 32, 33: Profile 34, 42, 51: Main area 40: Main service page 41: Category function table 43: Other content area 44: Profile change function 45: Content search function 46: Representative information area of ​​recommended content 47, 53: Play button 48: Content indicator 50: Recommended content page 52: Content detailed information area 54: Favorite button 55: Share button S61, S62, S63: Steps 71, 101, 111, 121: Machine learning model 72, 102, 112, 122: Feedback information 73: Confidence score S81 , S82, S83, S84: Steps S91, S92: Steps 103, 113: Prediction reward 114: Correct answer reward 123: Parameter value of probability distribution 124: Recommended content 131, 132, 133: Content 141, 151: User 142, 143, 152: Feedback information 153: False feedback judgment 161, 162, 171, 172: Existing content 163, 173: New content 181, 182, 183, 192, 193, 194: Context 184, 185, 186: Reinforcement learning model 191: Content 200: Computing device 201: Processor 202: Memory 203: Bus 204: Communication interface 205: Storage 206: Computer program FIG. 1 is an exemplary structural diagram illustrating a content service providing system according to some embodiments of the present disclosure. FIG. 2 is a schematic diagram for schematically illustrating the operation of a content recommendation system according to some embodiments of the present disclosure. 3 to 5 are schematic diagrams illustrating examples of methods for providing recommended content and user feedback information according to some embodiments of the present disclosure. FIG6 is an exemplary flow chart schematically illustrating a content recommendation method according to some embodiments of the present disclosure. FIG. 7 is a schematic diagram illustrating a probability distribution prediction method according to some embodiments of the present disclosure. FIG. 8 is an exemplary flowchart illustrating a detailed process of the content recommendation and update step S63 shown in FIG. 6 . FIG. 9 is an exemplary flowchart illustrating a detailed process of the recommended content determination step S81 shown in FIG. 8 . FIG. 10 is a schematic diagram illustrating a reward calculation method according to some embodiments of the present disclosure. FIG11 is a schematic diagram for illustrating a reward calculation method according to other embodiments of the present disclosure. FIG. 12 is a schematic diagram illustrating a reward reflection method according to some embodiments of the present disclosure. FIG13 illustrates an updating process of a reinforcement learning model according to some embodiments of the present disclosure. FIG. 14 is a schematic diagram illustrating a delayed update method according to some embodiments of the present disclosure. FIG. 15 is a schematic diagram illustrating a false feedback determination method according to some embodiments of the present disclosure. FIG. 16 is a schematic diagram illustrating a method for initializing and discounting a cumulative reward according to some embodiments of the present disclosure. FIG17 is a schematic diagram illustrating a method for setting an initial reward according to some embodiments of the present disclosure. 18 and 19 are schematic diagrams for illustrating a context-based content recommendation method according to some embodiments of the present disclosure. FIG. 20 illustrates an exemplary computing device capable of implementing a content recommendation system according to some embodiments of the present disclosure. S81, S82, S83, S84: Steps

Claims

1. A content recommendation method, executed by at least one computing device, comprising the following steps: determining recommended content to be provided to a user using a reinforcement learning model; obtaining feedback information from the user regarding the recommended content; updating the reinforcement learning model by reflecting a reward for the feedback information; and re-executing content recommendation for the user using the updated reinforcement learning model, wherein the reinforcement learning model is a model that determines the recommended content based on the cumulative rewards of pre-registered content, the step of obtaining the user's feedback information comprising the following steps: obtaining feedback information from the user regarding a first piece of content in the recommended content and updating the reinforcement learning model in real time; and accumulating the received feedback information from the provision of a second piece of content remaining in the recommended content excluding the first piece of content until a preset time, wherein... The first content is new content in the recommended content or content with a cumulative reward below a standard value, and the reinforcement learning model updates at or after the preset time based on the accumulated feedback information.

2. The content recommendation method as described in request item 1, wherein, The update delay time from the provision of the second content to the preset time is determined based on at least one of the length of the second content or the update cycle of the reinforcement learning model.

3. The content recommendation method as described in request item 1, wherein, The steps for accumulating received feedback information include the following: Gradually increasing the update delay time over time.

4. The content recommendation method as described in request item 1, wherein, The steps for accumulating received feedback information include the following: receiving specific feedback information; obtaining monitoring information on the user's actions after receiving the specific feedback; determining whether the specific feedback is false feedback based on the monitoring information; and deleting the specific feedback information based on the determination result of false feedback.

5. The content recommendation method as described in request item 4, wherein, The steps to determine whether a specific feedback is false feedback include the following: If the user's action after receiving the specific feedback matches a predefined action pattern, then the specific feedback is determined to be false feedback. The predefined action pattern includes the action pattern of returning to the previous page within a preset time after clicking on the recommended content.

6. The content recommendation method as described in request item 4, wherein, The steps to determine whether a specific feedback is false feedback include the following: inputting the user's action after receiving the specific feedback into a machine-learning model to output a confidence score for false feedback; and determining whether the specific feedback is false feedback based on the output confidence score.

7. The content recommendation method as described in request item 1, wherein, The steps to update the reinforcement learning model include the following: updating the parameters of the reinforcement learning model, which include the cumulative reward for each content, the upper confidence boundary (UCB) for each content, the number of recommendations for each content, and the probability distribution parameters for each content.

8. The content recommendation method as described in request item 1, wherein, The feedback information is multiple, and the steps to update the reinforcement learning model include the following: using a machine-learning model to predict the total reward of the multiple feedback information, wherein the machine-learning model is trained to output the total reward of the input feedback information; and updating the reinforcement learning model based on the predicted total reward.

9. The content recommendation method as described in request item 8, wherein, The training set of this machine learning model consists of different types of feedback information and correct answer rewards, and is generated using feedback information included in the recommendation history of the user and multiple users. The correct answer reward is calculated based on the total playback time of the recommended content and the time required for the multiple users to fully watch the recommended content.

10. A content recommendation system, comprising: One or more processors; and memory storing a computer program executed by the one or more processors, the computer program including instructions to perform the following operations: using a reinforcement learning model to determine recommended content to be provided to a user; obtaining feedback information from the user regarding the recommended content; updating the reinforcement learning model by reflecting a reward for the feedback information; and re-executing the content recommendation for the user using the updated reinforcement learning model, the reinforcement learning model being a model that determines the recommended content based on the cumulative rewards of pre-registered content, the operation of obtaining the user's feedback information including the following operations: obtaining feedback information from the user regarding a first piece of content in the recommended content and updating the reinforcement learning model in real time; and accumulating the received feedback information from the provision of a second piece of content in the recommended content other than the first piece of content until a preset time, wherein the first piece of content is new content in the recommended content or content with a cumulative reward below a standard value, and the reinforcement learning model is updated at or after the preset time based on the accumulated feedback information.

11. A computer program, stored on a computer-readable recording medium, coupled to a computing device to perform the following steps: determining recommended content to be provided to a user using a reinforcement learning model; obtaining feedback information from the user regarding the recommended content; updating the reinforcement learning model by reflecting rewards for the feedback information; and re-performing content recommendation for the user using the updated reinforcement learning model, the reinforcement learning model being a model that determines the recommended content based on accumulated rewards of pre-registered content, wherein the step of obtaining the user's feedback information includes the following steps: obtaining feedback information from the user regarding a first piece of content in the recommended content and updating the reinforcement learning model in real time; and accumulating the received feedback information from the provision of a second piece of content remaining in the recommended content excluding the first piece of content until a preset time, wherein... The first content is new content in the recommended content or content with a cumulative reward below a standard value, and the reinforcement learning model updates at or after the preset time based on the accumulated feedback information.

Images