Computer- implemented method based upon a machine-learning algorithm and a utility function for providing a recommendation regarding educational resources

Abstract

A computer-implemented method is suitable for providing a recommendation regarding educational resources to a teacher user for creating a personalized instructional course. The method involves performing on an electronic processor the step of calculating a utility value for each resource in a list of recommended educational resources by applying a utility function to preference values for each resource calculated by a probability distribution function of each resource parameter. Subsequently, the method provides, by means of a decision tree module, for predicting a resource rating for each resource in the list of recommended resources. Such decision tree module is trained by processing the calculated utility values of each resource in such a way as to provide said resource rating as a function of the utility values of each resource together with the ratings of each resource previously provided by the users and stored within an electronic storage device. Finally, the method provides for selecting one or more recommended resources on the basis of the predicted ratings for each resource and the calculated utility values.

Description

"COMPUTER-IMPLEMENTED METHOD BASED UPON A MACHINE-LEARNING ALGORITHM AND A UTILITY FUNCTION FOR PROVIDINGA RECOMMENDATION REGARDING EDUCATIONAL RESOURCES" DESCRIPTIONField of application

[0001] The present invention relates to a computer- implemented method for providing a recommendation re- garding educational resources to a teacher user for cre- ating a personalized instructional course.

[0002] Prior art

[0003] Recommendation systems attempt to predict el- ements (e.g. films, music, books, news, web pages, etc.) that a user may be interested in, based upon some infor- mation about their profile and / or data stored within a database. The extent to which an element affects a user is called relevance. A recommendation system therefore estimates the relevance of a user based upon a collection of elements. Such information is particularly useful in situations wherein the number of items is constantly growing (for example, the huge amount of content avail- able on the internet) and the amount of relevant data is relatively low.

[0004] These systems advise the user on the basis of a user profile model and the available data.

[0005] For example, in the context of film recommen- dation systems, if the article collection is composed offilms and the user to be advised is a child, the system will propose a list of cartoons because the child's pro- file corresponds to cartoons in some way and does not correspond to horror or action films.

[0006] Recommendation systems have become an im- portant area of research since the mid-1990s.

[0007] Much work has been carried out in the field over the last decade in order to develop new approaches to recommendation systems and new fields of application.

[0008] In the academic field, supporting teachers and experts in creating personalized learning paths through the selection and integration of heterogeneous educa- tional resources into training courses is a perceived problem. The overall context potentially relates to any type of organization and training community (companies, training centers, universities, educational institu- tions), which needs to significantly improve effective- ness and efficiency in the processes of creating and managing online courses or within the context of blended learning. In every organizational context, the demand for personalized training is increasingly central and demand-driven. According to a recent study, The Future of Jobs, conducted by the World Economic Forum in 2020, by 2025 at least 50% of employees will require reskilling and upskilling, i.e., operations aimed at increasing or updating the skills of the workforce.

[0009] In particular, with every type of training community, the problem of adapting the creation of courses and training paths based upon the specific struc- ture of the preferences of different teachers clearly emerges. The construction of an online course therefore requires an accurate selection of the various educa- tional resources, which must be selected, filtered, and integrated within the course in a targeted manner, ac- cording to the training, disciplinary, or organizational needs of different teachers.

[0010] Furthermore, the construction of different courses requires algorithms that are able to adapt the precise needs, constraints, or requirements of teachers with respect to the structure of the specific features of the educational resources considered, i.e., the metadata thereof, as well as considering other hetero- geneous information elements, such as user profiles or assessments provided by users (teachers) who have pre- viously benefited from the educational resources in con- sideration .

[0011] A precise and comprehensive analysis of infor- mation structures of this type, properly related accord- ing to the preferences expressed by teachers with respect to the various features of the resources, is difficult to find within the e-learning domain: precisely, the design of a recommendation system aimed at supportingteachers or experts in creating personalized content and training paths is not currently present on the market and within the e-learning sector. Most recommendation systems and, in general, smart systems within the e- learning domain, are currently focused upon supporting learners, within their training path, in particular for selecting their training paths based upon information elements such as their background, past skills, and fu- ture objectives.

[0012] As a result, the problem of creating courses and training paths, or educational materials, in a per- sonalized way that is suitable for different needs, is currently solved in an almost predominantly traditional way, through the fundamental contribution of human sup- port on the part of teachers.

[0013] In particular, the problem is generally re- solved by means of two complementary methods:

[0014] - the creation of courses ex-novo through the composition of educational material selected or produced by the teacher, in a manner consistent with their edu- cational and training experience, taking into account the needs of learners and / or of the organization for which the course is being implemented: in this case, the creation of the course is a considerably costly and time- consuming process for teachers and organizations;

[0015] - the use of proprietary educational material,typically periodically re-proposed by teachers, i.e., an already pre-existing knowledge base, which is often how- ever incomplete or inappropriate and does not meet all of the conceptual or disciplinary requirements for the specific course to be implemented, for the type of learner, or the precise training or organizational needs.

[0016] The first approach is typically used in the corporate context, as part of the training paths specif- ically organized by companies, for the training of their employees and stakeholders, with respect to the various corporate functions or specific courses organized by proprietary academies.

[0017] The second approach is typically used in aca- demic contexts; it suffers however from the general in- completeness and inadequacy of the proprietary educa- tional material periodically re-proposed to learners, often with limited adaptations or updates, if present, to the educational contents, with respect to the contin- uous evolution of the disciplines, especially in rela- tion to STEM, or to the specific needs of the course or the learners. With this approach, the instructor typi- cally tends to compensate for shortcomings by means of the lesson at the front of the class, integrating or correcting, if necessary, that proposed within the course materials during the lesson itself. Thisapproach, although traditionally valid for basic courses or those not subject to substantial modifications, sig- nificantly decreases the overall quality of the course, increases the cognitive effort, both by the instructor and the students, during lessons, and above all decreases the effectiveness of the teacher and training processes in the face of educational, disciplinary, or organiza- tional needs for greater customization of the educa- tional contents or the overall structure of the course.

[0018] Overall, the problem of generating accurate recommendations for a set of users interacting with a given set of resources is present within different do- mains of application, for example, E-commerce and mul- timedia content platforms. Such problem is addressed de- pending upon the available data, also taken into account is the complexity of space and time available in order to meet a request for a recommendation. However, deep learning recommendation systems are often preferred, which focus mainly on past interactions between users and resources, rather than on the precise structure of the users' preferences with respect to current needs. In the field of e-learning, such an approach is not effec- tive in generating adequate recommendations.

[0019] In this context, the document WO 2019 / 033423 Al describes a recommendation system for the automatic retrieval of learning content thatprovides the user with profiling that is useful for the recovery of documents that are of use to said user. In particular, such recommendation system proposes the rec- ommendation of content according to the aptitudes of a student. In particular, the student may upload their personal information, such as age, gender, preferred courses.

[0020] Solution of the invention

[0021] Specifically, within the e-learning domain, there is a strong need for a never-before-seen solution to the problem of recommending resources to a particular user as a function of heterogeneous information ele- ments, such as the structure of preferences with respect to the different metadata of the resources and the rat- ings provided (or missing) to each resource present within a given resource database.

[0022] Such needs are met by a computer-implemented method for providing a recommendation regarding educa- tional resources to a teacher user for creating a cus- tomized instructional course in accordance with the at- tached independent claim 1. The dependent claims de- scribe preferred or advantageous embodiments of the in- vention, comprising further advantageous features. Description of the drawings

[0023] The features and the advantages of the present invention shall be made readily apparent from thefollowing description of preferred exemplary embodiments thereof, provided purely by way of a non-limiting exam- ple, with reference to the accompanying figures, wherein:Figure 1 is a block diagram of an embodiment of the steps of the method according to the present invention;Figure 2 is a view of the screen for the user to enter preferences in accordance with an embodiment of step a) of the method described below;Figure 3 is a view of a screen displaying the re- sources recommended by the method according to the pre- sent invention after step e), which is detailed below, according to one embodiment of the present invention;Figure 4 is a view of a screen wherein a user has selected a resource from those shown by the method fol- lowing step e), according to one embodiment of the pre- sent invention.Detailed description

[0024] The computer-implemented method according to the present invention is suitable for providing a rec- ommendation regarding educational resources to a teacher user for creating a personalized instructional course by means of selecting educational resources, for example heterogeneous educational resources such as presenta- tions, documents, internet resources, and the like.

[0025] References to the following steps refer to theaforementioned accompanying figures.

[0026] It is clear that the fact that the method is implemented on a computer means that all of the steps of the method that will be described below may be carried out by an electronic processing unit, such as a CPU or a GPU or the like, by means of appropriate code that may be executed by such an electronic processing unit.

[0027] It is also clear that the method may also be executed by an electronic device that includes such an electronic unit.

[0028] The method according to the present invention involves performing the following steps on an electronic processor, illustrated by way of example in the diagram in Figure 1: a) receiving preference values for resource attributes that characterize each resource of a list of recommend- able educational resources, wherein said resource at- tributes include one or more disciplines whereto each of the educational resources belongs and one or more metadata characterizing the resources, for example type, language, difficulty, duration, format, minimum age, maximum age; b) converting, by means of a modeling module, the pref- erence values received in step a) into a probability distribution function for each of the resource parame- ters;c) calculating, by means of a utility function module, a utility value for each resource in the list of recom- mendable educational resources by applying a utility function to the preference values calculated from the probability distribution function of each of the re- source parameters, obtained in step b); d) predicting, by means of a decision tree module, a resource rating for each resource in the list of recom- mendable resources, said decision tree module being trained by processing the utility values of each resource calculated in step c), in such a way as to provide said resource rating as a function of the utility values of each resource together with the ratings of each resource previously provided by the users and stored within an electronic storage device; e) selecting one or more recommended resources based upon the ratings predicted for each resource in step d) and the utility values calculated in step c).According to one embodiment, step d) comprises the step of predicting a first rating, for example a positive value, for resources with utility above a first threshold, a second rating, for example a negative value, for resources with a utility value below a second threshold, and a third rating for resources with a utility between the first and second thresholds. Such third value is calculated by processing the ratings ofeach resource previously provided by the users and stored within an electronic storage device.According to one embodiment, the step of converting the preference values received in step a) into a probability distribution function for each of the resource attributes provides for a step of smoothing the probability distribution function by means of additive or Laplace smoothing, and wherein the probability values of the resource attributes for which the user has not expressed preferences have a low but non-zero probability value.According to one embodiment, the utility function is based upon the geometric mean of the preference values for resource attributes and the utility function is normalized to a range of between 0 and 1.According to one embodiment, the decision tree module predicts the resource ratings that the user may give to new resources, based upon the historical database data.

[0029] According to one embodiment, in step a), the method provides for a step wherein the user sends, through a graphical interface, the preference values to- wards the resource attributes that he / she want to see recommended. For example, they first send the preference values to the disciplines whereto the resources belong, and subsequently, for those metadata that characterize the resources present, for example one or more valuesrelating to one or more of the following metadata: type, resource, language, difficulty, duration, format, mini- mum age, maximum age.

[0030] In the remainder of the present description, unless otherwise specified, the term "preference" will, in an equivalent manner, signify a preference value.

[0031] For each resource attribute, including there- fore the disciplines, the user may send the preference values to one or more values that that resource attribute may take. For example, a user interested in resources with a specific difficulty level, such as "low", in that case selects the "low" value as a preference for the difficulty attribute. If, on the other hand, the user is interested in resources with a medium or lower difficulty level, he / she may select as preferred values the "low", "medium low" and "medium" difficulty values. The user may also decide not to express any preference value for the difficulty attribute, in which case all of the values assumed by the attribute will have the same weight in the calculation of the subsequent utility function which will be detailed hereinafter.

[0032] According to one embodiment, for the minimum age and maximum age resource attributes, the method pro- vides that the user specifies an age range, thus ex- pressing the preference for one and only one minimum age value and one and only one maximum age value. All ofthose minimum age values that are higher than the one expressed, and all of those maximum age values that are lower than the one expressed will therefore be considered to be preferred by the user. For example, the user ex- presses as a preference the age range 14-24 years, there- fore all of those minimum age values that are greater than or equal to 14 will have a greater weight in the calculation of the value in the utility function than age values that are less than 14, likewise those maximum age values that are less than or equal to 24 years will have a greater weight than maximum age values that are greater than 24 years.

[0033] According to one embodiment, the values of the disciplines are organized and stored on an electronic storage device in a hierarchical ontology and are then represented internally at the dataset level by means of a predefined number of distinct attributes, for example 4 distinct attributes, each representing a level of on- tology, from the most general (disciplinary macro-areas) to the most specific. The user, in order to make the interaction simpler and more immediate, is not shown this structure when expressing preferences towards dis- ciplines, but it is taken into account at the algorithmic level in the following way. The present method comprises a step wherein, if the user expresses a preference for a discipline belonging to a level of the ontology, thepreference is extended to all disciplines underlying the one explained by the user in the lower levels of the ontology (descendants) and to those disciplines of the upper levels of which the discipline described by the user is the daughter (ancestors). For example, if the user expresses a preference towards the "Biological" discipline, which is a discipline of the second level of the ontology, the preference is also extended to all sub-disciplines of "Biology" ("Microbiology", "Genet- ics", "Zoology", etc.) at the third level of the ontology and to the discipline at the first level of which "Bi- ology" is a sub-discipline, i.e., "Science and technol- ogy".

[0034] This propagation of preferences makes it pos- sible to give weight not only to those disciplines ex- pressed directly by the user but also to related disci- plines, i.e., belonging to the same scope, favoring the diversity of the recommendations that the system pro- poses and still making it possible to recommend a good number of resources even if those with the discipline specified by the user are limited in number. For example, if we assume that there are few biology resources in the system, some resources belonging to "Science and Tech- nology" could also be recommended in addition to the biology resources, which could still be of interest to the user.

[0035] According to one embodiment, once the user has expressed their preferences, i.e., once step a) of the method has been performed, for each resource attribute a list of preference values will be received and possibly stored.

[0036] According to one embodiment, step b) of the method provides for the following sub-step: if the user has expressed a preference for k values of a resource attribute which may take n values, a preference value equal to 1 / k is assigned for each of the k preference values from the user and 0 for all the other n-k values. Taking the case of difficulty, if the user expresses a preference for the "Low" and "Medium low" values, the following probability distribution for the preferences will occur:

[0037]

[0038] With this type of representation, the utility of any resource having difficulties other than "Low" and "Medium Low" will be 0 (see the formula described below), regardless of the value of the other metadata. Given that the user will be advised upon resources with thehighest utility function value, this representation es- sentially performs a resource filter keeping only those with "Low" and "Medium Low" difficulty.

[0039] According to one embodiment, the method pro- vides for the step wherein, for those preference values of the resource attributes for which the user has not expressed a preference, a low but non-zero probability value is attributed by means of an additive smoothing or Laplace smoothing technique. In particular, the method provides for adding an "a priori" pseudo-count to all of the values of the resource attribute. The probability distribution of the preference values for a given re- source attribute will then be calculated according to the following formula:

[0040] p(x) = (d(x)+a) / (k+na)

[0041] where x is the preference value of the resource attribute, d(x) is 1 if the value is present in the preferences expressed by the user and 0 otherwise, a is the smoothing parameter, k is the number of values for which the user has expressed preference and n is the total number of values that the resource attribute may take. Considering some extreme cases:

[0042] • if the smoothing parameter a is null, it returns to the case of "rigid" preferences, i.e., without smoothing: d(x) / k;

[0043] • if the smoothing parameter a tends to beinfinite, the distribution tends to be uniform: -> 1 / n for each x;

[0044] • if the user does not express preferences for any value of the resource attribute, then d(x) = 0 for each x and k = 0, therefore a uniform distribution is again obtained: 1 / n for each x;

[0045] • if the user expresses preference for all values of the metadata, then d(x) = 1 for each x and k = n, a uniform distribution will still occur: 1 / n for each x; since if there is a preference for each value of the resource attribute then there will be for each value (l+a) / (n+na) -> (1 / n)*(1+a) / (1+a) -> 1 / n.

[0046] According to one advantageous embodiment, the smoothing parameter a is selected to be equal to 1 / n. The a = 1 / n denotes the denominator k+1.

[0047] Considering the previous example on the "Dif- ficulty" resource attribute, the probability distribu- tion of the preferences after smoothing will therefore be as follows:

[0048] In this way, even for resources with difficul- ties other than Low and Medium Low, there will be autility that is not zero (but still strongly penalized).

[0049] According to one embodiment, after obtaining the mathematical representation of the user preferences for each resource attribute, i.e., after step b), the method provides for defining the utility function (step c)).

[0050] According to one embodiment, the utility func- tion is defined as follows:

[0051] where Xi are the values of the i-th resource attribute for a given resource R and Pref(Xi) is the probability distribution function representing the pref- erences of the use for the i-th resource attribute, cal- culated as described above. The function is defined as the geometric average of the user's preferences for the values of the resource attributes of a given resource. The reason for this choice was that, insofar as the function is based upon preference productivity, if the preference for the value of a given resource attribute is equal to zero, the overall utility of the resource is also zero.

[0052] According to one embodiment, the utility func- tion is defined as follows:

[0053] wherein Xi are the values of the i-th resource attribute for a given resource R, and Pref(Xi) is the probability distribution function representing the pref- erences of the user for the i-th resource attribute, calculated as described above, and a± is a weight param- eter which determines the weight attributed to each spe- cific i-th resource attribute. It may be noted that the preceding formula of the utility function is a particular case of the present formula wherein a± is identical for all the resource attributes.

[0054] According to one embodiment, in order to give greater weight to the attributes represented by the dis- ciplines with respect to the other resource attributes, a greater value is assigned for the weight parameters a± relating to disciplines with respect to the weight pa- rameters «i relative to the other attributes. For exam- ple, the method provides for assigning a value of 0.8 to the «i of the disciplines and equal to 0.2 to the a± of the other resource attributes.

[0055] Effectively, the utility function represents, by means of a numerical value, the extent to which aresource satisfies the preferences expressed by the user. The greater the utility of a resource, the more the resource attributes of that resource reflect those desired by the user. It thus makes it possible to sum- marize the preferences expressed by the user in a single value, thus allowing easy comparison between the various resources and allowing the best resources (from the metadata point of view) to be identified for the user.

[0056] Given that the order that the utility creates among the various resources, rather than the value per se of the utility for a given resource, is of particular interest, preferably, any monotonic transformation of the utility function may be applied, since it keeps the concerned properties unchanged. According to one embod- iment, the method therefore provides for the utility function to be normalized such that it takes a value of between 0 and 1, in such a way as to make it more inter- pretable.

[0057] Therefore, the method preferably provides for applying the following normalization to the utility value U obtained for each resource:

[0058] U = (U - min (U)) / (max (U) - min (U))

[0059] where the minimum and maximum of U are calcu- lated with respect to all resources.

[0060] In this way, the following interpretation is obtained: a resource with utility 1 is a resource which,among all those present within the dataset, best reflects the user's preferences, whilst a resource with utility 0 is a resource that, again amongst all of those present within the dataset, deviates as much as possible from the user's preferences.

[0061] It is again emphasized that such normalization has the effect of facilitating the interpretability of the utility and does not alter the operation of the system with respect to the use of the non-normalized utility.

[0062] Once the values of the utility function that represents the user's preferences towards the resource attributes have been calculated, i.e., once step c) has been performed, the method provides for step d) of pre- dicting a resource rating for each resource in the list of recommendable resources, by means of a decision tree module.

[0063] In particular, according to one embodiment, the method provides for constructing a decision tree as an algorithm for predicting the resource rating, i.e., the rating given by users to each resource. Such ratings represent the overall rating of the resource on the part of the users and therefore have a different meaning with respect to the utility: in fact, it is possible to im- agine a resource that satisfies the user's preferences regarding resource attributes but that is still ratednegatively because it is of poor quality. Conversely, it is possible to imagine having resources that are rated positively but that do not reflect the user's preferences towards the resource attributes. It is therefore im- portant to also have a predictive algorithm, based upon a decision tree, that makes it possible to predict rat- ings on new resources from the ratings present in a rating database. Indeed, the decision trees were chosen as the algorithm.

[0064] According to one advantageous embodiment, step d) of the method, related to predicting a resource rat- ing, comprises the following steps:

[0065] dl) pre-processing the ratings of the users present in a stored database, by increasing the rating of the resources with a high utility function value (i.e., above a first predetermined threshold), decreas- ing the rating of resources with a low utility function value (i.e., below a second predetermined threshold) and maintaining the rating of the resources with an unchanged intermediate utility function value (i.e., between the first and the second threshold).

[0066] According to one possible embodiment, step d) may comprise the following steps:

[0067] dll) binarizing the resource rating of each resource; for example, the original rating, on a scale of 1 to 5, is binarized as follows: ratings 1 to 3 aregiven a value of 0, and ratings 4 and 5 are given a value of 1; this binarization thus distinguishes highly rated resources from the rest of the resources;

[0068] dl2) as a function of the value of the utility function of each resource, for example the normalized utility value: dl21) if the value of the utility function is higher than a certain first threshold (for example 0.75, in the case of a normalized value), the resource rating is set to 1;

[0069] dl22) if the value of the utility function is less than a certain second threshold (for example at 0.25 in the normalized case), the resource rating is set to 0;

[0070] d23) if the value of the utility function is between the first and the second predetermined thresh- olds, the resource rating remains unchanged (therefore remains 0 if it was 0 or 1 if it was 1).

[0071] According to one embodiment, the method pro- vides for a step of determining the value of the first and second thresholds by an automated fine-tuning pro- cess.

[0072] According to one embodiment, following the pre-processing step dl) on the ratings of the users pre- sent in a stored database, the method provides for the step d2) of training a decision tree model on the storeddatabase. The model resulting from the training is a model that makes a prediction of a positive rating for resources with high utility for the user, a negative rating for resources with low utility for the user, and, for resources with a medium utility, will predict a pos- itive or negative rating depending upon the ratings of those users that are present within the database. Advan- tageously, such an approach implies that the user's pref- erences, expressed through the utility function, are di- rectly represented within the structure and rules of the decision tree that is trained, promoting explainability.

[0073] In order to further optimize the recommenda- tion process in such a way that it provides a limited number of recommended resources (for example no more than 10 resources), the method provides that, in step e), in addition to using the predicted ratings, the se- lection of the one or more recommended resources is also performed as a function of a confidence value in the prediction of the rating that the decision tree module returns in the form of probabilities.More particularly, according to a preferred embodiment, the method provides for selecting one or more recommended resources on the basis of the predicted ratings for each resource in step d), the utility values calculated in step c), and the confidence values in the prediction of the rating of each resource.

[0074] The term "select" means that the electronic processor performs operations to sort resources or unique indexes referring to resources, such that the resources are indexed according to a predetermined order (for example from the most recommendable resource to the least recommendable one).

[0075] According to the latter embodiment, the sort- ing of the resources for obtaining a list of recommen- dations for the user therefore follows the following logic:

[0076] - re-sorting the resources according to the predicted rating (whether this is binarized or not);

[0077] - for those resources having the same rating, the method provides for the step of sorting the resources according to the value of the utility function;

[0078] - for resources having the same rating and the same utility value, the method provides for the step of sorting the resources based upon the confidence in the prediction obtained from the decision tree, for example calculated using Smoothing Leaf Frequencies, for example according to the method described in Chawla, Nitesh V., and David A. Cieslak. "Evaluating probability estimates from decision trees." American Association for Artifi- cial Intelligence. 2006.

[0079] The list of resources that will be selected and possibly presented to the user on the graphicalinterface will therefore be a list of resources with predicted high rating, high utility, and high confidence in the predictions.

[0080] According to one embodiment, for the training step of the decision tree, the method provides for per- forming a sampling of the resources stored within the database in order to reduce the training time, without affecting the quality of the training. In this step, the method provides for performing sampling by selecting a predetermined number of resources with utility above a predetermined threshold (for example the first two hun- dred resources with higher utility), given that those resources with higher utility are the most important ones on which to train the predictive algorithm.

[0081] An even more detailed example of an embodiment of the method according to the present invention will be described below, in such a way as to facilitate the understanding of the description provided thus far by a person skilled in the art.

[0082] As a first step, the user inserts their own preferences via the interface:

[0083] 1) User input:'discipline': ['Coding', 'Programming', 'Programming & Programming Languages'], 'language' : [],'difficulty': ['Medium High', 'Medium Low'],'duration': ['0-30', '30-60'],'format': ['Presentation', 'Video', 'Website'],'type': [],'minimum_age ': 1,'maximum_age ': 30.Weights are then assigned to the user's preferences by dividing the unit by the number of preferences indicated and adding one unit, for each metadata. The addition of one unit (i.e., the number 1) indicates the null choice (i.e., no preference)The output of this assignment, therefore, generates the following user preferences: discipline_level_0’ :'Coding' 0.25'Programming' 0.25'Programming & Programming Languages' 0.25No preference 0.25'Difficulty' : No preference 0.333333Medium High 0.333333Medium Low 0.333333'discipline_level_l ':No preference 1 .0'Format':No preference 0.25Presentation 0.25Video 0.25Website 0.25'discipline_level_2 ':No preference 1 .0'Duration':No preference 0.3333330-30 0.33333330-60 0.333333'Language':No preference 1 .0'Type':No preference 1 .0'Maximum_age' :No preference 0.33333320 0.33333328 0.333333'Minimum age':No preference 0.218 0.214 0.211 0.26 0.2

[0084] The method subsequently provides for applying the utility function based upon the metadata and user preferences:for example, a first learning object with identifier ID 1 is as follows: link https: / / www.merlot.org / merlot / viewMaterial.htm... disciplineBusiness / Economics / Micro keywordsCost Concepts titleCost ConceptsDescriptionRead the lesson and try to answer the question... typePresentation languageEnglish difficultyMedium Low formatText duration 0-30 audience['College General Ed'] minimum_age18 maximum_age20 discipline_level_0Business discipline_level_l Economics discipline_level_2 Micro discipline_level_3 Absent discipline_level_4 Absent discipline_level_5 Absent resource_image https: / / www.merlot.org / merlot / getMaterialImage. .. average_rating 4.0.Calculated Utility: 0.10653474755728706

[0085] The utilities are then normalized, as de- scribed. For example, the following is an output of some first 25 resources, of the first 25 IDs: 0 0.4730161 1.0000002 0.0000003 0.4730164 0.4730165 1.0000006 0.4730167 0.4730168 0.4730169 0.00000010 0.47301611 0.00000012 0.47301613 0.47301614 0. 47301615 0.47301616 0.00000017 1.00000018 1.00000019 1.00000020 0.47301621 0.47301622 0.47301623 0.47301624 0.473016

[0086] Once the utility function is calculated and normalized, the rating is then modified as a value 1, if the utility is above a certain threshold (0.75), or as a value 0 if it is below another threshold (0.25). This ensures that a "useful" resource is rated as "good" even without user ratings or if users rated it as low, and auseless" resource is rated as "bad" even if users rated it positively. Those resources with utility between the two thresholds maintain the rating that users have given them.

[0087] For the generation of the decision tree, a function, such as a buildTree_std function, takes an input dataframe pandas df, where the last column repre- sents the data class labels. The goal is to construct a decision tree that may predict the class label of new data.

[0088] The aforesaid function first checks whether the dataset is already pure (contains only one class label), in which** case it creates a leaf node with that class label. If it is not then it selects the attribute with the highest information gain (using a find_winner), and creates a node for that attribute. It then recur- sively calls itself on each subset of the data defined by the values of the selected attribute.

[0089] Finally, the method also implements a function for setting a depth limit for the decision tree in order to avoid oversizing the training data. If the maximum depth has been reached, or if all attributes have been used to divide the data, a leaf node is created with the majority class label of the current data subset.

[0090] However, if the maximum depth has not been reached and there are still attributes to subdivide, itrecursively recalls itself on the subset of data corre- sponding to each value of the selected attribute.

[0091] Subsequently, the method then verifies whether the resulting subtree is pure (contains only one class label) or whether it satisfies a pre-pruning condition (all the attributes in the subset are the same). If one of these conditions is met, it creates a leaf node with the majority class label of the subset. Otherwise, it adds the subtree to the current node.

[0092] Here the method provides for the possibility of various calls of the find_winner function on the rat- ing dataset to obtawherein node has the highest infor- mation gain (the first is the complete call, the others represent only the last step).

[0093] Subsequently, a function, for example called return_top_k inputs four arguments: resources, tree, user_preferences, and optional parameters k, n_class, and alpha.

[0094] The resources argument is expected to be a Pandas DataFrame containing information about the re- sources that the user may select. The tree argument is a dictionary that contains a decision tree that is used to predict the adequacy of each resource. The user_pref- erences argument is a dictionary that contains the user preferences for each feature in the resources DataFrame. The parameter k is the number of primary resources to bereturned. The n_class parameter is the number of classes on which the decision tree has been trained, and alpha is a parameter used in calculating the utility of each resource.

[0095] The first function creates two empty lists, y_pred and y_prob, which will be used to store the pre- dicted labels and probabilities for each resource.

[0096] The function then scrolls every row in the resources DataFrame. For each row, it calls the predict function (defined later) to make a prediction using the decision tree. The predict function returns the pre- dicted label and probability for the current resource, which are added to the y_pred and y_prob lists, respec- tively.

[0097] If the utility column is not already present in the DataFrame of the resources, the function generates it using the generate_utility function (defined else- where). The generate_utility function takes the Data- Frame of the resources, the user_preferences dictionary, and the alpha parameter as input, and returns a Series containing the utility value for each resource.

[0098] The function then creates a new predictions DataFrame combining the y_pred, y_prob columns and util- ity. For each row in predictions, the apply method is called to adjust the probability value based upon the expected label. If the expected label is 0 (i.e., theresource is not expected to be suitable), the value of the probability is reversed (i.e., 1 - probability). This ensures that the final sort order of the resources considers both the predicted label and the probability.

[0099] Finally, the function sorts the DataFrame of the predictions for the y_pred, utility, and y_prob col- umns in descending order. Then it selects the first k resources based upon their index value and returns them. [000100] Subsequently, the inference phase of the pre- dictions takes place. A predict function takes various parameters and recursively traverses a decision tree to make predictions regarding a given input instance. The decision tree is represented as a nested dictionary, where each key represents a feature or attribute of the input instance, and the values represent the possible values that the feature may assume, along with the cor- responding subtree that should be traversed if the fea- ture assumes such value.[000101] The function accepts the following parameters: inst: a dictionary that represents a single input instance, where each key is a feature or attribute, and the value is the corresponding value of that feature in the input instance. tree: a nested dictionary that represents the decision tree to be traversed. get_rule : a Boolean indicator that reports whether thefunction should return the decision path (i.e., the rule sequence that led to the prediction) along with the prediction itself. rule: A list that tracks the decision path as the tree is traversed. return_prob: a Boolean indicator that reports whether the function should return the expected probability of the prediction. return_conf: a Boolean indicator that reports whether the function should return the confidence interval of the prediction. n_class: an integer that indicates the number of classes in the prediction problem.The function starts by checking if return_prob is False and return_conf is True. If so, it raises an error because the confidence may not be returned without the probabilities .The function then iterates through the keys of the decision tree, comparing them with the values in the input instance. If the input instance length is 1 and the only value in the instance does not match any of the keys in the decision tree, it returns a prediction of 0 with a low and high probability of 1. This is the stop condition for the recursive traversal function.If the input instance has multiple values, and a key in the decision tree matches one of the values, the functionadds the corresponding rule to the rule list and recursively calls itself on the subtree corresponding to that value. If the subtree is a dictionary, the function checks whether get_rule is True, and if it is, returns the prediction along with the decision path and expected probability or confidence interval (if required). If get_rule is False, the function simply returns the prediction (along with the expected probability or confidence interval, if required).If the subtree is not a dictionary, the function returns the prediction and expected probability and confidence interval (if required), along with the decision path.If the input instance has multiple values, and no key in the decision tree matches any of the values, the function returns a prediction of 0 with a low and high probability of 1, together with the decision path.If get_rule is True, the function returns the prediction along with the decision path and predicted probability or confidence interval (if required). If get_rule is False, the function simply returns the prediction (along with the predicted probability or confidence interval, if required).[000102] An example of a response generated is as fol- lows:{'id': ['190', '778', '1684', '1685', '1686', '1688', '1674', '1675', '1689', '313'],'average_rating ': ['4.0', '4.0', '4.0', '4.0', '4.0','4.0', '4.0', '4.0', '4.0', '5.0'],'explanations': [\ "This resource satisfies your preferences for the following 6 attributes:['discipline', 'format', 'duration', 'difficulty','minimum_age ', ’maximum_age ']. Looking at the votes of similar resources, the method predicted an above-average vote for this resource with a probability of 94% using the following association rule: discipline_level_2: Application Development -> Good Resourced',"This resource satisfies your preferences for the following 6 attributes: ['discipline', 'format', 'duration', 'difficulty', 'minimum_age ','maximum_age ']. Looking at the votes of similar resources, the system predicted an above-average vote for this resource with a probability of 94% using the following association rule: discipline_level_2:Application Development -> Good Resource\","This resource satisfies your preferences for the following 5 attributes: ['discipline', 'format', 'duration', 'difficulty', 'minimum_age ']. Looking at the votes of similar resources, the system predicted an above-average vote for this resource with a probability of 99% using the following association rule:discipline_level_2 : Absent, format: Presentation -> Good Resource\ ","This resource satisfies your preferences for the following 5 attributes: ['discipline', 'format', 'duration', 'difficulty', 'minimum_age ']. Looking at the votes of similar resources, the system predicted an above-average vote for this resource with a probability of 99% using the following association rule: discipline_level_2 : Absent, format: Presentation -> Good Resource\ ","This resource satisfies your preferences for the following 5 attributes: ['discipline', 'format', 'duration', 'difficulty', 'minimum_age ']. Looking at the votes of similar resources, the system predicted an above-average vote for this resource with a probability of 99% using the following association rule: discipline_level_2 : Absent, format: Presentation -> Good Resource\ ".[000103] Innovatively, the method according to the pre- sent invention successfully overcomes the typical prob- lems of the prior art.[000104] In particular, the method allows for an appro- priate selection of educational material within hetero- geneous educational resources and the adaptation ofresources to the specific preferences and needs of users. [000105] Advantageously, unlike known recommendation systems such as collaborative filtering or content-based filtering, the method according to the present invention considers both explicit preferences according to the at- tributes (the metadata) and the overall rating of the resources to be recommended, thereby obtaining an effec- tive recommendation for the creation of a course based upon the actual needs and quality required by the user. [000106] In an extremely advantageous way, the method according to the present invention provides recommenda- tions for completely new platform users, thus obviating the cold start problem that characterizes most recommen- dation systems, insofar as recommendations are based only upon the input offered by the user and upon a pre- existing database of resources rated by other users.[000107] Furthermore, due to the use of the decision trees, the method according to the present invention is intrinsically transparent (explainable), i.e., it is able to justify the recommendations provided. In fact, the method allows the developers of the system to easily find any problems and to carry out updates and at the same time allows a detailed explanation of the recommen- dations provided to the user, thus increasing the user's confidence in the system.[000108] Advantageously, contrary to that occurring inthe prior art, the method according to the present in- vention offers the possibility of optimizing the recom- mendation regarding adequate resources by adjusting the recommendation according to the resource ratings, updat- ing the ranking of the recovered resources taking into account the context of the entire community of users, analyzing the ratings that other users have assigned to a certain resource. This results in a more accurate final ranking.[000109] Furthermore, in a further advantageous way, the utility function that supports the selection and recommendation regarding educational resources, consid- ering the teacher's explicit preferences concerning at- tributes such as discipline, difficulty, duration, and format, together with the historical ratings of re- sources, allows these preferences to be aggregated and weighted for each attribute thereby making it possible to accurately calculate how much a resource satisfies them.[000110] Furthermore, the method according to the pre- sent invention makes it possible to provide highly cus- tomized recommendations, integrating the teacher's pref- erences with past ratings, whilst managing complex pref- erences and optimizing the ranking of the recommended resources.[000111] In particular, contrary to that whichtypically occurs in the prior art, wherein recommenda- tion methods are used for classifying courses or for making personalized recommendations only from students to students, the method according to the present inven- tion implements a decision tree algorithm for more ac- curately suggesting educational resources for teachers, given that it takes into account the teacher's prefer- ences and the historical ratings of other teachers with respect to an educational resource, and, consequently, uses this set to infer the specific rating of the re- source to be suggested to the teacher, also depending upon the student who will benefit from the instructional course.[000112] Furthermore, where the methods of the prior art are intended to recommend to users courses followed previously, using SVM models or neural networks with the aim of classifying the courses that a given user has already taken, the method according to the present in- vention is instead useful for recommending educational resources to support teachers in creating customized training paths. In other words, whilst the most relevant systems of the prior art are constructed to recommend pre-existing courses, the recommendation method accord- ing to the present invention involves combining hetero- geneous educational resources present within databases similar to the search to create a bespoke course, alsoby virtue of the utility function.[000113] Furthermore, it is precisely the utility func- tion that plays an advantageous role in the selection of resources in a holistic manner. It is in this way pos- sible to identify those resources considered to be of interest given the input of the teacher user, such as the language, difficulty, duration, and format of the resource, but at the same time it also takes into account the feedback of previous users. This feedback is repre- sented by the evaluations that are explicitly provided, for example on a scale of 1 to 5, ensuring a more robust and reliable recommendation.[000114] It is clear that a person skilled in the art may make changes to the invention in order to meet con- tingent needs, said changes all falling within the scope of protection as defined in the following claims.

Claims

CLAIMS1. A computer-implemented method for providing a recommendation regarding educational resources to a teacher user for creating a personalized instructional course, said method comprising performing the following steps on an electronic processor: a) receiving preference values for resource attributes characterizing each resource of a list of recommendable educational resources, wherein said resource attributes include one or more disciplines to which each of the educational resources belongs and one or more metadata characterizing the resources, for example type, language, difficulty, duration, format, minimum age, maximum age; b) by means of a modeling module, converting the preference values received in step a) into a probability distribution function for each of the resource parameters; c) by means of a utility function module, calculating a utility value for each resource in the list of recommendable educational resources by applying a utility function to the preference values calculated from the probability distribution function of each of the resource parameters, obtained in step b); d) by means of a decision tree module, predicting a resource rating for each resource of the list ofrecommendable resources, said decision tree module being trained by processing the utility values of each resource calculated in step c), so as to provide said resource rating as a function of the utility values of each resource together with ratings of each resource previously provided by the users and stored in an electronic storage device; e) selecting one or more recommended resources based upon the ratings predicted for each resource in step d) and the utility values calculated in step c).

2. Method according to claim 1, wherein step d) comprises the step of predicting a first rating, for example a positive value, for resources with a utility above a first threshold, a second rating, for example a negative value, for resources with a utility value below a second threshold, and a third rating for resources with a utility between the first and the second thresholds, said third value being calculated by processing the ratings of each resource previously provided by the users and stored in an electronic storage device.

3. Method according to claim 1 or 2, wherein the step of converting the preference values received in step a) into a probability distribution function for each of the resource attributes provides for a step of smoothing of the probability distribution function by means ofadditive or Laplace smoothing, and wherein the probability values of the resource attributes for which the user has not expressed preferences have a low but non-zero probability value.

4. Method according to any one of the preceding claims, wherein the utility function is based upon the geometric mean of the preference values for resource attributes and the utility function is normalized to a range from 0 to 1.

5. Method according to any one of the preceding claims, wherein the decision tree module predicts the resource ratings that the user may give to new resources, based upon historical database data.

6. Method according to any one of the preceding claims, wherein step e) provides for selecting one or more recommended resources based upon the ratings predicted for each resource in step d), the utility values calculated in step c), and confidence values in predicting the rating of each resource, calculated by the decision tree module.

7. Method according to any one of the preceding claims, wherein step b) of the method provides for the following sub-step: if the user has expressed a preference for k values of a resource attribute which may take n values, a preference value equal to 1 / k is assigned for each of the k preference values from the user and 0 for all theother n-k values.

8. Method according to claim 3, wherein for the preference values of the resource attributes for which the user has not expressed a preference, the probability distribution of the preference values for a given resource attribute is calculated according to the following formula: p(x) = (d (x)+a) / (k+na) where x is the preference value of the resource attribute, d(x) is 1 if the value is present in the preferences expressed by the user and 0 otherwise, a is the smoothing parameter, k the number of values for which the user has expressed a preference, and n is the total number of values that the resource attribute may take, wherein the smoothing parameter a is selected to be 1 / n.

9. Method according to any one of the preceding claims, wherein the utility function is defined as follows:where Xi are the values of the i-th resource attribute for a given resource R and Pref(Xi) is the probability distribution function representing the preferences of the user for the i-th resource attribute.

10. Method according to any one of claims 1 to 9, wherein the utility function is defined as follows:wherein Xi are the values of the i-th resource attribute for a given resource R and Pref(Xi) is the probability distribution function representing the preferences of the user for the i-th resource attribute, and ai is a weight parameter which determines the weight attributed to each specific i-th resource attribute.

11. Method according to any one of the preceding claims, wherein in step d) the method comprises the following sub-step: dl) pre-processing the ratings of the users present in a stored database, by increasing the rating of the resources with a high utility function value (i.e., above a first predetermined threshold), decreasing the rating of the resources with a low utility function value (i.e., below a second predetermined threshold), and maintaining the rating of the resources with an intermediate utility function value (i.e., with a value between the first and the second thresholds) unchanged.

12. Method according to claim 11, wherein in step dl) the method comprises the following sub-steps:dl2) as a function of the value of the utility function of each resource (for example, the normalized utility value): dl21) if the value of the utility function is higher than a certain first threshold (for example, 0.75, in the case of a normalized value), the resource rating, i.e., the rating of the resource, is set to 1; dl22) if the value of the utility function is less than a certain second threshold (for example, 0.25 in the normalized case), the resource rating, i.e., the rating of the resource, is set to 0; d23) if the value of the utility function is between the first and the second predetermined thresholds, the resource rating remains unchanged.

13. Method according to claim 11 or 12, wherein, following the pre-processing step dl), the method provides for the step d2) of training a decision tree model on the stored database, wherein the model resulting from the training is a model which calculates a prediction of a positive rating for resources with a high utility for the user, a negative rating for resources with a low utility for the user, and, for resources with a medium utility, it calculates a positive or negative rating depending on the ratings of the users present in the database.

14. Method according to claim 13, wherein, for the stepof training the decision tree, the method includes sampling the resources stored in the database by selecting a preset amount of resources with a utility above a preset threshold.

15. A computer program executable by a computing device, comprising code instructions which, when executed by the computing device, cause steps a) to e) of the method according to any one of claims 1 to 14 to be performed.

16. A computer product in which a computer program according to claim 15 is stored, suitable for being executed by a computing device to implement steps a) to e) of the method according to any one of claims 1 to 14.

Images