Device and method for determining the maturity level of user models

Abstract

The present invention relates to a computer-implemented method for determining maturity level for implementing an artificial intelligence model (1). Step a) consists in accessing a knowledge database (4) that includes a plurality of assessment questions (13) and answers (7) related to the artificial intelligence model (1). Step b) consists in executing an automatic reinforcement learning-based assessment process (12), said step comprising sub-steps b1), b2), b3), b4), and b5), where the final step b5) comprises repeating the steps b1) to b4). The invention further discloses embodiments of a device configured to execute the computer-implemented method, a computer-readable medium, and a computer program with instructions which, once read by a computer, make the device execute the method.

Description

[0001] APPARATUS AND METHOD FOR DETERMINING THE MATURITY LEVEL OF A USER

[0002] Related technical field

[0003] The present invention relates to computer-implemented methods for assessing the maturity of computational models. In particular, it relates to computer-implemented methods for assessing the maturity of artificial intelligence models.

[0004] Description of the state of the art

[0005] The prior art discloses in document US8019640B2 a method, an apparatus and a computer program product for implementing an automation computer evaluation scale configured to generate recommendations.

[0006] This document mentions that the method has a step of presenting, via the data processing system, automated computing educational material to a client user. Also, the method has a step of presenting to the client user an evaluation survey that includes a plurality of evaluation survey questions for a plurality of automated computing evaluation categories regarding a current level of self-management capabilities of the client's automated computing system.

[0007] This document further discloses a step of receiving responses to the plurality of assessment survey questions for each of the plurality of assessment categories about the current level of the customer's capabilities, and a step of assigning a ranking of computer automation according to an automation computing assessment scale that includes a plurality of predetermined levels of automation computing capabilities.

[0008] The plurality of predetermined levels of automation computing capabilities include a basic maturity level indicating that the customer uses manual analysis and problem resolution, a managed maturity level indicating that the customer uses centralized tools and performs manual actions, a predictive maturity level indicating that the customer monitors, correlates data, and recommends actions, an adaptive maturity level indicating that the customer uses a system that monitors, correlates data, and performs actions, and an autonomic maturity level indicating dynamic management based on business policies.

[0009] This document further mentions that the method may include using an artificial intelligence application to analyze historical information using a set of rules to create an analytical output, to achieve the target level of self-management capabilities of the client's automated computing system.

[0010] Document US1 1093885B2 discloses a computer-implemented method with a step of determining a change in a maturity score of an enterprise architecture assessment domain model associated with an enterprise architecture. The change in the maturity score is indicative of a change in a maturity state of the enterprise architecture assessment domain model.

[0011] The document also mentions that the method has a step of determining one or more maturity model types that exceed a predetermined degree of change, and determining at least one assessment capability group for each maturity model type.

[0012] The document additionally discloses a step of selecting one or more assessment capabilities for each assessment capability group based on one or more selection criteria. The selection of one or more recommendation templates for each selected assessment capability based on one or more rules.

[0013] The document also mentions the compilation of a first set of recommendations based on the selected recommendation templates, and the output of the first set of recommendations.

[0014] On the other hand, document CA3052256A1 discloses a method for assessing and optimizing the master data maturity of an enterprise data architecture. The method has a step of presenting to the plurality of users a maturity interface configured to generate a maturity matrix comprising a plurality of master data dimensions for each user. This document also mentions a step of presenting to each single user an evaluation interface for entering master data into a master data maturity engine.

[0015] This document discloses that the method further has a step of generating master data maturity units for each of the plurality of dimensions, and a step of generating, for each of the users, a maturity matrix comprising the plurality of master data dimension maturity units and an overall master data maturity.

[0016] This document also mentions a step of presenting, through the evaluation interface, a survey that includes, for each master data dimension, an entry for an ordinal measure of the dimension.

[0017] However, the methods and apparatus described in the prior art do not allow for an efficient evaluation of the implementation of an artificial intelligence model in an entity, for example, in a company or public institution.

[0018] Brief description of the disclosure

[0019] The present disclosure describes embodiments of a computer-implemented method for evaluating and determining the level of maturity for the implementation of an artificial intelligence model through an evaluation process based on machine reinforcement learning.

[0020] The method comprises step a) in which a knowledge database is accessed, including evaluation questions and answers for each question. These questions allow for the evaluation of a dimension of the artificial intelligence model. In addition, the method has step b) of executing an evaluation process based on reinforcement learning.

[0021] Step b) includes a sub-step b1) of extracting an assessment question corresponding to a dimension of the artificial intelligence model from the knowledge database. Furthermore, step b) includes a sub-step b2) of receiving an answer entered by a user on an input / output device on which the assessment question is displayed, and a sub-step b3) of updating a rating score associated with at least one dimension through a machine learning-based classification process. The rating score comprises a maturity level value of the dimension and a confidence level value of the maturity level value.

[0022] Stage b) also includes a substage b4) of obtaining a group of assessment questions through a machine learning-based selection process that takes the rating score and records associated with the dimension as input. The selection process is configured to reduce the number of assessment questions displayed on the input / output device and answered by the user.

[0023] In addition, stage b) has a stage b5) of repeating sub-stages b1) to b4) by selecting new assessment questions from the group of questions, until obtaining maturity level values ​​for the plurality of dimensions with a confidence level value greater than a predetermined value.

[0024] In any of the modalities of the method disclosed herein, in step a), the knowledge database (4) contains a plurality of evaluation questions (13) and answers (7) for each question. This knowledge database (4) provides a centralized and structured resource to obtain relevant information for the evaluation of the artificial intelligence model (1). The knowledge database (4) can be built by one or more experts in the area of ​​​​implementation of artificial intelligence models, and take as input heuristic study cases of other models implemented in other entities, companies or institutions.

[0025] The knowledge database may have 150 or more questions per dimension, and five or more dimensions may be assessed. However, from a practical standpoint, it would be difficult to apply up to 1,500 assessment questions to fully ensure the maturity level of the artificial intelligence model. This could even lead to users becoming fatigued or bored, failing to respond conscientiously. As for step b), running an assessment process based on reinforcement learning allows the artificial intelligence model to improve its performance and decision-making as it receives and analyzes the responses provided by users.

[0026] The method described here allows to reduce the number of assessment questions answered by each user through stage b) and its substages b1) to b6). The reinforcement learning-based assessment process allows to analyze the user's answers in real time to determine the maturity and confidence level values ​​for each dimension, and to generate groups of selected questions based on previous reinforcement training. This process continues until maturity level values ​​are obtained with a confidence level value higher than a predetermined threshold value.

[0027] In this way, the method can determine the required confidence level and maturity level for each dimension of the AI ​​model, using, for example, between 5 and 15 questions per dimension, instead of the 150 or more assessment questions available in the knowledge database.

[0028] The present disclosure also describes embodiments of an apparatus for determining the level of maturity for implementing an artificial intelligence model. The apparatus comprises a computing unit configured to execute the steps of the computer-implemented method described herein.

[0029] For example, the apparatus comprises a computing unit configured to execute step a) of accessing a knowledge database that includes evaluation questions and answers for each question. These questions allow evaluating a dimension of the artificial intelligence model. In addition, the apparatus comprises a computing unit configured to execute step b) of executing an evaluation process based on reinforcement learning.

[0030] The computing unit is also configured to execute stage b) and its sub-stages. Stage b) has a sub-stage b1) of extracting an assessment question corresponding to a dimension of the artificial intelligence model from the knowledge database. Furthermore, stage b) includes a sub-stage b2) of receiving an answer entered by a user on an input / output device on which the assessment question is displayed, and a sub-stage b3) of updating a rating score associated with at least one dimension by means of a machine learning-based classification process that takes as input the answer entered by the user. The rating score comprises a maturity level value of the dimension and a confidence level value of the maturity level value.

[0031] Stage b) also includes a substage b4) of obtaining a group of assessment questions through a machine learning-based selection process that takes the rating score and records associated with the dimension as input. The selection process is configured to reduce the number of assessment questions displayed on the input / output device and answered by the user.

[0032] In addition, stage b) has a stage b5 of repeating sub-stages b1) to b4) by selecting new assessment questions from the question pool, until obtaining maturity level values ​​for the plurality of dimensions with a confidence level value greater than a predetermined value.

[0033] The present disclosure also describes embodiments of a computer program, and a computer-readable medium, which include instructions that when read by a computing unit, cause the computing unit to execute any of the embodiments of the computer-implemented method described herein.

[0034] Brief description of the figures

[0035] FIG. 1 illustrates an application of one embodiment of the computer-implemented method for assessing and determining the maturity level for implementing an artificial intelligence model.

[0036] FIG. 2 shows a flowchart of an embodiment of the computer-implemented method for evaluating and determining the level of maturity for implementing an artificial intelligence model. FIG. 3 shows a flowchart of an embodiment of the computer-implemented method as in FIG. 2, further including a step b21) of obtaining a positive or negative rating data.

[0037] FIG. 4 shows a flow diagram of one embodiment of the computer-implemented method as in FIG. 2, further including a step of generating a report data.

[0038] FIG. 5 illustrates an application of one embodiment of the computer-implemented method for assessing and determining the maturity level for implementing an artificial intelligence model. This embodiment also includes a step for generating a report data.

[0039] Detailed description

[0040] The present disclosure describes embodiments of a computer-implemented method for evaluating and determining the level of maturity for the implementation of an artificial intelligence model (1) through an evaluation process (12) based on machine reinforcement learning.

[0041] The method comprises a step a) of accessing a knowledge database (4) that includes a plurality of evaluation questions (13) and answers (7) for each evaluation question (13). The evaluation questions (13) allow evaluating a dimension (10) of the artificial intelligence model (1). Also, the method has a step b) of executing an evaluation process (12) based on reinforcement machine learning.

[0042] Stage b) has a sub-stage b1) of extracting, from the knowledge database (4), an evaluation question (13) corresponding to a dimension (10) of the artificial intelligence model (1). Furthermore, stage b) includes a sub-stage b2) of receiving an answer (14) entered by a user (2) into an input / output device (15) on which the evaluation question (13) is displayed, and a sub-stage b3) of updating a rating score (20) associated with at least one dimension (10) by means of a classification process (21) based on machine learning that takes as input the answer (14) entered by the user (2). The rating score (20) comprises a maturity level value (25) of the dimension (10), and a confidence level value (26) of the maturity level value (25).

[0043] Stage b) further includes a sub-stage b4) of obtaining a group of evaluation questions (18) through a selection process (22) based on machine learning that takes as input the rating score (20) and the records (9) associated with the dimension (10). The selection process (22) is configured to reduce the number of evaluation questions (13) that are displayed on the input / output device (15) and answered by the user (2).

[0044] Stage b) also has a stage b5) of repeating stages b1) to b4) by selecting a new evaluation question (13) from the group of evaluation questions (18), until obtaining maturity level values ​​(25) for the plurality of dimensions (10) with a confidence level value (26) greater than a predetermined value.

[0045] In any of the modalities of the method disclosed herein, in step a), the knowledge database (4) contains a plurality of evaluation questions (13) and answers (7) for each question. This knowledge database (4) provides a centralized and structured resource to obtain relevant information for the evaluation of the artificial intelligence model (1). The knowledge database (4) can be built by one or more experts in the area of ​​​​implementation of artificial intelligence models, and take as input heuristic study cases of other models implemented in other entities, companies or institutions.

[0046] The knowledge database (4) can have 150 or more questions for each dimension (10), and five or more dimensions (10) can be assessed. Examples of dimensions (10) are strategy, tactics, organizational culture, security, ethics, technology and data governance, circular economy, environmental impact, accident rate and risk prevention.

[0047] Each dimension (10) has a plurality of maturity levels associated with it, which can be categorical values ​​(e.g., inactive, active, experimental, integrative, and transformative) or numerical (e.g., score according to a predetermined scale or hierarchy). However, from a practical point of view, it would be difficult to apply up to 1500 evaluation questions (13) to be completely sure of the maturity level for the artificial intelligence model (1). This could even cause users (2) to not respond conscientiously due to fatigue or boredom.

[0048] As for stage b), the execution of an evaluation process (12) based on reinforcement machine learning allows to improve its performance and decision making as it receives and analyzes the responses provided by users.

[0049] The method described here allows to reduce the number of evaluation questions (13) that each user (2) answers through stage b) and its substages b1) to b6). The evaluation process (12) based on reinforcement machine learning allows to analyze on the fly the answers (14) of the user (2) and based on whether they answer them correctly or incorrectly, determine the maturity level value (25) and a confidence level value (26) for each dimension (10), and generate groups of questions (18). These groups of questions (18) are selected according to the previous reinforcement training. The method generates groups of questions (18) until maturity level values ​​(25) are obtained with a confidence level value (26) higher than a predetermined value. The predetermined value can be, for example, a value between 90% and 99%. The higher the predetermined value, the more likely it is that the user (2) needs to answer more evaluation questions (13).

[0050] For example, when the default value is between 95% and 96%, the user (2) answers on average between 8 and 15 questions for each dimension.

[0051] In this way, the method can determine with the required level of confidence the maturity level of each dimension (10) of the artificial intelligence model (1), using on average a maximum of 10 questions per dimension (10), instead of the 150 or more evaluation questions (13) available in the knowledge database (4).

[0052] In the present disclosure, a database will be understood as a set of data stored in a memory record systematically for later use.

[0053] Knowledge databases (4) may include one or more databases selected from the group comprising hierarchical databases, network databases, transactional databases, relational databases, multidimensional databases, object-oriented databases, document databases, deductive databases, combinations thereof, and other databases known to a person of ordinary skill in the art.

[0054] In hierarchical databases, data is organized in a hierarchical tree structure, where each record has a single parent and one or more children. This approach is suitable for representing one-to-many relationships. In network databases, data is organized in a grid structure, where records can have multiple relationships with other records. This model allows for the representation of complex relationships and is used in applications that require greater flexibility in data structure.

[0055] Regarding relational databases, this is one of the most common types of databases. It uses tables with fields and records, or rows and columns, to store and organize data. It is based on the relational model, which establishes relationships between tables using primary keys and foreign keys. Multidimensional databases, on the other hand, are used to store and analyze data that can be represented in multiple dimensions. They are suitable for analyzing complex data, such as in business intelligence and data analytics. In object-oriented databases, data is modeled and stored as objects, which contain both the data and the associated methods or functions.

[0056] In some embodiments of the method disclosed herein, the knowledge database (4) may be constructed from a thorough literature review and expert consultations. For example, pilot tests may be conducted to validate the questions, and the information may be organized into a database for easy access during execution of the method described herein.

[0057] The answers (7, 14) to the evaluation questions (13) of the knowledge database (4) can be of the Boolean type. Preferably, the answers (7, 14) also include a third value different from the Boolean values. This allows the user (2) to answer yes, no or I don't know to each question, according to his knowledge and experience in relation to the dimension (10) being evaluated. The choice of binary answers (7, 14) allows to simplify and objectify the evaluation process. In this way, the subjectivity of open answers is avoided and it is guaranteed that the answers are clear and precise. Furthermore, the evaluation process (12) based on reinforcement machine learning is designed to adapt to the user's answers (2) and adjust the evaluation based on them, so that the binary answers are sufficient to obtain an objective and precise evaluation of the maturity level for the artificial intelligence model (1).

[0058] On the other hand, one of the advantages of having a third value for the answers (7, 14) is that the user (2) can indicate that he or she does not know. This prevents him or her from randomly selecting a Boolean answer and generating noise and inaccuracies in the evaluation of the artificial intelligence model (1).

[0059] In step a), the computing unit (3) accesses the knowledge database (4). To do so, it may use a communications protocol based on SQL queries, application programming interfaces (APIs), web services, database access protocols, and combinations thereof.

[0060] SQL queries use structured queries in the SQL (Structured Query Language). SQL provides specific commands and syntax for retrieving, inserting, updating, and deleting data from a relational database. The computing unit (3) can send SQL queries to the knowledge database (4) to retrieve specific information and perform data manipulation operations.

[0061] Regarding the APIs, these allow the computing unit (4) to access and manipulate the data stored in the knowledge database (4). These APIs provide sets of functions and methods that can be used within programs written in different programming languages, such as Python, Java, C#, among others. Through these APIs, the computing unit (3) can establish connections with the knowledge database (4), send queries and receive the corresponding results.

[0062] Database access protocols allow access to databases from computing units. Some examples are the Open Database Access Protocol (ODBC), the Microsoft Database Access Protocol (ADO.NET), and the Java Data Access Protocol (JDBC). These protocols allow the computing unit (3) to establish secure and efficient communication with the knowledge database (4), exchanging information and performing specific operations.

[0063] On the other hand, RESTful web interfaces and API services provide defined access points and use secure communication protocols, e.g., HTTP(S), to allow the computing unit (3) to make requests and receive structured responses in formats such as JSON or XML. The computing unit (3) can interact with these interfaces or services using HTTP requests to access and manipulate data in the knowledge database (4).

[0064] In some embodiments of the method disclosed herein, the knowledge database (4) may include records (9) grouped according to a plurality of record classes (8). Each record (9) includes at least one evaluation question (13) and one answer (7) for each evaluation question (13). In addition, each record class (8) groups records (9) associated with a dimension (10) of the artificial intelligence model (1). In this case, the knowledge database (4) is relational. These modalities have advantages in terms of organization, efficient search, improved analysis, scalability and flexibility.

[0065] By grouping records (9) into record classes (8), a structured organization of the knowledge database (4) is achieved. This facilitates the management and access to information, since records (9) related to a specific dimension (10) can be accessed more efficiently. The relational structure of the knowledge database (4) allows establishing relationships and associations between records (9) and dimensions (10) of the artificial intelligence model (1), which provides greater coherence and cohesion in the storage and retrieval of information. Also, this allows the computing unit (3) to access records (9) more quickly and accurately. This reduces the time and resources needed to retrieve relevant information and allows more efficient processing of assessment questions (13) and answers (7, 14).Additionally, the relational structure of the knowledge database (4) allows records (9) to be grouped according to record classes (8), which makes it possible to identify patterns, trends and relationships between the evaluation questions (13) and the answers (7, 14) associated with a specific dimension (10). This provides valuable information to improve the evaluation process (12), adjust the question selection policy (13) and optimize the overall performance of the evaluation process (12) based on reinforcement machine learning.

[0066] Also, the related knowledge database (4) can be scaled as more records (9) and record classes (8) are added. This provides flexibility to adapt to changes in the artificial intelligence model (1) and in the evaluation requirements determined by the evaluation process (12). For example, new dimensions (10) and related records (9) can be added without compromising the integrity and consistency of the knowledge database (4). Furthermore, the related structure allows complex queries and modifications to the database to be performed more efficiently, compared to other database structures.

[0067] In stage b) of any of the modalities of the method disclosed here, an evaluation process (12) based on machine reinforcement learning is executed.

[0068] Reinforcement learning is a technique in which one or more computer-implemented agents are trained to perform actions in an environment. As these agents interact with the environment, they receive feedback in the form of rewards or penalties, allowing them to learn to make decisions that maximize cumulative rewards over time.

[0069] In the process of machine reinforcement learning, computer-implemented method agents explore the environment and take actions based on their current knowledge. These actions can have different consequences and lead to different states of the environment. As agents interact, the quality of their actions is evaluated using a reward function, which provides a measure of how well they are performing the task. The computer-implemented method agent can be represented as an entity that perceives the current state of the environment, chooses an action to perform, and acts accordingly. It can use information about the current state and the history of past actions and rewards to make optimal decisions.

[0070] The goal of machine reinforcement learning is to train computer-implemented agents to learn a policy—a strategy that determines what action to take in each state of the environment. This policy is adjusted over time as the agents explore and learn from the feedback they receive.

[0071] To improve their performance, agents use reinforcement learning algorithms, such as Q-Learning-based learning methods or function approximation algorithms. These algorithms allow agents to estimate the expected value of each action based on the environmental states and adjust their policies accordingly.

[0072] As the computer-implemented method agents continue to interact with the environment and receive feedback, they refine their knowledge and improve their performance on the assigned task. This iterative process of exploration, evaluation, and policy adjustment is repeated until the agents achieve an optimal level of task performance.

[0073] Reinforcement learning has applications in a wide range of fields, from robotics and gaming to resource optimization in complex systems. By allowing computer-implemented agents to learn through interaction with their environment, this technique offers a powerful way to tackle problems where prior training data is unavailable or where environmental conditions may change over time.

[0074] To train the evaluation process (12) based on reinforcement learning, the knowledge database (4) can be used. In this case, the knowledge database (4) is divided into a training data set and a test data set. In addition, it is advisable to ensure that both data sets are statistically representative and cover a wide range of situations.

[0075] In some embodiments of the method disclosed herein, the evaluation process (12) based on reinforcement machine learning is pre-trained with a database of evaluation questions (13) and answers (7) from users (2) constructed in a heuristic manner. In training the evaluation process (12), at least one evaluation method and at least one evaluation metric are applied. The evaluation process (12) can be made to compete with the heuristic results. Cross-validation can also be performed to find the appropriate hyperparameters.

[0076] The evaluation method is selected from the group comprising cross-validation methods, test set retention methods, and combinations thereof. The evaluation metric is selected from the group comprising precision, recall, and F1 -score. This allows the evaluation of the performance of the artificial intelligence model (1 ) in classifying entities (e.g., companies, public institutions, educational institutions) at different maturity levels (25).

[0077] An advantage of using metrics such as precision, recall and F1 -score lies in their ability to comprehensively evaluate and measure the performance of the AI ​​model (1 ) in classifying users (2) according to different maturity levels (25). Each of these metrics addresses specific aspects of performance and provides a more complete and balanced view of the classification ability of the AI ​​model (1 ). Considering these metrics together results in a more robust and reliable evaluation, compared to using only one of them, or other evaluation metrics.

[0078] Precision assesses the proportion of cases correctly classified as belonging to a specific category (maturity level (25) of dimension (10)). It is especially useful when seeking to minimize false positives, i.e., misclassifying an entity as belonging to the wrong maturity level (25). High precision indicates that the model has a low false positive rate. Recall, also known as sensitivity or exhaustiveness, measures the model's ability to correctly identify positive cases, i.e., entities that truly belong to a specific maturity level (25). Recall is important when seeking to minimize false negatives, i.e., misclassifying an entity as not belonging to a specific maturity level (25). High recall indicates that the model has a low false negative rate.

[0079] The F1 score is a measure that combines precision and recall into a single value. It represents the balance between the two metrics and provides a more comprehensive measure of the model's overall performance. By combining precision and recall, the F1 score is useful when seeking a balance between a model's accurate and comprehensive classification ability.

[0080] Referring to FIG. 1 and FIG. 2, in sub-stage b1) of step b), an evaluation question (13) corresponding to a dimension (10) of the artificial intelligence model (1) is extracted from the knowledge database (4). This selection can be random. This has advantages in terms of fairness, variety, adaptability, elimination of hidden biases, efficiency and simplicity.

[0081] For example, by using a random approach, any bias or preference in the selection of assessment questions (13) is avoided. This means that all assessment questions (13) have an equal chance of being selected, ensuring an unbiased and fair assessment. The random selection of assessment questions (13) ensures greater diversity in the assessments. Furthermore, this prevents the process from becoming monotonous or predictable, as each assessment may feature different assessment questions (13). Additionally, sometimes, by selecting assessment questions (13) intentionally or based on specific preferences, hidden biases can be introduced into the assessment process (12). Random selection reduces the possibility of these biases influencing the results, as the assessment questions (13) are selected neutrally and without any predisposition.On the other hand, in step b2) the user (2) interacts with the input / output device (15) to enter his / her answer (14). The input / output device (15) is a set of electronic devices that facilitate the interaction between the user and the system. The input / output device (15) may include screens, keyboards, touch screens, mice, microphones, speakers, headphones, cameras, motion sensors, gesture sensors, and any other electronic device that allows the question (13) to be displayed and detects the user's (2) selection of an answer (14) to the question (13). The input / output device (15) may also be a terminal, for example, a computer, smartphone, or tablet.

[0082] Once the computing unit (3) receives the response (14), it validates in the knowledge database (4) whether it is correct. According to this validation, in sub-stage b3) it updates the rating score (20) associated with at least one dimension (10) through a classification process (21) based on machine learning that takes as input the response (14) entered by the user (2). The rating score (20) has at least one maturity level value (25) of the dimension (10), and a confidence level value (26) of the maturity level value (25).

[0083] The classification process (21 ) is a sub-process of the evaluation process (12) that uses reinforcement learning techniques to adjust the rating score (20). This improves the performance in assessing the user's maturity level (25) in a certain dimension (10) of the artificial intelligence model (1 ). In the classification process (21 ), a reward function is used to evaluate the quality of the answers and the rating score is adjusted accordingly.

[0084] The rating score (20) comprises at least a maturity level value (25) of the dimension (10) evaluated and a confidence level value (26) associated with said maturity level (25). The maturity level value (25) represents the degree of development or competence of the user (2) in that specific dimension (10) of the artificial intelligence model (1), while the confidence level value (26) indicates the certainty or reliability of the maturity level (25) assigned. Once the rating score (20) is obtained, the computing unit (3) executes stage b4) and obtains a group of evaluation questions (18) through a selection process (22) based on machine learning that takes as input the rating score (20) and the records (9) associated with the dimension (10). The selection process (22) is configured to reduce the number of evaluation questions (13) that are displayed on the input / output device (15) and answered by the user (2).

[0085] In this case, the loss function is defined as the difference between the prediction of the assessment process (12) and the heuristic maturity level (25) assigned to users (2) of an entity or company under assessment. A regression loss function such as the mean squared error (MSE) and the mean absolute error (MAE) can be used, which calculates the average of the squared and absolute differences, respectively, between the model predictions and the actual values ​​of the maturity levels (25).

[0086] The policy function, for its part, defines how the selection process (22) of the assessment process (12) decides which question to ask next to obtain the company's maturity level with the greatest possible confidence.

[0087] For example, an exploration-exploitation policy function can be used, where the selection process (22) of the assessment process (12) chooses the question that maximizes its uncertainty (exploration) in the answers (7, 14) of previously assessed companies, or the one that has given the best results in the past (exploitation). This helps the model to obtain the maturity level with a minimum of questions and a maximum of confidence.

[0088] The policy function determines the next evaluation question (13) to be asked in order to obtain the maturity level (25) of the user (2), and consequently, evaluating a plurality of users (2) of the company, with the greatest possible confidence. To achieve a balance between obtaining information and making decisions based on prior knowledge, a policy function based on exploration-exploitation can be used. This strategy allows exploring new options to improve learning, while taking advantage of the knowledge acquired so far to make better decisions. In some modalities of the method described here, an Adam optimizer can be used, which is an adaptive optimizer used in problems with large data sets and high dimensionality.Adam uses adaptive estimates of first- and second-order moments of gradients to efficiently update the neural network weights for the evaluation process (12). Thanks to this adaptability, the Adam optimizer is able to tune the network parameters effectively, which contributes to improved performance in the task of determining the maturity level (25).

[0089] The selection process (22) deploys its ability to adapt and make decisions based on the available data. It can use different approaches, such as decision trees, neural networks, or other classification methods. These algorithms are pre-trained with a variety of data to learn meaningful patterns and relationships.

[0090] The selection process (22) analyzes the rating score (20) and the records associated with the dimension (10) to identify the most relevant and effective evaluation questions (13) based on previous results. The objective is to reduce the number of questions needed to obtain accurate information about the maturity level (25) of the artificial intelligence model (1).

[0091] Reducing the number of questions (13) involves advantages in terms of computational efficiency of the computing unit (3), time savings, improved user experience, reduction of bias and improvement of the accuracy of the assessment of the maturity level (25) of the artificial intelligence model (1) compared to the case of evaluating one by one the questions (13) stored in the knowledge database (4).

[0092] Furthermore, by presenting only the most relevant and meaningful questions (13), the user experience (23) is improved. Users (2) are not overwhelmed by a large number of unnecessary questions (13), which makes it easier for them to participate and collaborate in the evaluation process.

[0093] As for step b5), this can be a conditional that the computing unit (3) executes each time the rating score (20) is updated. Sub-step b5) can be started in series with sub-step b4) or in parallel. When executed in series, this allows sub-step b4) to be stopped. However, since this usually happens in milliseconds, it can also be executed once sub-step b4) finishes without significantly affecting the performance of the computing unit (3).

[0094] In any of the embodiments of the method described herein, the method may further include a sub-step b21) of obtaining a rating data that is selected from a positive rating data (16) and a negative rating data (17) by comparing the response (14) of the user (2) with the response (7) associated with the evaluation question (13). Sub-step b21) is executed between sub-step b2) and sub-step b3) of step b).

[0095] Referring to FIG. 3, in sub-step b21 ), the computing unit (3) may query the knowledge database (4) and store the positive rating data (16) or negative rating data (17) in a field associated with the user (2), and the time at which the user (2) undergoes the assessment. This facilitates historical tracking and more detailed analysis of the individual performance of the user (2) over time. In addition, it allows for continuous and refined assessment of the performance of the user (2), thereby improving the reliability and quality of the assessment results.

[0096] In some of the modalities of the method described here that include sub-stage b21), the classification process (21) based on machine learning of sub-stage b3) is programmed to take as input the responses (14) classified as positive rating data (16) and discard the responses (14) classified as negative rating data (17).

[0097] This avoids unnecessarily executing the classification process (21 ) of sub-stage b3) for incorrect answers. In this way, a significant saving of computational resources is obtained in terms of processing time and storage capacity. Unnecessary calculations and operations are avoided in cases where the user's answer (2) does not match the expected answer (14). Now, in order to avoid biases in the evaluation, the classification process (21 ) can also take as input, for the calculation of the rating score (20), the proportion of the number of answers (14) classified as positive rating data (16) with respect to the number of evaluation questions (13) formulated. Also, the proportion of the number of answers (14) classified as negative rating data (17) with respect to the number of evaluation questions (13) formulated can be used.

[0098] In this way, the classification process (21) can adjust the values ​​of maturity level value (25) of dimension (10), and confidence level value (26) of maturity level value (25) according to the user performance (2).

[0099] In any of the modalities of the method described here, the method may also include step c) of obtaining a report data (5) of the maturity level for the implementation of the artificial intelligence model (1), taking as input the maturity level values ​​(25) with a confidence level value (26) greater than the predetermined value of each dimension (10).

[0100] Referring to FIG. 4 and FIG. 5, step c) allows to close the evaluation process and obtain a summary of results that can be used by those who need to know the maturity level for the implementation of the artificial intelligence model (1 ), for example, executive or management personnel. The report data (5) can include graphs, dashboards, tables, and any type of data structure that allows to display the results of maturity level values ​​(25) and / or their confidence level (26).

[0101] Preferably, in step c) the report data (5) is generated taking into account the results of the execution of steps a) and b) (together with their sub-steps b1 -b5)) in a plurality of users (2). The results of the users (2) can be grouped into classes, for example, according to their role in the organization, or according to their level of maturity (25) determined in said steps.

[0102] The report data (5) provides a concise and structured summary of the results of the maturity level assessment for the AI ​​model (1 ). This allows decision-makers and executives to gain a clear overview of the AI ​​model's (1 ) status in terms of its implementation and capability.

[0103] Furthermore, when the reporting data (5) includes reports and dashboards, these provide valuable and relevant information that supports informed decision-making. By having a visual representation of the maturity level values ​​(25) and their confidence level (26), decision makers can quickly assess the current situation and determine the necessary actions to improve and optimize the implementation of the artificial intelligence model (1).

[0104] Also, the reports in the reporting data (5) make it possible to identify specific areas where the artificial intelligence model (1 ) may require improvements or adjustments. By grouping the results of the users (2) into classes, for example, according to their role in the organization or maturity level, patterns and trends in the results can be detected. This makes it easier to identify specific areas for improvement and allocate adequate resources to address those areas. The reports in the reporting data (5) also make it possible to track the progress of the artificial intelligence model (1 ) over time, for example, when the users (2) of the entity or company are evaluated periodically (e.g., annually, semi-annually, quarterly). By comparing the reporting data from different evaluations, the evolution of the maturity level can be observed and it can be determined if improvements and progress are being made.

[0105] The present disclosure also describes embodiments of an apparatus for determining the level of maturity for the implementation of an artificial intelligence model (1). The apparatus comprises a computing unit (3) configured to execute the steps of any of the embodiments of the computer-implemented method disclosed herein.

[0106] For example, the apparatus comprises a computing unit (3) configured to execute step a) of accessing a knowledge database (4) that includes a plurality of evaluation questions (13) and answers (7) for each evaluation question (13). The evaluation questions (13) allow evaluating a dimension (10) of the artificial intelligence model (1). Also, the method has a step b) of executing an evaluation process (12) based on reinforcement machine learning.

[0107] The computing unit (3) is also configured to execute stage b) and its sub-stages. Stage b) has a sub-stage b1) of extracting, from the knowledge database (4), an evaluation question (13) corresponding to a dimension (10) of the artificial intelligence model (1). Furthermore, stage b) includes a sub-stage b2) of receiving an answer (14) entered by a user (2) into an input / output device (15) on which the evaluation question (13) is displayed, and a sub-stage b3) of updating a rating score (20) associated with at least one dimension (10) by means of a classification process (21) based on machine learning that takes as input the answer (14) entered by the user (2). The rating score (20) comprises a maturity level value (25) of the dimension (10), and a confidence level value (26) of the maturity level value (25).

[0108] Stage b) further includes a sub-stage b4) of obtaining a group of evaluation questions (18) through a selection process (22) based on machine learning that takes as input the rating score (20) and the records (9) associated with the dimension (10). The selection process (22) is configured to reduce the number of evaluation questions (13) that are displayed on the input / output device (15) and answered by the user (2).

[0109] Stage b) also has a stage b5 of repeating stages b1) to b4) by selecting a new evaluation question (13) from the group of evaluation questions (18), until obtaining maturity level values ​​(25) for the plurality of dimensions (10) with a confidence level value (26) greater than a predetermined value.

[0110] The computing unit (3) can be selected from a variety of options, including microcontrollers (e.g., Arduino, Raspberry Pi), microprocessors, DSCs (Digital Signal Controller), FPGAs (Field Programmable Gate Array), CPLDs (Complex Programmable Logic Device), ASICs (Application Specific Integrated Circuit), SoCs (System on Chip), PSoCs (Programmable System on Chip), computers, servers, tablets, cell phones, smartphones, signal generators, and other similar computing units known to a person of average skill in the art.

[0111] The computing unit (3) may include a memory module that covers options such as RAM memory (cache, SRAM, DRAM, DDR), ROM memory (Flash, cache, hard drives, SSD, EPROM, EEPROM, removable ROM memories such as SD, MMC, Compact Flash, SMC, SDC, MS, among others), CD-ROM, DVD or other optical storage, magnetic tapes, magnetic cassettes and any other data storage medium known to a person moderately versed in the matter.

[0112] In addition, the computing unit (3) may have a communications module that includes wired and wireless communication options. Examples of wireless communications are Bluetooth, WiFi, RF ID, UWB, GPRS, Konnex or KNX, DMX, WiMax and other equivalent technologies. For wired communications, connection ports such as I2C, CAN, Ethernet, SPI, SCI, QSPI, 1 -Wire, DDB, Profibus and other standards known to a person moderately versed in the subject can be used.

[0113] The computing unit (3) can communicate with other computing devices through communication protocols such as AS-¡, BSAP, CC-Link, CIP, CAN bus, among others. These protocols allow efficient and reliable communication between the computing unit (3) and other devices.

[0114] Additionally, the computing unit (3) can be connected to input / output devices (15) that include storage devices, display devices and Human Interface Devices (HID). The display devices can be CRT monitors, LCD displays, LED displays, projectors, televisions, OLED displays, AMOLED displays, quantum dot displays, among others. The HID devices can be keyboards, mice, trackballs, touchpads, joysticks, touch screens and other devices that allow user interaction with the computing unit (3).

[0115] In some embodiments of the apparatus disclosed herein, the computing unit (3) may be a server, whether physical or virtual, while the computing unit (15) is a terminal used by the user (2). The terminal may be a mobile phone, a personal computer, or other similar device.

[0116] The terminal establishes a communications protocol with the computing unit (3) to allow interaction between the user (2) and the method executed by the computing unit (3). A common way to establish this communication is through a web service. The terminal accesses a specific application or interface provided by the computing unit (3) through a network connection, such as the Internet. This allows the user (2) to access the application from his terminal and use it to interact with the method implemented in the computing unit (3).

[0117] The present disclosure also describes embodiments of a computer program, and a computer-readable medium, which include instructions that when read by a computing unit, cause the computing unit to execute any of the embodiments of the computer-implemented method described herein.

[0118] The computer program can be written in any programming language or framework known to a person of average skill in the art. For example, the computer program can be written in Java, JavaScript, Perl, PHP, C++, C#, Python, R-studio SQL, Swift, Ruby, Delphi, Visual Basic, D, HTML, HTML5, CSS, NodeJs, Angular, React, Go, RPA, .NET, Scala, Drupal, Ember, Labview, and other programming languages ​​and / or frameworks known to a person of average skill in the art.

[0119] In the field of machine learning, there are several programming languages ​​and frameworks. Some of the most popular programming languages ​​for machine learning are Python, R, and Julia. Python is known for its specialized machine learning libraries and frameworks, such as TensorFlow, Keras, PyTorch, and scikit-learn. R also offers packages and libraries, such as caret, randomForest, and ggplot2, that facilitate data analysis and the implementation of machine learning algorithms. Julia, meanwhile, has been noted for its capacity for numerical processing and high-performance scientific programming.

[0120] As for frameworks, in addition to the aforementioned TensorFlow, Keras, and PyTorch, other popular frameworks include scikit-learn, Caffe, Theano, and Spark MLlib. These frameworks provide a wide range of tools and features for developing, training, and deploying machine learning models.

[0121] Furthermore, the present invention also relates to a computer-readable medium comprising instructions, which when executed by a system according to any of the above-described embodiments cause said system to carry out steps of a method according to any of the embodiments of the methods described above in this disclosure.

[0122] The computer-readable medium may be selected from executable files, installable files, compact discs, RAM (cache, SRAM, DRAM, DDR), ROM (Flash, Cache, hard drives, SSD, EPROM, EEPROM, removable ROM (e.g. SD (miniSD, microSD, etc.), MMC (MultiMedia Card), Compact Flash, SMC (Smart Media Card), SDC (Secure Digital Card), MS (Memory Stick), among others), CD-ROM, digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic tapes, storage or any other medium that can be used to store information and that can be accessed by a processing unit.

[0123] The computer-readable medium provides a convenient and versatile way to distribute and implement the method in various computing environments, allowing its application to a wide range of systems and devices. The computer-readable medium stores the instructions necessary to execute the computer-implemented method to determine the implementation maturity level of an artificial intelligence model (1 ) in a computing system, allowing the system to carry out the corresponding steps. These instructions can be written in various programming languages ​​and be designed to be compatible with the target system. The computer-readable medium allows the instructions to be accessed and processed by the processing unit of the system, for example, by the computing unit (3) which implies that the operations necessary to carry out the method are executed efficiently and accurately.

[0124] Examples

[0125] Example 1: Computer-implemented method for determining the maturity level for the implementation of an artificial intelligence model (1) A first example of the method has a step a) of accessing a knowledge database (4) that includes a plurality of evaluation questions (13) and answers (7) for each evaluation question (13).

[0126] The knowledge database (4) is a relational database. The knowledge database (4) includes records (9) grouped according to a plurality of record classes (8). Each record (9) includes at least one assessment question (13) and one answer (7) for each assessment question (13). Each record class (8) groups records (9) associated with a dimension (10) of the artificial intelligence model (1). The answers (7) have possible values ​​of “yes”, “no” and “don't know”.

[0127] The database has 1000 assessment questions (13). At least 150 questions were used for each dimension (10). The dimensions (10) that have been considered to assess the maturity level (25) are: strategy, tactics, organizational culture, security, ethics, technology and data governance.

[0128] In this example, in step b) of executing an evaluation process (12) based on reinforcement machine learning, the evaluation process (12) is trained by taking as input a set of training data, which includes a plurality of records (9) from the knowledge database (4). The evaluation process (12) is validated against heuristic data.

[0129] The data were prepared by collecting the responses to the evaluation questions (13) from the knowledge database (4) and the heuristic maturity levels (25) assigned to 10,000 heuristically evaluated samples.

[0130] Cross-validation was also performed to find the appropriate hyperparameters. Model performance was evaluated using a test dataset using metrics such as precision, recall, and F1 score. In this case, the loss function was defined as the difference between the assessment process prediction (12) and the heuristic maturity level (25) assigned to a company under assessment. A regression loss function such as mean square error (MSE) and mean absolute error (MAE) was used. In addition, an exploration-exploitation based policy function was used, where the selection process (22) of the assessment process (12) chooses the question that maximizes its uncertainty (exploration) in the answers (7, 14) of previously assessed companies, or the one that has given the best results in the past (exploitation).

[0131] In addition, the Adam optimizer was used. Adam uses adaptive estimates of first and second order moments of the gradients to efficiently update the neural network weights of the selection process (22) and the evaluation process (12). Thanks to this adaptability, the Adam optimizer is able to tune the network parameters effectively.

[0132] As for the sub-stages of stage b), this example includes sub-stage b1) of extracting, from the knowledge database (4), an evaluation question

[0133] (13) corresponding to a dimension (10) of the artificial intelligence model (1 ). In this example, the evaluation question (13) is randomly selected. Data extraction and access to the knowledge database (4) is done through SQL queries.

[0134] In this example, sub-stage b2) of receiving a response is also executed.

[0135] (14) that a user (2) enters into an input / output device (15) on which the evaluation question (13) is displayed. In this example, the input / output device (15) is a user terminal (2) that establishes a connection with the computing unit (3) through a web service. The user terminal (2) connects to the Internet, and accesses a web application.

[0136] Then, the computing unit (3) executes sub-stage b3) of updating a rating score (20) associated with at least one dimension (10) by means of a classification process (21) based on machine learning that takes as input the answer (14) entered by the user (2). The rating score (20) comprises a maturity level value (25) of the dimension (10), and a confidence level value (26) of the maturity level value (25).

[0137] Then, the computing unit (3) executes sub-stage b4) of obtaining a group of evaluation questions (18) through a selection process (22) based on machine learning that takes as input the rating score (20) and the records (9) associated with the dimension (10). As mentioned above, the selection process (22) is configured to reduce the number of evaluation questions (13) that are displayed on the input / output device (15) and answered by the user (2).

[0138] In this example, sub-step b5) is executed by repeating steps b1) to b4) by selecting a new evaluation question (13) from the group of evaluation questions (18), until obtaining maturity level values ​​(25) for the plurality of dimensions (10) with a confidence level value (26) greater than a predetermined value.

[0139] As the example shows, a maximum of 10 questions per dimension are required on average to achieve a maturity level with a minimum confidence level of 95%. This means the assessment process is simplified and accelerated, without compromising the accuracy of the results.

[0140] Example 2: Computer-implemented method to determine the maturity level for the implementation of an artificial intelligence model (1) with report generation

[0141] The method of example 1 was modified by including a step c) of obtaining a report data (5) of the maturity level for the implementation of the artificial intelligence model (1), taking as input the maturity level values ​​(25) with a confidence level value (26) greater than the predetermined value of each dimension (10).

[0142] The report data (5) allows a dashboard to be displayed in the web application that establishes communication with the computing unit (3) and the terminal, with the results of a plurality of users (2) who have been evaluated in the company. This dashboard can be accessed by (2) with predetermined roles and permissions, for example, executive and management personnel.

[0143] It should be understood that the invention claimed herein is not limited to the modalities disclosed and illustrated herein, since as a person moderately versed in the art will understand, there are possible variations and modifications that do not deviate from the spirit of the invention, which is only defined by the following claims.

Claims

CLAIMS 1. A computer-implemented method for determining the maturity level for implementing an artificial intelligence model (1 ), CHARACTERIZED in that it comprises executing on a computing unit (3) the steps: a) accessing a knowledge database (4) that includes a plurality of evaluation questions (13) and answers (7) for each evaluation question (13); where the plurality of evaluation questions (13) allows evaluating a dimension (10) of the maturity level for implementing the artificial intelligence model (1 ); b) executing an evaluation process (12) based on machine learning by reinforcement, comprising: b1) extracting, from the knowledge database (4), an evaluation question (13) corresponding to a dimension (10) of the maturity level for implementing the artificial intelligence model (1 );b2) receiving a response (14) entered by a user (2) into an input / output device (15) on which the evaluation question (13) is displayed; b3) updating a rating score (20) associated with at least one dimension (10) by means of a classification process (21) based on machine learning that takes as input the response (14) entered by the user (2), where the rating score (20) comprises:; - a maturity level value (25) of dimension (10), and - a confidence level value (26) of the maturity level value (25); b4) obtaining a group of evaluation questions (18) by means of a selection process (22) based on machine learning that takes as input the rating score (20) and the records (9) associated with the dimension (10); where the selection process (22) is configured to reduce the number of evaluation questions (13) that are displayed on the input / output device (15) and that the user (2) answers; b5) repeating steps b1) to b4) selecting a new evaluation question (13) from the group of evaluation questions (18), until obtaining maturity level values ​​(25) for the plurality of dimensions (10) with a confidence level value (26) greater than a predetermined value.

2. The method of claim 1, CHARACTERIZED in that it also includes a step c) of obtaining a report data (5) of the level of maturity of implementation of the artificial intelligence model (1), taking as input the maturity level values ​​(25) with a confidence level value (26) greater than the predetermined value of each dimension (10).

3. The method of claim 1, CHARACTERIZED in that the knowledge database (4) includes records (9) grouped according to a plurality of record classes (8), where each record (9) includes at least one evaluation question (13) and an answer (7) for each evaluation question (13), and where each record class (8) groups records (9) associated with a dimension (10) of the artificial intelligence model (1).

4. The method of claim 1, CHARACTERIZED in that the evaluation process (12) based on machine learning by reinforcement, is previously trained with a database of a plurality of evaluation questions (13) and responses (7) of users (2) constructed in a heuristic manner; where in the training of the evaluation process (12) at least one evaluation method and at least one evaluation metric are applied.

5. The method of claim 4, CHARACTERIZED in that in the training of the evaluation process (12): the evaluation method is selected from the group comprising cross-validation methods, test set retention methods, and combinations thereof, and the evaluation metric is selected from the group comprising precision, recall and F1 -score.

6. The method of claim 1, CHARACTERIZED in that between sub-stage b2) and sub-stage b3) of stage b) a sub-stage b21) is executed to obtain a rating data that is selected between a positive rating data (16) and a negative rating data (17) by comparing the response (14) of the user (2) with the response (7) associated with the evaluation question (13).

7. The method of claim 1, CHARACTERIZED in that the responses (7, 14) are of Boolean type.

8. The method of claim 2, CHARACTERIZED in that the responses (7, 14) also include a third value different from the Boolean values.

9. The method of claim 1, CHARACTERIZED in that the dimensions (10) are selected from the group comprising strategy, tactics, organizational culture, security, ethics, technology and data governance, circular economy, environmental impact, accident rate and risk prevention.

10. A device for determining the level of implementation maturity of an artificial intelligence model (1), CHARACTERIZED in that it comprises a computing unit (3) configured to execute the stages of: a) accessing a knowledge database (4) that includes a plurality of evaluation questions (13) and answers (7) for each evaluation question (13); where the plurality of evaluation questions (13) allows evaluating a dimension (10) of the artificial intelligence model (1 ); b) executing an evaluation process (12) based on machine learning by reinforcement, comprising: b1) extracting, from the knowledge database (4), an evaluation question (13) corresponding to a dimension (10) of the maturity level for the implementation of the artificial intelligence model (1 ); b2) receiving a response (14) entered by a user (2) in an input / output device (15) in which the evaluation question (13) is displayed;b3) update a rating score (20) associated with at least one dimension (10) by means of a classification process (21) based on machine learning that takes as input the response (14) entered by the user (2), where the rating score (20) comprises:; - a maturity level value (25) of dimension (10), and - a confidence level value (26) of the maturity level value (25); b4) obtaining a group of assessment questions (18) by means of a selection process (22) based on machine learning that takes as input the rating score (20) and the records (9) associated with the dimension (10); where the selection process (22) is configured to reduce the number of assessment questions (13) that are displayed on the input / output device (15) and answered by the user (2); b5) repeat steps b1) to b4) by selecting a new evaluation question (13) from the group of evaluation questions (18), until obtaining maturity level values ​​(25) for the plurality of dimensions (10) with a confidence level value (26) greater than a predetermined value. 1 1. A computer program, CHARACTERIZED in that it comprises instructions, which when executed by a computing unit (3) cause the computing unit (3) to execute the steps of the method of claims 1 to 10.

12. A computer-readable medium, CHARACTERIZED in that it comprises instructions, which when executed by a computing unit (3) cause the computing unit (3) to execute the steps of the method of claims 1 to 10.

Images