System and method for combinatorial optimization over generic search space comprising both solutions and rules for searching the solutions

Abstract

The present invention belongs to the field of computer programs, more precisely to the field of methods related to machine learning and combinatorial optimization. The invention discloses a system and method for searching over generic search space comprising both solutions and rules for searching the solutions. The problem of choosing the rules of searching is treated in the same way as the original problem of combinatorial optimization by applying the same search techniques recursively. The method builds a hierarchy on relations between the more specific and more abstract rules. This second-order hierarchy does not fit into the search space and is processed separately for the assessment of more abstract rules against the quality criterion defined for the original problem using machine learning techniques. The method allows shifting from the use of specific a priori assumptions about the search space to the general assumption of the existence of effective search rules.

Description

System and method for combinatorial optimization over generic search space comprising both solutions and rules for searching the solutionsField of the invention

[0001] The present invention belongs to the field of computer programs, more precisely to the field of methods related to machine learning and combinatorial optimization. The invention relates to a system and method for combinatorial optimization over generic search space comprising both solutions and rules for searching the solutions, as well as to a computer program comprising executable instructions for performing said method as well as to a computer readable medium comprising said computer program.Background of the invention, prior art and the technical problem

[0002] Combinatorial optimization problems consist in construction of arrangements of elements from a given finite collection according to given restrictions to find a solution, which is possibly the best arrangement against some objective function or quality criterion. In contrast to problems for which straightforward procedures that require polynomial time and resource amounts to find the solutions are known, what is straightforward for most of combinatorial optimization problems is only the procedure of assessment of the quality of arrangements. The performance of a combinatorial optimization algorithm is defined by the complexity of search in the given space of arrangements. The complexity of brute force search in general case is an exponent from the length of the solutions, which is impractical, so additional problems consist in determining conditions of existence and suggesting rules for fast construction of candidate solutions which are close to the best arrangement. Different algorithmic approaches and heuristic restrictions of the search space can be tried for finding optimal or suboptimal solutions for the problems in polynomial time or resource amounts, but none of them is guaranteed to be polynomial for all problems. No-free-lunch theorems in machine learning, search, and optimization prohibit the existence of an effective general algorithm that fit all problems equally. This implies that even when some conditions of existence of the fast rules to narrow down the search are theoretically proved as enough, these conditions work only as assumptions or hypotheses in practice as their relevance to a particular problem domain must be checked and confirmed. Researches not focused on solving specific applied problems are always about seeking a compromise between finding assumptions that shape up a scope broad enough to cover a variety of practical problems on one hand and elaborating effective methods able to solve the problems from the scope on the other hand. What is even more complex is another process that takes place in operation of human intelligence that consists in choosing characteristics of the practical problems as candidates for the conditions of existence of the fast rules of searching solutions, associating them to the fast rules that can be found separately and proving these associations statistically, and then searching for the rules of reasoning that connect the conditions of existence of the fast rules and the fast rules themselves where possible.These reasoning rules are established on a higher level of abstraction as they work with representations of fast rules and their prerequisites and not address the data the fast rules process. The applications of the reasoning rules are broader than just the scope defined by the fast search rules, but limited again as they cannot overcome the limitations of the no-free-lunch theorems, so they need to get grounds in other conditions, statistically first and then logically by drawing new rules of even higher abstraction. The search for abstract reasoning rules continues recursively until it converges to math logic that needs nothing else than itself to define its own limitations.

[0003] There are a number of approaches to solving combinatorial optimization problems on a higher level of abstraction.

[0004] One of the approaches is evolutionary algorithm used for searching solutions recorded in some notation system that are optimal according to a given criterion, which in the method of evolutionary algorithm is defined by a fitness function (Goldberg, 1989: Generic algorithms in search, optimization and machine learning. Addison-Wesley, Reading). In the method, a plurality of candidate solutions is maintained, which is called population. New candidates are formed by applying mutations and crossover operations to the candidate solutions from the population. The quality of the new candidate solutions is evaluated against the quality criterion and the best of them get to the new generation of the population. The notation system in evolutionary algorithm can be a formal language, possibly a programming language, the syntax of which determines some restrictions on possible solutions. The language can be interpretable by an execution unit for practical application of the search results and assessment of the quality of the candidate solutions for selecting the best for reproduction and mutation. The language can be chosen to match the problem, providing means of expression for different types of relations, hierarchies, and abstractions, which makes evolutionary algorithm a method of combinatorial optimization in different applications, see augmented neural networks in patent application US2019080240 and amended machine learning methods (Khadka and Turner, 2018: Evolution-guided policy gradient in reinforcement learning, NeurlPS) as examples.

[0005] Evolutionary algorithm is an example that demonstrates how the assumptions underlying fast rules in combinatorial optimization work. The assumption under evolutionary algorithm is that the record of the best solution differs from the records of the good candidate solutions not much and that the best solution can be obtained by iteratively applying mutation and crossover operators to the candidates that become better in average on each iteration. By this assumption it becomes possible to develop a search algorithm boosted in depth and suppressed in breadth. Other hypotheses are proposed in some variations of evolutionary algorithm that accelerate the search in depth even further, e.g. the hypotheses that higher-quality solutions or their elements can be reused as elements for building even better solutions. These assumptions form the basis for elaborating an algorithm to perform the search faster but at the same time they restrict the scope of its application.

[0006] Other methods like neural networks and machine learning are becoming closer to combinatorial optimization and each other in attempts of defining common characteristics of problems that can be solved effectively and the general approach to overcome the limitations of the no-free-lunch theorems despite some terminological and historical differences as now they encounter the problems known before in combinatorial optimization but on a new level of technological development.

[0007] Neural networks are generally known as realizations of the gradient descent method that take advantage of fast parallel computation on GPU. The gradient descent works for fixed discrete parameters of the model defining the number of layers in the network, their functions and the quantity of neurons in each. When the optimal discrete parameters for a problem are not known in advance, finding them becomes a combinatorial problem. Some neural networks are augmented by long shortterm memory (LSTM) introducing attention layers into the model structure, which can be considered as a built-in approach to solving the combinatorial optimization problem on a subset of the model discrete parameters. In general case, the use of LSTM expectedly leads to a combinatorial explosion as data volume and time required for training such models grow exponentially on the number of attention layers. State-of-the-art neural network models do not keep the track of the combinatorial optimization process and has no built-in means to search how to run it more effectively for a particular problem domain. There is a cause for that deeper than just the state of the art, which lies in the close relationship of neural networks as a method with GPU as the platform with the limitations it imposes on the properties of algorithms for fast execution on GPU. Many effective approaches involving the use of abstractions developed for solving combinatorial optimization problems are hard to implement in neural networks alone. For this reason the effects of machine understanding and abstract thinking associated with e.g. adapting the trained models to new data representations are not realized within the framework of the neural networks.

[0008] Fixing the number of attention layers in neural networks is another example of how assumptions work. Again, this is the necessity determined by the GPU architecture. The hypothesis is that a model of meager hierarchy depth is enough to construct an optimal transformation of input data to output and it results in limiting the size of the search space, which helps finding the solutions faster in one hand but restricts the application of neural networks to problems that allow simpler solutions on the other hand. The hypotheses in these and other examples are always a compromise between the generality of the method and its performance expressed in terms of time and amount of data needed to train the model.

[0009] Although CPU, in theory, provides unlimited flexibility for developing deep models of variable structure and for the automation of their processing, evolutionary algorithm, which is usually implemented on CPU, does not make the most of the opportunities offered by the platform because of methodological limitations that have not yet been overcome. The analogy to the natural evolution underlying the specific depth-first search of the evolutionary algorithm, even when it proves effectiveon some problems, is a mere set of assumptions that should be revised and tried on each other problem before using. For this reason, developments in the evolutionary algorithm follow the path of partial improvements for individual problems, which cannot claim generality, including those classified as synergistic optimization of reinforcement learning and evolutionary algorithms by Pengyi et al. (2024; Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey. arXiv:2401.11963v1) and hierarchical evolutionary algorithm (Zhang et al., 2020: AutoAlpha: an Efficient Hierarchical Evolutionary Algorithm for Mining Alpha Factors in Quantitative Investment. arXiv:2002.08245; Lai and Chang, 2009: A hierarchical evolutionary algorithm for automatic medical image segmentation. Expert Systems with Applications, V36, N1).

[0010] Machine learning refers to a broad group of methods for building utility relations between input and output data not necessarily bound to implementation on GPU. In many applications of machine learning, there is also a need to find optimal values for both discreet and continuous model parameters, and though machine learning may be slower that neural networks in the speed of optimization in the continuous part of the model, it is potentially more effective in the discreet part as it offers different ways to combine algorithms and model elements automatically by implementing hypotheses that can be statistically confirmed or refuted for a given scope. On the other hand, machine learning applied to finding the best search algorithm for a given problem domain needs solving combinatorial optimization problem but on a new level. Combining the two approaches in one method and system to make them work in applications requiring not only solutions are but also the construction of rules of finding them, including the construction of abstract rules that shape the landscape of the applied rules, is the problem, which can be probably solved in various ways. An additional requirement for the combination of the two approaches is that it must be generic for the solutions and the rules of different levels of abstraction in order to reduce the involvement of developers into working out the specifics of the method at each level of abstraction and make the method fully automated even when applying in different problem domains.

[0011] The idea how to combine the two approaches to optimization can be outlined as follows. The depth-first search boosted by the reuse of parts of known solutions as in some variants of evolutionary algorithm can be described in terms of constructing a hierarchy over relations that define what each solution consists of, which will be referred to as hierarchy of the first order, and the corresponding relation can be referred to as relation of composition. The rules that build such relations are based on assumptions about the problem domain the rules are derived for. These rules in their turn can be searched as combinations of procedures just in the same way as the solutions they construct, so the relation between the rules and the procedures they call also fits into the framework of the first order hierarchy. Now when the first order hierarchy applies to rules of searching the solutions, a need in additional more abstract layer of rules arise to avoid combinatorial explosion, which otherwise goes on stage, now for the search for rules. The first order hierarchy thus describes the composition of all rules along with the solutions generated by the system. What the first order hierarchy does not fit for isanother type of relation, which exists between a solution and the rule applied for its production, as well as between the applied rule and a more abstract rule used to construct or shape the applied one. The abstract rule can also be considered as applied in relation to the rule it constructed, and so on. This type of relation is here referred to as relation of application and the hierarchy built on this relation will be referred to as second order hierarchy. Note that neural networks even when augmented by LSTM build only the first order hierarchy as they has no means to watch how they make decisions and influence this process by deriving abstract rules of making decisions, they just make them and evaluate them not evaluating the ways how they make them. It’s certainly an assumption that a method and a system implementing the process of building the two types of hierarchy can be effectively used in some applications, but this the way to evade the theoretical restrictions of the no- free-lunch theorem while still complying with its statement.

[0012] The technical problem in focus of this invention is to create complementary data structures for maintaining the hierarchies on the relations of composition and application and to organize a uniform routine of combinatorial optimization on different levels of the hierarchies. When it proves true that general rules can be found at some level of abstraction for the problem domain, the method presented in this invention is effective in time and resource amount needed for searching solutions to problems where either problems or solutions or both have descriptions which are necessarily long.Brief description of the invention

[0013] The present invention addresses the technical problem identified above. The technical problem is solved as defined in the independent claims, wherein the preferred embodiments are defined in dependent claims.

[0014] For a system of one or more computers to be configured to perform particular actions or operations means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the actions or operations. For an environment of the system to be configured to perform particular actions or operations means that the environment includes one or more computers with installed software, firmware, hardware, or a combination of them that in operation cause the actions or operations. For one or more computer programs to be configured to perform particular actions or operations means that one or more computer programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions or operations. For a system interacting with environment means that the system and the environment are configured to exchange messages, where some of the messages from the environment to the system may be rewards that have prior positive or negative meaning for the system. For a system or environment being in a state means that the system or environment in a particular moment in time possesses properties that can be expressed by values of variable parameters of the system or environment.

[0015] In the following description, the words will have a specific meaning defined in the table below:

[0016] The method according to the invention uses statistical data processing techniques of machine learning and search techniques of combinatorial optimization.

[0017] The main consideration under the method according to the invention is that the combinatorial search built on default assumptions for narrowing down the search space cannot be useful in general, that is, for any rules of searching solutions to problems, there are problems where the assumptions underlying the use of the rules hurdle the search or the way how the assumptions affect the search requires adaptation. The problem of choosing or adapting the rules by its character is similar to the original problem of combinatorial optimization, which may require a new round of search, which is now the search for rules how to solve the original problem. By reasoning in this way by induction, one can trace a hierarchy emerging on relations between specific solutions and abstract rules for producing them, while the abstract rules are, in turn, can be considered as solutions corresponding to a more abstract problem. This second order hierarchy built on relation of application for rules to produce rules is distinguished from the first-order hierarchy built on relation of composition for rules to comprise rules.

[0018] The present invention uses the concept and provides an implementation of the first and the second order hierarchies in a single embodiment and also provides a method based on the use of the hierarchies for searching solutions to problems along with the automated creation and usage of rules that recursively guide the search. The essence of the invention can also be described as augmenting the known method of evolutionary algorithm with techniques that enable the production of meta rules that apply to the production of other rules for the fast formal transformation of the solutions and rules into forms in which they can be used for combination with other elements, for substitutions into other elements, and for substitutions of other elements in them, where the fitness of the meta rules is statistically confirmed during the evolution as helping to form solutions and rules of high fitness.

[0019] The present invention allows shifting from the use of specific a priori assumptions about the first-order hierarchies and corresponding algorithms elaborated for specific combinatorial optimization problems to the general assumption on the ability to describe the problem domain by the hierarchies of the two orders and the corresponding search algorithm unified on each level of the first and second order hierarchy. This approach evades the limitations of the no-free-lunch theorems while not contradicting them as long as some search rules common for the whole problem domain on the higher levels of the second order hierarchy can be found, in which case the automated creation and usage of solutions and abstract rules to guide the search becomes possible.

[0020] The method presented in this invention is effective in time and resource amount needed for searching solutions to problems where either problems or solutions or both have descriptions which are necessarily long as long as the assumption defined above in article

[0019] is valid.

[0021] The implementation of the disclosed method has two main aspects. The first is related to the choice of language to define the search space equally suitable for both solutions and rules used to produce them. The second aspect is associated with the search process that links the production and assessment of solutions to the rules producing them that does not lead to a new combinatorial explosion caused by the necessity to build the second order hierarchy.

[0022] An example language that can be used to describe both solutions and rules for producing them uniformly is any from the LISP group of programming languages. The peculiarity of LISP is that it makes no syntactic distinction between data and algorithms, so that any algorithm can be treated as data for another algorithm, or the data input or output of an algorithm can be executed as an algorithm, if necessary. In addition to this peculiarity of being generic for algorithms and data, LISP implements features that are common for other programming languages, such as calling of an algorithm with certain parameters from the context of the execution of another algorithm, transferring data from the output of an algorithm to the input of another algorithm, and joint processing of data by several algorithms. The alphabet, or “standard library” of the chosen LISP dialect as internal language for the implementation of the method must include both input and output tokens associated with observing the state of the environment and actions directed to the environment as well as internal tokens associated with default procedures that check and / or change the states of the process of interpretation of the algorithms recorded in the internal language and return results also represented by tokens.

[0023] The choice of the internal language for the system is not limited to LISP dialects. Any language is suitable for the purpose if it allows constructing records from individual tokens and addressing to these records in the same way as to tokens and provides interpretation of the records as combinations of input and output operations and calls to the default procedures in the system that check or change the parameters that affect the process of interpretation of tokens. The internal language chosen asexample for this description is not a fully functional LISP dialect though it makes use of some of its ideas.

[0024] Building relations of application for the second order hierarchy is based primarily on establishing the facts of producing certain records by certain rules also described by records in the system. These facts of the relation of application are of two types:- a rule produces records possibly under influence of some variable parameters, in which case it is here referred to as combinatorial rule,- a rule determines the variable parameters that influence the rules that produce records, in which case it is referred to as introspective rule here or meta rule as in literature.Generally, a record can be produced in the system by a default procedure of mutation, which is defined in the system before the interaction of the system with the environment starts and does not change during the interaction, or by a combinatorial rule the record of which is derived by the system during the interaction with the environment, which can be influenced by introspective rules through varying the parameters of the interpretation of the combinatorial rule. Rules can be considered as combinatorial or introspective only in the context of their interaction with other rules, since a rule can simultaneously produce a record while being influenced by introspective rules and influence other rules. An introspective rule does not necessarily bring about interaction with the environment; instead, figuratively speaking, the process of interpretation of another rule become its new own environment. Observing and acting in this own environment becomes possible due to the use of the internal tokens in the records of the introspective rules, which are mentioned previously in article

[0022] .

[0025] When the quality of a token sequence in the output of the system is assessed, this assessment is shared recursively with the rules used to produce the sequence according to the established facts of the relations of application. There are three cases how these facts can be established and used for sharing the assessments of the quality of solutions and rules with the corresponding introspective and combinatorial rules:- The solution or rule is interpreted and assessed immediately after it’s produced, in which case the assessment is also shared immediately,- The rule is produced and recorded for later interpretation and assessment, in which case its assessment is also shared later,- The rule is produced but it is already recorded in the system as it was obtained in a different way, interpreted earlier, and already has an assessment, in which case the assessment is shared with the introspective and combinatorial rules according to the new fact of the relation of application.The first of the listed cases corresponds to the assessment of combinatorial and introspective rules for producing solutions on-the-fly. The second case corresponds to planning the application of the constructed rules. The third case corresponds to the assessment of the consistency of introspective and combinatorial rules with the experience obtained in the past.

[0026] In the method according to the invention, the functions of effective production of rules is assigned to other rules formed in the higher levels of the second-order hierarchy. To transfer this functionality from the default procedures that introduce changes into the existing rules, the default procedures like mutations and crossover operations in evolutionary algorithm are configured to make only rare changes to the rules under interpretation, which is still necessary to start the search process and to ensure some search in breadth, while most of the work of forming new rules is on the combinatory and introspective rules that were formed and assessed previously. The introspective rules can observe the parameters of the process of interpretation of another rule and change them or directly apply evolution operators to the rule under interpretation by calling default procedures identified by internal tokens used in the record of the introspective rule. The use of combinatorial and introspective rules results in more effective depth-first search as they represent algorithms that are derived from the experience of problem solving and confirmed statistically just as the rules used as solutions to the applied problems.

[0027] The functions of evolution operators such as abstraction and embedding, which provide for the reuse of ready-made candidate solutions as elements of other solutions to speed up depth-first search in the first order hierarchy, are achieved as described in the known method of hierarchical evolutionary algorithm. Crossover operators are not used as they are in the state-of-the-art evolutionary algorithm, because crossover is usually a complex procedure with a large number of influencing parameters, the effects of which, in the case of using crossover as a black-box operator, are hidden from the observers. Instead of the crossover operator, the disclosed method proposes spread as a default built- in mechanism that constitutes rules from such tokens and preferably in such sequences that actually takes place in the input and output as well as in the process of interpretation of the rules either due to the presence of some patterns in input data in response to solutions in the output of the system, or due to the existence of patterns in the output as reactions to the input data, which are default or acquired during the training. The action of the spread is similar to the action of crossover, the difference is that the spread does not regard assessments of the quality of an element or the quality of the solution the element in its decision about including this element in solutions but regards only the probability of appearance or use of this element.

[0028] In order to avoid another round of combinatorial explosion while establishing the facts of relations of application, the use of two types of synchronization is provided by default in the disclosed combinatorial method, which are temporal and spatial. The temporal synchronization prohibits simultaneous use of multiple solutions to generate system output. To implement this limitation, only one of the solutions from those interpreted in parallel is used to generate output in every moment, that is, it is interpreted continuously until it enters the terminal state, or until a threshold value of some interpretation parameters of this solution is reached indicating that achieving the terminal condition is probably impossible, or temporarily blocked by interpretation of one of the tokens in its record. Thespatial synchronization prohibits for an introspective rule each time it is interpreted to affect the interpretation of more than one rule.

[0029] Abstract rules formed in the system, the interpretation of which leads to the production of other rules and solutions with the high value of assessment against the quality function, allows reducing the amount of time and data required for learning. The formation of the abstract rule is exponential in time on the length of its record, while it replaces learning by memorizing a set of solutions, which is exponential in the volume of data and time on the size of the space of such solutions. Abstract rules are especially fruitful in case when the set of solutions that can be obtained using an abstract rule is not finite but countable, and though countable search spaces are not practical, it’s clear that in such case the gain in terms of time and resource amount needed for the search can be exponential if the length of the description of an abstract solution is less than the supposed size of the search space for a specific solution. Under the condition of a multilayer second-order hierarchy of the model, a number of the exponential replacements of the type of search lead to the possibility of searching practically utile solutions in sub-exponential or even polynomial time and amount of data on the supposed size of the solution search space. When the rules on the higher levels of abstraction are consistent with the problem domain, they can be used for obtaining solutions for new problems from this problem domain on-the-fly, i.e. in no extra search time.

[0030] Besides applied problems, the problem domain may include communication problems related to explaining ways used to produce the solutions by the system and explaining reasons for choosing the ways. Solving the communication problem, which will be referred to as training, involves the formation of rules for designating abstract algorithms and their properties as well as properties of data elements and other terms in the form of model rules and statistical relationships between them. In the disclosed method, the rules can be connected to their designations also in the form of rules. As the default algorithms work uniformly on each level of the second order hierarchy, the training can also be polynomial in the amount of data and polynomial or sub-exponential in time. The specifics of raining in comparison to searching solutions to the applied problems is that if the system implementing the disclosed method is a black box and there is no way to identify explicitly the rules formed in the system and the moments of their applications, their designations must be introduced right in the moments of their forming or application. To make the formation and application of the rules in the model predictable in order to make the training possible, the training must be performed stage-by-stage from simple to more and more complex problems, where the complexity is measured in the number of levels in the second-order hierarchy necessary to solve the problem in polynomial time and data volume. The stage-by-stage training for each successive level of the second order hierarchy can introduce structurally more complex problems formulated in the language terms already learned on previous stages with just some additions and assume the reuse of ways for solving problems formed by this training stage with just some additions, so that the system on each training stage not only findssolutions to new problems but also learns to designate the features of the new problems and the ways used to solve them.

[0031] As a result of training, machine understanding as the ability of introspection of the process of forming solutions and abstract thinking as the ability to choose and apply the right abstract rules for solving a problem become observable as the ability to provide a description of the process of solving a particular problem, justify the choice of ways used for solving, and explain how new ways of problem solving were formed in a form that can be perceived and evaluated by users or automated training means in the environment.

[0032] The method according to the invention can be used in applications where the use of abstraction and schemas are the key capabilities required for a system to derive complex patterns and causereasoning relationships that take place in processing digital models in various representations, especially under conditions of limited amounts of data, which is typically the case when using proprietary data, and to ensure the explainability and transparency of decision-making in learning systems. Examples of such applications exist in research and development processes, seamless CAx product design, virtual twin creation, functional verification and testing, predictive maintenance, etc.

[0033] In the detailed description, the method is used in the framework of a learning system interacting with environment described in terms as actions, states of the environment, rewards, model, utility function, and others used in machine learning and reinforcement learning in particular. In case of applying the method in the framework of a system searching a solution to a problem, the function of the same method could be described in terms of search space, fitness or loss function, algorithm population, etc. used in evolutionary algorithm.

[0034] The invention also relates to a computer program comprising executable instructions for performing the procedure as described above, as well as to a computer readable medium comprising said computer program.Brief description of drawings

[0035] The invention will be described in further detail based on exemplary embodiments and figures, which show:Figure 1 Scheme of exemplary embodiment of system and method for combinatorial optimization over generic search space comprising both solutions and rules for searching the solutionsFigure 2 Exemplary state of an instance running in the token interpretation unitFigure 3 Exemplary state of an instance with the cursor at an input tokenFigure 4 Exemplary state of an instance with the cursor at an output tokenFigure 5 Exemplary state of an instance binding the corresponding rule to another instance by interpretation of the passive internal token in the rule recordFigure 6 Exemplary state of an instance with the cursor at an active internal token that changes the state of the instanceFigure 7 Exemplary state of an instance corresponding to an introspective rule bound to another instance with the cursor on an active internal token in the rule recordDetailed description of the invention

[0036] Fig. 1. shows an example system 100 implementing the method of combinatorial optimization over generic search space comprising both solutions and rules for searching the solutions, configured to interact with the environment 101 in the representation of input and output sequences of tokens, 118 and 119 in Fig.1 correspondingly, in order to maximize the average value assessed according to a given quality criterion and possibly by receiving rewards from the environment 101 , whereas the environment 101 is configured to interact with the system 100 by transforming the parameters of its state into the input sequences of tokens 118 for the system 100, transforming the output sequences of tokens 119 from the system 100 into actions in the context of the state of the environment, and possibly making rewards to the system computed in accordance to given one or more utility functions. The system 100 and the environment 101 are examples of a system and environment implemented as a computer program wherein the components and techniques below are implemented. The example embodiment of the system 100 receives inputs as sequences of input tokens corresponding to environment state characteristics in the input buffer 111 and produces outputs as sequences of output tokens corresponding to system’s actions in the output buffer 110. For example, the system receives an input sequence of input tokens 118 at each of multiple iterations and generates an output sequence of tokens 119 after the input sequence of tokens at each of the iterations after the input sequence of tokens is received.

[0037] The method and the system described below uses tokens as elements to construct sequences of tokens referred to as records. Some of the tokens are input and output tokens that can appear in the system input 118 or output 119 correspondingly. Other tokens referred to as internal tokens are used as identifiers for some of the default procedures implemented in the system 100 that can check or change some parameters of the system that affect its operation. The input, output, and internal tokens constitute the alphabet 104 of the system 100. Some tokens referred to as extension tokens 105 are used as identifiers for the records. While the plurality of tokens in the alphabet 104 and the entities they identify is fixed for the whole period of interaction of the system 100 with the environment 101 , the plurality of extension tokens 105 can vary as new records are produced in the system and get identifiers or the records that have identifiers get low assessments against the quality criterion and the system 100 eliminates them along with the identifying tokens.

[0038] The example embodiment of the system 100 comprises a token interpretation unit 103, which can run simultaneously a number of instances of token interpretation. An instance can be considered as an object that interprets a token or tokens of a record and encapsulates some parameters that affect the way of the interpretation. The encapsulation makes the interpretation parameters not accessible for arbitrary processes running in the system. Running an instance corresponds to system following a rule of interaction with the environment by exchanging tokens or with another running instance by checking or changing its parameters. The following description will use the concept of instance for the sake of simplicity only and assuming that its implementation is not necessary.

[0039] In some embodiments, the token interpretation unit 103 is an interpreter for records in a Turing- complete language. In order to avoid infinite looping that occurs in such arrangements, means are needed in the system to limit the time for interpretation of each instance. The algorithm for the means can be probabilistic. The time allowed for the interpretation of an instance can be measured not necessarily in time units but, for example, as the number of time-requiring operations as described below in article

[0040] , and grow steadily as the system interacts with the environment.

[0040] The cycle of interaction of the example embodiment of the system described in this specification with the environment comprises a number of phases. On the first phase, the system generates a number of output tokens in the input-output buffer. On the next phase, the output tokens are processed by the environment to return a sequence of input tokens that characterize the state of the environment and make rewards to the system based on evaluation of the previous interaction according to the utility functions. On the next phase, the system processes the rewards and the input tokens from the input-output buffer to prepare for the next output. One iteration of the described cycle is referred to as iteration. Processing of each token placed in the input-output buffer is referred to as tact. Processing of each token interpreted by the instances in the token interpretation unit is referred to as tic. Iterations, tacts and tics are different measures of time used in the system for the purpose of the statistical assessment against the quality criterion and for the purpose of avoiding infinite loops that can be potentially generated by the method as described in the article

[0039] . For the latter purpose, the system maintains limits on the maximum duration of running instances and maximum duration of continuous running in passive regime expressed in iterations, tacts and tics. Ending of the interpretation of a record within the allowed is referred to as normal termination, otherwise it’s referred to as abnormal. Some other reasons for the abnormal ending will be described below.

[0041] Each instance runs in active or passive regime, which synchronizes the access of the instances to the output buffer 112. Fig. 1 represents an example state of the token interpretation unit that runs one instance in active regime 107 and simultaneously a number of instances in passive regime 108, 109. Running only one instance in any moment of time in active regime is used to ensure that only one of the rules at a time is used to generate the sequence of output tokens 119 and to avoid overlappingbetween different sequences of output tokens in the output buffer that could otherwise be generated by different running instances. When the sequence 118 of input tokens is formed in the input buffer 111 , each token from the sequence is used in series to update the states of the instances 107, 108, 109 and then only the instance running in active regime 107 is used to generate a series of output tokens 119 in the output buffer 110, which are also used to update the instances running in passive regime 108, 109 in series before they are transformed into actions for the environment 101 .

[0042] The number of the instances running in passive regime can vary according to the configuration of the token interpretation unit 103. Generally, the larger the average number of instances run in passive regime, the higher the average computation amount needed to process each input token but the less amount of input data and rewards are needed to find the utility solutions.

[0043] When interpreting a record, input tokens serve as conditions that need to be satisfied by finding the tokens in the input sequence to continue interpreting the record whereas an output token is placed into the output buffer without any condition and the interpretation of the record containing the output token continues. In this sense the input tokens can be considered as blocking the interpretation unless some condition is satisfied, while the output tokens are guaranteed to be not blocking. As output tokens are associated with actions performed by the system in the environment and input tokens are associated with passive waiting for a response or a new task from the environment, the internal tokens identifying default procedures can be also divided into active and passive by the guarantee of not blocking the interpretation of the records containing them. The guarantee of being not blocking is not a property of the default procedure but the property of the type of its call: the interpretation of an active internal token just calls the default procedure it identifies and does not wait for the result of its execution, while the interpretation of a passive token waits for the execution of the called default procedure to complete. For this reason, the interpretation of a passive internal token may have results significant for the further interpretation of the record containing the passive token, which are also expressed in terms of tokens for the purpose of keeping the method generic, while the interpretation of an active internal token usually has no result that can additionally change the state of the instance interpreting the record containing the active token.

[0044] Only one of those instances that are not currently blocked by the interpretation of the tokens of their records can run in active regime, the others run in passive regime. The state of the token interpretation unit 103 is shown in Fig. 1 for example purposes as running a number of instances in passive regime 108, 109. The purpose of running instances in passive regime is to have some instances updated with the recent information on the environment state characteristics in the moment when the instance running in active regime 107 is blocked and a different instance must be chosen to switch to running in active regime.

[0045] The extension tokens can be used as elements of the records in the same way as the alphabet tokens as described earlier in

[0037] . The use of the extension tokens in the records forms a hierarchy of calls of the rules when their records are instantiated for interpretation by the token interpretation unit 103. In the proposed example of embodiment of the system 100, when the interpretation for a rule makes a call, a new instance is created with the original record corresponding to the called rule. The regime of interpretation for the called rule depend on the type of the call which can be defined by the first tokens in the record of the called rule, i.e. the alphabet may include tokens referred to as releasing tokens that make the regime of interpretation for the called rules the records of which contain the releasing tokens active and correspondingly non-blocking the interpretation for the calling rules in accordance with the following principles:- By default, the called rule blocks the interpretation of the calling rule,- When the interpretation for the calling rule runs in passive regime, the interpretation for the called rule also starts in passive regime with blocking the interpretation for the calling rule regardless of the use of the release tokens. If the called rule terminates abnormally as described in article

[0039] , the calling rule also terminates abnormally. This is an example of a passive call.- When the interpretation for the calling rule is in active regime, the interpretation for the called rule starts in active regime, and the interpretation of the calling rule is blocked. In this case, the calling rule has a priority to continue running in active regime after the run of the called instance ends normally as part of the synchronization means. If the called rule terminates abnormally, the calling rule also terminates abnormally. This is another example of a passive call as it’s blocking.- When the interpretation for the calling rule is in active regime, the interpretation for the called rule is performed in passive regime until the cursor of the instance for the called rule comes over a release token. When this happens, the release token unblocks the interpretation of the calling rule, which may continue in active regime if no other token is chosen by the token interpretation unit for interpretation in active regime, while the called rule continues in passive regime. In this case, if the calling rule terminates abnormally, the called rule may continue running within its own time limit. The token that identifies the called rule in such a case can be considered as active as it’s non-blocking.

[0046] The example embodiment of the system 100 contains a model 102 comprising the alphabet 104 and the plurality of extension tokens 105 that identify the records produced in the system 100 during its interaction with the environment 101. Running instances in the token interpretation unit 103 can be considered as following rules kept in the model 102 in the form of records. The range of records in the model 102 is produced by means of default procedures implemented in the system or other rules that were produced by the system 100. The model 102 and the token interpretation unit 103 exchange by the records 113 where the model 102 saves new records, which are produced in the token interpretation unit 103 while performing interpretation of other records and tokens, and returns the records identified by extension tokens when these tokens are chosen for interpretation in unit 103.

[0047] The instance 108 running in passive regime in the example state of the token interpretation unit 103 depicted in Fig. 1 is introspective as it is updated not only by the input and output tokens, but also tokens 121 that characterize the internal state of the instance running in active regime 107. The introspective instances get bound with the instance running in active regime 107 to which they apply tokens 121 that characterize the internal state of the instance running in active regime 107 and can introduce changes to the internal state of the instance they are bound to when the instance running in active regime switches to passive regime and the introspective instance switches to running in active regime. The instance the introspective instance is bound to continues running from a changed internal state when it is switched back to active regime. Running some introspective instances 108 along with the instance running in active regime 107 can be means to introduce on-the-fly changes to the rules used to generate the output sequence of tokens 119 based on the input.

[0048] The use of the introspective rules like 108 can be a basis for implementation of machine understanding in the embodiment described in this example as the introspective instances like 108 work in a kind of debug mode to collect information on the application of other rules to generating output tokens in the context of the state of the environment defined by the input tokens and influencing interpretation of tokens in other instances by varying the parameters of the instances. Gathering the introspective information in the way proposed in this example or other ways can play accelerating role for the evolution of the records 105 in the model 102 and increase the speed of mutation-driven search for solutions by the system 100.

[0049] In the example embodiment in Fig. 1 , the choices 117 of records for interpretation in the token interpretation unit and the choice of the instance for running in active regime from the range of all instances running in the moment when blocking the current instance running in active is made in the model 102 using statistical data 106 on the assessments of the previous such choices. In some embodiments of the system, the choice of the instance for running in active regime from the range of running instances can take into account the measure of completion of each instance running, assigning higher weights to the instances that are closer to the terminal state.

[0050] Some tokens in the alphabet when encountered during the interpretation of a record can switch the instance into a state that requires choosing another token for interpretation. In the example embodiment in Fig. 1 , these conditional choices 115 are made using the statistical data 106 in the model 102 that takes into account additional conditions 116 determined by the state of the instance.

[0051] The statistical data 106 in the model 102 in the example embodiment in Fig. 1 is gathered by averaging rewards 122 coming from the environment and computed according to one or more utility function in the context of the recent interaction of the system with the environment, which is determined by the conditional choices 115 and choices 117 made recently. In some embodiments of the system, the choices 115 and 117 can be made using a function in analytical form, so thatgathering the statistical data 106 is not required, or the assessments against quality criterion for the choices 115 and 117 can be made using a function in analytical form, so that the rewards 122 are not required or taken into account. In some embodiments of the system, only a weighted part of the quality criterion can be given in analytical form. In some embodiments of the system, the quality criterion can be designed in a way where the facts of abnormal termination of rules can affect their assessments as mentioned in

[0040] .

[0052] In some embodiments of the system, the rewards 122 can be implemented by means of tokens.

[0053] Fig. 2 shows an example instance 200 run by the token interpretation unit 103 as described earlier in article

[0038] , where the instance is shown in an example state. The instance 200 is initialized by a original record 201 chosen as described earlier in article

[0049] . As the original record 201 is interpreted, the cursor 205 moves sequentially from one token of the record to another starting from the first token in the record. In some embodiments of the token interpretation unit, each instance200 has a number of flags 204 that characterize of the state of the instance along with the position of the cursor 205 in the original record. The original record is the record of the rule that is chosen for interpretation initially to initialize the instance, but it’s not necessary the exact rule that applies because of the mutations performed by the system to the original rule and the influence of introspective rules interpreted in parallel in the token interpretation unit.

[0054] The tokens T1 , T2, T4, T5, T3.1 , T3.2.1 , T3.2.2 in the example original record 201 are alphabet tokens described earlier in article

[0037] . The other two tokens T3 and T3.2 are extension tokens described in articles

[0037] and

[0045] that identify the sequence of tokens T3.1 , T3.2, T3.3 and the sequence of tokens T3.2.1 , T3.2.2 respectively.

[0055] In some embodiments of the described system, each instance 200 may have an internal memory 203, which contains tokens obtained as results of interpretation of the individual tokens during the interpretation of the record. In the example embodiment in Fig. 2, the internal memory is organized as first-in-first-out container. In the embodiments where rules perform calls of other rules as described previously in article

[0045] , the internal memory 203 can be used as means of interoperation between the calling and the called rules: when the instance for the called rule terminates normally and has tokens in the internal memory, the tokens are moved to the internal memory of the calling instance.

[0056] In some embodiments, each instance 200 may comprise means to record the modified rule 202, where the modification is due to mutations performed by the default procedures of mutation implemented in the system and application of introspective rules as described in article

[0046] . In the example in Fig. 2, the record of the modified rule may contain the same tokens as the original record201 initially chosen for interpretation but the token T6, which replaced the token T2 in the original record due to mutation. In variations of the proposed method, the modification of the original rule 201takes place before the instance 200 starts running, after the instance terminates normally, or during the run of the instance. In the proposed example embodiment of the system, the modifications takes place during the interpretation of the original rule 201 , where each token in the record of the original rule 201 has a chance to be excluded from the interpretation, delayed because of adding other tokens or substituted by a different token in the modified rule. As a result, what’s actually interpreted in not the original record 201 but one of its possible alternatives, which is constructed on-the-fly and recorded as the modified rule like 202. This approach is chosen because the modified rule gets its first assessment against the quality criterion right after it was generated and because the modification process can be observed and influenced by other instances for introspective rules in the context of the characteristics of the state of the environment 101 and the system 100.

[0057] In some embodiments of the proposed system, additional means to fix the state of the instance used as conditions for making the conditional choices as described in article

[0050] can be implemented. Such means referred to as actual state 206 represent a sequence of tokens that either coincide with the tokens from the modified record that were actually interpreted or their substitutes that took place in cases when the interpreted tokens identify the necessity of such substations including but not limited to the case described in article

[0050] .

[0058] The alphabet tokens described in

[0037] can be atomic identifiers or identify records constituted from tokens that determine their characteristics. For example, a token A that corresponds to a characteristic of the state of the environment can identify a sequence of two tokens B and C, where the first token B in the sequence is atomic and has a meaning that the token A corresponds to a characteristic of the environment state and the second token C is also atomic and corresponds to the value of the characteristic. The alphabet tokens can be organized in pre-defined hierarchies, in which case their interpretation is a subject for the introspective rules described previously in article

[0047] .

[0059] In the example implementation of the proposed method, when the cursor 205 reaches the end of the original record 201 that was not called from other rule and the internal memory container 203 is not empty, the sequence of tokens from the container is interpreted unless the contrary is defined by the state of the instance. For this purpose, in the example implementation of the proposed method, a new instance is specially initialized with the sequence of tokens from the internal memory as the original rule.

[0060] Fig.3-7 show example states of interpretation of rules comprising different tokens in the moment when the cursor of the instance is at the token under consideration. In all cases the interpretation of the token shifts the cursor to the next token in the record unless the contrary is not defined by the flags of the instance. Other specific effects of the interpretation that depend on the token type are shown for the example implementation of the proposed method. In the described implementation of the method, the interpretation of the tokens can affect the modified record 202 asdescribed earlier in article in

[0056] , e.g. arrow 303 in Fig. 3, and the actual state as described in

[0057] , e.g. arrow 301 in Fig.3.

[0061] Fig. 3 shows an example where the cursor 205 is at the input token T4 associated with a characteristic of the state of the environment. The cursor 205 of the instance 200 running in either active or passive regime proceeds to the next tokens in the original rule 201 in the example if the token T4 is encountered in the input buffer 111 , which is the case in the example, here the connection line 309 shows the comparison operation. If the instance is running in active regime and the token is not in the input buffer, the instance is blocked and switched to passive regime and a new instance for running in active regime is chosen as described in

[0049] .

[0062] Fig. 4 shows an example of the instance 200 where the cursor 205 is at the output token T5. The instance 200 running in active regime interprets the token T5 in the example by placing it into the output buffer 110 as described in

[0043] . When the instance is running in passive regime, it is blocked until it gets into active regime to interpret the output token or until the same token is interpreted by other instance running in active regime.

[0063] Fig. 5 shows an example instance for introspective rule 108 where the cursor 505I is at the passive internal token IT2 of a original record 5011. The instance 108 is running in passive regime and is not bound yet to other instance. It proceeds to the next token IT3 of the original record if the state of the instance running in active regime 107 satisfies the condition associated with the token IT2. When the condition is satisfied, the instance 108 becomes bound to the instance 107 as described earlier in article

[0047] . When an introspective instance running in active or passive regime interprets is already bound to another instance, the cursor proceeds to the next tokens in the original record if the state of the bound instance satisfies the condition associated to the token under the cursor. If the introspective instance is running in active regime and the token condition under the cursor is not satisfied, the introspective instance is switched to passive regime and a new instance for running in active regime is chosen as described in

[0049] .

[0064] Fig. 6 shows an example state of the instance 600 running in either active or passive regime where the cursor 605 is at the active internal token T2. In the example implementation of the proposed method, the instance 600 performs the action 608 immediately and the corresponding state in flags 604 of the instance 600 changes immediately. In the example implementation of the proposed method, the interpretations of active internal tokens do not affect the actual state of the instances described in article

[0057] .

[0065] Fig. 7 shows an example state of an introspective instance 108A running in active regime where the cursor 505I is at the token IT3 associated with an action 700 that changes the state in the flags 504A of the bound instance 107P. In the example, the instance 108A performs the action 700immediately and the internal state of bound instance in flags 504A changes immediately as the instance is running in active regime. In other cases, when an introspective instance is running in passive regime, it is blocked by the token IT3 until it gets into active regime to interpret the token or until the action identified by the token IT3 is performed in respect to the bound instance 107P by another instance running in active regime.

[0066] In the following articles, a number of example internal alphabet tokens are presented. For the purpose of convenience, individual alphabet tokens introduced below in articles

[0067] -

[0074] ,

[0076] are referred to by names in capital letters.

[0067] An example of the internal alphabet token is abstract token referred to as INPUT. When interpreting the token, the instance running in either active or passive regime gets blocked until any input token described in

[0037] appears in the input buffer. The modified rule as described in article

[0056] , in the case when no mutation is applied, gets complemented by INPUT while the actual state as described in

[0057] gets complemented by the token from the input buffer that unblocked the interpretation of the record caused by INPUT, which is referred to as substitute token.

[0068] An example of the internal alphabet token is abstract token referred to as OUTPUT. When the cursor of an instance running in active regime comes over OUTPUT, the token is interpreted by placing a substitute output token into the output buffer. The choice of the substitute token is described previously in article

[0050] . When the instance is running in passive regime, the instance waits until it gets into active regime to interpret the token or until any output token is interpreted by other instance running in active regime, which becomes the substitute token for OUTPUT. The modified rule as described in article

[0056] , in case when no mutation is applied, gets complemented by OUTPUT while the actual state as described in

[0057] gets complemented by the substitute token.

[0069] An example of the internal alphabet token is abstract token referred to as NEST. When the cursor of an instance comes over NEST, the token is interpreted by calling a substitute rule in a new instance. The choice of the substitute rule is described earlier in article

[0050] . The regimes for running the instances for the calling and called rules are chosen in accordance to the general guidelines described previously in article

[0045] . The calling and called rules use internal memory as means of interoperation as described earlier in article

[0055] . In case when no mutation is applied, the modified rule of the calling instance described in

[0056] , gets complemented by NEST while the actual state as described in

[0057] gets complemented by the token identifying the modified record from the instance corresponding to the called rule.

[0070] An example of the internal alphabet token is active token referred to as GEN. In the example embodiment of the system described in this specification, the flags of a running instance mentioned earlier in

[0053] comprise a generalization flag. GEN sets the generalization flag in the boundinstance, if any, or the generalization flag in that very instance with the record containing GEN. Then, if the generalization flag of an instance is set, the interpretation of any input token that follows in the original record is replaced, in case when no mutation is applied, by the interpretation of INPUT, the modified rule as described in

[0056] gets complemented by INPUT, the actual state as described in

[0057] gets complemented by the substitute token as described earlier in

[0067] , and the generalization flag resets. The interpretation of an output token, if the generalization flag of an instance is set, in case when no mutation is applied, is replaced by interpretation of OUTPUT, the mutated rule as described in

[0056] gets complemented by OUTPUT, the actual state as described in

[0057] gets complemented by the substitute token, and the generalization flag resets. The interpretation of a token identifying a rule, if the generalization flag of an instance is set, in case when no mutation is applied, is replaced by interpretation of NEST, the mutated rule as described in

[0056] gets complemented by NEST, the actual state as described in

[0057] gets complemented by the modified record from the instance for the called substitute for NEST, and the generalization flag resets. The generalization flag can be set not only due to the interpretation of GEN but also due to mutations performed by the system.

[0071] An example of the internal alphabet token is active token referred to as NOGEN. In the example embodiment of the system described in this specification, NOGEN resets the generalization flag of the bound instance, if any, if the flag is set or prohibit future setting the generalization flag if not set. When the instance with the record containing NOGEN is not bound to other instance, it performs the same in the instance with the record containing NOGEN. The default procedure identified by NOGEN can be called not only due to the interpretation of NOGEN but also due to mutations performed by the system or the interpretation of other tokens as described below.

[0072] An example of the internal alphabet token is active token referred to as LOOP. In the example embodiment of the system described in this specification, the flags of a running instance mentioned earlier in article

[0053] comprise a loop flag. LOOP sets the loop flag in the instance that interprets LOOP. If the loop flag of an instance is set, a new instance is initialized and called to interpret the next token or the substitute in loop. The regimes for running the instances for the calling and called rules are chosen in accordance to the general guidelines described previously in article

[0045] . In the example implementation of the described method, the normal termination condition for the looped call is that the cursor is at an input token or INPUT in the instance for the called rule or any other instance called from the instance running in loop and there are no such token in the input buffer. The calling and called rules use internal memory as means of interoperation as described earlier in article

[0055] . It is generally useful to prohibit mutations in looped instances to make sure that the rule on each iteration of the loop is the same, for example, by setting a specific flag for the looped instance.

[0073] An example of the internal alphabet token is active token referred to as POSTPONE. In the example embodiment of the system described in this specification, the flags of a running instancementioned earlier in article

[0053] comprise a counter set to zero at initialization of the instance. POSTPONE increments the counter in the bound instance, if any, or in that very instance the record of which contains POSTPONE otherwise. If the counter of an instance in active regime is greater than zero, the output token under the cursor or the substitute for OUTPUT chosen as described in

[0068] , in case there is no mutation, is placed into the internal memory of the instance as described earlier in article

[0055] decrementing the counter and complementing the modified rule and the actual state by POSTPONE and the memorized token instead of its immediate interpretation. The effect of POSPONE can be combined with the effect of GEN described earlier in article

[0070] if the instance is running in active regime, the counter is greater than zero, the generalization flag is set, and the cursor is at an output token.

[0074] An example of the internal alphabet token is active token referred to as NOPOSTPONE. In the example embodiment of the system described in this specification, NOPOSTPONE decrements the counter described in

[0073] if it is greater than zero or prohibits the future increments of the counter if it equals to zero in the bound instance, if any, or in that very instance the record of which contains NOPOSTPONE otherwise.

[0075] The details of the arrangement of interpretation of tokens as described in articles

[0067] -

[0074] are proposed for example purposes. Other configurations are possible; for example, there can be tokens identifying a necessity to call a number of default procedures, e.g. affect both the internal state of the system and the state of the environment. The effects the default procedures identified by tokens produce on the environment and the internal states of the system and the blocking ability of the tokens can be affected by the internal states, e.g. by the states of the flags in the instances they are interpreted in. These minor variations to the method can improve its performance in the beginning of the interaction of the system with the environment as they can be chosen to match the assumptions about the specifics of application, but the role of these variations for the computational complexity, the amount of data needed to train the model with regard of the problem domain as a whole including the communication problems may be hard to predict. When an embodiment of the system implements the means for operations described in

[0039] -

[0065] , all variations proves losing in performance on some problems if they gain on other problems, so that all variations are equal in terms of the no-free-lunch theorems even over a diverse enough problem domain. What can improve the performance of the described method further is the stage-by-stage training technique an example of which is presented below.

[0076] Consider an example where a system is able to perform some actions and gets characteristics of states from environment and rewards for its activity, all of which is expressed as synchronized token exchange between the system and the environment. The event of reward is identified by the input token REWARD. In the example environment, there is an infinite board where a piece is placed. The system can move the piece on the board. To move the piece, the system has to take it, shift the piecein vertical and lateral directions and put it into the new position. There are board rules that define how the piece can move laterally and vertically, which are changed regularly and the system is informed about the change. For example, the actions of the system that shifts the piece in lateral and vertical directions can be expressed by a pair of tokens from the range that corresponds to keywords aO, a1, a2, a3, a4, a5. When the board rules change, the environment informs the system about that by sending a pair of symbols transformed into a pair of tokens from the range that corresponds to symbols 0, 1_, 2, 3, 4, 5. To take the piece and put it back the system uses take and put keywords correspondingly. There is also an execution unit in the environment the system can program. The programming language of the execution unit contains only two keywords, these are begin to begin a program and end to end, and any symbols that the system might need to move the piece can be used in the program body between the begin and end. There is launch keyword the system can use to execute the program it wrote earlier on the execution unit, if any. The keywords take, put, begin, end and launch also correspond to atomic tokens from the alphabet of the system.

[0077] The total reward for writing and launching a useful program is split into smaller rewards for simpler action patterns of the system. The smaller rewards are paid for a keyword corresponding to a symbol received from the environment, e.g. for a1 in the output in response to 1_ in the input, for a pair of keywords in the output corresponding to a pair of symbols received from the environment, e.g. aO a4 in response to 04 in the input, for moving the piece on the board by taking it, shifting in accordance with the board rules, and putting it into a new position, e.g. for take a 2 a3 put in the output in response to 23 in the input, for writing any syntactically correct program, that is, any sequence of symbols within the brackets begin...end, e.g. for begin a1 take end, for writing the shortest program having an effect of moving the piece according to the current board rules, e.g. for begin take a1 a5 put end in response to 1 5, and for launching a previously written program, if any, e.g. begin a1 end launch.

[0078] Consider an exemplary problem of training the system to move the piece on the board, to use information from the environment on the changing board rules to move the piece accordingly, to write useful programs for the execution unit, and to launch the program it has written before. The system has to rewrite the program each time the board rules change. To train the system in the example embodiment and configuration described in articles

[0036] -

[0076] , a training program that trains the system to perform more and more complex operations from the very simple operations can be used. The training program is divided into stages, and the training moves to the next stage when it’s statistically ascertained that the system has solved the training problem of the current stage. The role of the training program is to contribute to shortening the training time for the system and the amount of data required by the system for training.

[0079] The exemplary training program is divided into four stages. In the first stage, the system learns the relation between the symbols used to inform it about the board rules and the actions that will comply with the board rules. The environment sends a symbol from the range 0, 1_, 2, 3, 4, 5 and thetraining problem for the stage is to make the system to reply by a corresponding keyword from the range aO, a1_. a2. a3. a4. a5. In the second stage, the system is trained to reply with a correct pair of actions corresponding to a pair of symbols using the relation learned on the first stage, e.g. to produce a1 a4 in the output for 1 4 in the input. In the third stage the system is trained to move the piece according to the board rules, e.g. to produce take a2 a3 put in the output in response to 2 3 in the input. In the fourth stage the system is trained to write and launch a program that moves the piece, e.g. to produce begin take a2 aO put end launch in the output in response to 2 0 in the input.

[0080] The examples of records formed in the model by the example implementation of the method of described in this specification that solve the training problems on the stages described in article

[0079] areStage 1 : REWARD INPUT OUTPUTStage 2: REC1 : INPUT POSTPONE OUTPUTREWARD LOOP REC1Stage 3: REWARD POSTPONE TAKE NEST POSTPONE PUTREWARD POSTPONE BEGIN NEST POSTPONE END POSTPONE LAUNCHStage 4: REWARD POSTPONE TAKE NEST POSTPONE PUT where REC1 is a designation token for the rule INPUT POSTPONE OUTPUT, TAKE is the token associated to the keyword take, PUT is the token associated to the keyword put, BEGIN is the token associated to the keyword begin, END is the token associated to the keyword end, LAUNCH is the token associated to the keyword launch. The rule that solves the problem for the fourth stage is also the solution for the training problem described in 61 . The rules are example abstract structures or meta rules that allow the system to produce rewarding output for the input data on the fly as they do not contain tokens associated to 0, 1_, 2, 3, 4, 5, a0, a1Ja2, a3, a4, a5. Though solutions that contain such tokens can also be formed during the training, their use is mostly temporal. To make the correct substitutions for OUTPUT and NEST and produce the rewarding output, the corresponding instances has to appeal to the statistical data which is formed in the model along with the rules. Due to the use of the meta rules, statistics required to make the correct substitutions for OUTPUT is formed on the first stage where the size of the solution search space is small and is used afterwards on all other stages. Similarly, statistics required to make the correct substitutions for NEST is formed on the stages where the size of the solution search space is relatively small and is used afterwards on the stages where the size of the solution search space grows larger.

Claims

Patent claims1 . System of combinatorial optimization over generic search space comprising both solutions and rules for searching the solutions, wherein the system transforms sequences of identifiers referred to as tokens in the input into sequences of tokens in the output assess the transformations against a given quality criterion, and wherein the search space is defined as a variety of all possible records, which are linear lists of tokens and implemented as plurality of some of the records the system has constructed during its operation, wherein some of the tokens are of one or more of the following types:• identifier for a constant value, which is referred to as data token,• identifier for a record, which in the case when the record is constructed by the system during its operation is referred to as extension token,• identifier of a data fragment that can appear in the system input, which is referred to as input token,• identifier of a data fragment that can appear in the system output, which is referred to as output token,• identifier for a default procedure, which is referred to as internal token, where the said default procedure is an algorithm implemented in the system and not expressed as record, whereas the said default procedure checks and / or changes the values of some of the discrete parameters of the system referred to as internal state characteristics and is either returning result, in which case the result is represented as token, or is not returning result, wherein the system comprises means for interpretation of tokens and records in active or passive regime and means for abnormal termination of the interpretations, wherein the interpretation of a record comprises sequential interpretation of each token in the record, wherein the interpretation of each token in the record can be either blocking the interpretation of the record by transferring it into a standby mode or not blocking, where in an active regime• the interpretation of an extension token comprises interpretation of the record it identifies in the active regime and is blocking until the interpretation of the record terminates normally or releases the block,• the interpretation of an input token comprises comparing the token to the current reading in the system input, possibly iteratively, and is blocking until the comparison result is true,• the interpretation of an output token comprises placing the token into the system output and is not blocking,• the interpretation of an internal token identifying a default procedure comprises calling the default procedure and is either not blocking or blocking and returning result in case it is blocking, and in a passive regime• the interpretation of an extension token identifying a record comprises interpretation of the record in the passive regime and is blocking until the interpretation of the record is complete,• the interpretation of an input token comprises comparing the token to the current readings in the system input, possibly iteratively, and is blocking until the comparison result is true,• the interpretation of an output token or an internal token that identifies a not blocking default procedure is blocking until the token is interpreted as part of interpretation of another record in the system in active regime, after which the token is assumed as interpreted and no longer blocking,• the interpretation of an internal token that identifies a blocking default procedure comprises calling the default procedure it identifies, and is blocking until the event when the default procedure returns its result, wherein the effects of the said default procedures comprises any combination of the following:• changing the parameters of the interpretation of the tokens that follow the token identifying the default procedure in the interpretation of the same record,• binding the interpretation of the record containing the token to the interpretation of another token that is performed in active regime, whereas after the interpretation of a token referred to as introspective rule is bound to the interpretation of another token, the further interpretation of internal tokens of the introspective rule checks or changes the parameters of the interpretation of the token the introspective rule is bound to, and wherein some of the the default procedures perform choosing tokens for interpretation, which is referred to as decision making, on the basis of the assessments of previous decision making in the context of the parameters of the interpretation of tokens referred to as conditions against the quality criterion.

2. The system according to claim 1 , wherein the interpretation of some extension tokens implemented in the system forms a result as a record, which is constructed recursively from the results of interpretation of the tokens of the record the extension token identifies, in which case the interpretation of the extension token is referred to as combinatorial rule, wherein the system has means to save the result, assign an extension token to identify the result, perform the interpretation of the result, and means for abnormal termination of the interpretation of the result.

3. The system according to any of the preceding claims, wherein the plurality of tokens set in the system by default includes tokens that are identifiers of records.

4. The system according to any of the preceding claims, wherein the depth of the hierarchy of records comprising tokens that identify other records in a recursive manner is limited by a fixed number of levels.

5. The system according to any of the preceding claims with means to add input and output tokens into the pluralities of input and output tokens after the start of the system operation.

6. The system according to any of the preceding claims implemented as a computer program, wherein some of its configuration parameters are checked and / or changed by default procedures identified by internal tokens.

7. The system according to any of the preceding claims, wherein the plurality of tokens in the system includes tokens associated to numbers and Boolean values as well as arithmetic and logical operations that produce results identified by tokens when applied to other tokens in a record, as well as:• operations of abstraction that transfers the interpretation of a record into a state where the interpretation of a token in a record is replaced by interpretation of a token that is identifier of a default procedure that performs decision making,• operations of memorizing that transfers the interpretation of a record into a state where the tokens of the record are not interpreted immediately but and treated as data for interpretation of other tokens or interpreted later,• operators of abnormal termination of the interpretation of a record,• conditional operators that transfer the interpretation of a record into a state where the interpretation of some of the tokens after the conditional operator is skipped or performed depending on the result of the interpretation of other token of the record,• looping operators that transfer the interpretation of a record into a state where some tokens the record are interpreted repeatedly until satisfying a condition defined by other tokens in the record.

8. The system according to any of the preceding claims, wherein the parameters of the interpretation of a record are encapsulated in a data object associated with the interpretation of the record and can be checked or changed only by the interpretation of internal tokens of the same record or by introspective rules bound to the interpretation of the record, in which case such data object is referred to as instance.

9. The system according to any of the preceding claims, wherein the plurality of input tokens allows formulating queries about the reasons and ways for choosing the decision rules, and where the plurality of output tokens allow formulating answers to such queries, in one or more languages from the programming languages, formal logical languages, markup languages, natural languages.

10. The system according to any of the preceding claims, wherein the system comprises means for representing data referred to as model including, but not limited to, one or more of the following elements:- data on the plurality of tokens, the types of the tokens and the records, or default procedures, or input or output data they are identifiers of;- data on the extension tokens saved in the system during the interaction with the environment, the records they identify,- data on the extension tokens saved in the system during the interaction with the environment, the records they identify, and recursively the same data for the extensions tokens used in these records possibly in a hierarchical form;- data on the assessments used for the decision making in the context of the conditions they were made for.

11. Method of combinatorial optimization over generic search space comprising both solutions and rules for searching the solutions implemented in the system according to claim 1 and performing interpretation of at least two tokens simultaneously, wherein the said method comprises performing the following steps repeated in a loop:- updating assessments against the quality criterion for the decisions on choosing tokens the interpretation of which terminated on previous iterations,- constructing new records by changing the records for which the interpretation is performed or combining records from tokens,- extending the plurality of tokens that can be used for constructing records by adding extension tokens and assigning them as identifiers to the constructed records,- updating the plurality of simultaneously interpreted tokens referred to as pool in order to match the pool size that determines the maximum quantity of tokens that can be interpreted simultaneously by steadily increasing the pool size, choosing tokens to add to the pool and initiating their interpretation to match the pool size, and choosing tokens from those interpreted in passive regime to terminate them abnormally and remove them from the pool to match the pool size, whereas the decisions on choices are made using the assessments of previous such decisions computed against the quality criterion,- choosing the tokens from the pool to perform interpretation in active regime using the assessments of previous such decisions computed against the quality criterion,- reading the tokens in the input to compare them to the input tokens in the records from the pool, where each reading is followed again by updating the pool and choosing the tokens from the pool for interpretation in active regime, wherein the order in which the steps are performed is arbitrary.

12. The method according to claim 11 , implemented in the system according to claim 2, performing the interpretation of the results produced by the combinatorial rules, if any, saving the results, if not saved on previous iterations, in the form of records and assigning extension tokens to identify the results, and performing assessments of choosing the results for interpretation against the quality criterion.

13. The method according to any of claims 11 or 12, wherein the interpretation of at least one and only one token is performed in active regime and, if the interpretation of the token in active is blocked, it continues in passive regime and another token is chosen for interpretation in active regime using the assessments previous such decisions computed against the quality criterion.

14. The method according to any of claims from 10 to 13, wherein the step of choosing the tokens to add to and remove from the pool and choosing the tokens from the pool to perform interpretation for in active regime are implemented using probabilistic algorithms.

15. The method according to any of claims from 10 to 14, wherein the new rules can be generated by default procedures implemented in the system in one or more ways, including the following:- mutation of records before their interpretation;- mutation of records during their interpretation;- mutation of records after their interpretation;- combination of tokens in a record;- changing the parameters of the interpretation of a rule before the interpretation begins;- changing the parameters of the interpretation of a rule during the interpretation.

16. The method according to any of claims from 10 to 15, wherein the choices of tokens for interpretation in default procedures implemented in the system on the basis of the assessments of previous such choices in the context of system parameters is made by means of neural networks.

17. The method according to any of claims from 10 to 16, wherein the computation of the assessments of choices against the quality criterion uses reward signals on the input of the system.

18. A computer program comprising executable instructions for performing the method according to any claim from 10 to 17.

19. A computer readable medium comprising the computer program according to claim 18.

Images