Machine learning model arrangement identification

Abstract

Described is a system for arrangement identification by accessing reference data that indicates reference initial states of reference data items and indicates reference transformed states of the reference data items, identifying a reference arrangement that exhibits a fixed set of proportional relationships, inferring a transformation rule, accessing candidate data, identifying a candidate arrangement that exhibits the fixed set of proportional relationships, and generating an output that indicates the candidate arrangement of data items in a candidate transformed state determined based on the transformation rule.

Description

MACHINE LEARNING MODEL ARRANGEMENT IDENTIFICATIONTECHNICAL FIELD

[0001] The present disclosure relates generally to machine learning and, more particularly, to identifying a reference arrangement using a machine learning model.BACKGROUND

[0002] Machine learning models are algorithms and techniques performed by computer hardware and are designed to enable computers to learn from data and make predictions or decisions without explicit programming. These models utilize statistical methods, mathematical optimization, and pattern recognition to identify patterns and relationships within data, allowing them to generalize and adapt to new information. Machine learning models are widely used across various domains, such as image and speech recognition, natural language processing, recommendation systems, and anomaly detection. In traditional models, the model is trained using historical and labeled information, and then, the trained model is applied to new unlabeled data to generate inferences.BRIEF DESCRIPTION OF THE DRAWINGS

[0003] To easily identify the discussion of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which that element is first introduced. Some non-limiting examples are illustrated in the figures of the accompanying drawings.

[0004] FIG. 1 illustrates an example method for identifying and transforming reference arrangements, according to some examples.

[0005] FIG. 2 illustrates an architecture for training a machine learning model using reference states and performing inferences on a new candidate initial state, according to some examples.

[0006] FIG. 3 illustrates a transformation causing flipping of a reference arrangement in data items indicating an initial state to a transformed state, according to some examples.

[0007] FIG. 4 illustrates a transformation causing a rotation in the reference arrangement, according to some examples.

[0008] FIG. 5 illustrates a reference arrangement that can be transformed in multiple ways, according to some examples.

[0009] FIG. 6 illustrates a transformation causing scaling of a reference arrangement, according to some examples.

[0010] FIG. 7 illustrates the generation of a template arrangement of data items, according to some examples.

[0011] FIG. 8 illustrates the generation of a mask capturing the reference arrangement, according to some examples.

[0012] FIG. 9 illustrates another example of a mask capturing the background, according to some examples.

[0013] FIG. 10 illustrates an offset position in the virtual space for a reference arrangement, according to some examples.

[0014] FIG. 11 illustrates the machine learning model's determination between multiple possible transformation rules, according to some examples.

[0015] FIG. 12 illustrates a machine-learning pipeline, providing context for examples described herein.

[0016] FIG. 13 illustrates training and use of an example machine-learning program, providing context for examples described herein.

[0017] FIG. 14 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein, according to some examples.DETAILED DESCRIPTION

[0018] Traditional machine learning models have several pitfalls when it comes to solving puzzles or problem sets (e.g., in a virtual space).Traditional machine learning models struggle to generalize from a limited set of training data. These models may not perform well when faced withpuzzle or problem variations or scenarios that were not part of their training data. This limits the models' ability to adapt to new scenarios.

[0019] Moreover, traditional models have difficulties handling spatial relationships and location information in a space (such as a two dimensional or three dimensional space, whether real or virtual) effectively. These models often rely on fixed, handcrafted spatial representations.

[0020] Traditional models can also overfit the training data, resulting in poor performance on unseen data. Furthermore, many traditional machine learning models require large amounts of labeled training data to perform well.

[0021] Furthermore, traditional models do not understand the context of a puzzle or problem scenario and are not be able to link goals with behaviors, such as the goal of moving the grey boxes toward the red box.

[0022] Examples of the Artificial Intelligence (Al) system described herein mitigate or eliminate the pitfalls described above. The Al system inputs reference initial and transformed states to a machine learning model described herein to generate transformation rules. These reference data items in reference states are used to train the machine learning model. The initial state corresponds to data items before a particular transformation and a transformed state corresponds to data items after a transformation. These transformation rules can include behavior information for individual data items, such as movement or characteristic changes, or determining to reach a certain location with a minimum number of steps.

[0023] In some cases, the Al system identifies arrangements of data items. These arrangements exhibit a fixed set of proportional relationships. These proportional relationships in the arrangements are exclusively maintained by a portion of the first reference data items with each other both in the first reference initial state and in the first reference transformed state. For example, the Al system can identify an arrangement with a fixed set of proportional relationships whereby three data items are adjacent to form a line of three data items.

[0024] The Al system identifies transformation rules applied to (e.g., used to generate or followed by) reference data items that transformed the dataitems from an initial state to a transformed state and tests the performance of such transformations. In some cases, the Al system identifies transformation rules applied to (e.g., used to generate or followed by) arrangements, such as a line of data items rotating 90 degrees or scaling from a one by three data item arrangement to a 2 by six data item arrangement. The Al system can then generate transformed states from new candidate data using the identified transformation rules in reference data. The trained machine learning model(s) are used to generate data items in candidate transformed states from data items in candidate initial states.

[0025] The Al system described herein inputs into a machine learning model directly such that the model learns from data items in initial states and transformed states and their corresponding characteristics, eliminating the need for manual feature engineering. This can save time and improve the accuracy of feature representation.

[0026] Moreover, the machine learning model described herein is designed to identify transformations of data items in initial states to transformed states. Each transformation is controlled (such as defined or specified) by a corresponding rule (such as a transformation rule). A transformation rule can specify one or more transformation rule parameters and have corresponding values (such as moving a data item 3 values to the left).

[0027] The machine learning model identifies optimal transformation rule parameters and associated values from reference states. This approach enables better generalization, as the model adapts to new puzzles, problem sets, environments, and associated variations in scenarios that were not explicitly part of the reference data.

[0028] The machine learning model can operate in a virtual space (such as a 2D space) with XY coordinates and data item characteristics, allowing the model to handle spatial relationships and location information more effectively. Moreover, the machine learning model captures temporal dependencies between prior and subsequent states by learning from reference states. This ability to understand how a system or states evolve over time is used by the model to solve puzzles or generate responses to environmental data, which traditional models may not handle as effectively.In some cases, the machine learning model operates in multi-dimensional spaces, such as data of the n-th dimension, where each dimension is representative of some characteristic of the corresponding data item. In some cases, the dimensions include data of other associated data items, such as adjacent or other similar data items.

[0029] The machine learning model, by learning behaviors from reference states, is less prone to overfitting the reference data, resulting in better performance on unseen data. Traditional models can be more susceptible to overfitting to training data, leading to poorer generalization.

[0030] In summary, the machine learning model described herein addresses many of the limitations and pitfalls associated with traditional machine learning systems when it comes to solving puzzles in a virtual space by offering automated feature learning, improved generalization, spatial and temporal understanding, reduced overfitting, and the like, making it a promising solution for solving puzzles and problem sets efficiently and effectively.IDENTIFYING AND TRANSFORMING REFERENCE ARRANGEMENTS

[0031] FIG. 1 illustrates an example method 100 for identifying and transforming reference arrangements, according to some examples. Although the example method 100 depicts a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the method 100. In other examples, different components of an example device or Artificial Intelligence (Al) system that implements the method 100 may perform functions at substantially the same time or in a specific sequence.

[0032] FIG. 1 and other examples described herein are described as being performed by certain Al systems or certain processes, such as a particular machine learning model, but the processes described herein can be performed by one or more machine learning models and the Al system. In some cases, certain portions of the Al system or certain steps are disclosedas being performed by the machine learning model. However, it is appreciated that other portions of the Al system or other steps can be performed by the machine learning model or the Al system.

[0033] An arrangement can include a layout, configuration, formation, shape, or figure, or other collection of data items. Reference data includes training data (e.g., ground truth data) for supervised or unsupervised machine learning model training. The Al system inputs reference data into a machine learning model to identify transformation rules that are later used to generate data items in transformed states. Once the machine learning model is trained, the Al system inputs candidate data items indicating initial states into the machine learning model, and the machine learning model generates candidate data items indicating transformed states based on the transformation rules.

[0034] At block 102, the Al system accesses reference data. The reference data indicates reference initial states of reference data items and reference transformed states of the reference data items. The reference data includes a first reference initial state of first reference data items and a first reference transformed state of the first reference data items at block 102.

[0035] FIG. 2 illustrates an architecture for training a machine learning model using reference states and performing inferences on a new candidate initial state, according to some examples. The Al system receives reference initial states, such as reference initial states 202, 204, and 206. The Al system receives reference transformed states, such as reference transformed states 208, 210, and 212.

[0036] In some cases, the reference states include reference data items, such as as colored boxes (e.g., made of one or more colored pixels) within a virtual area. The virtual area can include data items, such as on a two dimensional grid with X and Y coordinates or three dimensional area with X, Y, and Z coordinates.

[0037] In some cases, the reference data items are shapes (circles, squares, triangles) in a 2D grid, 3D cubes with colors in a 3D grid, nodes in a graph with node features (color, size, edges), image patches or segments, video segments with objects that have features like color, motion, time series datapoints with features like value, slope, game board positions with features like piece types, locations, or the like.

[0038] In some cases, the reference data items include training data for training the machine learning model to then perform inferences on candidate data.

[0039] In some cases, the reference initial states and the reference transformed states include reference data items placed across a virtual space. In some cases, every point of the virtual space includes (e.g., occupied by) at least one data item. In some cases, the data items are placed in certain portions of the virtual space.

[0040] In some cases, the virtual space includes a two dimensional area, where each reference data item includes an X coordinate, a Y coordinate, and a characteristic for the corresponding reference data item. In other cases, the virtual space is a multi-dimensional area, such as a three dimensional area where each reference data item includes an X coordinate, a Y coordinate, Z coordinate, and one or more characteristics for the corresponding reference data item.

[0041] In some cases, reference data items can have an implied characteristic. For example, the Al system can infer that there are two objects, one object being on top of another object. As such, the data item that is on top can be included in the characteristic of the reference data item, whereas the data item for the object underneath the top object is stored in the implied characteristic.

[0042] In some cases, the reference data items are pixels or boxes that make up the virtual area. The reference data items can include nodes in a graph or network with features such as connections, centrality, atoms or molecules with features such as element type, charge, bonds, words in a sequence or document with features like part-of-speech, relationships to other words, or the like.

[0043] In some cases, reference data items include portions of an image, the portions corresponding to features like color, texture, edges, segments in a time-series with features like value, slope, seasonality, characters or tokens in a string with features like case, punctuation, position, vectors in a vectorspace with features like direction, magnitude, distance to other vectors, entries in a matrix with features like value, row, column, or the like.

[0044] In some cases, reference data items include vertices in a 3D model with features like position, connections, color, agents in a simulation with features like health, inventory, relationships, positions on a game board with features like piece type, possible moves, value, people in a social network with features like demographics, connections, interests, organisms with characteristics like genotype, phenotype, or the like.

[0045] In some cases, at least some of the reference data items correspond to a single object, such as a single pixel or a single box. In some cases, at least some of the reference data items correspond to a group of objects, such as a group pixels or a group of boxes. Reference data items could represent clusters or segments identified within the data, for example, pixels clustered together based on color or other features.

[0046] Reference data items can represent composites built from smaller objects, such as based on identifying meaningful shapes or patterns built from multiple pixels or boxes. Reference data items can be identified at higher levels of abstraction, such as characters formed from multiple pixels, words from multiple characters, or sentences from words.

[0047] Reference data items can represent the relationships between objects rather than just the objects themselves. For example, edges in a graph linking nodes or adjacent borders between objects. Reference data items can be defined in a hierarchical or nested manner, such as pixels within larger image patches within whole images. Reference data items can represent overlaps or combinations of other items, such as groups sharing common objects or properties.

[0048] In some cases, the size of the virtual space for the reference initial state and the reference transformed state is the same. For example, the reference initial state and the reference transformed state can both be a 10 by 10 grid of data items. In some cases, the size of the virtual space for the candidate initial state and the candidate transformed state is the same. In some cases, the reference and candidate states are of the same size.

[0049] In other cases, the states are of a different size (such as different sizes for reference initial and transformed states, for candidate initial and transformed states, or for reference states and transformed states). For example, the machine learning model can be trained on reference state data of a 10 by 10 grid of boxes. The machine learning model can then make inferences on candidate data that includes a 12 by 12 grid.

[0050] In some cases, the reference data items include size or dimensions (e.g. width and height of an object) information, orientation or direction (e.g. the angle or facing of an object), texture or pattern (e.g. visual patterns within an object) information, motion vectors (e.g. speed and direction of motion), a bounding box (e.g. enclosing rectangle around an object), type or category (e.g. labeling an object as a specific type) of an object, or the like.

[0051] In some cases, reference data items include relationships to other objects (e.g. distance to other objects), hidden state values (e.g. not directly observable but learned features such as overlap as further described herein), node connections in a graph, sequence position for characters / words, game state information like health, score, or the like.

[0052] At block 104, the Al system identifies a reference arrangement that exhibits a fixed set of proportional relationships exclusively maintained by a portion of the first reference data items with each other among the first reference data items both in the first reference initial state and in the first reference transformed state.

[0053] In the example of FIG. 2, the reference data items indicating the initial states and transformed states are inputted into a machine learning model. The machine learning model assesses the data items indicating the initial states and identifies a reference arrangement, such as a first reference arrangement 248 of the first reference initial state 206.

[0054] The first reference arrangement 248 includes a fixed set of proportional relationships such as a center data item with a first value, and the adjacent horizontal and adjacent vertical data items of the center data item sharing a same second value. The adjacent horizontal and vertical data items are adjacent to the center data item in a particular direction. In some cases, the proportional relationships are in a single direction, only in avertical direction, only in a horizontal direction, a diagonal direction, or the like.

[0055] The machine learning model identifies a similar or the same reference arrangement in the transformed state, such as a second reference arrangement 252 in the first reference transformed state. The machine learning model identifies that the second reference arrangement 252 has the same center value and the same values in vertically and horizontally adjacent data items of the center data item.

[0056] At block 106, the Al system infers a transformation rule that controlled a transformation of the reference arrangement of data items of the multiple reference data items from the first reference initial state to the first reference transformed state.

[0057] For example, in FIG. 2, the Al system assesses the first reference initial state 206 and a first reference transformed state 212. The Al system inputs these states to a machine learning model to generate a transformation rule. The machine learning model is trained to receive as input an initial state and a transformed state, and generate a transformation rule to transform a data item from its initial state to its transformed state. The transformation rule can then be used by the Al system to generate reference data items in transformed states to fulfill a behavior outlined in the transformation rule based on the transformation rule parameter.

[0058] As shown in the first reference initial state, there is a first reference arrangement 248 and a single data item 250. The machine learning model identifies that the reference arrangement 248 does not change in character or location from the initial state to the transformed state. Thus, the first transformation rule can be to identify reference arrangement that exhibit a fixed set of proportional relationships exclusively maintained by a portion of the first reference data items with each other among the first reference data items both in the first reference initial state and in the first reference transformed state.

[0059] The Al system determines that the single data item 250 is transformed to exhibit the fixed set of proportional relationships exclusively maintained by a portion of the first reference data items with each otheramong the first reference data items both in the first reference initial state and in the first reference transformed state as the first reference arrangement.

[0060] As mentioned above, there is a reference arrangement 248 that stays the same from initial state to transformed state and a single data item 250 that transforms into a reference arrangement 254, and the machine learning model infers a transformation rule using this transformation.

[0061] The Al system then verifies whether the transformation rules work with the other reference states. For example, as shown in FIG. 2, the Al system accesses the second reference initial state 202 and the second reference transformed state 208 and generates a generated second reference transformed state based on the transformation rule on data items in the second reference initial state 202.

[0062] The Al system verifies the transformation rule based on a comparison of the generated second reference transformed state with the second reference transformed state.

[0063] The Al system then compares the generated second reference transformed state with the received second reference transformed state. If the two states are the same or meet a certain similarity threshold, then the Al system saves an indication in a database that the first and second transformation rules worked with the second set of reference state data.

[0064] In some cases, the Al system determines that the generated second reference transformed state is not the same with the received second reference transformed state. In such cases, the Al system saves an indication that the transformation did not work with the second set of reference state data.

[0065] In some cases, the Al system saves an indication of how close the generated second reference transformed state is to the received second reference transformed state. For example, the Al system can find a means squared error, a percentage correct or incorrect, a number of correct or incorrect transformations, a measure of how many sub-components were correct, or the like.

[0066] The Al system iteratively repeats this check across the other reference states, such as via the third reference initial state 204 and thesecond reference transformed state 210. Upon confirming that the transformation rule generated using the first reference states work with the other reference states (e.g., using the transformation rule on the reference initial state generates the correct reference transformed state), the system proceeds to block 108 of FIG. 1.

[0067] In some cases, if the transformation rule did not produce the correct result with the second reference states, then the Al system immediately iterates back to the first reference states and identifies another transformation rule. In other cases, the Al system continues to test the other reference states for verification.

[0068] In some cases, the Al system proceeds to block 108 if all (or at least some) of the reference states are confirmed to work using the transformation rule. In other cases, the Al system proceeds to block 108 if the number of reference states that are confirmed using the transformation rule meet or exceed a certain threshold amount or percentage.

[0069] If the transformation rule does not accurately transform some of the other reference initial states into their corresponding reference transformed states, this indicates the initial transformation rule was incomplete or incorrect. In this case, the Al system can return to the first reference state to re-evaluate the initial rule generation process and identify why the rule failed for those states.

[0070] In some cases, the Al system regenerates the transformation rule from scratch using different parameters, such as based on different characteristics of the data items. In some cases, the Al system identifies modifications that can be made to the transformation rule based on the closeness to the correct reference transformation state.

[0071] In some cases, the Al system regenerates a rule which represents a subset or superset of the transformation rule. In some cases, the Al system accesses a rule, such as the rule to turn all blue squares into green triangles. The Al system can access such a rule based on other reference data items in certain reference states. For example, there could be a first set of reference data where data items in an initial state have blue squares and the transformed states have the blue squared turned into green triangles. The Alsystem regenerates the rule to be of a subset of the changes (such as turning all blue squares of at least 9 pixels in size into green triangles) or a superset (such as turning all blue objects of any shapes into green triangles).

[0072] In some cases, the Al system determines that the scope of the rule was too narrow and needs to be expanded, such as by considering another characteristic or a different characteristic of the reference data items. The Al system then performs another iteration of identifying values for the other or different characteristic identified from one set of reference states, and tests with other reference states.

[0073] In some cases, the Al system determines to assess another characteristic based on a determination that the transformation is too complex for the simplicity of the initial rule and more nuance is needed, or vice versa, that incorrect assumptions were made about the nature of the transformation, or the like, which can trigger another set of iterative reference state transformation rule generation and testing.

[0074] Through this iterative process of rule generation, validation, and refinement, the Al system can incrementally improve and expand the database of transformation rules until the Al system reliably identifies one or more transformation rules that can transform all reference states or transforms the highest number of initial states correctly (or the closest to the correct transformation).

[0075] In some cases, there are multiple distinct transformation behaviors and the initial rule only captured one subset of them. In such cases, the Al system checks the other rules with the reference state dataset to confirm which of the rules perform the best to transform the initial reference states to the transformed reference states.

[0076] At block 108, the Al system accesses candidate data that indicates a candidate initial state of candidate data items without indicating any transformed states of the candidate data items. The candidate data items can be inputted to the machine learning model to generate candidate transformed states. The Al system can receive the candidate initial states without indicating any transformed states of the candidate data items.

[0077] At block 110, the Al system identifies a candidate arrangement that exhibits the fixed set of proportional relationships, the candidate arrangement exclusively maintained by a portion of first candidate data items with each other among the first candidate data items in the first candidate initial state. For example, the Al system identifies a first candidate arrangement 256 indicative of the candidate initial state 214, and generates the first candidate arrangement 264 that maintains the fixed set of proportional relationships in the candidate transformed state 216.

[0078] The Al system identifies single data items 258, 262, and 260 and generates candidate arrangements 270, 266, and 268 exhibiting the same fixed set of proportional relationships as the candidate arrangement 256.

[0079] At block 112, the Al system generates an output that indicates the candidate arrangement of data items in a candidate transformed state determined based on the transformation rule. The Al system transforms the candidate data items that are associated with the candidate arrangement and single data items from their candidate initial states to candidate transformed states based on the transformation rules. For example, the Al system can receive as input candidate data items in candidate initial states and transforms these data items to data items in candidate transformed states. The Al system generates an output with data items that indicate the candidate transformed states. In other cases, the Al system generates candidate data items in candidate initial states and transforms these initial states to candidate transformed states.

[0080] Once the transformation rule is generated using the first reference initial state 206 and the first reference transformed state 212, and the transformation rule is tested with the second and third reference initial states 202, 204 and the second and third reference transformed states 208, 210, the Al system receives candidate initial state 214.

[0081] The Al system identifies that certain characteristics of the candidate initial state are similar to the reference initial states, such as the number of data items and arrangements. The Al system generates an output that indicates the candidate transformed states of the candidate data items. TheAl system uses the transformation rule on the candidate initial state 214 to generate the candidate transformed state 216.

[0082] In some cases, the machine learning model is part of or connected to another machine. As such, the output of the machine learning model changes the operation of the other machine. For example, the machine is connected to an external data source that provides candidate and reference state information, whereby the machine inputs data to the Al system and the Al system responds with output from the machine learning model. The responses can affect the operations of the connected machine.

[0083] The machine learning model described herein dynamically generates transformation rules based on one or more characteristics, such as goals, behaviors, or characteristics of data items. The machine learning model can also generate transformation rules based on other characteristics such as certain characteristics of the network, environmental influences, Al system performance requirements and limitations, or changes thereof over time. As such, the machine learning model dynamically modifies and remodifies transformation rules to adapt to the needs and requirements of the machine.

[0084] For example, the external data source is connected to a physical real-world machine (such as a robot) that sends sensory inputs into the machine learning model and adapts its movements and actions based on output of the machine learning model. Machine learning models can be connected to self-driving cars, enabling the machine learning model to interpret sensor data, make real-time decisions, and send responses to selfdriving cars on changing road conditions.

[0085] In some cases, the machine learning models can be connected to surveillance cameras, whereby the machine learning model performs intelligent video analytics for identifying anomalies and detecting potential security threats. In some cases, machine learning models can be connected to a wide range of Internet of Things (loT) devices, enabling smart and adaptive automation in smart homes, cities, and industries.

[0086] In some cases, machine learning models can be connected to recommendation engines, providing personalized suggestions for products, content, or services. Machine learning models can be connected to computervision applications, enabling object detection, facial recognition, medical devices, and image understanding. Machine learning models can be connected to gaming environments, providing adaptive gameplay, opponent Al, and personalized experiences.

[0087] The machine learning models described herein can interact with other systems and machines, and the model adapts to its new or changing environment dynamically by restructuring its internal architecture during runtime. The model can adapt its structure to better suit new data or tasks by adding / pruning nodes and layers.

[0088] In such cases, the Al systems can provide historical data, such as reference state data, or data gathered from other machines that are working property, and the machine learning model can adapt or generate transformation rules to be used by the currently connected machine.

[0089] The Al system can determine reference and candidate arrangements based on one or more characteristics of the data items, such as the number of data items in the reference and candidate initial state dataset, behavior or identified goals of data items, dimensionality of the data for the data items, size of the virtual space, adjacent data items or data items that are of a certain number distance, color, neighboring data item color, contiguous colors, neighboring or adjacent non-black or background data items, changes occurring to nearby data items, a number of clustering, a distance from another data item or arrangement, or the like (such as the examples further described herein).TRANSFORMATIONS OF REFERENCE ARRANGEMENTS

[0090] FIG. 3 illustrates a transformation causing flipping of a reference arrangement in data items indicating an initial state to a transformed state, according to some examples. The reference arrangement 304 exhibits a fixed set of proportional relationships exclusively maintained by a portion of the first reference data items with each other among the first reference data items both in the first reference initial state and in the first reference transformed state. For example, the reference arrangement 304 includes 5 data items, whereby four of the data items are connected diagonally to eachother, with the last data item connected to one data item horizontally and another data item vertically.

[0091] The machine learning model uses the transformation rule to generate the reference arrangement 306 whereby the reference arrangement 304 of the initial state is flipped horizontally to generate the reference arrangement 306 in the transformed state. Because four of the data items are connected diagonally, these four data items do not change during flipping. However, the last data item that is connected horizontally / vertically (as described above) changes locations due to the flipping. Thus, the reference arrangement maintains the fixed set of proportional relationships from initial to transformed state.

[0092] FIG. 4 illustrates a transformation causing a rotation in the reference arrangement, according to some examples. The reference arrangement 402 includes four data items arranged vertically with the bottom three data items adjacent to three other data items also arranged vertically in the initial state. The transformation causes the reference arrangement 404 to be transformed by rotating the reference arrangement 402 by 90 degrees counter-clockwise. As such, the reference arrangement 404 in the transformed state has four data items arranged horizontally, the right three data items adjacent to three other data items also arranged vertically. Also in FIG. 4, the reference arrangement maintains the fixed set of proportional relationships from initial to transformed state.

[0093] FIG. 5 illustrates a reference arrangement that can be transformed in multiple ways, according to some examples. The machine learning model can identify a transformation rule for the reference arrangement 504 in the initial state 502 as flipping or rotating 180 degrees to the reference arrangement 508 in the transformed state 506. The machine learning model can

[0094] In some cases, the transformation rule changes the data items from initial state to transformed state by moving the data items or arrangements to a different location in the virtual space, scaling one or more data items or arrangements, offsetting the data item or arrangement, rotating or flipping a data item or arrangement, or the like. The machine learning model canverify which of the transformation rules is valid by generating data items indicating transformed states from second reference initial states and comparing the generated data items indicating transformed states with received data items indicating transformed states for the second reference data (comparison further described herein).

[0095] FIG. 6 illustrates a transformation causing scaling of a reference arrangement, according to some examples. The reference arrangement 604 in the initial state comprises of a block of two by four data items. The reference arrangement 606 in the transformed state comprises a block of two by one data items. The machine learning model identifies that the transformation to the reference arrangement 606 is scaling the reference arrangement 604 in the initial state to a smaller size. A block of two by two data items in the initial state is transformed to be a single data item in the transformed state.GENERATING A TEMPLATE ARRANGEMENT

[0096] FIG. 7 illustrates the generation of a template arrangement of data items, according to some examples. As shown in FIG. 7, the original reference arrangement 702 includes three large squares made up of nine squares each. In order to be able to find matching reference arrangements in initial states and transformed states, the machine learning model generates a template arrangement.

[0097] In some cases, the machine learning model scales down the reference arrangement in the initial state. For example, as noted above, the reference arrangement 702 includes three large squares made up of nine squares each. The reference arrangement 702 can be scaled down to three data items, such as the reference arrangement 704.

[0098] The reference arrangements found in the transformed state can also be formed into a template arrangement. For example, the reference arrangement 706 can be identified from the transformed state. This reference arrangement 706 can also be scaled down to three data items, where each data item represents a square of four data items.

[0099] In some cases, the machine learning model rotates or flips the reference arrangements such that certain conditions are met. For example,the reference arrangements can be rotated or flipped such that a longest or shortest side of the reference arrangement is set to one direction. In some cases, the second longest or shortest side is set to another direction. In some cases, the third longest or shortest side is set to a particular direction. For example, once the reference arrangements 702 and 706 are scaled down, the machine learning model sets the longest side to be at the bottom and the second longest side to be on the left.

[0100] Even though the scaled down reference arrangement 704 has two sides of two data items long, the machine learning model sets the two data item long sides on the left and the bottom, given the longest side and second longest side rules. As such, the reference arrangement 702 in the initial state (which is quite larger than the reference arrangement 706 in the transformed state) and the reference arrangement 706 in the transformed state can be compared and matched by the machine learning model.GENERATING A MASK CAPTURING THE ARRANGEMENT

[0101] FIG. 8 illustrates the generation of a mask capturing the reference arrangement, according to some examples. The machine learning model identifies a reference arrangement 802 and generates a monochrome mask 804. The monochrome mask 804 includes a background of the virtual space.

[0102] The monochrome mask 804 includes a background on the top right data item. Thus, the reference arrangement 802 is generated on the virtual space by adding the monochrome mask 804 with the arrangement mask 806 that has the reference arrangement data, such as a color for the reference arrangement that is different than the background color in the monochrome mask 804.

[0103] FIG. 9 illustrates another example of a mask capturing the background, according to some examples. As shown in FIG. 9, the reference arrangement 902 includes two vertical data items of one characteristic, such as one color. There are two data items adjacent to the bottom data item of the vertical data items, and the two data items are horizontally adjacent. These two data items are of a different color than the two vertical data item. The monochrome mask 904 is generated as a two by three block with background data items on the top right and top left corners of the block. Themachine learning model combines the monochrome mask 904 with the arrangement mask 906 that includes the two different colors for the two sets of data items.

[0104] FIG. 10 illustrates a mask and an offset position in the virtual space for a reference arrangement, according to some examples. As shown in the virtual space 1002, there is a reference arrangement of three data items. If the bottom left corner is represented as the starting point (0,0) of the virtual space, the reference arrangement 1004 is offset by one data item in the vertical direction. Thus, the machine learning model can generate an offset of (0,1) for the reference arrangement 1004. The offset position can be used by the machine learning model to, for example, identify the entire virtual space (such as a 10 x 10 grid) as a single arrangement, where the entire grid rotates rather than the arrangements within it. In some cases, the mask can also be used by the machine learning model for tasks which involve inverting the color, where the data items corresponding to a particular arrangement becomes the background, and vice versa.

[0105] DETERMINATION BETWEEN MULTIPLE TRANSFORMATION RULES

[0106] FIG. 11 illustrates the machine learning model's determination between multiple possible transformation rules, according to some examples. The machine learning model takes the first reference initial state 1102 and the first reference transformed state 1106 and generates a transformation rule.

[0107] The machine learning model verifies the transformation rule on a second reference initial state 1104 and a second reference transformed state 1108. However, in some cases, multiple transformation rules can be identified from the same reference dataset.

[0108] A first transformation rule can include, in the example of FIG. 11, identifying an intersection between two reference arrangements, the vertical reference arrangement and the horizontal reference arrangement, and to create a new characteristic for the surrounding blocks of the individual box or pixel of the intersection. The machine learning model can use the transformation rule on a candidate initial state 1110 to generate the correctcandidate transformed state 1114 by using the correct candidate transformation.

[0109] However, in some cases, the incorrect transformation rule can be identified. For example, the transformation rule could be to take the reference data item that is of a single row or column and extends to the entire width or height of the virtual space, take an individual box or pixel at around % of its length, and add a new characteristic to all surrounding blocks of that identified box or pixel at the % length mark. In this circumstance, the machine learning model could have determined that the transformation rule works given that the rule was verified using the second reference state input. However, the machine learning model incorrectly uses this on the candidate initial state to generate an incorrect candidate transformed state 1112.

[0110] In some cases, a transformation rule is generated using one reference dataset but failed to verify based on the use of the transformation rule on other reference datasets. In some cases, the transformation rule generated using one reference dataset is verified with other reference datasets, but still fails to generate an incorrect candidate transformed state. [OHl] In such cases, the Al system can iteratively return to the first reference dataset or another reference dataset and modify one or more parameters to obtain the transformation rule, such as by applying transformation rule parameters of a different pixel in the multi-dimensional representation of the transformation rule parameters. The Al system can test another pair of reference datasets to generate a different transformation rule. The Al system can create more or less groups of data items, such as based on characteristics, behaviors, or the like as further described herein.

[0112] In some cases, a non-transitory medium can be used to implement one or more features described herein. In some cases, a non-transitory medium includes a tangible medium.

[0113] Although examples described herein describe one type of model, such as a machine learning model, it is appreciated that such features apply to other types of models, such as an artificial intelligence model or a graphical machine learning model, and vice versa. For example, types of artificial intelligence models includes a graphical machine learning modeland a machine learning model. An artificial intelligence model includes models used to simulate human intelligence in machines, such as understanding natural language, recognizing patterns in data, making decisions, and solving complex problems. A machine learning model is a subset of artificial intelligence models that includes using algorithms and statistical techniques to enable computers to improve performance on a task through learning from data. A graphical machine learning model is a type of machine learning model that uses graphical representations to model dependencies between variables.

[0114] Although examples described herein describe one type of data items, it is appreciated that such features apply to other types of models such as a model of nodes, such as a cluster of nodes or a group of interconnected nodes, and vice versa.

[0115] Although characteristics are described herein to be applied to one feature, such as a color being used to map data items, it is appreciated that the characteristics can be applied to other features (such as using color to group data items or using color to identify behaviors of data items) and that features can be applied to other characteristics (such as identifying movement or distance to map data items).

[0116] Systems and methods described herein include training a machine learning network, such as training to make correct inferences on candidate data. The machine learning network can be trained to identify transformation rule parameters of transformation rules to be applied to data items in reference state data. The machine learning algorithm can be trained using reference initial states and transformed states that include reference data items.

[0117] Training of models, such as artificial intelligence models, is necessarily rooted in computer technology, and improves modeling technology by using training data to train such models and thereafter applying the models to new inputs to make inferences on the new inputs. Here, the new inputs can be new candidate state data, including candidate initial state data and candidate transformed state data that include candidate data items. The trained machine learning model can determinetransformation rule parameters of transformation rules to be applied to reference data items based on reference state data, and transform candidate data items using such transformation rules.

[0118] Such training involves complex processing that typically requires a lot of processor computing and extended periods of time with large training data sets, which are typically performed by massive server systems. Training of models can require logistic regression and / or forward / backward propagating of training data that can include input data and expected output values that are used to adjust parameters of the models. Such training is the framework of machine learning algorithms that enable the models to be applied to new and unseen data (such as new candidate data) and make predictions that the model was trained for based on the weights or scores that were adjusted during training. Such training of the machine learning models described herein reduces false positives and increases the performance by identifying transformation rule parameters of transformation rules to be applied to data items.MACHINE-LEARNING PIPELINE

[0119] FIG. 12 is a flowchart depicting a machine-learning pipeline 1300, and FIG. 13 is a schematic diagram illustrating the training and use of an example machine-learning program. FIG. 12 and FIG. 13 provide context for examples described herein by describing various techniques and approaches to structuring, training, validating, and or deploying a machine learning model. One or more features of the machine learning models described in FIG. 12 and FIG. 13 are example features can be applied to the examples described herein, where applicable. In some cases, the features described herein can be applied to one or more machine learning models described in FIG. 12 and FIG. 13.

[0120] The machine-learning pipeline 1300 may be used to generate a trained machine learning model, for example the trained machine-learning program 1302 of FIG. 13, to perform operations associated with searches and query responses.Overview

[0121] Broadly, machine learning may involve using computer algorithms to automatically learn patterns and relationships in data, potentially without the need for explicit programming. Machine learning algorithms can be divided into three main categories: supervised learning, unsupervised learning, and reinforcement learning.• Supervised learning involves training a model using labeled data to predict an output for new, unseen inputs. Examples of supervised learning algorithms include linear regression, decision trees, and neural networks.• Unsupervised learning involves training a model on unlabeled data to find hidden patterns and relationships in the data. Examples of unsupervised learning algorithms include clustering, principal component analysis, and generative models like autoencoders.• Reinforcement learning involves training a model to make decisions in a dynamic environment by receiving feedback in the form of rewards or penalties. Examples of reinforcement learning algorithms include Q- learning and policy gradient methods.

[0122] Examples of specific machine learning algorithms that may be deployed, according to some examples, include logistic regression, which is a type of supervised learning algorithm used for binary classification tasks. Logistic regression models the probability of a binary response variable based on one or more predictor variables. Another example type of machine learning algorithm is Naive Bayes, which is another supervised learning algorithm used for classification tasks. Naive Bayes is based on Bayes' theorem and assumes that the predictor variables are independent of each other. Random Forest is another type of supervised learning algorithm used for classification, regression, and other tasks. Random Forest builds a collection of decision trees and combines their outputs to make predictions. Further examples include neural networks, which consist of interconnected layers of nodes (or neurons) that process information and make predictions based on the input data. Matrix factorization is another type of machine learning algorithm used for recommender systems and other tasks. Matrix factorization decomposes a matrix into two or more matrices to uncover hidden patterns or relationships in the data. Support Vector Machines (SVM)are a type of supervised learning algorithm used for classification, regression, and other tasks. SVM finds a hyperplane that separates the different classes in the data. Other types of machine learning algorithms include decision trees, k-nearest neighbors, clustering algorithms, and deep learning algorithms such as convolutional neural networks (CNN), recurrent neural networks (RNN), and transformer models. The choice of algorithm depends on the nature of the data, the complexity of the problem, and the performance requirements of the application.

[0123] The performance of machine learning models is typically evaluated on a separate test set of data that was not used during training to ensure that the model can generalize to new, unseen data.

[0124] Although several specific examples of machine learning algorithms are discussed herein, the principles discussed herein can be applied to other machine learning algorithms as well. Deep learning algorithms such as convolutional neural networks, recurrent neural networks, and transformers, as well as more traditional machine learning algorithms like decision trees, random forests, and gradient boosting may be used in various machine learning applications.

[0125] Machine learning models are trained to perform specific types of inference tasks. Two example types of inference tasks in machine learning are classification tasks and regression tasks. Classification tasks, also referred to as categorization tasks, aim at classifying items into one of several category values (for example, is this object an apple or an orange?). Regression algorithms aim at quantifying some items (for example, by providing a value that is a real number) to perform a regression task.

[0126] Generating a trained machine-learning program 1302 may include multiple phases that form part of the machine-learning pipeline 1300, including for example the following phases illustrated in FIG. 12:• Data collection and preprocessing 1202: This phase may include acquiring and cleaning data to ensure that it is suitable for use in the machine learning model. This phase may also include removing duplicates, handling missing values, and converting data into a suitable format.• Feature engineering 1204: This phase may include selecting and transforming the training data 1306 to create features that are useful for predicting the target variable. Feature engineering may include (1) receiving features 1308 (e.g., as structured or labeled data in supervised learning) or (2) identifying features 1308 (e.g., unstructured or unlabeled data for unsupervised learning) in training data 1306.• Model selection and training 1206: This phase may include selecting an appropriate machine learning algorithm and training it on the preprocessed data. This phase may further involve splitting the data into training and testing sets, using cross-validation to evaluate the model, and tuning hyperparameters to improve performance.• Model evaluation 1208: This phase may include evaluating the performance of a trained model (e.g., the trained machine-learning program 1302) on a separate testing dataset. This phase can help determine if the model is overfitting or underfitting and determine whether the model is suitable for deployment.• Prediction 1210: This phase involves using a trained model (e.g., trained machine-learning program 1302) to generate predictions on new, unseen data.• Validation, refinement or retraining 1212: This phase may include updating a model based on feedback generated from the prediction phase, such as new data or user feedback.• Deployment 1214: This phase may include integrating the trained model (e.g., the trained machine-learning program 1302) into a more extensive system or application, such as a web service, mobile app, or loT device. This phase can involve setting up APIs, building a user interface, and ensuring that the model is scalable and can handle large volumes of data.

[0127] FIG. 13 illustrates further details of two example phases, namely a training phase 1304 (e.g., part of the model selection and trainings model selection and training 1206) and a prediction phase 1310 (part of prediction 1210). Prior to the training phase 1304, feature engineering 1204 is used to identify features 1308. This may include identifying informative,discriminating, and independent features for effectively operating the trained machine-learning program 1302 in pattern recognition, classification, and regression. In some examples, the training data 1306 includes labeled data, known for pre-identified features 1308 and one or more outcomes. Each of the features 1308 may be a variable or attribute, such as an individual measurable property of a process, article, system, or phenomenon represented by a data set (e.g., the training data 1306). Features 1308 may also be of different types, such as numeric features, strings, and graphs, and may include one or more of content 1312, concepts 1314, attributes 1316, historical data 1318 , or user data 1320, merely for example.

[0128] In training phase 1304, also referred to herein as a training period, the machine-learning pipeline 1300 uses the training data 1306 to find correlations among the features 1308 that affect a predicted outcome or predict! on / inference data 1322. The training data may also be referred to herein as reference data, because it is used as a reference for setting the trained parameter values of the model.

[0129] With the training data 1306 and the identified features 1308, the trained machine-learning program 1302 is trained during the training phase 1304 during machine-learning program training 1324. The machine-learning program training 1324 appraises values of the features 1308 as they correlate to the training data 1306. The result of the training is the trained machinelearning program 1302 (e.g., a trained or learned model).

[0130] Further, the training phase 1304 may involve machine learning, in which the training data 1306 is structured (e.g., labeled during preprocessing operations). The trained machine-learning program 1302 implements a neural network 1326 capable of performing, for example, classification and clustering operations. In other examples, the training phase 1304 may involve deep learning, in which the training data 1306 is unstructured, and the trained machine-learning program 1302 implements a deep neural network 1326 that can perform both feature extraction and classification / clustering operations.

[0131] In some examples, a neural network may be generated during the training phase 1304, and implemented within the trained machine-learningprogram 1302. The neural network 1326 includes a hierarchical (e.g., layered) organization of neurons, with each layer consisting of multiple neurons or nodes. Neurons in the input layer receive the input data, while neurons in the output layer produce the final output of the network. Between the input and output layers, there may be one or more hidden layers, each consisting of multiple neurons.

[0132] Each neuron in the neural network 1326 operationally computes a function, such as an activation function, which takes as input the weighted sum of the outputs of the neurons in the previous layer, as well as a bias term. The output of this function is then passed as input to the neurons in the next layer. If the output of the activation function exceeds a certain threshold, an output is communicated from that neuron (e.g., transmitting neuron) to a connected neuron (e.g., receiving neuron) in successive layers. The connections between neurons have associated weights, which define the influence of the input from a transmitting neuron to a receiving neuron. During the training phase, these weights are adjusted by the learning algorithm to optimize the performance of the network. Different types of neural networks may use different activation functions and learning algorithms, affecting their performance on different tasks. The layered organization of neurons and the use of activation functions and weights enable neural networks to model complex relationships between inputs and outputs, and to generalize to new inputs that were not seen during training.

[0133] In some examples, the neural network 1326 may also be one of several different types of neural networks, such as a single-layer feedforward network, a Multilayer Perceptron (MLP), an Artificial Neural Network (ANN), a Recurrent Neural Network (RNN), a Long Short-Term Memory Network (LSTM), a Bidirectional Neural Network, a symmetrically connected neural network, a Deep Belief Network (DBN), a Convolutional Neural Network (CNN), a Generative Adversarial Network (GAN), an Autoencoder Neural Network (AE), a Restricted Boltzmann Machine (RBM), a Hopfield Network, a Self-Organizing Map (SOM), a Radial Basis Function Network (RBFN), a Spiking Neural Network (SNN), a Liquid State Machine (LSM), an Echo State Network (ESN), a Neural Turing Machine (NTM), or a Transformer Network, merely for example.

[0134] In addition to the training phase 1304, a validation phase may be performed on a separate dataset known as the validation dataset. The validation dataset is used to tune the hyperparameters of a model, such as the learning rate and the regularization parameter. The hyperparameters are adjusted to improve the model's performance on the validation dataset.

[0135] Once a model is fully trained and validated, in a testing phase, the model may be tested on a new dataset. The testing dataset is used to evaluate the model's performance and ensure that the model has not overfitted the training data.

[0136] In prediction phase 1310, also referred to herein as a runtime period, the trained machine-learning program 1302 uses the features 1308 for analyzing query data 1328 to generate inferences, outcomes, or predictions, as examples of a predict! on / inference data 1322. For example, during prediction phase 1310, the trained machine-learning program 1302 generates an output. Query data 1328 is provided as an input to the trained machinelearning program 1302, and the trained machine-learning program 1302 generates the predict! on / inference data 1322 as output, responsive to receipt of the query data 1328. Query data processed by the model during the runtime phase may also be referred to herein as candidate data, because it is data that is a priori not associated with any particular inference output.

[0137] In some examples, the trained machine-learning program 1302 may be a generative Al model. Generative Al is a term that may refer to any type of artificial intelligence that can create new content from training data 1306. For example, generative Al can produce text, images, video, audio, code, or synthetic data similar to the original data but not identical.

[0138] Some of the techniques that may be used in generative Al are:• Convolutional Neural Networks (CNNs): CNNs may be used for image recognition and computer vision tasks. CNNs may, for example, be designed to extract features from images by using filters or kernels that scan the input image and highlight important patterns.• Recurrent Neural Networks (RNNs): RNNs may be used for processing sequential data, such as speech, text, and time series data, for example. RNNs employ feedback loops that allow them to capturetemporal dependencies and remember past inputs. The feedback loops of an RNN can be considered a cycle within its network.• Generative adversarial networks (GANs): GNNs may include two neural networks: a generator and a discriminator. The generator network attempts to create realistic content that can “fool” the discriminator network, while the discriminator network attempts to distinguish between real and fake content. The generator and discriminator networks compete with each other and improve over time.• Variational autoencoders (VAEs): VAEs may encode input data into a latent space (e.g., a compressed representation) and then decode it back into output data. The latent space can be manipulated to generate new variations of the output data. VAEs may use self-attention mechanisms to process input data, allowing them to handle long text sequences and capture complex dependencies.• Transformer models: Transformer models may use attention mechanisms to learn the relationships between different parts of input data (such as words or pixels) and generate output data based on these relationships. Transformer models can handle sequential data, such as text or speech, as well as non-sequential data, such as images or code.

[0139] In generative Al examples, the output prediction / inference data 1322 include predictions, translations, summaries or media content.MACHINE ARCHITECTURE

[0140] FIG. 14 is a diagrammatic representation of the machine 1400 within which instructions 1402 (e.g., software, a program, an application, an applet, an app, or other executable code) for causing the machine 1400 to perform any one or more of the methodologies discussed herein may be executed. For example, the instructions 1402 may cause the machine 1400 to execute any one or more of the methods described herein. The instructions 1402 transform the general, non-programmed machine 1400 into a particular machine 1400 programmed to carry out the described and illustrated functions in the manner described. The machine 1400 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1400 may operate in thecapacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1400 may comprise, but not be limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular telephone, a smartphone, a mobile device, a wearable device (e.g., a smartwatch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions 1402, sequentially or otherwise, that specify actions to be taken by the machine 1400. Further, while a single machine 1400 is illustrated, the term “machine" shall also be taken to include a collection of machines that individually or jointly execute the instructions 1402 to perform any one or more of the methodologies discussed herein. In some examples, the machine 1400 may comprise both client and server systems, with certain operations of a particular method or algorithm being performed on the server-side and with certain operations of the particular method or algorithm being performed on the client-side.

[0141] The machine 1400 may include processors 1404, memory 1406, and input / output I / O components 1408, which may be configured to communicate with each other via a bus 1410. In an example, the processors 1404 (e.g., a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) Processor, a Complex Instruction Set Computing (CISC) Processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Radio- Frequency Integrated Circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1412 and a processor 1414 that execute the instructions 1402. The term "processor" is intended to include multi-core processors that may comprise two or more independent processors (sometimes referred to as “cores”) that may execute instructions contemporaneously. Although FIG. 14 shows multiple processors 1404, the machine 1400 may include a single processor with a single-core, a single processor with multiple cores (e.g., a multi-coreprocessor), multiple processors with a single core, multiple processors with multiples cores, or any combination thereof.

[0142] The memory 1406 includes a main memory 1416, a static memory 1418, and a storage unit 1420, both accessible to the processors 1404 via the bus 1410. The main memory 1406, the static memory 1418, and storage unit 1420 store the instructions 1402 embodying any one or more of the methodologies or functions described herein. The instructions 1402 may also reside, completely or partially, within the main memory 1416, within the static memory 1418, within machine-readable medium 1422 within the storage unit 1420, within at least one of the processors 1404 (e.g., within the processor’s cache memory), or any suitable combination thereof, during execution thereof by the machine 1400.

[0143] The I / O components 1408 may include a wide variety of components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so on. The specific I / O components 1408 that are included in a particular machine will depend on the type of machine. For example, portable machines such as mobile phones may include a touch input device or other such input mechanisms, while a headless server machine will likely not include such a touch input device. It will be appreciated that the I / O components 1408 may include many other components that are not shown in FIG. 14. In various examples, the I / O components 1408 may include user output components 1424 and user input components 1426. The user output components 1424 may include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), other signal generators, and so forth. The user input components 1426 may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo- optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or another pointing instrument), tactile input components (e.g., a physical button, a touch screen that provides location and force oftouches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like.

[0144] Communication may be implemented using a wide variety of technologies. The I / O components 1408 further include communication components 1428 operable to couple the machine 1400 to a network 1430 or devices 1432 via respective coupling or connections. For example, the communication components 1428 may include a network interface component or another suitable device to interface with the network 1430. In further examples, the communication components 1428 may include wired communication components, wireless communication components, cellular communication components, Near Field Communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components to provide communication via other modalities. The devices 1432 may be another machine or any of a wide variety of peripheral devices (e.g., a peripheral device coupled via a USB).

[0145] Moreover, the communication components 1428 may detect identifiers or include components operable to detect identifiers. For example, the communication components 1428 may include Radio Frequency Identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., an optical sensor to detect onedimensional bar codes such as Universal Product Code (UPC) bar code, multi-dimensional bar codes such as Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph™, MaxiCode, PDF417, Ultra Code, UCC RSS- 2D bar code, and other optical codes), or acoustic detection components (e.g., microphones to identify tagged audio signals). In addition, a variety of information may be derived via the communication components 1428, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via detecting an NFC beacon signal that may indicate a particular location, and so forth.

[0146] The various memories (e.g., main memory 1416, static memory 1418, and memory of the processors 1404) and storage unit 1420 may store one or more sets of instructions and data structures (e.g., software)embodying or used by any one or more of the methodologies or functions described herein. These instructions (e.g., the instructions 1402), when executed by processors 1404, cause various operations to implement the disclosed examples.

[0147] The instructions 1402 may be transmitted or received over the network 1430, using a transmission medium, via a network interface device (e.g., a network interface component included in the communication components 1428) and using any one of several well-known transfer protocols (e.g., hypertext transfer protocol (HTTP)). Similarly, the instructions 1402 may be transmitted or received using a transmission medium via a coupling (e.g., a peer-to-peer coupling) to the devices 1432.EXAMPLES

[0148] In view of the above-described implementations of subject matter this application discloses the following list of examples, wherein one feature of an example in isolation or more than one feature of an example, taken in combination and, optionally, in combination with one or more features of one or more further examples are further examples also falling within the disclosure of this application.

[0149] Example 1 is a system comprising: at least one processor; and at least one memory component storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: accessing reference data that indicates reference initial states of reference data items and indicates reference transformed states of the reference data items, the reference data including a first reference initial state of first reference data items and a first reference transformed state of the first reference data items; identifying a reference arrangement that exhibits a fixed set of proportional relationships exclusively maintained by a portion of the first reference data items with each other among the first reference data items both in the first reference initial state and in the first reference transformed state; inferring a transformation rule that controlled a transformation of the reference arrangement of data items of the reference data items from the first reference initial state to the first reference transformed state; accessing candidate data that indicates a candidate initialstate of candidate data items without indicating any transformed states of the candidate data items, the candidate data including a first candidate initial state of first candidate data items; identifying a candidate arrangement that exhibits the fixed set of proportional relationships exhibited by the reference arrangement, the candidate arrangement being exclusively maintained by a portion of first candidate data items with each other among the first candidate data items in the first candidate initial state; and generating an output that indicates the candidate arrangement of data items in a candidate transformed state determined based on the transformation rule.

[0150] In Example 2, the subject matter of Example 1 includes, the inferring of the transformation rule is performed by inputting the first reference data items indicating the first reference initial state and the first reference data items indicating the first reference transformed state into a machine learning model, the machine learning model outputting inferred transformation rules based on inputted data items indicating initial and transformed states upon training of the machine learning model.

[0151] In Example 3, the subject matter of Example 2 includes, wherein a size of a virtual space occupied by the reference initial states and the reference transformed states and a size of a first virtual space of the candidate initial state and a second virtual space of the candidate transformed state are the same.

[0152] In Example 4, the subject matter of Examples 2-3 includes, wherein the reference data items indicate the reference initial states and the reference transformed states where reference data items are positioned within a virtual space.

[0153] In Example 5, the subject matter of Example 4 includes, wherein the virtual space includes a two dimensional area, wherein each reference data item includes an X coordinate, a Y coordinate, and a characteristic of the corresponding reference data item, wherein at least some of individual reference data items each correspond to at least one of a single pixel or box of the virtual space, wherein the characteristic includes a color of the corresponding reference data item.

[0154] In Example 6, the subject matter of Examples 1-5 includes, wherein the fixed set of proportional relationships include at least a portion of the data items that are adjacent to each other in a particular direction.

[0155] In Example 7, the subject matter of Examples 1-6 includes, wherein the operations further comprise: identifying a data item or a group of data items that do not exhibit the fixed set of proportional relationships in the first reference initial state; and identifying that the data item or the group of data items exhibit the fixed set of proportional relationships in the first reference transformed state, wherein the inferring of the transformation rule is based on the identification that data item or the group of data items do not exhibit the fixed set of proportional relationships in the first reference initial state and exhibit the fixed set of proportional relationships in the first reference transformed state.

[0156] In Example 8, the subject matter of Examples 1-7 includes, wherein the operations further comprise: generating generated reference data items indicating a generated second reference transformed state from a second reference initial state by transforming a second reference data items based on the transformation rule; and verifying the transformation rule based on a comparison of the generated second reference transformed state with a received second reference transformed state.

[0157] In Example 9, the subject matter of Examples 1-8 includes, wherein the fixed set of proportional relationships include a number of adjacent data items.

[0158] In Example 10, the subject matter of Example 9 includes, wherein the fixed set of proportional relationships further include a number of adjacent data items in one or more particular directions.

[0159] In Example 11, the subject matter of Examples 9-10 includes, wherein the fixed set of proportional relationships include a number of contiguous colors on adjacent data items.

[0160] In Example 12, the subject matter of Examples 9-11 includes, wherein the operations further comprise: identifying another transformation rule that controlled the transformation of the reference arrangement of data items of the reference data items from the first reference initial state to thefirst reference transformed state; and determine the validity of the transformation rule and the other transformation rule based on second reference data items indicating second reference initial states and second reference transformed states.

[0161] In Example 13, the subject matter of Examples 1-12 includes, the operations further comprising: generating a first template reference arrangement from the reference arrangement identified in the initial state; generating a second template reference arrangement from the reference arrangement identified in the transformed state; and matching the reference arrangement identified in the initial state and the reference arrangement identified in the transformed state based on a comparison between the first template reference arrangement and the second template reference arrangement, wherein inferring the transformation rule is in response to matching the reference arrangement.

[0162] In Example 14, the subject matter of Example 13 includes, wherein the generating of the first template reference arrangement is based on scaling a size of the reference arrangement identified in the reference initial state.

[0163] In Example 15, the subject matter of Examples 13-14 includes, wherein the generating of the first template reference arrangement is based on rotating or flipping of the reference arrangement identified in the reference initial state.

[0164] Example 16 is a method comprising: accessing reference data that indicates reference initial states of reference data items and indicates reference transformed states of the reference data items, the reference data including a first reference initial state of first reference data items and a first reference transformed state of the first reference data items; identifying a reference arrangement that exhibits a fixed set of proportional relationships exclusively maintained by a portion of the first reference data items with each other among the first reference data items both in the first reference initial state and in the first reference transformed state; inferring a transformation rule that controlled a transformation of the reference arrangement of data items of the reference data items from the first reference initial state to the first reference transformed state; accessing candidate datathat indicates a candidate initial state of candidate data items without indicating any transformed states of the candidate data items, the candidate data including a first candidate initial state of first candidate data items; identifying a candidate arrangement that exhibits the fixed set of proportional relationships exhibited by the reference arrangement, the candidate arrangement being exclusively maintained by a portion of first candidate data items with each other among the first candidate data items in the first candidate initial state; and generating an output that indicates the candidate arrangement of data items in a candidate transformed state determined based on the transformation rule.

[0165] In Example 17, the subject matter of Example 16 includes, identifying a data item or a group of data items that do not exhibit the fixed set of proportional relationships in the first reference initial state; and identifying that the data item or the group of data items exhibit the fixed set of proportional relationships in the first reference transformed state, wherein the inferring of the transformation rule is based on the identification that data item or the group of data items do not exhibit the fixed set of proportional relationships in the first reference initial state and exhibit the fixed set of proportional relationships in the first reference transformed state.

[0166] In Example 18, the subject matter of Examples 16-17 includes, generating generated reference data items indicating a generated second reference transformed state from a second reference initial state by transforming a second reference data items based on the transformation rule; and verifying the transformation rule based on a comparison of the generated second reference transformed state with a received second reference transformed state.

[0167] In Example 19, the subject matter of Examples 16-18 includes, wherein the fixed set of proportional relationships include a number of adjacent data items.

[0168] Example 20 is a computer-readable storage medium (e.g., a non- transitory computer-readable storage medium) storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising: accessing reference data that indicatesreference initial states of reference data items and indicates reference transformed states of the reference data items, the reference data including a first reference initial state of first reference data items and a first reference transformed state of the first reference data items; identifying a reference arrangement that exhibits a fixed set of proportional relationships exclusively maintained by a portion of the first reference data items with each other among the first reference data items both in the first reference initial state and in the first reference transformed state; inferring a transformation rule that controlled a transformation of the reference arrangement of data items of the reference data items from the first reference initial state to the first reference transformed state; accessing candidate data that indicates a candidate initial state of candidate data items without indicating any transformed states of the candidate data items, the candidate data including a first candidate initial state of first candidate data items; identifying a candidate arrangement that exhibits the fixed set of proportional relationships exhibited by the reference arrangement, the candidate arrangement being exclusively maintained by a portion of first candidate data items with each other among the first candidate data items in the first candidate initial state; and generating an output that indicates the candidate arrangement of data items in a candidate transformed state determined based on the transformation rule.

[0169] Example 21 is at least one machine-readable medium including instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement any of Examples 1- 20.

[0170] Example 22 is an apparatus comprising means to implement any of Examples 1-20.

[0171] Example 23 is a system to implement any of Examples 1-20.

[0172] Example 24 is a method to implement any of Examples 1-20.CONCLUSION

[0173] Unless the context clearly requires otherwise, throughout the description and the claims, the words “comprise,” “comprising,” and the like are to be construed in an inclusive sense, as opposed to an exclusive orexhaustive sense, i.e., in the sense of “including, but not limited to.” As used herein, the terms “connected,” “coupled,” or any variant thereof means any connection or coupling, either direct or indirect, between two or more elements; the coupling or connection between the elements can be physical, logical, or a combination thereof. Additionally, the words “herein,” “above,” “below,” and words of similar import, when used in this application, refer to this application as a whole and not to any particular portions of this application. Where the context permits, words using the singular or plural number may also include the plural or singular number respectively. The word “or” in reference to a list of two or more items, covers all of the following interpretations of the word: any one of the items in the list, all of the items in the list, and any combination of the items in the list. Likewise, the term “and / or” in reference to a list of two or more items, covers all of the following interpretations of the word: any one of the items in the list, all of the items in the list, and any combination of the items in the list.

[0174] Although some examples, e.g., those depicted in the drawings, include a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the functions as described in the examples. In other examples, different components of an example device or system that implements an example method may perform functions at substantially the same time or in a specific sequence.

[0175] The various features, steps, and processes described herein may be used independently of one another, or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of this disclosure. In addition, certain method or process blocks may be omitted in some implementations.

Claims

CLAIMSWhat is claimed is:

1. A system comprising: at least one processor; and at least one memory component storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: accessing reference data that indicates reference initial states of reference data items and indicates reference transformed states of the reference data items, the reference data including a first reference initial state of first reference data items and a first reference transformed state of the first reference data items; identifying a reference arrangement that exhibits a fixed set of proportional relationships exclusively maintained by a portion of the first reference data items with each other among the first reference data items both in the first reference initial state and in the first reference transformed state; inferring a transformation rule that controlled a transformation of the reference arrangement of data items of the reference data items from the first reference initial state to the first reference transformed state; accessing candidate data that indicates a candidate initial state of candidate data items without indicating any transformed states of the candidate data items, the candidate data including a first candidate initial state of first candidate data items; identifying a candidate arrangement that exhibits the fixed set of proportional relationships exhibited by the reference arrangement, the candidate arrangement being exclusively maintained by a portion of first candidate data items with each other among the first candidate data items in the first candidate initial state; and generating an output that indicates the candidate arrangement of data items in a candidate transformed state determined based on the transformation rule.

2. The system of claim 1, the inferring of the transformation rule is performed by inputting the first reference data items indicating the first reference initial state and the first reference data items indicating the first reference transformed state into a machine learning model, the machine learning model outputting inferred transformation rules based on inputted data items indicating initial and transformed states upon training of the machine learning model.

3. The system of claim 2, wherein a size of a virtual space occupied by the reference initial states and the reference transformed states and a size of a first virtual space of the candidate initial state and a second virtual space of the candidate transformed state are the same.

4. The system of claim 2, wherein the reference data items indicate the reference initial states and the reference transformed states where reference data items are positioned within a virtual space.

5. The system of claim 4, wherein the virtual space includes a two dimensional area, wherein each reference data item includes an X coordinate, a Y coordinate, and a characteristic of the corresponding reference data item, wherein at least some of individual reference data items each correspond to at least one of a single pixel or box of the virtual space, wherein the characteristic includes a color of the corresponding reference data item.

6. The system of claim 1, wherein the fixed set of proportional relationships include at least a portion of the data items that are adjacent to each other in a particular direction.

7. The system of claim 1, wherein the operations further comprise: identifying a data item or a group of data items that do not exhibit the fixed set of proportional relationships in the first reference initial state; and identifying that the data item or the group of data items exhibit the fixed set of proportional relationships in the first reference transformed state, wherein the inferring of the transformation rule is based on the identification that data item or the group of data items do not exhibit thefixed set of proportional relationships in the first reference initial state and exhibit the fixed set of proportional relationships in the first reference transformed state.

8. The system of claim 1, wherein the operations further comprise: generating generated reference data items indicating a generated second reference transformed state from a second reference initial state by transforming a second reference data items based on the transformation rule; and verifying the transformation rule based on a comparison of the generated second reference transformed state with a received second reference transformed state.

9. The system of claim 1, wherein the fixed set of proportional relationships include a number of adjacent data items.

10. The system of claim 9, wherein the fixed set of proportional relationships further include a number of adjacent data items in one or more particular directions.

11. The system of claim 9, wherein the fixed set of proportional relationships include a number of contiguous colors on adjacent data items.

12. The system of claim 9, wherein the operations further comprise: identifying another transformation rule that controlled the transformation of the reference arrangement of data items of the reference data items from the first reference initial state to the first reference transformed state; and determine the validity of the transformation rule and the other transformation rule based on second reference data items indicating second reference initial states and second reference transformed states.

13. The system of claim 1, the operations further comprising: generating a first template reference arrangement from the reference arrangement identified in the initial state;generating a second template reference arrangement from the reference arrangement identified in the transformed state; and matching the reference arrangement identified in the initial state and the reference arrangement identified in the transformed state based on a comparison between the first template reference arrangement and the second template reference arrangement, wherein inferring the transformation rule is in response to matching the reference arrangement.

14. The system of claim 13, wherein the generating of the first template reference arrangement is based on scaling a size of the reference arrangement identified in the reference initial state.

15. The system of claim 13, wherein the generating of the first template reference arrangement is based on rotating or flipping of the reference arrangement identified in the reference initial state.

16. A method comprising: accessing reference data that indicates reference initial states of reference data items and indicates reference transformed states of the reference data items, the reference data including a first reference initial state of first reference data items and a first reference transformed state of the first reference data items; identifying a reference arrangement that exhibits a fixed set of proportional relationships exclusively maintained by a portion of the first reference data items with each other among the first reference data items both in the first reference initial state and in the first reference transformed state; inferring a transformation rule that controlled a transformation of the reference arrangement of data items of the reference data items from the first reference initial state to the first reference transformed state; accessing candidate data that indicates a candidate initial state of candidate data items without indicating any transformed states of the candidate data items, the candidate data including a first candidate initial state of first candidate data items;identifying a candidate arrangement that exhibits the fixed set of proportional relationships exhibited by the reference arrangement, the candidate arrangement being exclusively maintained by a portion of first candidate data items with each other among the first candidate data items in the first candidate initial state; and generating an output that indicates the candidate arrangement of data items in a candidate transformed state determined based on the transformation rule.

17. The method of claim 16, further comprising: identifying a data item or a group of data items that do not exhibit the fixed set of proportional relationships in the first reference initial state; and identifying that the data item or the group of data items exhibit the fixed set of proportional relationships in the first reference transformed state, wherein the inferring of the transformation rule is based on the identification that data item or the group of data items do not exhibit the fixed set of proportional relationships in the first reference initial state and exhibit the fixed set of proportional relationships in the first reference transformed state.

18. The method of claim 16, further comprising: generating generated reference data items indicating a generated second reference transformed state from a second reference initial state by transforming a second reference data items based on the transformation rule; and verifying the transformation rule based on a comparison of the generated second reference transformed state with a received second reference transformed state.

19. The method of claim 16, wherein the fixed set of proportional relationships include a number of adjacent data items.

20. A computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:accessing reference data that indicates reference initial states of reference data items and indicates reference transformed states of the reference data items, the reference data including a first reference initial state of first reference data items and a first reference transformed state of the first reference data items; identifying a reference arrangement that exhibits a fixed set of proportional relationships exclusively maintained by a portion of the first reference data items with each other among the first reference data items both in the first reference initial state and in the first reference transformed state; inferring a transformation rule that controlled a transformation of the reference arrangement of data items of the reference data items from the first reference initial state to the first reference transformed state; accessing candidate data that indicates a candidate initial state of candidate data items without indicating any transformed states of the candidate data items, the candidate data including a first candidate initial state of first candidate data items; identifying a candidate arrangement that exhibits the fixed set of proportional relationships exhibited by the reference arrangement, the candidate arrangement being exclusively maintained by a portion of first candidate data items with each other among the first candidate data items in the first candidate initial state; and generating an output that indicates the candidate arrangement of data items in a candidate transformed state determined based on the transformation rule.

Images