Training AI by refining model outputs

Abstract

Improved training of artificial intelligence. Raw output data is obtained by applying an input data set to artificial intelligence (AI). Such raw output data is sometimes difficult to interpret. The principles defined herein provide a systematic approach to refine the output for various AI models. The AI ​​model set representation structure is used for the purpose of refining the AI ​​model output so that it is more useful. For each AI model in multiple and perhaps a large number of AI models, the representation structure represents the refinement of the output data generated by the application of the AI ​​model to the input data. After obtaining the output data from the AI ​​model, appropriate refinement can be applied. The refined data can then be semantically indexed to provide a semantic index. The representation structure can also provide customized information to allow intuitive queries against the semantic index.

Description

Background Art

[0001] Computing systems and associated networks have dramatically changed our world. Computing systems are now capable of participating in various levels of artificial intelligence. Artificial intelligence is a process by which a non-living entity (such as a speech system, speech device, or a combination thereof) receives and interprets data to add structure to at least a portion of the data.

[0002] Artificial intelligence can classify the data it receives. As relatively intuitive examples, "image examples" and "video examples" will often be mentioned, where the data input to the artificial intelligence is an image or a video, respectively. In the image example, the artificial intelligence can obtain the original image data, determine what the object represented in the image is, identify the object, and may determine the properties of these objects. For example, artificial intelligence can determine the position, orientation, shape, size, etc. of the object. Artificial intelligence can also determine the relationship between the object and other objects (such as relative position), and / or organize objects with similar properties. Artificial intelligence can also output confidence about its determination. In the video example, artificial intelligence may also make predictions, such as whether two objects will collide, and may also have confidence in these predictions. Artificial intelligence can also estimate the position of objects.

[0003] Technology has not yet reached a level of general intelligence, where any data can be interpreted in any way. However, the AI ​​models used are tailored to make specific kinds of determinations based on specific kinds of data. Some AI models may be very specific in their function, such as determining whether a weld will break based on X-ray data. Some AI models may be more general, such as identifying objects in an image. Traditionally, there are many tools available for developing new AI models. Currently, there are a large number of conventional AI models available, each tailored to varying specificities, and each with varying qualities. Furthermore, the number of available AI models is growing rapidly.

[0004] The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as described above. Rather, this background is merely provided to illustrate one exemplary technology area in which some embodiments described herein may be exercised. Summary of the invention

[0005] At least some embodiments described herein relate to improved training of artificial intelligence. Raw output data is obtained by applying an input data set to an artificial intelligence (AI) model. Such raw output data is sometimes difficult to interpret. For example, an AI model customized for video recognition may recognize a list of objects, relationships, confidence levels, etc. over time. Some information (e.g., the presence of a pen) may not have any relevance at all. In fact, the raw output may contain a large amount of data that is irrelevant or not so relevant. The principles defined herein provide a systematic approach to refine the output for various AI models.

[0006] Artificial intelligence (AI) model set representation structure is used for the purpose of refining AI model output, so as to be more useful. For each AI model in and perhaps a large number of AI models, the representation structure represents the refinement of the output data generated by applying the input data set to the AI ​​model. Refinement can involve removing, transforming or prioritizing the original output data. For example, refinement can involve filtering out some AI model outputs. Refinement can involve truncating, converting, combining and / or otherwise transforming parts of the AI ​​model output. Refinement can involve prioritizing parts of the output by possibly sorting or ranking the output, marking parts of the AI ​​model output, and so on. Different refinements can be specified for each AI model or model type. For each model / data combination, different refinements can even be specified, including AI models or model types with associated input data sets or input data set types. After obtaining the output data from the AI ​​model, appropriate refinement can be applied. Refinement can, for example, bring about the most relevant content that a typical user will find from a given AI model applied to given data. The refinement actually performed can be enhanced or modified by prompts specific to the AI ​​model and / or by the learned data.

[0007] In some embodiments, the refined data may then be semantically indexed to provide a semantic index that may then be queried by a user or may be used to suggest queries to a user. The representation structure may also include a set of one or more operators and / or terms that may be used by a query engine to query against the semantic index or may be included in suggested queries presented to a user. By providing these operators and / or terms to a query engine, a user may more efficiently use the query engine to extract desired information from the semantic index.

[0008] The representation structure may also include a collection of one or more visualizations that can be used by the visualization engine to visualize responses to users to query against a semantic index. Such a visualization may be a visualization that most effectively and intuitively represents the output of a user's query for a given semantic index. Thus, the representation structure may also provide a mechanism for effectively interacting with a semantic index generated from the refined output of an AI model. The representation structure may be easily extended as new AI models and / or data set types become available.

[0009] This Summary is provided to introduce a selection of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to describe the manner in which the above and other advantages and features of the present invention are obtained, a more particular description of the invention briefly described above will be given by reference to specific embodiments of the invention illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of the scope of the invention, the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0011] Figure 1 An example computer system is shown in which the principles described herein may be employed;

[0012] Figure 2 An environment is shown in which an input data set is applied to an AI model to generate output data;

[0013] Figure 3 An environment is shown, which shows multiple input dataset types and multiple AI model types;

[0014] Figure 4 An artificial intelligence (AI) model ensemble representation structure that can be used to refine raw output data resulting from being applied to an AI model's input data set(s) according to the principles described herein is shown;

[0015] Figure 5 A method for training artificial intelligence for a computing system according to the principles described herein is shown; and

[0016] Figure 6 Shown by executing Figure 5 The processing flow is completed by this method. DETAILED DESCRIPTION

[0017] At least some embodiments described herein relate to improved training of artificial intelligence. Raw output data is obtained by applying an input data set to an artificial intelligence (AI) model. Such raw output data is sometimes difficult to interpret. For example, an AI model customized for video recognition may recognize a list of objects, relationships, confidence levels, etc. over time. Some information (e.g., the presence of a pen) may not have any relevance at all. In fact, the raw output may contain a large amount of data that is irrelevant or not so relevant. The principles defined herein provide a systematic approach to refine the output for various AI models.

[0018] Artificial intelligence (AI) model set representation structure is used for the purpose of refining AI model output, so as to be more useful. For each AI model in and perhaps a large number of AI models, the representation structure represents the refinement of the output data generated by applying the input data set to the AI ​​model. Refinement can involve removing, transforming or prioritizing the original output data. For example, refinement can involve filtering out some AI model outputs. Refinement can involve truncating, converting, combining and / or otherwise transforming parts of the AI ​​model output. Refinement can involve prioritizing parts of the output by possibly sorting or ranking the output, marking parts of the AI ​​model output, and so on. Different refinements can be specified for each AI model or model type. For each model / data combination, different refinements can even be specified, including AI models or model types with associated input data sets or input data set types. After obtaining the output data from the AI ​​model, appropriate refinement can be applied. Refinement can, for example, bring about the most relevant content that a typical user will find from a given AI model applied to given data. The refinement actually performed can be enhanced or modified by prompts specific to the AI ​​model and / or by the learned data.

[0019] In some embodiments, the refined data may then be semantically indexed to provide a semantic index that may then be queried by the user or may be used to suggest queries to the user. The representation structure may also include a set of one or more operators and / or terms that may be used by the query engine to query against the semantic index or may be included in suggested queries presented to the user. By providing these operators and / or terms to the query engine, the user may more efficiently use the query engine to extract desired information from the semantic index.

[0020] The representation structure may also include a collection of one or more visualizations that can be used by the visualization engine to visualize responses to users to query against a semantic index. Such a visualization may be a visualization that most effectively and intuitively represents the output of a user's query for a given semantic index. Thus, the representation structure may also provide a mechanism for effectively interacting with a semantic index generated from the refined output of an AI model. The representation structure may be easily extended as new AI models and / or data set types become available.

[0021] Because the principles described herein operate in the context of a computing system, reference will be made to Figure 1 Describes the computing system. Then, reference is made to Figures 2 to 6 Describe the principles of training an AI.

[0022] Computing systems are now increasingly taking a variety of forms. For example, a computing system may be a handheld device, a home appliance, a laptop computer, a desktop computer, a mainframe, a distributed computing system, a data center, or even a device that is not traditionally considered a computing system, such as a wearable device (e.g., glasses, watches, bracelets, etc.). In this specification and claims, the term "computing system" is broadly defined as any device or system (or combination thereof) including at least one physical and tangible processor and a physical and tangible memory on which computer executable instructions that can be executed by the processor can be located. The memory may take any form and may depend on the nature and form of the computing system. The computing system may be distributed over a network environment and may include multiple constituent computing systems.

[0023] like Figure 1 As shown, in its most basic configuration, computing system 100 generally includes at least one hardware processing unit 102 and memory 104. Memory 104 can be physical system memory, which can be volatile, non-volatile, or some combination of the two. The term "memory" may also be used herein to refer to non-volatile mass storage devices, such as physical storage media. If the computing system is distributed, then processing, memory, and / or storage capabilities may also be distributed.

[0024] The computing system 100 has a number of structures thereon, which are generally referred to as "executable components." For example, the memory 104 of the computing system 100 is shown as including the executable component 106. The term "executable component" is a name for a structure known to those of ordinary skill in the computing arts as a structure that can be software, hardware, or a combination thereof. For example, when implemented in software, one of ordinary skill in the art will understand that the structure of the executable component can include a software object, routine, method that can be executed on the computing system, regardless of whether such executable component is present in the computing system's stack or whether the executable file is present on a computer-readable storage medium.

[0025] In this case, one of ordinary skill in the art will recognize that the structure of the executable component exists on a computer-readable medium so that when executed by one or more processors of the computing system (e.g., by a processor thread), the computing system is caused to perform a function. Such a structure can be directly computer-readable by the processor (which is the case when the executable component is a binary file). Alternatively, the structure can be constructed to be interpretable and / or compiled (whether in a single stage or in multiple stages) to generate such a binary file that is directly interpretable by the processor. When the term "executable component" is used, such an understanding of the example structure of the executable component is fully within the scope of understanding of those of ordinary skill in the computing field.

[0026] The term "executable component" is also known to those of ordinary skill in the art as including structures that are implemented exclusively or almost exclusively in hardware, such as in a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or any other special circuit. Therefore, the term "executable component" is a term for a structure known to those of ordinary skill in the field of computing, whether implemented in software, hardware, or a combination. In this specification, the term "component" or "vertex" may also be used. As used in this specification, and in this case, the term (whether or not the term is modified by one or more modifiers) is also intended to be synonymous with the term "executable component", or a specific type of such an "executable component", and therefore also has a structure known to those of ordinary skill in the field of computing.

[0027] In the following description, embodiments are described with reference to actions performed by one or more computing systems. If such actions are implemented in software, then one or more processors (of the associated computing system performing the actions) direct the operation of the computing system in response to having executed the computer-executable instructions constituting the executable component. For example, such computer-executable instructions may be embodied on one or more computer-readable media forming a computer program product. One example of such operations involves the manipulation of data.

[0028] Computer-executable instructions (and manipulated data) may be stored in memory 104 of computing system 100. Computing system 100 may also contain communication channels 108 that allow computing system 100 to communicate with other computing systems over network 110, for example.

[0029] Although not all computing systems require a user interface, in some embodiments, the computing system 100 includes a user interface 112 for interfacing with a user. The user interface 112 may include an output mechanism 112A and an input mechanism 112B. The principles described herein are not limited to precise output mechanisms 112A or input mechanisms 112B, as this will depend on the nature of the device. However, the output mechanism 112A may include, for example, a speaker, a display, a tactile output, a hologram, virtual reality, etc. Examples of the input mechanism 112B may include, for example, a microphone, a touch screen, a hologram, virtual reality, a camera, a keyboard, a mouse of other indicator inputs, any type of sensor, etc.

[0030] The embodiments described herein may include or utilize a dedicated or general-purpose computing system including computer hardware (such as, for example, one or more processors and system memory), as discussed in more detail below. The embodiments described herein also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media can be any available media that can be accessed by a general or dedicated computing system. The computer-readable medium storing computer-executable instructions is a physical storage medium. The computer-readable medium carrying computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, an embodiment may include at least two significantly different types of computer-readable media: a storage medium and a transmission medium.

[0031] Computer-readable storage media includes RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other physical and tangible storage media that can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general or special purpose computing system.

[0032] A "network" is defined as one or more data links that enable the transmission of electronic data between computing systems and / or components and / or other electronic devices. When information is transmitted or provided to a computing system over a network or another communications connection (hardwired, wireless, or a combination of hardwired or wireless), the computing system will correctly view the connection as a transmission medium. Transmission media may include networks and / or data links that can be used to carry desired program code means in the form of computer-executable instructions or data structures and that can be accessed by general-purpose or special-purpose computing systems. Combinations of the above should also be included within the scope of computer-readable media.

[0033] Furthermore, upon reaching various computing system components, program code means in the form of computer executable instructions or data structures may be automatically transferred from a transmission medium to a storage medium (or vice versa). For example, computer executable instructions or data structures received over a network or data link may be buffered in RAM within a network interface component (e.g., a "NIC") and then ultimately transferred to computing system RAM and / or a less volatile storage medium at the computing system. Thus, it should be understood that readable media may be included in computing system components that also (or even primarily) utilize transmission media.

[0034] Computer executable instructions include, for example, instructions and data that, when executed on a processor, cause a general purpose computing system, a special purpose computing system, or a special purpose processing device to perform a specific function or group of functions. Alternatively or additionally, the computer executable instructions may configure a computing system to perform a specific function or group of functions. Computer executable instructions may be, for example, binary files or even instructions that undergo some conversion (such as compilation) before being directly executed by a processor, such as intermediate format instructions such as assembly language, or even source code.

[0035] Those skilled in the art will recognize that the present invention can be practiced in a network computing environment with many types of computing system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, pagers, routers, switches, data centers, wearable devices (such as glasses or watches), etc. The present invention can also be practiced in a distributed system environment, in which local and remote computing systems linked by a network (by a hardwired data link, a wireless data link, or by a combination of hardwired and wireless data links) all perform tasks. In a distributed system environment, program components can be located in local and remote memory storage devices.

[0036] Those skilled in the art will also appreciate that the present invention can be practiced in a cloud computing environment supported by one or more data centers or portions thereof. The cloud computing environment can be distributed, although this is not required. When distributed, the cloud computing environment can be distributed internationally within an organization, and / or have components owned across multiple organizations.

[0037] In this specification and the appended claims, "cloud computing" is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). The definition of "cloud computing" is not limited to any of the other numerous advantages that can be derived from such a model when properly deployed.

[0038] For example, cloud computing is currently used in the market to provide ubiquitous and convenient on-demand access to a shared pool of configurable computing resources. In addition, the shared pool of configurable computing resources can be quickly provisioned via virtualization and released with less management effort or service provider interaction, and then scaled accordingly.

[0039] The cloud computing model may include various features, such as on-demand, self-service, broad network access, resource pooling, rapid elasticity, measured services, etc. The cloud computing model may also take the form of various application service models, such as software as a service ("SaaS"), platform as a service ("PaaS"), and infrastructure as a service ("IaaS"). The cloud computing model may also be deployed using different deployment models, such as private cloud, community cloud, public cloud, hybrid cloud, etc. In this specification and claims, a "cloud computing environment" is an environment in which cloud computing is employed.

[0040] Figure 2 An environment 200 is shown in which an input data set 201 is applied to an AI model 210 to generate output data 211. An example of an AI model is a machine learning model, where the AI ​​model learns through experience. Another example of an AI model is a rule-based model, where the AI ​​model does not learn itself, but responds to rules and parameters.

[0041] Output data 211 is represented as a cloud to indicate that the AI ​​model 210 may generate a large amount of data that is difficult to sift through to find relevant information. For example, an AI model customized for video recognition may recognize a list of objects, relationships, confidence levels, etc. over time. Some information (e.g., the presence of a pen) may not have any association at all. In fact, the raw output may contain a large amount of data that is irrelevant or not so relevant. The principles defined in this article provide a systematic approach to refine the output for various AI models. This is critical because it will help interpret data output from numerous enumerable and rapidly growing AI models. Therefore, the efficiency of the AI ​​model is improved.

[0042] Figure 3An environment 300 is shown that shows a plurality of input data set types 310 and a plurality of AI model types 320. In the illustrated embodiment, three input data set types are shown, including input data set types 311, 312, and 313. However, ellipsis 314 indicates that there may be any number (and perhaps an enumerable and rapidly growing number) of input data set types available for use in environment 300. Additionally, AI model types 320 are shown to include four AI model types 321, 322, 323, and 324. However, ellipsis 325 indicates that there may be any number (and perhaps an enumerable and rapidly growing number) of AI model types available for use in environment 300.

[0043] In each input data set type 310, one or more input data sets or input data set subtypes may be available. For example, within the first input data set type 311, there may be multiple input data sets 311A, 311B, and 311C. The ellipsis 311D indicates that the first input data set type 311 may have any number of input data sets. Alternatively, any one of the input data sets 311A, 311B, or 311C may be an input data set subtype representing a more specific type in the input data set type 311. Therefore, the input data set type 311 may typically be a root node in a conceptual hierarchy of input data set types, where a leaf node in the hierarchy may contain one or more input data sets. This may apply to any input data set type 320.

[0044] In all figures, an input dataset type is represented by a larger shape, and input datasets (or input dataset subtypes) of that input dataset type are represented by smaller versions of the same shape. For example, input dataset type 311 is represented as a larger upward pointing triangle, and input datasets 311A, 311B, and 311C of that type 311 are represented by smaller upward pointing triangles.

[0045] Within the second input dataset type 312 (represented as a larger square), there are also multiple input datasets 312A and 312B or input dataset subtypes (represented as smaller squares). Within the third input dataset type 313 (represented as a larger circle), there are also multiple input datasets 313A to 313D or input dataset subtypes (represented as smaller circles). The ellipses 311D, 312C, and 313E indicate that for any given input dataset type, there can be any number of input datasets or input dataset subtypes.

[0046] Turning now to AI model types 320, within each AI model type, there may be one or more AI models or AI model subtypes available. For example, within a first AI model type 321, there may be multiple AI models 321A and 321B. Ellipsis 321C indicates that the first AI model type 321 may have any number of AI models. Alternatively, any one of AI models 321A and 321B may be an AI model subtype representing a more specific type within AI model type 321. Thus, AI model type 321 may generally be a root node in a conceptual hierarchy of AI model types, where a leaf node in the hierarchy may contain one or more AI models. This may apply to any AI model type 320.

[0047] Likewise, in all figures, an AI model type is represented by a larger shape, while an AI model (or AI model subtype) of that AI model type is represented by a smaller version of the same shape. For example, AI model type 321 is represented as a larger downward pointing triangle, while AI models 321A and 321B are represented by smaller downward pointing triangles.

[0048] Within the second AI model type 322 (represented as a parallelogram), there are also multiple AI model models 322A and 322B or AI model subtypes (represented as smaller parallelograms). Within the third AI model type 323 (represented as an ellipse), there are also multiple AI models 323A to 323D or AI model subtypes (represented as smaller ovals). Within the fourth AI model type 324 (represented as a diamond), there is AI model 324A or AI model subtype (represented as a smaller diamond). Ellipses 321C, 322C, 323E, and 324B indicate that for any given AI model type, there can be any number of AI models or AI model subtypes.

[0049] The point is that there is an environment 300 in which there are a large number of available input data sets and a large number of AI models. The input data sets may be classified by type or sub-type. Furthermore, the AI ​​models may be classified by type or sub-type. Thus, the hierarchy of input data set types is merely conceptual, as is the hierarchy of AI models. The environment 300 may be, for example, a global environment such as the Internet. However, the environment 300 may also be any environment in which multiple AI models are available for application to an input data set.

[0050] Arrow 330 indicates that for any given input dataset type, the input dataset of the input dataset type can be applied to an AI model of an AI model type. In this example, the input dataset of input dataset type 311 can be applied to an AI model of AI model type 321 (as shown by arrow 331). Alternatively or additionally, the input dataset of input dataset type 312 can be applied to an AI model of AI model type 321 (as shown by arrow 332). The input dataset of input dataset type 312 can also be applied to an AI model of AI model type 322 (as shown by arrow 333), as well as to an AI model of AI model type 323 (as shown by arrow 334). The input dataset of input dataset type 313 can be applied to an AI model of AI model type 323 (as shown by arrow 335), as well as to an AI model of AI model type 324 (as shown by arrow 336).

[0051] More generally, arrow 330 represents an operational combination of an input dataset type and an AI model type. At a finer granularity, there may also be an operational combination of an input dataset and an AI model type, where for each AI model type, an operational combination is available for each input dataset type. Similarly, there may also be an operational combination of an input dataset type and a specific AI model. At the finest combination granularity, there may be an operational combination of an input dataset and an AI model. In general, in environment 300, there are operational combinations of (on the one hand) input datasets, input dataset subtypes, and / or input dataset types, and (on the other hand) AI models, AI model subtypes, and / or AI model types. Such operational combinations may also be generally referred to as "data / model combinations" hereinafter.

[0052] Figure 4 An artificial intelligence (AI) model set representation structure 400 (hereinafter also referred to as a "representation structure") that can be used to refine raw output data generated by being applied to (multiple) input data sets of the AI ​​model is shown. For example, also refer to Figure 2 , the representation structure 400 can be used to refine the output data 211 generated by applying the input data set 201 to the AI ​​model 210.

[0053] For each AI model in the plurality of AI models, the representation structure 400 represents a refinement of the results of being applied to the input data set of the AI ​​model. This can be performed by having what will be referred to herein as an "operational AI model representation" (the meaning of this term will be further described below) based on the representation structure. Figure 4The representation structure 400 is shown to include a plurality of operational AI model representations 410, including operational AI model representations 411 through 414. However, the ellipsis 415 indicates that the representation structure 400 may include any number (and possibly an enumerable number) of operational AI model representations.

[0054] Representation structure 400 also includes a refinement definition 420 for each operational AI model representation. Refinement definition 420 thus includes refinement definition 421 associated with operational AI model representation 411, refinement definition 422 associated with operational AI model representation 412, refinement definition 423 associated with operational AI model representation 413, and refinement definition 424 associated with operational AI model representation 414. Ellipsis 425 indicates that a refinement definition may exist for each operational AI model representation 410.

[0055] AI model representations 411 to 414 are provided as examples only, and only as a starting point to describe some breadth of the term "operational AI model representation." For example, operational AI model representation 411 is shown as identifying a single AI model type (in this case, from Figure 3 AI model type 321). This means that the operational AI model representation is not a data / model combination, but merely an AI model expression. In this case, the AI ​​model expression is an identifier of the AI ​​model type without referencing any input dataset type. This means that regardless of the input dataset of the AI ​​model applied to the AI ​​model of AI model type 321, the refined definition 421 is associated with the AI ​​model. The AI ​​model expression may include one or more AI model types, one or more AI model subtypes, and / or one or more AI models. Therefore, the AI ​​model expression can very precisely and compactly define a set of any number of AI models.

[0056] On the other hand, the operational AI model representation 412 is shown as identifying a single input data set type (in this case, from Figure 3 ). This means that the operational AI model representation 412 is also not a data / model combination, but merely an input dataset expression. In this case, the input dataset expression is an identification of the input dataset type without referencing any AI model expression. This means that no matter which AI model the input dataset of the input dataset type 312 is applied to, the refinement definition 422 is associated with the input dataset. The input dataset expression may include one or more input dataset types, one or more input dataset subtypes, and / or one or more input datasets. Therefore, the input dataset expression may very precisely and compactly define a set of any number of input datasets.

[0057] The operational AI model representation 413 is represented by a data / model combination. Specifically, the AI ​​model representation 413 includes a combination of an input data set type 313 and an AI model type 323. Recall that in Figure 3 , this is an operational data / model combination as indicated by arrow 335. The operational AI model representation 413 indicates that refinement 423 is associated with a data / model combination including all combinations of input data sets of input data set type 313 applied to an AI model of AI model type 323.

[0058] The operational AI model representation 414 is also represented by a data / model combination. Specifically, the AI ​​model representation 414 includes a combination of the input data set type 313 and the AI ​​model type 324. Recall that in Figure 3 , this is an operational data / model combination as indicated by arrow 336. The operational AI model representation 414 indicates that refinement 424 is associated with a data / model combination including all combinations of input data sets of input data set type 313 applied to an AI model of AI model type 324.

[0059] More generally, an operational AI model representation may include a data / model combination where (multiple) input data sets are defined by specific input data set expressions and (multiple) AI models are defined by specific AI model expressions. In this case, when an input data set that satisfies the input data set expression is applied to an AI model that satisfies the AI ​​model expression, an associated refinement may be applied. Thus, even with a large number of input data sets and AI models, the data / model combination may be defined very precisely and compactly. Furthermore, the operational AI model representation 410 may be defined compactly and precisely.

[0060] Now refer to Figure 5 Let's continue Figure 4 Description. Figure 5 A method 500 for training artificial intelligence in a computing system is shown. The method 500 may be performed by Figure 1 The computing system 100 is Figure 3 Used in environment 300 Figure 4 The method 500 may be performed by the computing system 100 by means of a computer program product including one or more computer-readable storage media on which executable instructions are stored, which are configured to cause the computing system 100 to perform the method 400 when executed by the processor(s) 102 of the computing system 100.

[0061] Method 500 includes accessing at least a portion of an artificial intelligence (AI) model collection representation structure that represents, for each of a plurality of AI models, a refinement of a result of applying data to the AI ​​model (act 501). For example, a computing system may access Figure 4 At least a portion of the representation structure 400.

[0062] Additionally, method 500 obtains results of being applied to the input data set(s) of the AI ​​model (act 502 ). Figure 6 The process flow 600 completed by the execution of the method 500 is shown. The process flow starts with the results 601 obtained from the AI ​​model. These obtained results 601 are represented as clouds because it is Figure 2 Again, the obtained results 601 may be difficult to interpret to extract the desired information.

[0063] Accessing the representation structure (act 501) and obtaining results from the AI ​​model (act 502) are shown in parallel. This does not mean that these actions must occur in parallel. Instead, this is done to indicate that no time order or relationship is required when performing actions 501 and 502. Accessing the representation structure (act 501) can occur before obtaining results from the AI ​​model (act 502), during obtaining results from the AI ​​model, after obtaining results from the AI ​​model, or any or all of the above. Certain portions of the representation structure can be accessed, cached, and processed as needed to complete method 500.

[0064] Then, method 500 identifies an applicable operational AI model representation to be applied to the output data (act 503). For example, assume that output data 601 (e.g., Figure 2 ). If AI model 210 has AI model type 321, then operational AI model representation 411 is applicable. If input dataset 201 has input dataset type 312, then operational AI model representation 412 is applicable. If input dataset 201 has input dataset type 313, and AI model 210 has AI model type 323, then operational AI model representation 413 is applicable. If input dataset 201 has input dataset type 313, and AI model 210 has AI model type 324, then operational AI model representation 414 is applicable.

[0065] The obtained results are then refined based at least in part on the refinements represented in the representation structure (action 504). Refinement may involve any process calculated to make the AI ​​model output data more relevant. Since the relevance may be specific to the input data set representation and / or the AI ​​model representation (i.e., a given operational AI model representation), different refinements may be expressed for each operational AI model representation. By way of example only, refinement may involve removal, transformation, or prioritization of raw output data. For example, refinement may involve filtering out some AI model outputs. Refinement may involve truncating, converting, combining, and / or otherwise transforming portions of the AI ​​model output. Refinement may involve prioritizing portions of the output by possibly sorting or ranking the output, labeling portions of the AI ​​model output, and the like.

[0066] Refinement may be performed based on applicable refinement definitions. For example, if the operational AI model representation 413 is applicable, an associated refinement of the refinement definition 423 may be performed. However, the refinement may be modified based on AI model-specific hints. These hints may be provided by the author of the particular AI model. In one embodiment, when the author creates the AI ​​model, the AI ​​model is associated with a wrapper in which refinement hints are provided, which may be as specific as hints that specify expressions about the input data set. Refinement may also be modified or enhanced through machine learning analysis based on previous refinements of the results obtained on the input data set applied to the AI ​​model. This learned information may have broad applications, even globally, but may be very granular from the user level to even per-user and per-case levels.

[0067] The refined data contains more relevant and refined information than the original output data of the AI ​​model. Then, the refined data is semantically indexed to generate a semantic index (action 505). Figure 6 In the method 500, the indexer 620 operates on the refined data to generate a semantic index 621. The semantic index 621 is in a semantic space that the user can understand and reason about. In addition, due to the refinement process, the information represented within the semantic index is very relevant. In addition, because the method 500 can be performed on a variety of input data sets / AI model combinations, the method 500 can be repeated to provide users with highly refined semantic access to artificial intelligence results regardless of the number of available input data sets and AI models.

[0068] In any case, once the semantic index is created (act 505), it can be used (act 510). There are many ways in which the semantic index can be used, especially when the semantic index is generated based on refined intelligence. By way of example only, the semantic index can be used to cause at least a portion of a set of one or more operators / terms to be transmitted to a query engine (act 511). Then, when a user issues a query against the query engine, the query engine can enable these operators and understand these terms. For example, in Figure 6 In the representation structure 400, the query engine 630 can interact with the semantic index through the query interface 625. These operators and terms can also be included as operators / terms 430 in the representation structure 400. For example, the operation AI model representations 411 to 413 are associated with operator / term sets 431, 432, and 433, respectively. Therefore, the representation structure 400 can be used to quickly determine the appropriate operations and terms for interfacing with the semantic index.

[0069] As another example, the semantic index can be used to cause at least a portion of the set of one or more operators / terms to be transmitted to a suggestion engine (act 512), which can then form relevant suggestion queries to the user. Figure 6 In, suggestion engine 650 can receive operators and terms and determine appropriate suggested queries for presentation to the user.

[0070] As yet another example, a semantic index can be used to determine appropriate visualizations for the results of a particular query. These (multiple) visualizations can also be included in the representation structure 400 as visualizations 440. For example, operational AI model representations 412 to 414 are associated with visualization sets 442, 443, and 444, respectively. Thus, the representation structure 400 can be used to quickly determine appropriate visualizations for presenting query results to a user.

[0071] Without departing from the spirit or essential features of the present invention, the present invention may be implemented in other specific forms. The described embodiments are considered to be illustrative and non-restrictive in all respects. Therefore, the scope of the present invention is indicated by the appended claims rather than the preceding description. All changes falling within the equivalent meanings and scope of the claims should be included within their scope.

Claims

1. A computing system for data query, comprising: one or more processors; as well as one or more computer-readable hardware storage devices having computer-executable instructions thereon, the computer-executable instructions being executable by the one or more processors to cause the computing system to at least: Obtain results for input datasets applied to artificial intelligence (AI) models; Accessing at least a portion of an AI representation structure, the AI ​​representation structure comprising: Multiple operational AI model representations, and A plurality of refined definitions determine the input dataset type of the input dataset and the AI ​​model type of the AI ​​model; Identifying the operational AI model representations included in the plurality of operational AI model representations, wherein one operational AI model representation is identified based on the following determination: determining that the one operational AI model representation can be applied to the obtained result based on a combination of the input dataset type and the AI ​​model type associated with the obtained result; selecting a refined definition included in the plurality of refined definitions based on the identified one operational AI model representation, wherein the one operational AI model representation is identified as a result of the obtained result being associated with the input dataset type and the AI ​​model type; refining the obtained results using the selected refinement definition, wherein refining the obtained results using the selected refinement definition is further enhanced by a machine learning analysis for determining at what granularity to refine the obtained results; and Perform semantic indexing on the refined results to generate a semantic index. The AI ​​representation structure includes information for querying, and the information for querying includes: a set of one or more operator terms that a query engine can use to query against the semantic index; and / or a set of one or more operators and / or terms that a query engine can use to query against the semantic index; and wherein execution of the computer executable instructions further causes the computing system to at least: At least a portion of the information is caused to be transmitted to a query engine so that the query engine performs a query based on the information for querying.

2. The computing system of claim 1 , wherein an operational AI model representation is associated with a combination of an input dataset type included in a plurality of input dataset types and an AI model type included in a plurality of AI model types, wherein the input dataset type is a data type identified for the input dataset applied to the AI ​​model, and the AI ​​model type is a model type identified for the AI ​​model, such that a subsequently selected refined definition is selected based on the combination of the input dataset type and the AI ​​model type identified for the input dataset and the AI ​​model; and wherein each refined definition is associated with at least one operational AI model representation included in the plurality of operational AI model representations.

3. The computing system of claim 1 , wherein the AI ​​representation structure further comprises a set of one or more visualizations that a visualization engine can use to visualize responses to a user to query against the semantic index, and execution of the computer executable instructions further causes the computing system to at least: At least a portion of the set of one or more visualizations is caused to be transmitted to the visualization engine.

4. A method for a computing system to process data output of an artificial intelligence (AI) model, the method comprising: Obtain results for input datasets applied to artificial intelligence (AI) models; Accessing at least a portion of an AI representation structure, the AI ​​representation structure comprising: Multiple operational AI model representations, and Multiple detailed definitions; Determining the input dataset type of the input dataset and the AI ​​model type of the AI ​​model; Identifying the operational AI model representations included in the plurality of operational AI model representations, wherein one operational AI model representation is identified based on the following determination: determining that the one operational AI model representation can be applied to the obtained result based on a combination of the input dataset type and the AI ​​model type associated with the obtained result; selecting a refined definition included in the plurality of refined definitions based on the identified one operational AI model representation, wherein the one operational AI model representation is identified as a result of the obtained result being associated with the input dataset type and the AI ​​model type; refining the obtained results using the selected refinement definition, wherein refining the obtained results using the selected refinement definition is further enhanced by a machine learning analysis for determining at what granularity to refine the obtained results; and Perform semantic indexing on the refined results to generate a semantic index. The AI ​​representation structure includes information for querying, and the information for querying includes: a set of one or more operator terms that a query engine can use to query against the semantic index; and / or a set of one or more operators and / or terms that a query engine can use to query against the semantic index; And wherein the method further comprises: At least a portion of the information is caused to be transmitted to a query engine so that the query engine performs a query based on the information for querying.

5. The method according to claim 4, further comprising: Suggested queries are presented to the user using the semantic index.

6. The method of claim 4, wherein the operating AI model representation is associated with a combination of an input dataset type included in a plurality of input dataset types and an AI model type included in a plurality of AI model types, wherein the input dataset type is a data type identified for the input dataset applied to the AI ​​model, and the AI ​​model type is a model type identified for the AI ​​model, so that a subsequently selected refined definition is selected based on the combination of the input dataset type and the AI ​​model type identified for the input dataset and the AI ​​model. 7 . The method of claim 4 , wherein each refinement definition is associated with at least one operational AI model representation included in the plurality of operational AI model representations.

8. The method of claim 4, wherein the AI ​​representation structure further comprises a set of one or more visualizations that a visualization engine can use to visualize responses to a user to query against the semantic index, the method further comprising: At least a portion of the set of one or more visualizations is caused to be transmitted to the visualization engine.

9. According to the method of claim 4, the AI ​​representation structure also represents a refinement of the result of the data applied to the AI ​​model for each of multiple AI model and input data set type combinations, wherein the refinement of the obtained result is based at least on the refinement represented in the representation structure for the combination of the AI ​​model and the input data set.

10. The method of claim 4, the refinement of the obtained results further being based on prompts specific to the AI ​​model.

11. The method of claim 10, the hints specific to the AI ​​model being within a model-specific data structure associated with the AI ​​model.

12. The method of claim 4, the refinement of the obtained results further being based on a machine learning analysis of previous refinements of the obtained results of data applied to an AI model.

13. The method of claim 4, wherein the refinement of the obtained results is also based on a machine learning analysis of previous refinements of the obtained results of the input data set applied to an AI model.

14. The method of claim 4, wherein the refinement of the obtained results is also based on a machine learning analysis of previous refinements of the obtained results of data applied to the AI ​​model when those obtained results are provided for a specific user so that the refinement is specific to the specific user.

15. The method of claim 4, wherein the AI ​​model comprises a machine learning model.

16. The method according to claim 4, wherein the obtained result is a first obtained result, the input data set is a first input data set of a first data set type, the obtained result is a first obtained result, and the method further comprises: Obtain a result of a second input data set of a second data set type applied to the AI ​​model to obtain a second obtained result; as well as The second obtained result is refined based at least on the refinement for the AI ​​model represented in the AI ​​representation structure.

17. The method of claim 4, wherein the obtained result is a first obtained result, the input data set is a first input data set of a first data set type, the refinement represented in the representation structure for the AI ​​model is a first refinement applicable to the AI ​​model and the input data set of the first data set type, the obtained result is a first obtained result, and the method further comprises: Obtain a result of a second input data set of a second data set type applied to the AI ​​model to obtain a second obtained result; as well as The second obtained result is refined based on at least a second refinement of an input data set represented in the AI ​​representation structure for the AI ​​model and the second data set type, the second refinement being different from the first refinement.

18. The method according to claim 4, wherein the AI ​​model is a first AI model, and the method further comprises: Obtaining a result of a second input data set applied to a second AI model, the second AI model also being one of the plurality of AI models; as well as The obtained results from the second AI model are refined based at least on the refinement represented in the AI ​​representation structure for the second AI model.

19. A computer program product comprising one or more computer-readable storage media having computer-executable instructions thereon, the computer-executable instructions being executable by the one or more processors to cause the computing system to perform a method for querying data by performing at least the following: Obtain results for input datasets applied to artificial intelligence (AI) models; Accessing at least a portion of an AI representation structure, the AI ​​representation structure comprising: Multiple operational AI model representations, and A plurality of refined definitions determine the input dataset type of the input dataset and the AI ​​model type of the AI ​​model; Identifying the operational AI model representations included in the plurality of operational AI model representations, wherein one operational AI model representation is identified based on the following determination: determining that the one operational AI model representation can be applied to the obtained result based on a combination of the input dataset type and the AI ​​model type associated with the obtained result; selecting a refined definition included in the plurality of refined definitions based on the identified one operational AI model representation, wherein the one operational AI model representation is identified as a result of the obtained result being associated with the input dataset type and the AI ​​model type; refining the obtained results using the selected refinement definition, wherein refining the obtained results using the selected refinement definition is further enhanced by a machine learning analysis for determining at what granularity to refine the obtained results; and Perform semantic indexing on the refined results to generate a semantic index. The AI ​​representation structure includes information for querying, and the information for querying includes: a set of one or more operator terms that a query engine can use to query against the semantic index; and / or a set of one or more operators and / or terms that a query engine can use to query against the semantic index; And wherein the computer is further caused to: At least a portion of the information is caused to be transmitted to a query engine so that the query engine performs a query based on the information for querying.

Images