Recognition Model in Knowledge Management Artificial Intelligence System

Abstract

An artificial intelligence device for identifying objects in a data set, comprising a processing circuit for receiving the data set and a query including an object. The processing circuit selects one or more models through an entity knowledge database, and the entity knowledge database includes multiple entities corresponding to the objects to be identified. Each of the multiple recognition models is linked to multiple entities of the entity knowledge database, so that the processing circuit can select multiple recognition models. Then, the processing circuit processes the data set through the selected one or more recognition models to indicate whether the data set includes at least one object. The entity can be coupled with an ontology in the database, so that even if the object has no corresponding entity in the database, the object can be identified by a model selected based on the ontology.

Description

[0001] Cross-application of related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 734,016, filed on September 20, 2018, entitled “Recognition Module in Knowledge Management-Based Artificial Intelligence System,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to artificial intelligence (AI) systems, and more particularly to knowledge-based tools for organizing and using recognition models to respond to queries involving related objects in a data set. Background Art

[0004] Entity recognition of objects in a dataset, such as items in an image or video file or text in an audio file, is a widely used AI application. The dataset may include multiple entities, each of which may have multiple granularities. Each type of entity is usually recognized by one or more dedicated models. However, as more and more models appear on AI platforms, it is difficult for users to determine which models are available and which models should be used to recognize specific objects. Summary of the invention

[0005] On the one hand, a device for identifying at least one object in a data set includes: a memory storing instructions; one or more processors communicating with the memory, wherein the one or more processors execute the instructions to receive a data set and a query as to whether the data set includes at least one object; select at least one recognition model through an entity knowledge database including multiple entities corresponding to the objects to be identified, wherein each of the multiple recognition models is linked to multiple entities of the entity knowledge database; and process the data set through the selected at least one recognition model to indicate whether the data set includes at least one object.

[0006] Optionally, in combination with the foregoing aspects, the one or more processors also execute instructions to receive the multiple recognition models, wherein each recognition model includes at least one annotation; for each recognition model: based on at least one annotation of each recognition model, identify at least one entity in the entity knowledge database corresponding to the recognition model; and link the recognition model to at least one entity in the entity knowledge database.

[0007] Optionally, in combination with any of the preceding aspects, the at least one recognition model includes multiple recognition models, each of the multiple recognition models includes at least one parameter indicating computing resources for processing the data set using the recognition model; the one or more processors execute the instructions to process the data set through at least one recognition model of the multiple recognition models, and the at least one recognition model has indicated computing resources that are compatible with resources available to the one or more processors.

[0008] Optionally, in combination with any of the preceding aspects, the one or more processors execute the instructions to obtain additional resources from a network connection service before selecting the recognition model.

[0009] Optionally, in combination with any of the foregoing aspects, the entity knowledge database is a graph database; the entities of the entity knowledge graph database are ontology-coupled nodes of the entity knowledge graph database, so that a recognition model directly linked to a node in the entity knowledge database is linked to all nodes in the entity knowledge database that are ontology-coupled to the one node; the entity knowledge graph database includes nodes corresponding to the at least one object, and the nodes corresponding to the at least one object are not directly linked to the recognition models; the one or more processors execute the instructions to select one or more recognition models as the at least one recognition model, the one or more recognition models are directly linked to at least one node in the entity knowledge database, and the at least one node ontology is coupled to the node corresponding to the at least one object.

[0010] Optionally, in combination with any of the foregoing aspects, the one or more processors execute the instructions to select multiple recognition models of corresponding multiple node links coupled to a node ontology corresponding to the at least one object as one or more recognition models associated with the ontology coupling nodes; process the data set through the selected multiple recognition models; and merge the results of processing through the selected multiple recognition models to indicate whether the data set includes the at least one object.

[0011] Optionally, in combination with any of the preceding aspects, the entity knowledge database is a graph database, the entity knowledge graph database includes multiple ontology organization nodes corresponding to the at least one object with different general levels, the one or more processors execute the instructions to select multiple recognition models associated with the multiple nodes corresponding to different general levels of the at least one object as the at least one recognition model; process the data set through the selected multiple recognition models; and merge the results of the processing through the selected multiple recognition models to indicate whether the data set includes the at least one object.

[0012] On the one hand, a device for identifying at least one object in a data set includes: a memory storing instructions; one or more processors communicating with the memory, wherein the one or more processors execute the instructions to receive multiple recognition models, wherein each recognition model includes multiple annotations; based on the annotations of each recognition model in the received multiple recognition models, establishing links between each recognition model and multiple corresponding entities in an entity knowledge database; receiving a data set and a query about identifying whether at least one object is included in the data set; selecting at least one recognition model from the entity knowledge database to identify the at least one object, the at least one recognition model being linked to at least one entity in the entity knowledge database corresponding to the at least one object; processing the data set through the selected at least one recognition model to indicate whether the data set includes at least one object.

[0013] Optionally, in combination with the foregoing aspects, the entity knowledge database is a graph database; entities in the entity knowledge graph database are ontologically coupled, so that a recognition model directly linked to a node in the entity knowledge graph database is linked to all nodes in the entity knowledge graph database that are ontologically coupled to the one node; the entity knowledge database includes nodes corresponding to the at least one object, and the nodes corresponding to the at least one object are not directly linked to any of the multiple recognition models; the one or more processors execute the instructions to select at least one recognition model as the at least one recognition model, and the at least one recognition model ontology is coupled to the node corresponding to the at least one object and is linked to at least one corresponding recognition model among the multiple recognition models.

[0014] Optionally, in combination with any of the foregoing aspects, the one or more processors execute the instructions to identify multiple identified recognition models of multiple node links coupled to a node body corresponding to the at least one object as the at least one identified recognition model; process the data set through the selected multiple identified recognition models; and merge the results of processing through the selected multiple identified recognition models to indicate whether the data set includes the at least one object.

[0015] Optionally, in combination with any of the preceding aspects, the entity knowledge database includes multiple ontology organization nodes corresponding to the at least one object having different general levels, and the one or more processors execute the instructions to select multiple corresponding models associated with the multiple nodes corresponding to the different general levels of the at least one object as multiple identified recognition models; process the data set through the selected multiple recognition models; and merge the results of processing through the selected multiple identified recognition models to indicate whether the data set includes the at least one object.

[0016] Optionally, in combination with any of the preceding aspects, the data set includes image frames, and the at least one object includes an entity of the entity knowledge database.

[0017] Searching an entity knowledge database including a plurality of nodes corresponding to an object to be identified, wherein each of a plurality of recognition models is linked to a plurality of nodes of the entity knowledge database; in response to searching the entity knowledge database, selecting at least one recognition model from the plurality of recognition models for identifying at least one object; processing the data set through the selected at least one recognition model to indicate whether the data set includes at least one object.

[0018] Optionally, in combination with any of the preceding aspects, the method further includes: receiving the multiple recognition models, wherein each recognition model includes at least one annotation; based on the at least one annotation of each recognition model, identifying at least one entity in the entity knowledge database corresponding to the recognition model; and linking the recognition model to at least one entity in the entity knowledge database.

[0019] Optionally, in combination with any of the preceding aspects, the at least one recognition model includes multiple recognition models, each of the multiple recognition models includes at least one parameter indicating computing resources for processing the data set using the recognition model; the method further includes: processing the data set by at least one recognition model of the multiple recognition models, the at least one recognition model having indicated computing resources that are compatible with resources available to one or more processors.

[0020] Optionally, in combination with any of the preceding aspects, the method includes: before selecting the recognition model, obtaining additional resources from a network connection service.

[0021] Optionally, in combination with any of the foregoing aspects, the entity knowledge database is a graph database, which includes entities coupled to node ontology in the entity knowledge graph database, so that a recognition model directly linked to a node in the entity knowledge database is linked to all nodes in the entity knowledge database coupled to the one node ontology, and the entity knowledge graph database includes nodes corresponding to the at least one object, and the nodes corresponding to the at least one object are not directly linked to the recognition model; the method also includes: selecting one or more recognition models as the at least one recognition model, the one or more recognition models are directly linked to at least one node in the entity knowledge database, and the at least one node ontology is coupled to the node corresponding to the at least one object.

[0022] Optionally, in combination with any of the preceding aspects, the method includes: selecting multiple recognition models of corresponding multiple node links coupled to a node ontology corresponding to the at least one object as one or more recognition models associated with the ontology coupling nodes; processing the data set through the selected multiple recognition models; and merging the results of processing through the selected multiple recognition models to indicate whether the data set includes the at least one object.

[0023] Optionally, in combination with any of the preceding aspects, the entity knowledge database is a graph database, which includes multiple ontology organization nodes corresponding to the at least one object with different general levels; the method further includes: selecting multiple recognition models associated with multiple nodes corresponding to different general levels of the at least one object as the at least one recognition model; processing the data set through the selected multiple recognition models; and merging the results of processing through the selected multiple recognition models to indicate whether the data set includes the at least one object.

[0024] In one aspect, a method for identifying at least one object in a data set includes: receiving multiple recognition models, wherein each recognition model includes multiple annotations; establishing links between each recognition model and multiple corresponding entities in an entity knowledge database based on the annotations of each recognition model in the received multiple recognition models; receiving a data set and a query about identifying whether at least one object is included in the data set; selecting at least one recognition model from the entity knowledge database to identify the at least one object, the at least one recognition model being linked to at least one entity in the entity knowledge database corresponding to the at least one object; and processing the data set through the selected at least one recognition model to indicate whether the data set includes at least one object.

[0025] Optionally, in combination with the foregoing aspects, the entity knowledge database is a graph database, which includes entities coupled to node ontology in the entity knowledge graph database, so that a recognition model directly linked to a node in the entity knowledge graph database is linked to all nodes in the entity knowledge graph database coupled to the one node ontology, and the entity knowledge graph database includes nodes corresponding to the at least one object, and the nodes corresponding to the at least one object are not directly linked to any of the multiple recognition models; the method also includes: selecting at least one recognition model as the at least one recognition model, the at least one recognition model ontology is coupled to the node corresponding to the at least one object, and is linked to at least one corresponding recognition model among the multiple recognition models.

[0026] Optionally, in combination with any of the preceding aspects, the method further includes: identifying multiple identified recognition models of multiple node links coupled to a node body corresponding to the at least one object as the at least one identified recognition model; processing the data set through the selected multiple identified recognition models; and merging the results of processing through the selected multiple identified recognition models to indicate whether the data set includes the at least one object.

[0027] Optionally, in combination with any of the preceding aspects, the entity knowledge database includes multiple ontology organization nodes corresponding to the at least one object with different general levels; the method further includes: selecting multiple corresponding models associated with multiple nodes corresponding to different general levels of the at least one object as multiple identified recognition models; processing the data set through the selected multiple recognition models; and merging the results of processing through the selected multiple identified recognition models to indicate whether the data set includes the at least one object.

[0028] Optionally, in combination with any of the preceding aspects, the data set includes image frames, and the at least one object includes an entity of the entity knowledge database.

[0029] In one aspect, a computer-readable medium stores computer instructions for identifying an object in a data set. When one or more processors execute the instructions, the one or more processors: search an entity knowledge database including multiple nodes corresponding to the objects to be identified, wherein each recognition model of multiple recognition models is linked to multiple nodes of the entity knowledge database; in response to the search of the entity knowledge database, select at least one recognition model from the multiple recognition models for identifying at least one object; and process the data set through the selected at least one recognition model to indicate whether the data set includes at least one object.

[0030] Optionally, in combination with the aforementioned aspects, when the one or more processors execute the computer instructions, the one or more processors: receive the multiple recognition models, wherein each recognition model includes at least one annotation; based on at least one annotation of each recognition model, identify at least one entity in the entity knowledge database corresponding to the recognition model; and link the recognition model to at least one entity in the entity knowledge database.

[0031] Optionally, in combination with any of the preceding aspects, the at least one recognition model includes multiple recognition models, each of the multiple recognition models includes at least one parameter indicating computing resources for processing the data set using the recognition model; when the one or more processors execute the computer instructions, the one or more processors process the data set through at least one recognition model of the multiple recognition models, and the at least one recognition model has indicated computing resources that are compatible with the resources available to the one or more processors.

[0032] Optionally, in combination with any of the preceding aspects, when the one or more processors execute the computer instructions, the one or more processors obtain additional resources from a network connection service before selecting a recognition model.

[0033] Optionally, in combination with any of the foregoing aspects, the entity knowledge database is a graph database, which includes entities coupled to node ontology in the entity knowledge graph database, so that a recognition model directly linked to a node in the entity knowledge database is linked to all nodes in the entity knowledge database coupled to the one node ontology, and the entity knowledge graph database includes nodes corresponding to the at least one object, and the nodes corresponding to the at least one object are not directly linked to the recognition model; when the one or more processors execute the computer instructions, the one or more processors select one or more recognition models as the at least one recognition model, and the one or more recognition models are directly linked to at least one node in the entity knowledge database, and the at least one node ontology is coupled to the node corresponding to the at least one object.

[0034] Optionally, in combination with any of the preceding aspects, when the one or more processors execute the computer instructions, the one or more processors: select multiple recognition models of corresponding multiple node links coupled to the node ontology corresponding to the at least one object as one or more recognition models associated with the ontology coupling nodes; process the data set through the selected multiple recognition models; and merge the results of processing through the selected multiple recognition models to indicate whether the data set includes the at least one object.

[0035] Optionally, in combination with any of the preceding aspects, the entity knowledge database is a graph database, which includes multiple ontology organization nodes corresponding to the at least one object with different general levels; when the one or more processors execute the computer instructions, the one or more processors: select multiple recognition models associated with multiple nodes corresponding to different general levels of the at least one object as the at least one recognition model; process the data set through the selected multiple recognition models; and merge the results of processing through the selected multiple recognition models to indicate whether the data set includes the at least one object.

[0036] In one aspect, a computer-readable medium stores computer instructions for identifying objects in a data set. When one or more processors execute the computer instructions, the one or more processors: receive multiple recognition models, wherein each recognition model includes multiple annotations; establish links between each recognition model and multiple corresponding entities in an entity knowledge database according to the annotations of each recognition model in the received multiple recognition models; receive a data set and a query for identifying whether at least one object is included in the data set; select at least one recognition model from the entity knowledge database to identify the at least one object, wherein the at least one recognition model is linked to at least one entity in the entity knowledge database corresponding to the at least one object; process the data set through the selected at least one recognition model to indicate whether the data set includes at least one object.

[0037] Optionally, in combination with the aforementioned aspects, the entity knowledge database is a graph database, which includes entities coupled to node ontology in the entity knowledge graph database, so that a recognition model directly linked to a node in the entity knowledge graph database is linked to all nodes in the entity knowledge graph database coupled to the one node ontology, and the entity knowledge graph database includes nodes corresponding to the at least one object, and the nodes corresponding to the at least one object are not directly linked to any one of the multiple recognition models; when the one or more processors execute the computer instructions, the one or more processors select at least one recognition model as the at least one recognition model, and the at least one recognition model ontology is coupled to the node corresponding to the at least one object and is linked to at least one corresponding recognition model among the multiple recognition models.

[0038] Optionally, in combination with any of the preceding aspects, when the one or more processors execute the computer instructions, the one or more processors are further caused to: identify multiple identified recognition models of multiple node links coupled to a node body corresponding to the at least one object as the at least one identified recognition model; process the data set through the selected multiple identified recognition models; and merge the results of processing through the selected multiple identified recognition models to indicate whether the data set includes the at least one object.

[0039] Optionally, in combination with any of the preceding aspects, the entity knowledge database includes multiple ontology organization nodes corresponding to the at least one object with different general levels; when the one or more processors execute the computer instructions, the one or more processors are further caused to: select multiple corresponding models associated with multiple nodes corresponding to different general levels of the at least one object as multiple identified recognition models; process the data set through the selected multiple recognition models; and merge the results of processing through the selected multiple identified recognition models to indicate whether the data set includes the at least one object.

[0040] Optionally, in combination with any of the preceding aspects, the data set includes image frames, and the at least one object includes an entity of the entity knowledge database.

[0041] In one aspect, an apparatus for identifying objects in a data set comprises: a component for searching an entity knowledge database including multiple nodes corresponding to objects to be identified, wherein each of multiple recognition models is linked to multiple nodes of the entity knowledge database; a component for selecting at least one recognition model from the multiple recognition models for identifying at least one object in response to searching the entity knowledge database; and a component for processing the data set through the selected at least one recognition model to indicate whether the data set includes at least one object.

[0042] Optionally, in combination with any of the preceding aspects, the device includes: a component for receiving the multiple recognition models, wherein each recognition model includes at least one annotation; a component for identifying at least one entity in the entity knowledge database corresponding to the recognition model based on at least one annotation of each recognition model; and a component for linking the recognition model to at least one entity in the entity knowledge database.

[0043] Optionally, in combination with any of the preceding aspects, the at least one recognition model includes multiple recognition models, each of the multiple recognition models includes at least one parameter indicating computing resources for processing the data set using the recognition model; the device also includes: a component for processing the data set through at least one recognition model of the multiple recognition models, the at least one recognition model having indicated computing resources that are compatible with resources available to the one or more processors.

[0044] Optionally, in combination with any of the preceding aspects, the apparatus includes: a component for obtaining additional resources from a network connection service before selecting the recognition model.

[0045] Optionally, in combination with any of the foregoing aspects, the entity knowledge database is a graph database, which includes entities coupled to node ontology in the entity knowledge graph database, so that a recognition model directly linked to a node in the entity knowledge database is linked to all nodes in the entity knowledge database coupled to the one node ontology, and the entity knowledge graph database includes nodes corresponding to the at least one object, and the nodes corresponding to the at least one object are not directly linked to the recognition model; the device also includes: a component for selecting one or more recognition models as the at least one recognition model, the one or more recognition models are directly linked to at least one node in the entity knowledge database, and the at least one node ontology is coupled to the node corresponding to the at least one object.

[0046] Optionally, in combination with any of the preceding aspects, the device includes: a component for selecting multiple recognition models of corresponding multiple node links coupled to a node body corresponding to the at least one object as one or more recognition models associated with the body coupling node; a component for processing the data set through the selected multiple recognition models; and a component for merging the results of processing through the selected multiple recognition models to indicate whether the data set includes the at least one object.

[0047] Optionally, in combination with any of the preceding aspects, the entity knowledge database is a graph database, which includes multiple ontology organization nodes corresponding to the at least one object with different general levels; the device also includes: a component for selecting multiple recognition models associated with multiple nodes corresponding to different general levels of the at least one object as the at least one recognition model; a component for processing the data set through the selected multiple recognition models; and a component for merging the results of processing through the selected multiple recognition models to indicate whether the data set includes the at least one object.

[0048] Optionally, in combination with any of the preceding aspects, the device includes: a component for receiving multiple recognition models, wherein each recognition model includes multiple annotations; a component for establishing links between each recognition model and multiple corresponding entities in an entity knowledge database based on the annotations of each recognition model in the received multiple recognition models; a component for receiving a data set and a query for identifying whether at least one object is included in the data set; a component for selecting at least one recognition model from the entity knowledge database to identify the at least one object, the at least one recognition model being linked to at least one entity in the entity knowledge database corresponding to the at least one object; and a component for processing the data set through the selected at least one recognition model to indicate whether the data set includes at least one object.

[0049] Optionally, in combination with any of the foregoing aspects, the entity knowledge database is a graph database, which includes entities coupled to node ontology in the entity knowledge graph database, so that a recognition model directly linked to a node in the entity knowledge graph database is linked to all nodes in the entity knowledge graph database coupled to the one node ontology, and the entity knowledge graph database includes nodes corresponding to the at least one object, and the nodes corresponding to the at least one object are not directly linked to any of the multiple recognition models; the device also includes: a component for selecting at least one recognition model as the at least one recognition model, the at least one recognition model ontology is coupled to the node corresponding to the at least one object, and is linked to at least one corresponding recognition model among the multiple recognition models.

[0050] Optionally, in combination with any of the preceding aspects, the device includes: a component for identifying multiple identified recognition models of multiple node links coupled to a node body corresponding to the at least one object as the at least one identified recognition model; a component for processing the data set through the selected multiple identified recognition models; and a component for merging the results of processing through the selected multiple identified recognition models to indicate whether the data set includes the at least one object.

[0051] Optionally, in combination with any of the preceding aspects, the entity knowledge database includes multiple ontology organization nodes corresponding to the at least one object with different general levels; the device also includes: a component for selecting multiple corresponding models associated with multiple nodes corresponding to different general levels of the at least one object as multiple identified recognition models; a component for processing the data set through the selected multiple recognition models; and a component for merging the results of processing through the selected multiple identified recognition models to indicate whether the data set includes the at least one object. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A functional block diagram of an AI recognition system provided for an example embodiment.

[0053] Figure 2 A system diagram of an exemplary AI operating environment.

[0054] Figure 3 A schematic diagram for describing visual entity recognition is provided for example embodiments.

[0055] Figure 4A and Figure 4B A flow chart of an example AI recognition process is provided for an example embodiment.

[0056] Figure 5A A schematic diagram of an entity knowledge graph database provided for example embodiments.

[0057] Figure 5B A schematic diagram of a method for associating entities used in a knowledge graph through concept subspaces associated with high-dimensional space objects is provided for example embodiments.

[0058] Figure 6 A schematic diagram of an object recognition process is provided for an example embodiment.

[0059] Figure 7 A block diagram of circuitry of clients, servers, and network connection resources for implementing algorithms and executing methods is provided for example embodiments. DETAILED DESCRIPTION

[0060] The following is a detailed description in conjunction with the accompanying drawings, which are part of the description and show specific embodiments that can be implemented by way of diagrammatic illustration. These embodiments are described in sufficient detail so that those skilled in the art can practice the disclosed content. It is understood that other embodiments may be used and structural, logical or electrical changes may be made without departing from the scope of the appended claims. Therefore, the following description of exemplary embodiments should not be considered as a limitation to the appended claims.

[0061] The functions or algorithms described herein may be implemented by software of some embodiments. The software may include computer executable instructions stored on a computer readable medium or on a computer readable storage device, such as one or more non-transient memories or other types of local or networked hardware storage devices. In addition, these functional corresponding modules may be software, hardware, firmware or any combination thereof. Multiple functions may be implemented in one or more modules as required, and the described embodiments are only examples. The software may be executed by a processing circuit, which may include one or more of the following: a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-core or multi-core microprocessor, or other types of processors running on a computer system such as a personal computer, a server, or other computer systems, thereby converting these computer systems into a dedicated programming machine.

[0062] In the following embodiments, the problem of the increasing number of available recognition models on AI platforms is solved. These embodiments use entity knowledge databases such as entity knowledge graph databases to organize models, so that users can arbitrarily describe the objects to be recognized in the data set, and the system can automatically select a model or a group of models based on the description to perform entity recognition on the data set. Exemplary entity knowledge databases may include commercially available knowledge databases such as DBPEDIA, and / or Or generated from these commercial knowledge databases. For example, a commercial AI system can use its own dedicated knowledge database similar to one of the commercial databases. In this case, the entity knowledge graph database can be a dedicated knowledge database. In another example, the entity knowledge database can be generated from one or more of the commercial databases to provide a database dedicated to a specific recognition task or a group of recognition tasks.

[0063] In some of the embodiments described below, the models used to identify objects in a data set are organized by the AI ​​platform in a way that all recognition models are stored in a model plane in a data storage device. The models in the model plane are linked to an entity knowledge database, so that the ontology inherent in the knowledge database can be used to identify a suitable recognition model for a received query from a flexible and / or approximate description of an object to be identified in the query.

[0064] In some of the following embodiments, a single recognition model is automatically selected or a group of recognition models are used in coordination to improve the recognition of objects in an arbitrary data set. In these embodiments, a knowledge database and / or automatically determined conceptual relationships between objects can be used to select the single or group of models to be used in the recognition process.

[0065] The following embodiments describe the context of an AI image recognition system. In the AI ​​image recognition system, a user provides or indicates a dataset including an image, a set of images, or a video to be found, and also provides a query for asking whether a particular object or class of objects is represented in the dataset. The query can be a specific query, such as, "Does the dataset include a German Shepherd", or it can be a more general query, such as, "Identify all animals in the dataset".

[0066] Figure 1 Functional block diagram of an AI recognition system 100 provided for an example embodiment. The exemplary system 100 includes a set of recognition models 102, each recognition model 102 including a model code 104, a model annotation 105, and a model parameter 106. The model code 104 is executed by a processing system (e.g., a model selection and service process 126) to recognize or classify a specific object, and the recognition model 102 is used to recognize a specific object in a data set provided to the system 100. The model annotation 105 is used to describe an object or object category that can be recognized by the model code 104, and the model annotation 105 may also include other data, such as features of a feature vector for use by the recognition model 102.

[0067] The system 100 also includes a recognition model acquisition process 108 and a data storage 110, which includes a model repository 112 and an entity knowledge graph database 114. The exemplary model repository 112 includes five recognition models, labeled 1-5. The exemplary entity knowledge graph database 114 includes a first-level entity, labeled A; two second-level entities, labeled B and C; and four third-level entities, labeled D, E, F, and G. The nodes of the database are arranged according to the ontology, so the granularity of the entities represented by nodes AG increases from top to bottom. In this way, nodes B and C represent different subsets of node A, nodes D and E represent subsets of node B, and nodes F and G represent subsets of node C.

[0068] Combined with the following FIG. 4A to FIG. 5BTo describe that the recognition models 1-5 in the model repository 112 are linked to the entity knowledge graph database 114, so when searching for an object in the database 114, one or more linked models are returned from the model repository 112. The links between the recognition models 1-5 in the model repository 112 and the entities AG in the knowledge graph database 114 are generated by the knowledge graph update process 116. When the recognition model 102 is collected into the model repository 112 by the model collection process 108, the process 116 updates the entity knowledge graph database 114 linked to the recognition model 102 according to the model annotation 105 of the recognition model 102.

[0069] like Figure 1 As shown, some entities in the exemplary entity knowledge graph database 114 may be linked to multiple models 1-5, and some models may be linked to multiple entities. Some entities may not be linked to any model, and some models may not be linked to any entity. For example, entity D is linked to model 1 and model 4, and entity C is linked to models 3 and 5. Model 1 is linked to entities B, D, and E, and model 5 is linked to entities F and C. Model 2 is not linked to any entity, and entities A and G are not linked to any model. Ideally, each model in the model repository 112 is linked to at least one entity in the entity knowledge graph database 114. In model acquisition 108, when an entity cannot be found based on the annotation 105 of the model 102, the knowledge graph update process 116 can, for example, obtain an entity knowledge service such as Figure 2 The service 210 shown adds entity data so that each model collected is directly linked to at least one entity in the entity knowledge graph database 114 .

[0070] The system 100 includes a block 118 for receiving a plurality of queries. Data sets associated with the plurality of queries may be provided to a model selection and serving process 126. As described above, in the example, a data set may be an image file or a video file, and an example query may be determining whether an object or object class is represented in the data set. Block 120 of the system 100 is for determining whether a query object exists in the entity knowledge graph 114. If the object does not exist in the knowledge graph 114, the exemplary system 100 includes a process 122, which may access a graph embedding process 124, in conjunction with Figure 4B , 5Aand 5B, to identify entities in the entity knowledge graph that have similar characteristics to the query object. Example blocks 120, 122, and / or 124 illustrate using the entity knowledge graph database 114 to identify one or more recognition models 1-5 in the model repository 112 that can satisfy the query. These recognition models are then provided to the model selection and service process 126, which uses the selected model to process the provided or indicated data set to provide a prediction result 128. FIG. 4A to FIG. 5B The operation of system 100 is described in more detail.

[0071] Figure 2 2 is a system diagram of an exemplary AI operating environment 200. The exemplary environment 200 includes a user terminal 202, which is coupled to a local server 204. The exemplary local server 204 may also be coupled to a data storage 206, which may include the above-mentioned entity knowledge graph database 114 and the above-mentioned model repository 112. The local server 204 may also be coupled to a network 208, which couples the local server 204 to an entity knowledge service 210, a graph embedding service 212, and a network connection service 214. Another user terminal 218 may also be coupled to the network 208 via an access point 216. The network 208 may be a local area network (LAN), a wide area network (WAN), or a global information network (e.g., the Internet).

[0072] Figure 2 Two environments in which the system 100 can be implemented are shown. In the first environment, the system 100 is primarily implemented on a local server 204. In this embodiment, the local server 204 can receive a data set and a query, retrieve one or more recognition models 1-5 from a data storage 206, execute the recognition model on the data set, and display the determined prediction result 128 through the user terminal 202. However, this embodiment can access the graph embedding service 212 through the network 208. In combination Figure 4A and Figure 4BAs shown, some recognition models 102 may require more resources than the local server 204 has available resources. In this case, the system 100 implemented on the local server 204 can request additional resources from the network connection (cloud) service 214. Similarly, the graph embedding process 124 may require more resources than the local server 204 has available resources. In this case, the system 100 implemented on the local server 204 can access the graph embedding service 212 through the network 208 to identify entities in the entity knowledge graph database 114 that are similar to the target object concept in the query, so that the query can be satisfied. The local server 204 can also obtain or establish the entity knowledge graph database 114 in the data storage 206 through the entity knowledge service 210.

[0073] exist Figure 2 In the second environment shown, the system 100 is mainly implemented through resources available to the network connection service 214. In this embodiment, the user terminal 218 that accesses the network 208 through the access point 216 can control the allocated resources of the network connection service 214. These resources may include one or more processors, memories, and data storage (not shown), and can implement functions similar to the local server 204 and the data storage 206. In this case, the resources obtained through the network connection service 214 can obtain the recognition model 102 from the user terminal 218 or from other services (not shown) coupled to the network 208 to build the model repository 112. Similarly, the network connection service 214 can build the entity knowledge graph database 114 through the entity knowledge service 210. Alternatively, the network connection service 214 can access the entity knowledge service 210 as the entity knowledge graph database 114, and can link the entities in the entity knowledge graph database 114 to the model repository service (not shown), which is collocated or separated from the entity knowledge service 210 and / or the network connection service 214. The system 100 or at least a portion of the system 100 runs on allocated resources of the network connection service 214 and can use the graph embedding service 212 to identify conceptually similar models 102, such as FIG. 4B to FIG. 5B described.

[0074] Figure 3 is a diagram provided by an example embodiment for describing visual entity recognition. Figure 3Included is an image 302 and an entity knowledge graph database 340. Although not shown in the figure, entities in the entity knowledge graph database 114 are linked to the recognition model 102 in the model repository 112. Included in the image 302 are images of a pickup truck 304, a dog 306, a horse 308, and two people 310 and 312. The exemplary entity graph database 114 includes nodes or vertices connected by edges. The exemplary database 340 is organized by ontology so that nodes corresponding to specific entities are child nodes of nodes corresponding to more general entities.

[0075] exist Figure 3 In the example shown, the first-level node 350 corresponds to a broad entity category. The node 350 is coupled to three second-level nodes, including a person node 352, an animal node 354, and a vehicle node 356, through corresponding edges. The person node 352 is coupled to a third-level child node 358 and a third-level adult node 360. The adult node 360 ​​is coupled to two fourth-level nodes 366 and 368 corresponding to specific characters Isaac Newton and Marie Curie. Similarly, the second-level animal node 354 is coupled to the person node 352, the horse node 362, and the dog node 364. The dog node 364 is coupled to three specific dog type nodes, including Chihuahua 370, Husky 372, and Hound 374. The database 340 is greatly simplified and is provided only as an illustration. The actual entity knowledge graph database 114, for example, DBPEDIA, Microsoft Satori, and / or Google Graph, may include more categories, each of which includes more entities. Further, the entity knowledge database 114 may not have a tree structure, for example, the entity knowledge graph database 114 may be an acyclic graph database. With respect to the entity knowledge graph database 340, when traversing each node from bottom to top, the edge connecting the node meets the statement "is a", for example, the statement "Chihuahua is a dog" meets the edge connecting the node 372 and the node 364.

[0076] As described above, in operation, each node shown in the entity knowledge graph database 340 may be linked to one or more recognition models ( Figure 3 362 ). A model may be directly or indirectly linked to a node in the entity knowledge graph database 340. For example, a horse recognition model may be directly linked to the horse node 362 and indirectly linked to the animal node 354 through the ontology of the graph database 340.

[0077] Through database 340, a query to find a dog in image 302 can use ontological knowledge to determine that a dog is an animal and an animal is an entity. The search for a model corresponding to "dog" in the database 344 can follow a path from node 350 to node 354 to node 364. The system then determines whether node 364 is coupled to a recognition model. If node 364 is coupled to a recognition model, the recognition model code 104 is returned to the model selection and service box 126 for determining whether image 302 includes a dog. At this point, image 302 is processed through canine model 364, thereby identifying dog 306 with a confidence value of 0.997.

[0078] System 100 can identify multiple objects in a data set. For example, Figure 3 The image 302 shown may be the result of a query to identify all entities in the image 302 through the entity knowledge graph database 340. The response to the query may be to identify all entities in the image 302 through one or more recognition models 102 associated with the entity node 350. The recognition model may be a general classification / detection service that provides entry-level classification of the image 302 as a whole and part detection to resolve objects in the image 302. The service may also provide general characteristics of the scene in the image 302 (e.g., "outdoors" and / or "horse farm") and outline the detected parts of the image 302 through bounding boxes 304, 306, 308, 310, and 312. An exemplary classification / detection system may also include a general model (e.g., a secondary model of the database 340) that provides a broad classification of multiple categories that include objects that can be found and recognized in the image 302. These are entry-level classifications, such as animals, people, vehicles, fashion, food and drink, plants, sports, and other broad categories. The results returned by each recognition model 102 may also include a confidence value indicating the likelihood that the image 302 belongs to a certain category. The model / service associated with node 350 may return multiple categories corresponding to nodes 352, 354, and 356, each category being associated with a corresponding confidence value.

[0079] Once the entities in image 302 are isolated and classified, system 100 can use the broad categories returned by the model / service to look up nodes connected to the broad entity node from knowledge graph database 340 to determine whether a more specific entity can be identified. For example, if the recognition model / service associated with node 350 only isolates the entity and generates bounding boxes 304, 306, 308, 310, and 312, system 100 can process each bounded image indicated by bounding boxes 304, 306, 308, 310, and 312 using each secondary recognition model associated with nodes 352, 354, and 356. For example, each bounded image can be processed by the model associated with vehicle node 356 to identify pickup truck 304 in image 302 with a confidence value of 1.000, as shown by dashed line 320. Similarly, the system 100 may process each bounded image using the recognition model 102 associated with the animal node 354, and recognize the dog 306, the horse 308, the person 310, and the person 312 as animals. Tracing further down the entity knowledge database 340, the system 100 may process the bounded images recognized as animals using the recognition model 102 associated with the person node 352, the horse node 362, and the dog node 364, thereby recognizing the person 310 indicated by the dotted line 328, whose confidence value is 0.992, the person 312 indicated by the dotted line 330, whose confidence value is 0.979, the horse 308 indicated by the dotted line 326, whose confidence value is 0.993, and the dog 306 indicated by the dotted line 324, whose confidence value is 0.997.

[0080] The ontology structure of the knowledge graph database 340 can improve the accuracy of the system 100. For example, only bounded images 304, 306, 308, 310, and 312 are applied to the secondary recognition models corresponding to nodes 352, 354, and 356, removing elements not related to the image 302 from the recognition process. Similarly, the secondary recognition model further filters the bounded images, such that bounded image 304 of pickup truck 304 is not applied to models 364, 362, and 352 used to recognize dog 306, horse 308, person 310, and / or person 312.

[0081] Figure 4A is a flowchart of an exemplary AI recognition process provided by an exemplary embodiment, combined with Figure 1 and Figure 3 The system 100 shown is described. Figure 1The exemplary system shown is capable of accessing multiple recognition models 102, each recognition model 102 including a model code 104, a model annotation 105, and a model parameter 106. In an exemplary embodiment, the model annotation 105 is used to describe the characteristics of a target object that can be recognized using the model code 104, and the model parameters 106 are used to indicate the resources for running the model 102. For example, the model annotation 105 may indicate multiple names of the target object and / or the characteristics of the feature vector used by the recognition model 102. The model parameters 106 may include an indication of hardware resources, including a processor and memory for running the model code 104. For example, the parameters 106 of a trained neural network recognition model may indicate multiple processing elements (e.g., processors and / or processing cores) to be configured as a neural network in order to efficiently execute the model. The model 102 can be based on a variety of different AI recognition technologies, including but not limited to support vector machines, trained neural networks, Bayesian networks, genetic networks, and / or decision trees.

[0082] The recognition model 102 can be generated in a variety of different ways. In one approach, an expert defines the properties of the model 102, such as the features of a feature vector for identifying a target object, and then trains the model 102 to recognize the target object using labeled training data. In another approach, the model 102 can be generated by applying labeled training data to a neural network, which adjusts the coefficients of the neural network nodes through back propagation to improve its ability to recognize the target object. However, since multiple recognition models 102 are generated, Figure 1 The illustrated system 100 requires access to a large number (eg, hundreds to tens of thousands) of different recognition models 102 .

[0083] In block 402, the system 100 obtains an available recognition model 102. In block 404, the system 100 stores the recognition model 102 in the model repository 112 through the model acquisition process 108. The process 108 stores the model 102, including the model code 104 and / or annotations 105 and parameters 106, in the model repository 112 of the data storage 110. In some embodiments, the model repository 112 may include an indexed or unindexed database for storing multiple recognition models 102. Figure 1 As shown, the exemplary model repository 112 includes five models 102 numbered 1 - 5. It should be appreciated that the repository 112 may include many more models 102 organized via an entity knowledge graph database 114, as described below.

[0084] In block 406, the example process 400 includes linking the models 1-5 stored in the model repository 112 to nodes in the entity knowledge graph database 114. For example, when the model acquisition process 108 stores the model 102 in the model repository 112, the knowledge graph update process 116 can use the model annotations 105 to look up the model 102 in the entity knowledge graph database 114 to identify a node corresponding to the model annotation 105. The corresponding node corresponds to the type of entity identified by the model 102, or has auxiliary data (e.g., feature data) that matches the corresponding model annotation 105 of the model 102. When the knowledge graph update process 116 identifies one or more nodes in the entity knowledge graph based on the model annotations 105 of a particular identified model 102, a link to the model 102 can be stored in the identified node of the entity knowledge graph database 114. When the annotation data matches the auxiliary data of the model 102, the knowledge graph update process 116 can link the recognition model 102 in the model repository 112 to the node of the entity knowledge graph database 114 only when there are multiple matches between the model annotation 105 and the auxiliary data associated with the node of the entity knowledge graph database 114.

[0085] Blocks 402, 404, and 406 are performed until all recognition models 102 available to the system 100 have been stored in the model repository 112 and linked to nodes of the entity knowledge graph database 114. In block 408, the process 400 includes receiving a query for the current data set, or receiving a new data set and a query for the new data set. In block 408, also by Figure 1 The query entity process 118 shown extracts the objects in the query to identify the target object in the query. Then, in box 410, the model corresponding to the target object is obtained from the data storage 110 using the entity knowledge graph 114, as follows Figure 4BIn box 412, if no model is found in box 410, process 400 includes obtaining one or more new recognition models 102 of the object (box 414), for example, through an Internet search, and can branch back to box 404 to collect new models. In box 412, if process 400 includes determining that a model has been obtained from the model repository 112 in box 410, the obtained model is provided to the model selection and service process 126. The model selection and service process 126 can optionally analyze the parameters 106 of the obtained model as indicated by the dashed box 416 to determine the resources for running the model 102, obtain any such resources, and dockerize the model 102 using the obtained resources. Alternatively, when multiple models 102 are returned in box 410, one or more models 102 can be selected in box 416 based on the resources specified in the model parameters 106 compared with the resources available to the model service and selection process 126.

[0086] Figure 4B About Figure 4A 4. The flowchart of the example AI recognition process of block 410 shown in FIG. 4 shows a flowchart of an example AI recognition process of block 410, that is, finding and obtaining the recognition model 102. In block 452, the process of block 410 receives the extracted target object from the query. In block 454, the process obtains the extracted target object from the entity knowledge graph database 114 (such as Figure 1 If any of the target objects are not found in block 454, the process proceeds to block 456 to identify conceptually similar objects through graph embedding.

[0087] Figure 5A4 is an example diagram of an entity knowledge graph database 500 provided by an example embodiment, which is used to describe how to use graph embedding and the ontology of the entity knowledge graph database 114 in the example process 400. The example graph database 500 can be used to identify different types of animals. The database 500 contains ontology arrangements of nodes at different levels. For example, the database 500 includes a first-level node 502 that can be linked to the recognition model of all animals. The second-level node includes a pet node 504, a carnivore node 506, and a bird node 508, which are the ontology subordinate nodes of the animal node 502. The third-level node ontology is coupled to the pet node 504, including a dog node 520, a cat node 526, and a canary node 512. Similarly, the third-level node coupled to the carnivore node 506 ontology includes a canine node 510, a feline node 514, and a raptor node 516. The third-level node coupled to the bird node 508 ontology includes a canary node 512, a raptor node 516, and an ostrich node 518. The example entity knowledge database 500 also includes a fourth-level node coupled to the third-level node ontology. These nodes include: a poodle node 536 and a labrador node 538 linked to the dog node 520, a dog node 520 linked to the canine node 510, a wolf node 522 and a jackal node 524, a tabby cat node 540 and a Siamese cat node 542 linked to the cat node 526, a cat node 526 linked to the feline node 514, a lion node 528 and a tiger node 530, and a falcon node 532 and an eagle node 534 linked to the raptor node 516.

[0088] Ontologies can be used to identify objects that are similar to objects found in the entity knowledge graph database 114 but are not directly linked to the recognition model 102. Graph embeddings can be used to identify objects that are conceptually similar to objects that are not found in the entity knowledge graph database 114. For example, when the object extracted from the query is "Doberman Pinscher", the object is not found in the entity knowledge graph database 500 in block 454, such as Figure 5A As shown. Figure 4B As shown, the process in box 410 uses graph embedding to identify similar objects. Graph embedding is a process by which entities in a knowledge graph are linked based on conceptual similarities between entities in an ontology of a substitutable entity knowledge graph database. An example of graph embedding is provided in "Entity Embeddings with Conceptual Subspaces as a Basis for Plausible Reasoning" by S. Jameel et al., Proceedings of the 22nd European Conference on Artificial Intelligence, pp. 1353-1361 (2016). Briefly, the conference proceedings describe a method of extracting information from Wikipedia. A method for obtaining a high-dimensional vector space embedding of entities in knowledge services such as , and constraining the embedding so that entities of the same semantic type are located in a certain low-dimensional subspace. Features associated with the entity can be used to determine the semantic type of each entity.

[0089] Figure 5B is a schematic diagram of a method provided by an example embodiment for associating entities used in a knowledge graph through concept subspaces associated with high-dimensional space objects. Figure 5B Two spaces are shown, an entity space 550 and a relationship space 560. The feature vectors of the entities define a multidimensional space with one dimension for each feature. Entity h is in cluster 552 of other entities that have similar feature vectors as entity h. Therefore, these other entities are located near entity h in the multidimensional space. Similarly, entity t is in cluster 554 of other entities that have similar feature vectors as entity t, and these other entities are located near entity t in the multidimensional space. These groupings are based on the feature vectors in the high-dimensional space. When entity h and entity t are mapped to the relationship space 560, cluster 562 and cluster 564 are linked by a relationship r. For example, the relationship can be a simplified set of more important features in the feature vector and, therefore, can represent a lower dimensional space than the entity space 550.

[0090] For example, entity h may correspond to a Doberman Pinscher, which is located in cluster 552 like other domestic dogs. Entity t may be a wolf, which is located in cluster 554 like other wild canids. Clusters 552 and 554 share features and have features that distinguish them. The shared features may define a relationship r. Examples of these features may be "quadrupeds," "carnivores," and "cluster animals." Associating two clusters by these features may allow several models 102 related to the object Doberman Pinscher to be identified in process 400 in the entity knowledge database 500. These models include a dog node 520, and a pet node 504, a poodle node 536, and a Labrador node 538 coupled to the dog node 520 ontology. Due to the relationships defined by the graph embedding process 124, the identified nodes may also include a canine node 510, a wolf node 522, and a jackal node 524. Thus, although the entity knowledge graph database 500 does not include Doberman Pinscher as a node, the process 400 includes identifying six other nodes having recognition models 102 that can be used to indicate whether an image of a Doberman Pinscher is included in the image dataset.

[0091] Continue to refer Figure 4B, after the unknown entity is processed in block 456 by the graph embedding process 124, it is determined in block 458 whether the recognized object in the entity knowledge graph database 114 is linked to the recognition model 102 in the model repository 112. When an entity exists in the entity knowledge graph database 114 but the entity is not linked to one or more recognition models 102, similar objects in the entity knowledge graph database 114 are identified according to the graph ontology in block 460. For example, if the query object is a poodle, and the poodle node 536 is not directly linked to the recognition model 102, the database 500 may be traversed according to the ontology in block 460 to identify the dog node 520 and the pet node 504 as possibly related to the poodle node 536. In block 462, the process 400 includes determining whether the object recognized in block 460 using the graph ontology of the database 114 is linked to the model 102 in the model repository 112. If no object is linked to the model 102, an indication that the recognition model 102 was not found is returned in block 466. When the recognition model 102 is linked to the object recognized in block 458 or block 462, the recognized recognition model 102 is retrieved from the model repository 112 in block 464 and the model 102 is returned for use. Figure 4A Processing is performed in block 412 as described above.

[0092] Figure 6 600 is a schematic diagram of an object recognition process 600 provided by an example embodiment. The example process 600 is as follows: Figure 1 The model selection and service process 126 shown is executed. In box 602, the process 600 includes determining whether a recognition model 102 directly linked to the target object or the object extracted from the query is retrieved in the above process 400. If such a model 102 is found, one or more data sets are processed by the directly linked model in box 604. In this way, it is beneficial to run the identified recognition model 102 and all recognition models at higher levels in the ontology of the entity knowledge graph database 114, thereby identifying possible entities in the data set and reducing the number of entities that may correspond to the target object.

[0093] If, in block 602, it has been determined in process 600 that the retrieved recognition model is not directly linked to the target object extracted from the query, multiple related recognition models that are indirectly linked to the target object have been received in process 600. In this case, multiple models 102 are run on the related entities in the data set and the results are merged in block 606. For example, the results may be merged by averaging the results or by selecting the result indicating the highest probability of detection.

[0094] The result determined by box 604 or box 606 is processed by box 608 to determine whether the result indicates that the target object exists in the data set. For example, the confidence value returned by the recognition model 102 can be compared with a threshold value in the test, and an indication of whether the target object exists is indicated when the confidence value exceeds the threshold value. If the confidence value exceeds the threshold value, then in step 610, process 600 includes returning an indication that the target object has been identified. The result can be returned together with the data used to identify the target object and the corresponding confidence value. If the confidence value is below the threshold value, an indication that the target object has not been identified is returned in box 612.

[0095] Figure 7 Block diagram of circuitry of a client, server, and cloud-based resource for implementing algorithms and execution methods provided for example embodiments. Not all components need to be used in various embodiments. For example, the client, server, and network resources may each use different sets of components, or, for example, use a larger storage device in the server.

[0096] An example computing device in the form of a computer 700 may include a processing unit 702, a memory 703, a removable memory 710, and a non-removable memory 712. Although the example computing device is illustrated and described as a computer 700, in different embodiments, the computing device may have different forms. For example, the computing device may alternatively be a smartphone, a tablet computer, a smart watch, or a computer including Figure 7 Other computing devices with the same or similar elements as shown. Devices such as smart phones, tablet computers, smart watches, etc. are generally referred to as mobile devices or user devices. In addition, although various data storage elements are shown as part of the computer 700, the memory 710 can also or alternatively include cloud-based storage accessed through a network 208 such as the Internet, or server-based storage.

[0097] The memory 703 may include a volatile memory 714 and a non-volatile memory 708. The computer 700 may include or access a computing environment. The computing environment includes various computer-readable media, such as the volatile memory 714 and the non-volatile memory 708, a removable memory 710 and a non-removable memory 712. Computer memory includes random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD ROM), digital versatile disc (DVD) or other optical disk storage, cassette tape, magnetic tape, disk storage or other magnetic storage device, or any other medium capable of storing computer-readable instructions.

[0098] The computer 700 may include or access a computing environment including an input interface 706, an output interface 704, and a communication connection 716. The output interface 704 may provide an interface to a display device such as a touch screen, and may also be used as an input device. The input interface 706 may provide an interface to one or more of the following devices: a touch screen, a touchpad, a mouse, a keyboard, a camera, one or more device-specific buttons, one or more sensors integrated in the computer 700 or coupled to the computer 700 via a wired or wireless data connection, and other input devices. The computer 700 may operate in a networked environment via a communication connection to connect to one or more remote computers, such as a database server. The remote computer may include a personal computer (PC), a server, a router, a network PC, a peer device, or other public network node, etc. The communication connection may include a local area network (LAN), a wide area network (WAN), a cellular, WiFi, Bluetooth, or other network 208.

[0099] Computer readable instructions stored on a computer readable medium can be executed by a processing unit 702 of a computer 700. Hard drives, CD-ROMs, RAM, and flash memory are some examples of products that include non-transitory computer readable media such as storage devices. The terms "computer readable medium" and "storage device" do not include carrier waves if carrier waves are considered too transient. The memory may also include networked memory, such as a storage area network (SAN) indicated at 720.

[0100] Although several embodiments are described in detail above, other modifications may be made thereto. For example, in order to obtain the desired results, the logic flows depicted in the accompanying drawings do not need to follow the specific order or sequential order shown. Other steps may be provided or some steps may be removed from the described processes, and other components may be added to or removed from the described systems. Other embodiments are intended to be within the scope of the appended claims.

Claims

1. A device for identifying at least one object in a data set, characterized in that The device comprises: Memory, including instructions; One or more processors in communication with the memory, wherein the one or more processors execute the instructions so as to receiving a data set and a query as to whether at least one object is included in the data set; Selecting at least one recognition model through an entity knowledge database including a plurality of entities corresponding to the object to be recognized, wherein each of the plurality of recognition models is linked to a plurality of entities of the entity knowledge database, and the N recognition models associated with the entity are used to recognize the object in the data set, the recognition model is a trained neural network, and N is an integer greater than or equal to 1; The data set is processed through the selected at least one recognition model to indicate whether the data set includes at least one object.

2. The device according to claim 1, characterized in that The one or more processors also execute instructions to Receiving the plurality of recognition models, wherein each recognition model includes at least one annotation; For each recognition model: based on at least one annotation of each individual model, identifying at least one entity in the entity knowledge database corresponding to the recognition model; The recognition model is linked to at least one entity in the entity knowledge database.

3. The device according to claim 1 or 2, characterized in that The at least one recognition model includes a plurality of recognition models, each of the plurality of recognition models includes at least one parameter indicating a computing resource for processing the data set using the recognition model; The one or more processors execute the instructions to process the data set by at least one recognition model of the plurality of recognition models, the at least one recognition model having indicated computing resources that are compatible with resources available to the one or more processors.

4. The device according to any one of claims 1 to 3, characterized in that The one or more processors execute the instructions to obtain additional resources from a network connection service before selecting the recognition model.

5. The device according to any one of claims 1 to 4, characterized in that The entity knowledge database is a graphic database; The entities of the entity knowledge graph database are ontology-coupled nodes of the entity knowledge graph database, such that a recognition model directly linked to a node in the entity knowledge database is linked to all nodes in the entity knowledge database that are ontology-coupled to the one node; The entity knowledge graph database includes nodes corresponding to the at least one object, the nodes corresponding to the at least one object not directly linked to the recognition model; The one or more processors execute the instructions to select one or more recognition models as the at least one recognition model, the one or more recognition models are directly linked to at least one node in the entity knowledge database, and the at least one node ontology is coupled to a node corresponding to the at least one object.

6. The device according to any one of claims 1 to 5, characterized in that The one or more processors execute the instructions so that selecting a plurality of recognition models linked to corresponding plurality of nodes coupled to a node ontology corresponding to the at least one object as one or more recognition models associated with the ontology-coupled node; processing the data set through the selected plurality of recognition models; Results processed by the selected plurality of recognition models are merged to indicate whether the data set includes the at least one object.

7. The device according to any one of claims 1 to 6, characterized in that The entity knowledge database is a graphic database; The entity knowledge graph database includes a plurality of ontology organization nodes corresponding to the at least one object with different generality levels, and the one or more processors execute the instructions so as to Selecting a plurality of recognition models associated with a plurality of nodes corresponding to different general levels of the at least one object as the at least one recognition model; the different general levels are used to indicate different categories; processing the data set through the selected plurality of recognition models; Results processed by the selected plurality of recognition models are merged to indicate whether the data set includes the at least one object.

8. A device for identifying an object in a data set, characterized in that: The device comprises: Memory, including instructions; One or more processors in communication with the memory, wherein the one or more processors execute the instructions so as to Receiving a plurality of recognition models, wherein each recognition model includes a plurality of annotations; According to the annotation of each recognition model in the received multiple recognition models, a link is established between each recognition model and multiple corresponding entities in the entity knowledge database, the N recognition models associated with the entity are used to recognize the object in the data set, the recognition model is a trained neural network, and N is an integer greater than or equal to 1; receiving a data set and a query to identify whether at least one object is included in the data set; Selecting at least one recognition model from the entity knowledge database to recognize the at least one object, wherein the at least one recognition model is linked to at least one entity in the entity knowledge database corresponding to the at least one object; The data set is processed through the selected at least one recognition model to indicate whether the data set includes at least one object.

9. The device according to claim 8, characterized in that The entity knowledge database is a graphic database; Ontological coupling between entities of an entity knowledge graph database, such that a recognition model directly linked to a node in the entity knowledge graph database is linked to all nodes in the entity knowledge graph database that are ontologically coupled to the one node; The entity knowledge database includes a node corresponding to the at least one object, and the node corresponding to the at least one object is not directly linked to any one of the plurality of recognition models; The one or more processors execute the instructions to select at least one recognition model as the at least one recognition model, the at least one recognition model body being coupled to a node corresponding to the at least one object and being linked to at least one corresponding recognition model of the plurality of recognition models.

10. The device according to claim 8 or 9, characterized in that The one or more processors execute the instructions so that identifying a plurality of identified recognition models of a plurality of node links coupled to a node body corresponding to the at least one object as the at least one identified recognition model; processing the data set through the selected plurality of identified recognition models; Results processed by the selected plurality of identified recognition models are merged to indicate whether the data set includes the at least one object.

11. The device according to any one of claims 8 to 10, characterized in that The entity knowledge database includes a plurality of ontology organization nodes corresponding to the at least one object with different generality levels, and the one or more processors execute the instructions so as to selecting a plurality of corresponding models associated with a plurality of nodes corresponding to different general levels of the at least one object as the identified plurality of recognition models; the different general levels are used to indicate different categories; processing the data set through the selected plurality of recognition models; Results processed by the selected plurality of identified recognition models are merged to indicate whether the data set includes the at least one object.

12. The device according to any one of claims 8 to 11, characterized in that The data set includes an image frame, and the at least one object includes an entity of the entity knowledge database.

13. A method for identifying objects in a data set, characterized in that: The method comprises: Searching for an entity knowledge database including a plurality of nodes corresponding to the object to be identified, wherein each of the plurality of recognition models is linked to a plurality of nodes of the entity knowledge database, the N recognition models associated with the entity are used to identify the object in the data set, the recognition model is a trained neural network, and N is an integer greater than or equal to 1; In response to searching the entity knowledge database, selecting at least one recognition model from the plurality of recognition models for recognizing at least one object; The data set is processed through the selected at least one recognition model to indicate whether the data set includes at least one object.

14. The method according to claim 13, characterized in that Also includes: Receiving the plurality of recognition models, wherein each recognition model includes at least one annotation; For each recognition model: Based on at least one annotation of each recognition model, identifying at least one entity in an entity knowledge database corresponding to the recognition model; The recognition model is linked to at least one entity in the entity knowledge database.

15. The method according to claim 13 or 14, characterized in that The at least one recognition model includes a plurality of recognition models, each of the plurality of recognition models includes at least one parameter indicating a computing resource for processing the data set using the recognition model; The method further includes processing the data set by at least one recognition model of the plurality of recognition models, the at least one recognition model having indicated computing resources that are compatible with resources available to the one or more processors.

16. The method according to any one of claims 13 to 15, characterized in that The method also includes obtaining additional resources from a network connection service before selecting the recognition model.

17. The method according to any one of claims 13 to 16, characterized in that The entity knowledge database is a graph database, which includes entities coupled to node ontology in the entity knowledge graph database, so that a recognition model directly linked to a node in the entity knowledge database is linked to all nodes in the entity knowledge database coupled to the one node ontology, the entity knowledge graph database includes nodes corresponding to the at least one object, and the nodes corresponding to the at least one object are not directly linked to the recognition model; The method further includes: selecting one or more recognition models as the at least one recognition model, the one or more recognition models are directly linked to at least one node in the entity knowledge database, and the at least one node ontology is coupled to a node corresponding to the at least one object.

18. The method according to any one of claims 13 to 17, characterized in that Also includes: selecting a plurality of recognition models linked to corresponding plurality of nodes coupled to a node ontology corresponding to the at least one object as one or more recognition models associated with the ontology-coupled node; processing the data set through the selected plurality of recognition models; Results processed by the selected plurality of recognition models are merged to indicate whether the data set includes the at least one object.

19. The method according to any one of claims 13 to 18, characterized in that The entity knowledge database is a graph database including a plurality of ontology organization nodes corresponding to the at least one object having different generality levels; The method further comprises: Selecting a plurality of recognition models associated with a plurality of nodes corresponding to different general levels of the at least one object as the at least one recognition model; the different general levels are used to indicate different categories; processing the data set through the selected plurality of recognition models; Results processed by the selected plurality of recognition models are merged to indicate whether the data set includes the at least one object.

20. A method for identifying objects in a data set, characterized in that: The method comprises: Receiving a plurality of recognition models, wherein each recognition model includes a plurality of annotations; According to the annotation of each recognition model in the received multiple recognition models, a link is established between each recognition model and multiple corresponding entities in the entity knowledge database, the N recognition models associated with the entity are used to recognize the object in the data set, the recognition model is a trained neural network, and N is an integer greater than or equal to 1; receiving a data set and a query to identify whether at least one object is included in the data set; Selecting at least one recognition model from the entity knowledge database to recognize the at least one object, wherein the at least one recognition model is linked to at least one entity in the entity knowledge database corresponding to the at least one object; The data set is processed through the selected at least one recognition model to indicate whether the data set includes at least one object.

21. The method according to claim 20, characterized in that The entity knowledge database is a graph database, which includes entities coupled to node ontology in the entity knowledge graph database, so that a recognition model directly linked to a node in the entity knowledge graph database is linked to all nodes in the entity knowledge graph database coupled to the one node ontology, the entity knowledge graph database includes nodes corresponding to the at least one object, and the nodes corresponding to the at least one object are not directly linked to any one of the multiple recognition models; The method further includes selecting at least one recognition model as the at least one recognition model, the at least one recognition model body being coupled to a node corresponding to the at least one object and being linked to at least one corresponding recognition model of the plurality of recognition models.

22. The method according to claim 20 or 21, characterized in that Also includes: identifying a plurality of identified recognition models of a plurality of node links coupled to a node body corresponding to the at least one object as the at least one identified recognition model; processing the data set through the selected plurality of identified recognition models; Results processed by the selected plurality of identified recognition models are merged to indicate whether the data set includes the at least one object.

23. The method according to any one of claims 20 to 22, characterized in that The entity knowledge database includes a plurality of ontology organization nodes corresponding to the at least one object having different generality levels; The method further comprises: selecting a plurality of corresponding models associated with a plurality of nodes corresponding to different general levels of the at least one object as the identified plurality of recognition models; the different general levels are used to indicate different categories; processing the data set through the selected plurality of recognition models; Results processed by the selected plurality of identified recognition models are merged to indicate whether the data set includes the at least one object.

24. The method according to any one of claims 20 to 23, characterized in that The data set includes an image frame, and the at least one object includes an entity of the entity knowledge database.

25. A non-transitory computer readable medium, characterized in that The device stores computer instructions for identifying an object in a data set, which, when executed by one or more processors, cause the one or more processors to: Searching for an entity knowledge database including a plurality of nodes corresponding to the object to be identified, wherein each of the plurality of recognition models is linked to a plurality of nodes of the entity knowledge database, the N recognition models associated with the entity are used to identify the object in the data set, the recognition model is a trained neural network, and N is an integer greater than or equal to 1; In response to searching the entity knowledge database, selecting at least one recognition model from the plurality of recognition models for recognizing at least one object; The data set is processed through the selected at least one recognition model to indicate whether the data set includes at least one object.

26. The non-transitory computer readable medium of claim 25, wherein: When the one or more processors execute the computer instructions, the one or more processors: Receiving the plurality of recognition models, wherein each recognition model includes at least one annotation; For each recognition model: Based on at least one annotation of each recognition model, identifying at least one entity in an entity knowledge database corresponding to the recognition model; The recognition model is linked to at least one entity in the entity knowledge database.

27. The non-transitory computer readable medium according to claim 25 or 26, wherein: The at least one recognition model includes a plurality of recognition models, each of the plurality of recognition models includes at least one parameter indicating a computing resource for processing the data set using the recognition model; When the one or more processors execute the computer instructions, the one or more processors are caused to process the data set through at least one recognition model of the multiple recognition models, and the at least one recognition model has indicated computing resources that are compatible with resources available to the one or more processors.

28. The non-transitory computer readable medium according to any one of claims 25 to 27, wherein: Also included are computer instructions that, when executed by the one or more processors, cause the one or more processors to obtain additional resources from a network connection service before selecting a recognition model.

29. The non-transitory computer readable medium according to any one of claims 25 to 28, wherein: The entity knowledge database is a graph database, which includes entities coupled to node ontology in the entity knowledge graph database, so that a recognition model directly linked to a node in the entity knowledge database is linked to all nodes in the entity knowledge database coupled to the one node ontology, the entity knowledge graph database includes nodes corresponding to the at least one object, and the nodes corresponding to the at least one object are not directly linked to the recognition model; When the one or more processors execute the computer instructions, the one or more processors select one or more recognition models as the at least one recognition model, the one or more recognition models are directly linked to at least one node in the entity knowledge database, and the at least one node ontology is coupled to a node corresponding to the at least one object.

30. The non-transitory computer readable medium according to any one of claims 25 to 29, wherein: Also included are computer instructions, which, when executed by the one or more processors, cause the one or more processors to: selecting a plurality of recognition models linked to corresponding plurality of nodes coupled to a node ontology corresponding to the at least one object as one or more recognition models associated with the ontology-coupled node; processing the data set through the selected plurality of recognition models; Results processed by the selected plurality of recognition models are merged to indicate whether the data set includes the at least one object.

31. The non-transitory computer readable medium according to any one of claims 25 to 30, wherein: The entity knowledge database is a graph database including a plurality of ontology organization nodes corresponding to the at least one object having different generality levels; When the one or more processors execute the computer instructions, the one or more processors: Selecting a plurality of recognition models associated with a plurality of nodes corresponding to different general levels of the at least one object as the at least one recognition model; the different general levels are used to indicate different categories; processing the data set through the selected plurality of recognition models; Results processed through the selected plurality of recognition models are combined to indicate whether the data set includes an indication of the at least one object.

32. A non-transitory computer readable medium, characterized in that The computer program product stores computer instructions for identifying an object in a data set, and when one or more processors execute the computer instructions, the one or more processors: Receiving a plurality of recognition models, wherein each recognition model includes a plurality of annotations; According to the annotation of each recognition model in the received multiple recognition models, a link is established between each recognition model and multiple corresponding entities in the entity knowledge database, the N recognition models associated with the entity are used to recognize the object in the data set, the recognition model is a trained neural network, and N is an integer greater than or equal to 1; receiving a data set and a query to identify whether at least one object is included in the data set; Selecting at least one recognition model from the entity knowledge database to recognize the at least one object, wherein the at least one recognition model is linked to at least one entity in the entity knowledge database corresponding to the at least one object; The data set is processed through the selected at least one recognition model to indicate whether the data set includes at least one object.

33. The non-transitory computer readable medium of claim 32, wherein: The entity knowledge database is a graph database, which includes entities coupled to node ontology in the entity knowledge graph database, so that a recognition model directly linked to a node in the entity knowledge graph database is linked to all nodes in the entity knowledge graph database coupled to the one node ontology, the entity knowledge graph database includes nodes corresponding to the at least one object, and the nodes corresponding to the at least one object are not directly linked to any one of the multiple recognition models; When the one or more processors execute the computer instructions, the one or more processors select at least one recognition model as the at least one recognition model, the at least one recognition model body being coupled to a node corresponding to the at least one object and being linked to at least one corresponding recognition model among the multiple recognition models.

34. The non-transitory computer readable medium according to claim 32 or 33, wherein: When the one or more processors execute the computer instructions, the one or more processors are also caused to: identifying a plurality of identified recognition models of a plurality of node links coupled to a node body corresponding to the at least one object as the at least one identified recognition model; processing the data set through the selected plurality of identified recognition models; Results processed by the selected plurality of identified recognition models are merged to indicate whether the data set includes the at least one object.

35. The non-transitory computer readable medium according to any one of claims 32 to 34, wherein: The entity knowledge database includes a plurality of ontology organization nodes corresponding to the at least one object having different generality levels; When the one or more processors execute the computer instructions, the one or more processors are also caused to: selecting a plurality of corresponding models associated with a plurality of nodes corresponding to different general levels of the at least one object as the identified plurality of recognition models; the different general levels are used to indicate different categories; processing the data set through the selected plurality of recognition models; Results processed by the selected plurality of identified recognition models are merged to indicate whether the data set includes the at least one object.

36. The non-transitory computer readable medium according to any one of claims 32 to 35, wherein: The data set includes an image frame, and the at least one object includes an entity of the entity knowledge database.

37. A device for identifying a data set object, characterized in that: The device comprises: A component, configured to search an entity knowledge database including a plurality of nodes corresponding to an object to be identified, wherein each of a plurality of recognition models is linked to a plurality of nodes of the entity knowledge database, the N recognition models associated with the entity are used to identify the object in the data set, the recognition model is a trained neural network, and N is an integer greater than or equal to 1; A component for selecting at least one recognition model from among the plurality of recognition models for recognizing at least one object in response to searching the entity knowledge database; A component is configured to process the data set through the selected at least one recognition model to indicate whether the data set includes at least one object.

38. The device according to claim 37, characterized in that Also includes: A component for receiving the plurality of recognition models, wherein each recognition model comprises at least one annotation; For each recognition model: A component for identifying at least one entity in an entity knowledge database corresponding to the recognition model based on at least one annotation of each recognition model; A component is used to link the recognition model to at least one entity in the entity knowledge database.

39. The device according to claim 37 or 38, characterized in that The at least one recognition model includes a plurality of recognition models, each of the plurality of recognition models includes at least one parameter indicating a computing resource for processing the data set using the recognition model; The apparatus further includes a component for processing the data set by at least one recognition model of the plurality of recognition models, the at least one recognition model having indicated computing resources that are compatible with resources available to the one or more processors.

40. The device according to claim 39, characterized in that Also included is a component for obtaining additional resources from a network connection service before selecting the recognition model.

41. The device according to any one of claims 37 to 40, characterized in that The entity knowledge database is a graph database, which includes entities coupled to node ontology in the entity knowledge graph database, so that a recognition model directly linked to a node in the entity knowledge database is linked to all nodes in the entity knowledge database coupled to the one node ontology, the entity knowledge graph database includes nodes corresponding to the at least one object, and the nodes corresponding to the at least one object are not directly linked to the recognition model; The apparatus further comprises: a component for selecting one or more recognition models as the at least one recognition model, the one or more recognition models being directly linked to at least one node in the entity knowledge database, the at least one node ontology being coupled to a node corresponding to the at least one object.

42. The device according to any one of claims 37 to 41, characterized in that Also includes: a component for selecting a plurality of recognition models linked to a corresponding plurality of nodes coupled to a node ontology corresponding to the at least one object as one or more recognition models associated with the ontology-coupled node; A component for processing the data set through the selected plurality of recognition models; A component is used to merge the results processed by the selected multiple recognition models to indicate whether the data set includes the at least one object.

43. The device according to any one of claims 37 to 42, characterized in that The entity knowledge database is a graph database including a plurality of ontology organization nodes corresponding to the at least one object having different generality levels; The device also includes: A component for selecting a plurality of recognition models associated with a plurality of nodes corresponding to different general levels of the at least one object as the at least one recognition model; the different general levels are used to indicate different categories; A component for processing the data set through the selected plurality of recognition models; A component is used to merge the results processed by the selected multiple recognition models to indicate whether the data set includes the at least one object.

44. A device for identifying a data set object, characterized in that: The device comprises: A component for receiving a plurality of recognition models, wherein each recognition model includes a plurality of annotations; A component, configured to establish a link between each recognition model and a plurality of corresponding entities in an entity knowledge database according to the received annotations of each recognition model in the plurality of recognition models, wherein the N recognition models associated with the entities are used to recognize objects in the data set, the recognition model is a trained neural network, and N is an integer greater than or equal to 1; A component for receiving a data set and a query for identifying whether at least one object is included in the data set; a component for selecting at least one recognition model from the entity knowledge database to recognize the at least one object, the at least one recognition model being linked to at least one entity in the entity knowledge database corresponding to the at least one object; A component is configured to process the data set through the selected at least one recognition model to indicate whether the data set includes at least one object.

45. The device according to claim 44, characterized in that The entity knowledge database is a graph database, which includes entities coupled to node ontology in the entity knowledge graph database, so that a recognition model directly linked to a node in the entity knowledge graph database is linked to all nodes in the entity knowledge graph database coupled to the one node ontology, the entity knowledge graph database includes nodes corresponding to the at least one object, and the nodes corresponding to the at least one object are not directly linked to any one of the multiple recognition models; The apparatus further comprises a component for selecting at least one recognition model as the at least one recognition model, the at least one recognition model body being coupled to a node corresponding to the at least one object and being linked to at least one corresponding recognition model of the plurality of recognition models.

46. ​​The device according to claim 44 or 45, characterized in that Also includes: A component for identifying a plurality of identified recognition models of a plurality of node links coupled to a node body corresponding to the at least one object as the at least one identified recognition model; a component for processing said data set through said selected plurality of identified recognition models; A component is used to merge the results processed by the selected multiple recognized recognition models to indicate whether the data set includes the at least one object.

47. The device according to any one of claims 44 to 46, characterized in that The entity knowledge database includes a plurality of ontology organization nodes corresponding to the at least one object having different generality levels; The device also includes: A component for selecting a plurality of corresponding models associated with a plurality of nodes corresponding to different general levels of the at least one object as the identified plurality of recognition models; the different general levels are used to indicate different categories; A component for processing the data set through the selected plurality of recognition models; A component is used to merge the results processed by the selected multiple recognized recognition models to indicate whether the data set includes the at least one object.

Images