Systems and methods for conductive out-of-domain learning
By creating associations between basic and customized concepts on a base model, and combining ignore lists and voting methods, the efficiency and accuracy issues of image classification in different domains are solved, achieving efficient out-of-domain learning.
Patent Information
- Application Number
- CN202380020822.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-02-10
- Filing Date
- 2023-02-10
- Publication Date
- 2024-11-26
AI Technical Summary
Existing technologies struggle to perform image classification tasks efficiently and accurately across different domains, especially the conversion from catalog images to real-world images. Furthermore, existing zero- or few-time learning methods lose contextual knowledge and some relationships.
By using a base model to generate the basic concepts for prediction, and by using a mapping module to create the association between the basic concepts and custom concepts, out-of-domain classification is achieved by combining an ignore list and a voting method.
It improves the efficiency and accuracy of out-of-domain learning, reduces processing memory requirements, increases training and inference speeds, and enhances overall performance.
Smart Images

Figure CN119032357A_ABST
Abstract
Description
Background Art
[0001] The fields of artificial intelligence and machine learning are advancing rapidly, with significant improvements in accuracy and efficiency. Despite these advances, it is still difficult to perform certain tasks accurately and computationally efficiently. For example, it is still difficult to use models trained in different domains to perform image classification tasks (sometimes referred to as "out-of-domain image distribution"). It is even more difficult to perform such image classification using zero-shot or few-shot learning methods. The ability to recognize and / or distinguish objects from a small number of samples is extremely difficult. Current zero-shot or few-shot techniques replace the last fully connected layer when applying transfer learning, thereby losing some contextual knowledge and part of the relationship field that exists between objects and is captured in these layers, hoping that they can be re-established via shallow or deep training.
[0002] It would be desirable to provide systems and methods for conductive out-of-domain learning. It would further be desirable to provide systems and methods that allow for classification of items of different modalities and that use less computation and increase interpretability than existing methods. Summary of the invention
[0003] According to some embodiments, systems, methods, and computer program codes for classifying inputs are provided. According to some embodiments, inputs from one domain may be classified using a model trained with inputs from different domains. In some embodiments, systems, methods, and computer program codes for classifying the input by applying a base model to the input are provided. The base model generates predictions of one or more first base concepts associated with the input. A determination is made regarding the presence of at least a first customized concept mapped to the first base concept and outputting the at least first customized concept as a classification of the input. According to some embodiments, the input may be from a domain different from a set of inputs used to train the base model. For example, the input may be from an image of the real world, and the base model may have been trained on catalog images, as will be further described below.
[0004] The technical effect of some embodiments of the present invention is to apply improved, cost-effective and efficient out-of-domain learning methods using the base model. The embodiments are illustrative, at least because the use of ignore lists and voting methods allows the contribution of each base concept to be identified. The embodiments are architecture-independent, and the method does not require backpropagation and can be used with limited knowledge. With these and other advantages and features that will become apparent hereinafter, a more comprehensive understanding of the present invention can be obtained by reference to the following detailed description and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0005] Figure 1 is a block diagram of a system according to some embodiments.
[0006] Figure 2 A process according to some embodiments is shown.
[0007] Figure 3 A process according to some embodiments is shown.
[0008] Figure 4 A portion of a user interface that may be used in accordance with some embodiments is shown.
[0009] Figure 5 A portion of a user interface that may be used in accordance with some embodiments is shown.
[0010] Figure 6 is a block diagram of a classification platform according to some embodiments. DETAILED DESCRIPTION
[0011] Businesses, merchants, and other users often wish to use artificial intelligence ("AI") or machine learning models (generally referred to herein as "models") to identify or classify features or items in images or videos. Typically, images from one domain are used to train the models used. It is then difficult to use these models to classify images from another domain. As a specific example that will be used in the following disclosure to illustrate features of the present invention, many models are trained on catalog images or images of items (such as products) placed on a white background. These catalog-style images typically display or position the product in the center of a white background. Classification models trained on such catalog-style images can be trained to highly accurately classify images of the same style (where the product or item in the image is centered on a white background). However, such models may not be as accurate when classifying images from a different domain (e.g., real-world images of products in use or in other environments).
[0012] Applicants have recognized that it is desirable to provide systems and methods that allow for the classification of items of different domains or modalities that use less computing power and increase interpretability than other methods. The terms "automatic" or "automatically" as used herein may refer to actions that may be performed, for example, with little or no human intervention.
[0013] Now we will first refer to Figure 1 To describe the features of some embodiments, Figure 11 is a block diagram of a system 100 according to some embodiments of the present invention. As shown, the system 100 includes a classification platform 120 that receives an input 102 (e.g., an image, a video, etc.) and generates outputs (stored as output data 136) such as metadata, attributes, etc., which are generated by processing the input 102 using features of the present invention. For example, the output data 136 may include data associated with the classification of the input 102 even when the input 102 is from a different domain or type than the input on which the model was trained (e.g., where the input 102 is an image in a real-world setting and the model is a model trained on catalog class images).
[0014] In some embodiments, the system 100 allows one or more users operating the user device 104 to interact with the classification platform 120 to perform classification processing on these inputs 102, as further described herein. The classification platform 120 includes one or more modules configured to perform processing to classify the input 102, as further described herein. According to some embodiments, the classification platform 120 includes a base model 112 (or communicates with the base model 112). For example, the base model 112 can be a classification model that has been trained and configured to accept input (e.g., an image or video), run predictions, and output one or more "concepts" predicted by the model. The concept represents the word that the model predicts to be associated with the input. The concept can be a specific concept such as "dog" or "cat" or an abstract concept such as "love". According to some embodiments, applying the base model to the input 102 results in outputting a set of concepts as output data associated with the input. In some embodiments, each concept can have a unique identifier, a name (e.g., a label or tag), and a confidence score that represents the confidence level that the model has in the predicted concept.
[0015] The classification platform 120 may include a mapping module 116 associated with the mapping data 134. The mapping module 116 may be operable to create a plurality of associations between concepts predicted using the base model and a plurality of custom concepts associated with inputs in different domains. Figure 2 and Figure 3 The description of the process is used to further describe the application and use of the mapping module 116 in the classification of out-of-domain input.
[0016] According to some embodiments, the system 100 includes components and interfaces that allow the creation of mapping data 134, which can be used to perform classification processing on inputs from different domains. Embodiments of the present invention provide out-of-domain processing that is performed with desired accuracy and efficiency and in a manner that provides illustrative results. The system 100 may generally be referred to as a "machine learning system" (or a portion thereof) herein. The system 100 may include one or more models that may be stored in, for example, a model database 132 and interact via components or controllers such as a model module 112. In some embodiments, one or more models may be so-called "classification" models that are configured to receive and process input 102 and generate output data 136. As used herein, the term "classification model" may include various machine learning models, including but not limited to "detection models" or "regression models". The embodiments may be used in conjunction with other models, and the use of classification models as illustrative examples is intended to be illustrative rather than restrictive. Therefore, the term "model" as used herein is used to refer to a variety of different types of models (from classification models to segmentation models, etc.).
[0017] For the sake of clarity and ease of illustration, the term "concept" is used herein to refer to the predicted output of a model. For example, in the context of a classification model, a "concept" can be a predicted classification of an input. Embodiments are not limited to use with models that produce "concepts" as outputs - instead, embodiments can be used with other model output types that are stored or written to memory for further processing with desired results. For convenience and ease of illustration, in order to illustrate the features of some embodiments, the term "confidence score" is used to refer to an indication of the confidence of the accuracy of the output of a model (e.g., a "concept" output from a model such as a classification model). A "confidence score" can be any indicator of the confidence or accuracy of the output from a model, and a "confidence score" is used as an example in this article. In some embodiments, the confidence score is used as an input to one or more threshold models to determine further processing actions as will be further described herein.
[0018] According to some embodiments, an “automatic” classification platform 120 may use mapping data 134 to automatically classify input 102 that is in different domains as further described herein.
[0019] In some embodiments, the user device 104 can interact with the classification platform 120 via a user interface (e.g., via a web browser), where the user interface is generated by the classification platform 120, and more specifically, by the user interface module 114. In some embodiments, the user device 104 can be configured with an application (not shown) that allows the user to interact with the classification platform 120. In some embodiments, the user device 104 can interact with the classification platform 120 via an application programming interface (“API”) and more specifically via the interface module 118. For example, the classification platform 120 (or other system associated with the classification platform 120) can provide one or more APIs for submitting the input 102 for processing by the classification platform 120.
[0020] For the purpose of illustrating the features of some embodiments, the use of a web browser interface will be described; however, those skilled in the art will understand after reading this disclosure that an API can be used to implement similar interactions. Figure 4 and Figure 5 Illustrative (but non-limiting) examples of portions of a web browser interface according to some embodiments are further described.
[0021] The system 100 may include various types of computing devices. For example, the user device 104 may be a mobile device (such as a smart phone), a tablet computer, a laptop computer, a desktop computer, or any other type of computing device that allows a user to interact with the classification platform 120 as described herein. The classification platform 120 may include one or more computing devices, including the following reference Figure 6 In some embodiments, the classification platform 120 includes multiple server devices and / or applications running on one or more server devices. For example, the classification platform 120 may include an application server, a communication server, a network hosting server, etc.
[0022] The devices of system 100 (including, for example, user device 104, input 102, classification platform 120, and databases 132, 134, and 136) can communicate using any communication platform and technology suitable for transmitting data and / or communication signals (including any known communication technology, equipment, media, and protocols that support data communication). For example, the devices of system 100 can exchange information via any wired or wireless communication network such as the Internet, an intranet, or an extranet. Note that any device described herein can communicate via one or more such communication networks.
[0023] although Figure 11 and 2, but any number of such devices may be included. In addition, the various devices described herein may be combined according to embodiments of the present invention. For example, in some embodiments, the classification platform 120 and the databases 132, 134, and 136 may be co-located and / or may include a single device.
[0024] The system 100 can be operated to facilitate out-of-domain classification of inputs in a manner that relates real-world concepts to concepts predicted by a machine learning model (and also identifies each concept's contribution to that relationship). The systems and methods of the present invention are highly scalable, architecturally free, and do not require backpropagation (unlike other cross-domain approaches such as transfer learning, zero-shot / few-shot learning, or ensemble learning). Compared to current transfer learning techniques (which do require backpropagation), embodiments require more than two times less processing memory, provide more than ten times faster training speeds, one and a half times faster inference speeds, and one-third better overall performance.
[0025] The preprocessing or training process 200 (in conjunction with Figure 2 ) and then describes the reasoning (or classification) process 300 (combined Figure 3 ) to describe features of some embodiments. Before making these descriptions, a brief illustrative (but non-limiting) example will first be introduced. In this illustrative example, an organization uses the system 100 of the present invention to classify images using a base model that has been trained using images primarily from a first domain (e.g., a catalog domain). The images to be classified are from another domain, such as a real-world image domain. According to some embodiments, the organization first operates the system 100 to perform a preprocessing or training process that creates a mapping or association between concepts from the base model and a set of customized concepts. The organization can then operate the system 100 to perform reasoning on the input. The system 100 maps the concepts output from the base model to customized concepts associated with real-world images, and outputs these customized concepts as the final result.
[0026] The final result can be output after applying the ranking and distribution process so that a set of final concepts are presented in association with each input image. As a specific illustrative and non-limiting example, the input image is such as Figure 5 An image such as image 502 shown includes a light bulb in the foreground and a desktop in the background behind the light bulb. That is, image 502 is from the "real world" domain. The model may have used Figure 4The model can be trained on catalog-style images such as the illustrated illustrative image 402 (where a light bulb is shown on a white background). These images can be referred to as being from the "catalog domain". Embodiments allow such a model to be enhanced using a mapping of concepts from the model to a set of customized concepts. This allows a model trained on images from a different domain (such as a catalog domain) to be used to classify images from a real-world (or other) domain.
[0027] Reference now Figure 2 , wherein it is shown that according to some embodiments of the present invention, Figure 1 The present invention relates to a preprocessing or training process 200 performed by some or all elements of the system 100 described herein. The flowcharts and process diagrams described herein do not imply a fixed order of steps, and embodiments of the present invention may be practiced in any order that is practicable. Note that any method described herein may be performed by hardware, software, or any combination of these methods. For example, a computer-readable storage medium may have instructions stored thereon that, when executed by a machine, produce performance according to any embodiment described herein.
[0028] Figure 2 The training process 200 can be performed via a user interface (e.g., by a user interacting with a user interface of the user device 104), via an API associated with the classification platform 120, or in some embodiments, as an automated or partially automated process. For simplicity and ease of illustration, the process 200 will be described as being configured via a user interface (although those skilled in the art will understand that each step can be performed via an API, etc.).
[0029] Process 200 begins at 202, where input is received. For example, in the illustrative example presented above, an image 402 of a light bulb may be received at 202. The image may be a catalog-style image of a light bulb (such as Figure 4 402). Processing continues at 204 where the input is provided to a model (e.g., a base model or a general model trained to classify a wide variety of inputs). The model operates on the input and generates (at 206) one or more predictions for the concepts shown in the input. For example, in this illustrative example, when an input such as from Figure 4 When given an image such as image 402 , the model may generate a set of predictions such as {light: 0.9, lamp: 0.6, electricity: 0.5, lighting: 0.7, glass: 0.5, excitation: 0.8 ...} (where each predicted concept has an associated confidence score ranging from 0 to 1).
[0030] Process 200 continues at 208, where one or more labels are provided to map to the predicted concepts. For example, the label "light bulb" can be provided as a custom concept at 208. At 210, the concepts predicted by the base model are mapped to the custom concepts provided at 208. The mapping between these concepts and the custom concepts can be a bidirectional mapping (e.g., a custom concept of "light bulb" can be mapped to the concepts of "light", "lamp", and "inspiration", and vice versa). Process 200 continues at 212, where a final mapping is created.
[0031] Creating the final mapping at 212 may include filtering the custom concepts mapped to the base concepts to retain only those custom concepts with confidence scores greater than a threshold amount (e.g., a threshold of the top 50% of accepted custom concepts). The remaining custom concepts (i.e., concepts that passed the filtering) may be associated with new confidence scores ranging from 0 to 1. For each custom concept, processing may also include adding its predicted count and its confidence score to generate a mapping between the custom concept and the base concept.
[0032] In some embodiments, creating the final mapping at 212 may also include aggregating the results and creating an ignore list by filtering the results and creating a voting scheme. For example, in some embodiments, if a base concept is associated with a relatively large list of custom concepts (or a number of custom concepts exceeding a desired amount), a voting scheme may be run to create an ignore list of custom concepts mapped to the base concept (thereby reducing the number of custom concepts mapped to the base concept). The final mapping generated at 212 is then stored at 214 for use in, for example, Figure 3 The process 200 may be repeated for multiple inputs 202 and multiple labels to create a rich set of mapping data 214.
[0033] Now refer to Figure 3 Describe the use of the mapping data, Figure 3 is an inference process 300 that may be performed by the system 100 according to some embodiments. For example, the process 300 may be performed after a set of mapping data has been created for a particular application or use case (e.g., such as the illustrative but non-limiting application described above, where a base model trained on images from a catalog domain is to be used to perform classification of images in a real-world domain).
[0034] Figure 3The reasoning process 300 can be performed via a user interface (e.g., by a user interacting with a user interface of the user device 104), via an API associated with the classification platform 120, or in some embodiments, as an automated or partially automated process. For simplicity and ease of illustration, the process 300 will be described as being configured via a user interface (although those skilled in the art will understand that each step can be performed via an API, etc.).
[0035] The reasoning process 300 begins at 302 where an input is received. For example, continuing with the illustrative example presented above, the input may be a real-world image such as the image 502 of a light bulb. For purposes of illustrating features of the present invention, it may be assumed that the input received at 302 is an input from a different domain than the input used to train the base model (such as an image, video, etc.). Processing continues at 304 where the base model is operated 304 to operate on the input and generate (at 306) one or more predictions for the concepts shown in the input. For example, in this illustrative example, when an input such as from 502 is received at 302, the base model may generate (at 306) one or more predictions for the concepts shown in the input. Figure 5 When an image such as image 502 of FIG. 5 is viewed, the model may generate a set of predictions such as {light: 0.8, lamp: 0.3, electricity: 0.6, lighting: 0.4, glass: 0.9, excitation: 0.4} (where each predicted concept has an associated confidence score ranging from 0 to 1). This set of predictions may generally be referred to herein as predictions of “base concepts” (e.g., concepts predicted by the operation of the base model at 304).
[0036] Processing continues at 310 where the mapping data stored at 308 is used (e.g., via Figure 2 Continuing with this illustrative example, Figure 2 The process 200 may have resulted in the creation of mapping data indicating a relationship between the base concept of "light" and the custom concepts of "bulb" and "lamp". In addition, the mapping data may indicate a relationship between the base concept of "inspiration" and the custom concepts of "bulb" and "bookshelf" and a relationship between the base concept of "no one" and the custom concepts of "bulb", "lamp", and "bookshelf". In some embodiments, at 310, each concept predicted by the base model at 306 is mapped to any corresponding custom concept. The resulting set of custom concepts is then processed by re-ranking and filtering (at 314) to produce a final result set at 316.
[0037] According to some embodiments, re-ranking and filtering at 314 may include: identifying an ignore list associated with the base concept, and applying the ignore list to remove any predictions of the ignore list. The remaining set of customized concepts is then re-ranked based on their confidence scores to produce a set of customized concepts with associated confidence scores. In this way, an embodiment provides a cost-effective and efficient out-of-domain learning method applied using a base model. The embodiment is illustrative, at least because the use of ignore lists and voting methods can allow the contribution of each base concept to be identified. The embodiment is architecture-independent, and the method does not require back-propagation and can be used with limited knowledge.
[0038] Any number of different hardware configurations may be used to implement the embodiments described herein. For example, Figure 6 A classification platform 600 is shown, which can be used, for example, with Figure 1 The classification platform 600 includes a processor 610 (e.g., one or more commercially available central processing units (CPUs) in the form of microprocessors) coupled to a communication device 620, which is configured to communicate via a communication network ( Figure 6 The classification platform 600 may also include an input device 640 (e.g., a mouse and / or keyboard for defining rules and relationships) and an output device 650 (e.g., a computer monitor for displaying reports and results to an administrator).
[0039] The processor 610 also communicates with a storage device 630. The storage device 630 may include any suitable information storage device, including a combination of a magnetic storage device (e.g., a hard drive), an optical storage device, a mobile phone, and / or a semiconductor memory device. The storage device 630 stores a program 612 for controlling the processor 610 and / or one or more software modules 614 (e.g., Figure 1 The processor 610 executes the instructions of the programs 612 and 614 to operate according to any of the embodiments described herein. For example, the processor 610 may receive input data and then perform processing on the input data, such as combining Figure 2 and Figure 3 The procedures 612, 614 may access, update, and otherwise interact with data such as model data 616, mapping data 618, and label data 620 as described herein. In some embodiments, one or more ignore lists may also be stored at or accessible by the system (e.g., defining concepts to be ignored in different mapping relationships).
[0040] Programs 612, 614 may be stored in a compressed, uncompiled, and / or encrypted format. Programs 612, 614 may also include other program elements, such as an operating system, a database management system, and / or device drivers used by processor 610 to interface with peripheral devices.
[0041] As used herein, information may be “received” or “sent”, for example: (i) from another device to the classification platform 600 ; or (ii) from another software application, module, or any other source to a software application or module within the classification platform 600 .
[0042] Various additional embodiments of the present invention are described below. These additional embodiments do not constitute a limitation of all possible embodiments, and those skilled in the art will understand that the present invention is also applicable to many other embodiments. In addition, although the following embodiments are briefly described for clarity, those skilled in the art will understand how to make any changes (if necessary) to the above-mentioned apparatus and methods to adapt to these and other embodiments and applications.
[0043] Although specific hardware and data configurations have been described herein, note that any number of other configurations may be provided in accordance with embodiments of the present invention (eg, some of the information associated with the databases described herein may be combined or stored in external systems).
[0044] The present invention has been described by several embodiments for illustrative purposes only. Those skilled in the art will recognize from this description that the present invention is not limited to the embodiments described, but can be practiced with modifications and changes limited only by the spirit and scope of the appended claims.
Claims
1. A computer-implemented method for classifying an input, the method comprising: applying the base model to the input; predicting at least a first base concept associated with the input; determining that there is at least a first customized concept mapped to the at least first base concept; as well as The at least first customized concept is output as a classification of the input.
2. The computer-implemented method of claim 1 , wherein: Predicting at least a first base concept associated with the input further includes generating a confidence score associated with the at least first base concept.
3. The computer-implemented method of claim 1 , wherein: Predicting at least a first base concept associated with the input further comprises: generating a first confidence score associated with the at least first base concept; predicting at least a second base concept associated with the input; and A second confidence score associated with the at least a second base concept is generated.
4. The computer-implemented method of claim 3, further comprising: It is determined that there is at least a second custom concept mapped to the at least second base concept.
5. The computer-implemented method of claim 4, wherein: The output also includes: The at least second customized concept is output as a classification of the input.
6. The computer-implemented method of claim 5, further comprising: The at least first customized concept and the second customized concept are re-ranked to output the highest ranked concept first.
7. The computer-implemented method of claim 1 , wherein: Determining that there is at least a first customized concept mapped to the at least first base concept further comprises: querying a mapping data structure using the at least first base concept; and The at least first customization concept is received.
8. The computer-implemented method of claim 8, wherein: The mapping data structure includes a plurality of base concepts including the at least first base concept.
9. The computer-implemented method of claim 9, wherein: For each base concept of the plurality of base concepts, the mapping data structure includes information identifying one or more corresponding custom concepts.
10. The computer-implemented method of claim 9, wherein: For the one or more corresponding custom concepts, the mapping data structure further includes a confidence score indicating a confidence of a relationship between the one or more corresponding custom concepts and the associated base concept.
11. The computer-implemented method of claim 4, further comprising: The at least first customized concept and the at least second customized concept are compared to an ignore list to determine whether to ignore the at least first customized concept or the at least second customized concept.
12. A system comprising: Processing unit; as well as a memory storage device comprising program code which, when executed by the processing unit, causes the system to: Apply the base model to the input; predicting at least a first base concept associated with the input; determining that there is at least a first customized concept mapped to the at least first base concept; as well as The at least first customized concept is output as a classification of the input.
13. The system according to claim 12, wherein: The input is one of an image and a video.
14. The system according to claim 12, wherein: Predicting at least a first base concept associated with the input also includes program code for: generating a first confidence score associated with the at least first base concept; predicting at least a second base concept associated with the input; and A second confidence score associated with the at least a second base concept is generated.
15. The system of claim 14, further comprising program code that, when executed by the processing unit, causes the system to: It is determined that there is at least a second custom concept mapped to the at least second base concept.
16. The system of claim 15, further comprising program code that, when executed by the processing unit, causes the system to: The at least second customized concept is output as a classification of the input.
17. The system of claim 16, further comprising program code that, when executed by the processing unit, causes the system to: The at least first customized concept and the second customized concept are re-ranked to output the highest ranked concept first.
18. The system of claim 1, wherein: The program code for determining that there is at least a first customized concept mapped to the at least first base concept also includes program code for: querying a mapping data structure using the at least first base concept; and The at least first customization concept is received.
19. The system of claim 18, wherein: The inputs are from a different domain than a set of inputs used to train the base model.
