Method and system for energy-based sample-model retrieval

Abstract

The invention discloses a method and electronic equipment for generating search results. A method includes: obtaining a set of data samples from a sample database; obtaining a group of pre-training models from a model database; an energy score is generated for a given model in the set of pre-trained models using the set of data samples by an energy function. The method includes ranking the set of pre-trained models using the energy scores for the given model, thereby generating a ranked column of pre-trained models. The method comprises the steps that the column of ranked pre-training models is used for generating the search result, the search result comprises the given model, and the given model is the highest-ranked model in the column of ranked pre-training models. Another method includes a model-based sample search in which one or more data samples are obtained as search results for a given model by an energy score calculation.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This is the first application related to the technology described herein. Technical Field

[0003] The present technology relates to sample-model retrieval, and more particularly, to methods and systems for energy-based sample-model retrieval. Background Art

[0004] Due to the widespread use of deep neural networks, an increasing number of pre-trained models have been generated by training on datasets from various sources. These pre-trained models can be used to provide a good warm-start for fine-tuning the target task, especially in transfer learning. Broadly speaking, "warm-start" is a method in machine learning that allows training or fine-tuning to begin from a better initial point on the loss surface, potentially leading to the learning of a higher-performing model. Furthermore, "transfer learning" refers to a machine learning method in which a pre-trained model is reused as the starting point for another model on a new task.

[0005] In some cases, it is necessary to find a model that best suits a given dataset. Given a target task (or dataset) and many machine learning models, the "model search" problem is defined as the process of finding, ranking, and selecting the best model by evaluating all given models for the target task (or dataset). Model search can also be used to perform transferability assessments to predict fine-tuning results. A simple solution to this problem is brute-force fine-tuning, which requires fine-tuning all given pre-trained models on a given dataset and then selecting the fine-tuned model with the best performance for the target task. In other cases, it is necessary to find the data sample that best suits a given model. Given a database of input models and data samples, the "sample search" framework is used to search and find samples suitable for fine-tuning a given model.

[0006] Existing solutions to model search and sample search problems are time-consuming and computationally expensive, so there is a need to improve these techniques. Summary of the Invention

[0007] Developers have devised methods and apparatuses to overcome at least some of the shortcomings of prior art solutions.

[0008] It should be noted that searching and ranking pre-trained models is challenging because selecting the best pre-trained model is often task-specific. The developers of this technology have recognized that traditional approaches to "model search" can be divided into two types: (i) heuristic methods, such as the Log Expectation of the Empirical Predictor (LEEP) and Negative Conditional Entropy (NCE), among others; and (ii) learning-based methods, such as the Logarithm of Maximum Evidence (LogME), PacTran, and SFDA. In LEEP, an empirical predictor is constructed by empirically computing the joint distribution of pre-trained and target labels on a given dataset. The logarithm of the empirical predictor is then used as a transferability metric. In NCE, conditional entropy is used between the labeled sequences of the source task and the labeled sequences of the target task to study the transferability between these labeled sequences. In LogME, evidence is defined as the marginalized likelihood, and the logarithm of the maximum evidence is then used to evaluate pre-trained models for transfer learning. In other words, LogME estimates the maximum value of the labeled evidence based on the extracted features and uses this maximum value to evaluate the model. In PacTran, the PAC-Bayesian bound is used to measure transferability. The PAC-Bayesian bound uses cross-entropy loss to measure generalization error. In SFDA, the static features obtained from the pre-trained model are transformed into the Fisher space, and then the static features are optimized to enhance the separability between categories. Next, SFDA uses a self-challenge mechanism to encourage different pre-trained models to distinguish on samples that are difficult to evaluate.

[0009] Developers have recognized that the frameworks LEEP and NCE have at least some shortcomings when solving the model search problem. LEEP and NCE require the entire dataset to be labeled and cannot handle models trained in an unsupervised setting. These solutions are limited to specific architecture types (only ResNet), tasks (only image classification), and data types (only images). While these solutions have lower computational complexity, their performance in finding relevant models could be improved. Developers have recognized that the frameworks LogME, PacTran, and SFDA have at least some shortcomings when solving the model search problem. LogME, PacTran, and SFDA require training a neural network. This means that LogME, PacTran, and SFDA require the entire training dataset to be labeled and have high computational complexity.

[0010] It is important to note that some applications of example search include data evaluation, pruning, and example selection for unsupervised, self-supervised, and semi-supervised learning techniques. The developers of this technology are aware of traditional solutions to the "example search" problem, such as the variance of the pre-softmax activation gradient (VOG), GraNd, and the complexity gap score (CG score). In VoG, the difficulty of learning an example is determined by the variance of the pre-softmax activation gradient indexed at the predicted / true label of the input. In other words, it is believed that it is difficult for the model to learn examples with high VoG scores. In GraNd, the importance of each training example is scored using the expected value of the loss gradient norm. GraNd identifies a small set of training data that can achieve generalization. The CG score is a training-free data evaluation method used to measure the impact of individual examples on the optimization and generalization of a neural network.

[0011] Developers have recognized that the VoG framework has at least some shortcomings when solving the sample search problem. VoG requires labeling of the entire training dataset and requires a high-complexity learning process. Furthermore, VoG can only be applied to a limited set of architectures whose performance can be improved. Another disadvantage is that the VoG approach cannot be used as a plug-and-play module for search engines. Developers have recognized that the GraNd framework has at least some shortcomings when solving the sample search problem. In addition to the aforementioned shortcomings of VoG, GraNd is also limited to a certain type of task (i.e., classification). Developers have recognized that the CG score framework has at least some shortcomings when solving the sample search problem. CG methods require data labeling of the dataset and therefore cannot be used in unsupervised settings. Additionally, CG methods are only applicable to certain architectures, tasks (classification only), and data types (images only). Another disadvantage of the CG score approach is that it is not a plug-and-play solution, which makes its integration with existing search frameworks challenging.

[0012] In the context of the present technology, methods and electronic devices for performing "sample-based model" searches are provided. Broadly speaking, sample-based model search is a process where the goal is to search for and find the best performing model given one or more data samples. These data samples may be of different types, such as documents (text), images, audio, video, and so on. In one example, given some dog images, the goal is to find the top image classifier that classifies these images very accurately. As will be apparent from the further description below, in some embodiments of the present technology, the data samples used to perform sample-based model searches can be unlabeled data samples.

[0013] In the context of the present technology, methods and electronic devices for performing "model-based sample" search are provided. Broadly speaking, model-based sample search is a process that, given a pre-trained model for a task, searches for and identifies samples that have acceptable performance for that given model. In some embodiments, the retrieved samples can then be used to fine-tune the model's performance. In one example, given a pre-trained model for dog recognition, the goal is to find suitable unlabeled samples with high pseudo-labeling performance to enable semi-supervised learning.

[0014] As will be apparent from the further description below, a search engine is provided for performing at least one of the following methods: (i) sample-based model search and (ii) model-based sample search, wherein "energy scores" are used to rank pre-trained models and / or data samples. Figure 6 , Figure 6 A comparison table is shown between the functionality of at least some prior art solutions and the methods and electronic devices contemplated in at least non-limiting embodiments of the present technology.

[0015] In some embodiments, the search engine can be a high-performance energy-based solution for at least one of sample-based model search and model-based sample search. In other words, the search engine can include an energy-based decision module for finding the best matching model and / or sample.

[0016] In other embodiments, the search engine may be a dataset-free solution, meaning that performing a model search and / or a sample search does not necessarily require obtaining the entire training dataset of the data samples.

[0017] In another embodiment, the search engine can be a training-free solution, which means that the search engine can operate pre-trained models without the need for training / pre-training or fine-tuning of these models. The entire model and sample search process can be performed in the inference phase of the corresponding model.

[0018] In a further embodiment, the search engine can be a label-free solution, meaning that labels are not necessarily required to perform the model search and / or the sample search. The search engine can be used to perform the search in a completely unsupervised manner.

[0019] In some other embodiments, the search engine can be an architecture-independent solution, which means that the search engine can be used for various models regardless of the neural architecture of these models. For example, the search engine can be used for ResNet models, DenseNet models, Transformer-based models, and so on. In a broad sense, "neural network" can refer to a system of neurons that attempts to identify potential relationships in a set of data by mimicking the way the human brain works. In a broad sense, "ResNet" is an artificial neural network that introduces "identity shortcut connections" that allow the model to skip one or more layers. In a broad sense, "DenseNet" is a densely connected convolutional network that uses all previous outputs as input to future layers. In a broad sense, "Transformer" is a model that uses a self-attention mechanism and differentially weights each part of the input data. Computer vision and natural language processing are the two areas where transformers are most widely used.

[0020] In another embodiment, the search engine can be a task-independent and / or modality-independent solution, which means that the search engine can be used without considering: (i) a specific machine learning task, such as classification, detection, and generation, etc., and / or (ii) a specific data sample modality (type), such as image, video, audio, and text data, etc.

[0021] In some embodiments, a search engine can be used to operate on models that have been pre-trained using supervised and / or unsupervised methods. Broadly speaking, "machine learning" refers to a type of artificial intelligence that enables software programs to make more accurate predictions without being explicitly programmed. The process of machine learning is performed by taking historical data as input and predicting future outcomes. Broadly speaking, a "pre-trained model" can refer to a model or saved network that has been trained on a large dataset to solve a problem. This model undergoes a training process in which an algorithm uses historical data to extract patterns and learn to distinguish these patterns in underlying data. Broadly speaking, a "dataset" can refer to a collection of data used to train a model. A dataset is used as an example to teach a machine learning algorithm how to make predictions. Broadly speaking, "fine-tuning" can refer to adjusting a process to achieve a desired output or performance. For example, in deep learning, the weights of an existing deep learning algorithm can be fine-tuned to design another deep learning process.

[0022] In the context of the present technology, a search engine can be used to perform an energy function on one or more data samples and one or more pre-trained models to generate an energy score for ranking the one or more data samples and / or one or more pre-trained models. The developers of the present technology have recognized that using the energy function to perform the ranking operation can be used as a plug-and-play solution, meaning that the energy function can be integrated with existing model search engines and / or sample search engines. It can also be said that without departing from the scope of the present technology, the energy score calculation used by the search can be performed in parallel on a batch of samples.

[0023] In the context of this technology, "object detection" may refer to a computer technology related to computer vision and image processing that involves detecting instances of a class of semantic objects in digital images and / or videos. In the context of this technology, "downstream tasks" may refer to fine-tuning tasks that inherit models and parameters from pre-trained models. In the context of this technology, "transferability evaluation" may refer to the process of evaluating transferability—the ability to acquire, transfer, and reuse knowledge from one model to another. In the context of this technology, "generalization" may refer to the ability of a model to adapt to new, previously unavailable data drawn from the same distribution as the one used to create the model. In the context of this technology, "computer vision" may refer to the field of artificial intelligence (AI) that enables computers to acquire information from images and videos. In the context of this technology, "natural language processing" may refer to the interdisciplinary subfield of linguistics, computer science, and AI that involves the interaction between computers and human language. In the context of this technology, "data pruning" may refer to the process of removing suboptimal tuples from a dataset to improve the learning of a machine learning model. In the context of the present technology, "supervised learning" may refer to a machine learning paradigm used to solve problems where the available data consists of labeled examples, meaning that each data point includes features and an associated label. In the context of the present technology, "unsupervised learning" may refer to techniques used by machine learning algorithms to analyze and cluster unlabeled data sets. In the context of the present technology, "self-supervised learning" may refer to a machine learning process in which a model trains itself to learn about one part of the input from another part of the input. In the context of the present technology, "semi-supervised learning" may refer to a learning problem that involves a small set of labeled examples and a large set of unlabeled examples from which the model must learn and make predictions about new examples. In the context of the present technology, "image classification" may refer to the classification and labeling of different groups of images.

[0024] In the context of the present technology, "likelihood" can be used to find the best distribution of data for a feature or situation in the data given a specific value of the feature or situation. In other words, likelihood is an indicator of the probability of an event occurring, which can be used to find the best matching model / sample. In the context of the present technology, "density function" can refer to a tool used by machine learning for training, which can calculate the probability of a continuous random variable. In the context of the present technology, "softmax function" can refer to a function that transforms a vector of K real values ​​into a vector of K real values ​​that sum to 1. In the context of the present technology, "confidence score" can refer to a score that quantifies the confidence of a machine learning model in its predictions. In the context of the present technology, "softmax confidence score" can refer to a score calculated using a softmax function.

[0025] In a first broad aspect of the present technology, a method for generating search results is provided. The method is performed by an electronic device. The electronic device is communicatively coupled to a sample database and a model database. The method includes: the electronic device obtains a set of data samples from the sample database. The method includes: the electronic device obtains a set of pre-trained models from the model database. The method includes: the electronic device uses an energy function and the set of data samples to generate an energy score for a given model in the set of pre-trained models. The method includes: the electronic device uses the energy score of the given model to rank the set of pre-trained models, thereby generating a list of ranked pre-trained models. The method includes: the electronic device uses the list of ranked pre-trained models to generate the search results, wherein the search results include the given model, and the given model is the highest-ranked model in the list of ranked pre-trained models.

[0026] In some embodiments of the method, the set of data samples includes at least one of text samples, image samples, audio samples, and video samples.

[0027] In some embodiments of the method, the set of data samples is a set of unlabeled data samples.

[0028] In some embodiments of the method, the set of pre-trained models includes at least one of a ResNet model, a DenseNet model, and a Transformer-based model.

[0029] In some embodiments of the method, the method further includes: the electronic device fine-tuning the target model using the set of data samples.

[0030] In some embodiments of the method, the energy function is:

[0031] Where L = f i (x j )

[0032] Among them, f i is the given model, x j is a given data sample among the plurality of data samples, l is a pair (x j ,f i ).

[0033] In some embodiments of the method, the energy fraction is:

[0034]

[0035] Among them, s i is the energy score of the given model, X is the plurality of data samples, and N is the number of data samples in the plurality of data samples.

[0036] In some embodiments of the method, the method further includes: obtaining a text query by the electronic device; and generating a text-based score for the given model using the text query and text information associated with the given model. The ranking includes ranking the set of pre-trained models using the energy score and the text-based score.

[0037] In a second broad aspect of the present technology, a method for generating search results is provided. The method is performed by an electronic device. The electronic device is communicatively coupled to a sample database and a model database. The method includes: the electronic device obtains a set of data samples from the sample database. The method includes: the electronic device obtains a pre-trained model from the model database. The method includes: the electronic device uses an energy function and the pre-trained model to generate an energy score for a given sample in the set of data samples. The method includes: the electronic device uses the energy score of the given data sample to rank the set of data samples, thereby generating a list of ranked data samples. The method includes: the electronic device uses the list of ranked data samples to generate the search results, wherein the search results include the given data sample, and the given data sample is the highest-ranked data sample in the list of ranked data samples.

[0038] In some embodiments of the method, the set of data samples includes at least one of text samples, image samples, audio samples, and video samples.

[0039] In some embodiments of the method, the set of data samples is a set of unlabeled data samples.

[0040] In some embodiments of the method, the pre-trained model is at least one of a ResNet model, a DenseNet model, and a Transformer-based model.

[0041] In some embodiments of the method, the method further includes: the electronic device fine-tuning the pre-trained model using the given data sample.

[0042] In some embodiments of the method, the energy function is:

[0043] Where L = f i (x j )

[0044] Among them, E(x j ;f i ) is the given data sample x j The energy fraction, f i is the pre-training model, l is a pair (x j ,f i ).

[0045] In some embodiments of the method, the method further includes: obtaining a text query by the electronic device; generating a text-based score for the given data sample using the text query and text information associated with the given data sample. The ranking includes ranking the set of data samples using the energy score and the text-based score by the electronic device.

[0046] In a third broad aspect of the present technology, an electronic device for generating search results is provided. The electronic device is communicatively coupled to a sample database and a model database. The electronic device is used to obtain a set of data samples from the sample database. The electronic device is used to obtain a set of pre-trained models from the model database. The electronic device is used to generate an energy score for a given model in the set of pre-trained models using an energy function and the set of data samples. The electronic device is used to rank the set of pre-trained models using the energy scores of the given model, thereby generating a list of ranked pre-trained models. The electronic device is used to generate the search results using the list of ranked pre-trained models, wherein the search results include the given model, and the given model is the highest-ranked model in the list of ranked pre-trained models.

[0047] In some embodiments of the electronic device, the set of data samples includes at least one of a text sample, an image sample, an audio sample, and a video sample.

[0048] In some embodiments of the electronic device, the set of data samples is a set of unlabeled data samples.

[0049] In some embodiments of the electronic device, the set of pre-trained models includes at least one of a ResNet model, a DenseNet model, and a Transformer-based model.

[0050] In some embodiments of the electronic device, the method further includes: the electronic device fine-tuning the target model using the set of data samples.

[0051] In some embodiments of the electronic device, the energy function is:

[0052] Where L = f i (x j )

[0053] Among them, f i is the given model, x j is a given data sample among the plurality of data samples, l is a pair (x j ,f i ).

[0054] In some embodiments of the electronic device, the energy fraction is:

[0055]

[0056] Among them, s i is the energy score of the given model, X is the plurality of data samples, and N is the number of data samples in the plurality of data samples.

[0057] In some embodiments of the electronic device, the electronic device is further configured to: obtain a text query; and generate a text-based score for the given model using the text query and text information associated with the given model. The electronic device being configured to perform ranking includes the electronic device being configured to rank the set of pre-trained models using the energy score and the text-based score.

[0058] In a fourth broad aspect of the present technology, an electronic device for generating search results is provided. The electronic device is communicatively coupled to a sample database and a model database. The electronic device is configured to obtain a set of data samples from the sample database. The electronic device is configured to obtain a pre-trained model from the model database. The electronic device is configured to generate an energy score for a given sample in the set of data samples using an energy function and the pre-trained model. The electronic device is configured to rank the set of data samples using the energy scores of the given data samples, thereby generating a list of ranked data samples. The electronic device is configured to generate the search results using the list of ranked data samples, wherein the search results include the given data sample, and the given data sample is the highest-ranked data sample in the list of ranked data samples.

[0059] In some embodiments of the electronic device, the set of data samples includes at least one of a text sample, an image sample, an audio sample, and a video sample.

[0060] In some embodiments of the electronic device, the set of data samples is a set of unlabeled data samples.

[0061] In some embodiments of the electronic device, the pre-trained model is at least one of a ResNet model, a DenseNet model, and a Transformer-based model.

[0062] In some embodiments of the electronic device, the method further includes: the electronic device fine-tuning the pre-trained model using the given data sample.

[0063] In some embodiments of the electronic device, the energy function is:

[0064] Where L = f i (x j )

[0065] Among them, E(x j ;f i ) is the given data sample x j The energy fraction, f i is the pre-training model, l is a pair (x j ,f i ).

[0066] In some embodiments of the electronic device, the electronic device is further configured to: obtain a text query; and generate a text-based score for the given data sample using the text query and text information associated with the given data sample. Configuring the electronic device to perform ranking includes configuring the electronic device to rank the set of data samples using the energy score and the text-based score.

[0067] In the context of this specification, a "server" is a computer program running on suitable hardware that is capable of receiving requests (e.g., requests from devices) over a network and executing those requests, or causing those requests to be executed. The hardware may be a physical computer or a physical computer system, but for the purposes of the present technology, it does not have to be either. In this context, the use of the term "server" does not mean that every task (e.g., received instruction or request) or any particular task is received, executed, or caused to be executed by the same server (i.e., the same software and / or hardware); rather, it means that any number of software elements or hardware devices may be involved in receiving / sending, executing, or causing the execution of any task or request, or the consequences of any task or request; all of which may be a single server or multiple servers, both of which are included in the term "at least one server."

[0068] In the context of this specification, a "device" is any piece of computer hardware capable of running software appropriate for the task at hand. Thus, some (non-limiting) examples of devices include personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, as well as network devices such as routers, switches, and gateways. It should be noted that a device used as a device in this context does not exclude its use as a server for other devices. The use of the term "device" does not exclude multiple devices being used to receive / send, perform, or cause the performance of any task or request, or the consequences of any task or request, or the steps of any method described herein.

[0069] In the context of this specification, a "database" is any structured collection of data, regardless of the database's specific structure, the database management software, or the computer hardware that stores, implements, or otherwise presents the data for use. A database can be located on the same hardware as the processes that store or use the information stored in the database, or it can be located on separate hardware, such as a dedicated server or multiple servers. Suffice it to say that a database is a logically ordered collection of structured data stored electronically in a computer system.

[0070] In the context of this specification, the expression "information" includes any nature or kind of information that can be stored in a database. Thus, information includes, but is not limited to, audiovisual works (images, films, recordings, presentations, etc.), data (location data, numerical data, etc.), text (opinions, comments, questions, messages, etc.), documents, spreadsheets, word lists, etc.

[0071] In the context of this specification, the expression "component" is intended to include software (appropriate to the context of specific hardware) that is necessary and sufficient for realizing the specific function or functions mentioned.

[0072] In the context of this specification, the expression "computer-usable information storage medium" is intended to include media of any nature and kind, including RAM, ROM, disks (CD-ROM, DVD, floppy disks, hard disk drives, etc.), USB keys, solid-state drives, tape drives, etc.

[0073] In the context of this specification, the purpose of using words such as "first", "second", and "third" as adjectives is solely to distinguish the nouns they modify, and not to describe any specific relationship between these nouns. Thus, for example, it should be understood that the use of the terms "first server" and "third server" does not imply any particular order, type, chronological order, hierarchy, or ranking (for example) between the servers, nor does the use of these terms (by themselves) imply that there must be any "second server" in any given situation. Furthermore, as discussed in other contexts herein, reference to a "first" element and a "second" element does not preclude the two elements from being the same actual, real-world elements. Thus, for example, in some cases, the "first" server and the "second" server may be the same software and / or hardware, while in other cases, they may be different software and / or hardware.

[0074] Implementations of the present technology all have at least one of the above-mentioned objectives and / or aspects, but not necessarily all of them. It should be understood that some aspects of the present technology are generated in an attempt to achieve the above-mentioned objectives, and these aspects may not meet the objectives and / or may meet other objectives not specifically described herein.

[0075] Additional and / or alternative features, aspects, and advantages of various implementations of the present technology will be apparent from the following description, drawings, and appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] For a better understanding of the present technology and other aspects and further features of the present technology, reference is made to the following description used in conjunction with the accompanying drawings, in which:

[0077] Figure 1 An example of a computing device that may be used to implement any of the methods described herein is shown;

[0078] Figure 2 A system for performing sample-based model search provided by some non-limiting embodiments of the present technology is shown;

[0079] Figure 3 A system for performing model-based sample search provided by some non-limiting embodiments of the present technology is shown;

[0080] Figure 4 Some non-limiting embodiments of the present technology provide Figure 1 Detailed representation of the sample-based model search performed by the system in A using energy scores;

[0081] Figure 5 Some non-limiting embodiments of the present technology provide Figure 1 Detailed representation of the model-based sample search performed by the system in A using energy scores;

[0082] Figure 6 A comparison table showing the functionality of at least some prior art solutions and systems provided by at least some non-limiting embodiments of the present technology;

[0083] Figure 7A Another system for performing exemplar-based model validation provided by at least some non-limiting embodiments of the present technology is shown.

[0084] Figure 7B Another system for performing sample-based model clone detection provided by at least some non-limiting embodiments of the present technology is shown;

[0085] Figure 8 is a schematic block diagram of a method for generating search results provided by at least some non-limiting embodiments of the present technology;

[0086] Figure 9 is a schematic block diagram of another method for generating search results provided by at least some non-limiting embodiments of the present technology. DETAILED DESCRIPTION

[0087] The examples and conditional language described herein are intended primarily to help readers understand the principles of the present technology, and are not intended to limit its scope to these specific examples and conditions. It is understood that those skilled in the art can design various devices that, although not explicitly described or shown herein, embody the principles of the present technology and are included within the spirit and scope of the present technology.

[0088] In addition, to facilitate understanding, the following description may describe a relatively simplified implementation of the present technology. It will be appreciated by those skilled in the art that various implementations of the present technology may have greater complexity.

[0089] In some cases, examples of useful modifications to the present technology may also be listed. This is intended only to aid understanding and is not intended to define the scope of the present technology or to define its limits. These modifications are merely examples, and those skilled in the art may make other modifications without departing from the scope of the present technology. Furthermore, where no examples of modifications are listed, this should not be interpreted as implying that modifications are impossible and / or that the content described is the only way to implement elements of the present technology.

[0090] In addition, all statements herein reciting the principles, aspects, and implementations of the present technology, and specific examples thereof, are intended to encompass structural and functional equivalents thereof, whether currently known or developed in the future. Thus, for example, it will be understood by those skilled in the art that any block diagram herein is a conceptual view of an illustrative circuit embodying the principles of the present technology. Similarly, it will be understood that any flow charts, flow diagrams, state transition diagrams, pseudocode, etc. represent various processes that can be substantially represented in a computer-readable medium and thus executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0091] The functions of the various elements shown in the figure (including any functional blocks marked as "processor" or "processing unit") can be provided by using dedicated hardware and hardware capable of executing software in association with appropriate software. When a processor provides functions, these functions can be provided by a single dedicated processor, a single shared processor, or multiple separate processors, some of which can be shared. In some embodiments of the present technology, the processor can be a general-purpose processor, for example, a central processing unit (CPU), or a processor dedicated to a specific purpose, for example, a digital signal processor (DSP). In addition, the explicit use of the term "processor" should not be interpreted as referring specifically to hardware capable of executing software, and may also implicitly include, but are not limited to, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), read-only memories (ROMs), random access memories (RAMs), and non-volatile memories for storing software. Other traditional and / or custom hardware may also be included.

[0092] Software modules, or modules or units represented as software, may be represented herein as any combination of flow chart elements or other elements indicating execution process steps and / or textual descriptions. These modules may be executed by hardware, as explicitly or implicitly shown. Furthermore, it should be understood that modules may include, but are not limited to, computer program logic, computer program instructions, software, stacks, firmware, hardware circuits, or combinations thereof, that provide the desired capabilities.

[0093] With these basic principles in mind, some non-limiting examples are used below to illustrate various implementations of various aspects of the technology.

[0094] Figure 1 1 is a schematic diagram of a computing environment 100 provided by one embodiment of the present technology. In some embodiments, computing environment 100 can be implemented by a conventional personal computer, a computer dedicated to data center-related operation and / or monitoring systems, a controller and / or electronic device (for example, but not limited to, a mobile device, a tablet device, a server, a controller unit, a control device, a monitoring device, etc.), and / or any combination thereof suitable for the current relevant task. In some embodiments, computing environment 100 includes various hardware components, including one or more single-core or multi-core processors collectively represented by processor 110, solid-state drive 120, random access memory 130, and input / output interface 150.

[0095] In some embodiments, computing environment 100 may also be a subsystem of one of the above-mentioned systems. In some other embodiments, computing environment 100 may be an "off-the-shelf" general-purpose computer system. In some embodiments, computing environment 100 may also be distributed among multiple systems. Computing environment 100 may also be specifically used to implement the present technology. It will be appreciated by those skilled in the art that various ways of implementing computing environment 100 may be envisioned without departing from the scope of the present technology.

[0096] Communications between the various components in the computing environment 100 may be accomplished via one or more internal and / or external buses 160 (e.g., a PCI bus, a Universal Serial Bus, an IEEE 1394 "Firewire" bus, a SCSI bus, a Serial ATA bus, an ARINC bus, etc.) to which the various hardware components are electrically coupled.

[0097] The input / output interface 150 can implement networking capabilities, such as wired access or wireless access. For example, the input / output interface 150 can include a network interface, such as, but not limited to, a network port, a network socket, a network interface controller, etc. Multiple examples of how to implement a network interface will be apparent to those skilled in the art. For example, but not limited to, the network interface can implement a specific physical layer and data link layer standard, such as Ethernet, Fibre Channel, Wi-Fi, or Token Ring. The specific physical layer and data link layer can provide the basis for a complete network protocol stack, thereby enabling communication between small groups of computers on the same local area network (LAN), and enabling large-scale network communication through routable protocols (e.g., Internet Protocol (IP)).

[0098] According to an implementation of the present technology, the solid-state drive 120 stores program instructions suitable for loading into the random access memory 130 and executing by the processor 110. These program instructions are used to execute the data center operation according to the generated machine learning pipeline. For example, the program instructions can be part of a library or application.

[0099] refer to Figure 2 , Figure 2 1 shows a system 200 contemplated in at least some embodiments of the present technology. The system 200 includes an electronic device 201 for executing a Sample-Model Retrieval (SAMOR) search engine 202. In some embodiments, the electronic device 201 can be implemented similarly to the computing environment 100. It is contemplated that the SAMOR search engine 202 can be used to perform sample-based model searches, among other things.

[0100] System 200 also includes an electronic device 203 communicatively coupled to electronic device 201. Electronic device 205 is configured to store data set 204. In this example, data set 204 includes images, but this may not be the case in other embodiments of the present technology. In some embodiments of the present technology, electronic device 203 may be implemented as a repository for storing data set 204. In other embodiments, electronic device 203 may be implemented similarly to computing environment 100 and associated with end users. In further embodiments, it is contemplated that the electronic device storing data set 204 may be the same electronic device used to execute SAMOR search engine 202 without departing from the scope of the present technology.

[0101] In some embodiments, the electronic device 201 can be communicatively coupled to the electronic device 203 via a communication network (not shown). In some non-limiting embodiments of the present technology, the communication network can be implemented as the Internet. In other non-limiting embodiments of the present technology, the communication network can be implemented in different ways, for example, any wide area communication network, local area communication network, dedicated communication network, etc. As an example only and not a limitation, the communication link between the electronic device 201 and the electronic device 203 can be implemented as a wireless communication link (for example, but not limited to, a 3G communication network link, a 4G communication network link, a wireless fidelity or abbreviated wireless network). and In other examples, the communication link may be a wireless communication link (e.g., Wireless Fidelity or Wi-Fi). or etc.), or it may be a wired communication link (e.g., an Ethernet-based connection).

[0102] The system 200 also includes an electronic device 205 communicatively coupled to the electronic device 201. The electronic device 205 is used to store a model database 206. The model database 206 includes a plurality of machine learning (ML) models 207. In some embodiments of the present technology, the electronic device 205 can be implemented as a repository for storing the model database 206. In other embodiments, the electronic device 205 can be implemented similarly to the computing environment 100. In further embodiments, it is contemplated that the electronic device hosting the model database 206 can be the same as the electronic device 201 used to execute the SAMOR search engine 202. In some embodiments, the electronic device 205 can be communicatively coupled to the client device 201 via a communication network.

[0103] SAMOR engine 202 is used to obtain input from data set 204. Data set 204 can be used to store one or more data samples, which can be used as input to SAMOR engine 202. In some embodiments, the data sample in data set 204 can be a labeled data sample. In other embodiments, data set 204 can include unlabeled data samples. As will be apparent from the further description below, SAMOR engine 202 can be used to perform sample-based model search using unlabeled data. In one example, the data sample stored in data set 203 can include an image data sample. However, the type of the data sample stored in data set 204 is not particularly limited. It can be said that SAMOR engine 202 can be a modality-independent search engine, which means that the engine can use various data samples to perform searches, without considering the specific type of data sample. It can be said that SAMOR engine 202 can be a tagless search engine, which means that the engine can use unlabeled data samples to perform searches.

[0104] SAMOR engine 202 is configured to perform sample-based model search using model database 206. Broadly speaking, model database 206 is a collection of pre-trained ML models, including multiple ML models 207. SAMOR engine 202 is configured to obtain input including one or more data samples, perform a search on model database 206, and identify one or more ML models from multiple ML models 207 as top search results. It is contemplated that the search results may be represented as a list of ranked pre-trained models according to their corresponding ranking scores. How SAMOR engine 202 determines the ranking scores is discussed in further detail below.

[0105] exist Figure 2 In the example shown, the top search results include a first model 208 and a second model 210. In one implementation, the first model 208 may be a MobileNetV3 architecture pre-trained for dog image classification. Broadly speaking, MobileNetV3 is a deep neural network that is optimized to run on a mobile phone CPU. MobileNetV3 is capable of performing image classification, object detection, and semantic segmentation. In another implementation, the second model 210 may be a Yolov5 architecture pre-trained for "cat" and "dog" object detection. Broadly speaking, Yolov5 is a deep neural network that performs real-time object detection. Multiple versions of Yolov5 are currently known, with different trade-offs (e.g., accuracy-speed).

[0106] Optionally, in addition to using input data samples from the dataset 204, the SAMOR engine 202 can also be used to use text queries 212. As will be apparent from the further description below, the text queries 212 can be used to determine a text-based ranking score, which can be used in combination with other types of ranking scores to generate a combined ranking score for the corresponding pre-trained model.

[0107] Additionally or alternatively, the system 200 can be used to fine-tune one or more models (e.g., the first model 208 and / or the second model 210) using input samples in the dataset 204. It is contemplated that the electronic device 203 and / or the electronic device 201 can further train the first model 208 to perform dog image classification using data samples in the dataset 204. It is contemplated that the electronic device 203 and / or the electronic device 201 can further train the second model 210 to perform “cat” and “dog” object detection using data samples in the dataset 204.

[0108] In some embodiments, the SAMOR engine 202 can be configured to receive a query (optionally a text query 212) in the form of one or more data samples in the dataset 204, such as documents, images, audio, video, etc. The SAMOR engine 202 can be configured to search for and identify pre-trained models from the plurality of models 207 that perform relatively better on the provided data samples than other models in the plurality of models 207. For example, given an input data sample in the form of a dog image, the SAMOR engine 202 can be configured to search for and identify a top image classifier that classifies the dog image very accurately.

[0109] refer to Figure 3 , Figure 3 1 shows a system 300 contemplated in at least some embodiments of the present technology. System 300 includes an electronic device 301 for executing a SAMOR search engine 302. In some embodiments, electronic device 301 can be implemented similarly to computing environment 100 and / or electronic device 201. Broadly speaking, SAMOR search engine 202 is configured to perform model-based sample searches, among other things.

[0110] System 300 also includes an electronic device 303 communicatively coupled to electronic device 301. Electronic device 303 is used to store a third model 308. In one implementation, the third model 308 may have a Yolov5 architecture pre-trained for dog detection. In other embodiments, the implementation of electronic device 303 may be similar to the implementation of computing environment 100 and / or electronic device 203 and the manner in which it is associated with the end user. In some embodiments, electronic device 301 may be communicatively coupled to electronic device 303 via a communication network (not shown). In further embodiments, it is contemplated that, without departing from the scope of the present technology, the electronic device storing the third model 308 may be the same as the electronic device used to execute SAMOR search engine 302.

[0111] System 300 also includes an electronic device 305 communicatively coupled to electronic device 301. In some embodiments, electronic device 205 can be implemented similarly to computing environment 100. Electronic device 305 is used to store a sample database 306. Sample database 306 includes multiple data samples 307. In some embodiments, at least some of multiple data samples 307 can be labeled data samples and stored in association with corresponding labels. In other embodiments, at least some of multiple data samples 307 can be unlabeled data samples. It is contemplated that the electronic device hosting sample database 306 can be the same as electronic device 201 used to execute SAMOR search engine 302. In some embodiments, electronic device 305 can be communicatively coupled to client device 301 via a communication network.

[0112] The SAMOR engine 302 is configured to obtain input in the form of a pre-trained ML model. For example, the SAMOR engine 302 may obtain a third model 308 from the electronic device 303. Optionally, the SAMOR engine 302 may also obtain a text query 312 as input. The SAMOR engine 302 is configured to process the input and search the sample database 306 to identify search results in the form of a list of ranked data samples. To this end, the SAMOR engine 302 is configured to generate a ranking score for the corresponding data sample in the sample database 306 based on the input and the third model 308. How the SAMOR engine 302 determines the ranking score is discussed in further detail below.

[0113] Additionally or alternatively, the electronic device 303 may obtain one or more search results provided by the SAMOR engine 302 and fine-tune the third model 308 based on the data samples corresponding to the one or more search results. In an embodiment where the data samples obtained by the electronic device 303 from the SAMOR engine 302 are unlabeled data samples, pseudo labels may be generated for the corresponding unlabeled data samples. The pseudo labels may be generated using the third model 308.

[0114] In some embodiments, it can be said that the SAMOR engine 302 can be used to receive a query (which may also be a text query) in the form of a pre-trained model and search and identify data samples from the plurality of data samples 307, and the performance of the third model 308 on these data samples is relatively better than the performance on other data samples in the plurality of data samples 307. The electronic device 303 can use the identified data samples to further train the third model 308 to improve the performance of the third model 308 in a given task. For example, given a pre-trained model for dog recognition, the SAMOR engine 308 can be used to identify unlabeled data samples with the highest pseudo-labeling performance to achieve semi-supervised learning.

[0115] refer to Figure 4 , Figure 4 Shown Figure 2 Schematic representation of how the SAMOR engine 202 generates ranking scores (e.g., energy scores 410) for pre-trained models. As will be apparent from the further description below, the SAMOR engine 202 can be used to perform sample-based model search using the energy scores 410 for ranking pre-trained models.

[0116] For example, the end user 402 may use the electronic device 203 to provide the input data sample 204 to the SAMOR engine 202. In one implementation, the end user 402 may be associated with a traffic management office in a city, use a camera to capture data, and then use the SAMOR engine 202 to improve the car detection model based on the captured data. In another implementation, the end user 402 may be associated with a farm, capture farm images, and then use the SAMOR engine 202 to find a semantic segmentation model to automatically perform segmentation. Assume X: {x0, x1, ..., x N} is the input data sample 204, F:{f0,f1,…,f M} is a plurality of pre-trained models 207 in the model database 206. In this example, the SAMOR engine 202 is used to process and rank the models in the plurality of models 207, thereby outputting a list of ranked models 414 (denoted as F * ), these models are ranked from high to low according to their corresponding energy scores in the plurality of energy scores 410. It is contemplated that the SAMOR engine 202 can be used to rank only some of the plurality of models 207 without departing from the scope of the present technology.

[0117] In some embodiments of the present technology, the SAMOR engine 202 may be used to perform sample-based model search according to algorithm (1).

[0118]

[0119] It can be said that the SAMOR engine 202 is used to find a model that is "suitable" for the input data sample 404 in a sense, that is, the input sample has a model with a high likelihood in their corresponding density probability function. It can be seen that data points with a higher probability of occurrence are equivalent to having higher energy. The developers of this technology have realized that since a given energy score has a linear alignment relationship with the log-likelihood function, the energy score is very suitable for determining which models are most suitable for a given set of input data samples. It can be said that the linear alignment relationship refers to the linear relationship between log-likelihood and energy. It should be noted that the higher the energy score, the higher the likelihood of a given data point, and the more models that are suitable for the input data sample. The developers of this technology also realized that since the energy function 408 is aligned with the probability density of the input data sample 204, the energy score 410 is relatively less affected by the confidence problem, while other types of scores (e.g., softmax confidence score, etc.) have this problem.

[0120] To calculate each pre-trained model f i Given an energy score, each data sample x j All must be entered into f i To do this, a pair of (f i ,x j ) is fed into the energy function 408, which is represented as E(.) and executed by the SAMOR engine 202. The SAMOR engine 202 can implement the energy score calculation process according to the following algorithm (2).

[0121]

[0122] As shown above, the number of real-valued pairs (denoted as L) calculated by a given model (after receiving a data sample as input) is obtained. Then, according to equation (1), the number of pairs (f i ,x j ) corresponding energy score:

[0123] Where L = f i (x j )(1)

[0124] With each data sample x j For all energy scores corresponding to N samples (a total of N samples), the SAMOR engine 202 can calculate the model f according to equation (2): i Total energy fraction:

[0125]

[0126] With a list of all energy scores The SAMOR engine 202 may accordingly rank all models from highest to lowest total energy score: F * =Rank(F; S). Then, the ranked model F * (or at least a top-ranked model) is returned to the end user.

[0127] In summary, the SAMOR engine 202 is configured to input an input sample 204 into a plurality of pre-trained models 207 and apply an energy function 408 to corresponding sample-model pairs. The SAMOR engine 202 can use the scores generated by the energy function 408 to generate corresponding energy scores 410. The SAMOR engine 202 is then configured to perform a model ranking process 412 to generate a ranked list 414 in which a top pre-trained model 416 is associated with the highest energy score in the energy scores 410, and a bottom pre-trained model 418 is associated with the lowest energy score in the energy scores 410.

[0128] As mentioned above, SAMOR engine 202 can be used for using text input to provide text-based sample-model retrieval framework.It is contemplated that SAMOR engine 302 can be used for receiving text-based query and sample-based query as input.SAMOR engine 302 can be used for performing text-based search for the text description associated with the corresponding pre-trained model in model database 206.For example, text description can include model card information.These information can be associated with the corresponding model in a plurality of pre-trained models 207 and stored in model database 206.Then, text-based search results can be combined with the search results based on sample (for example, the score of these search results is averaged) to provide search results.In certain embodiments, known technology can be used to generate text-based ranking scores by comparing text description and text input.Then, SAMOR engine 202 can use these text-based scores and energy score combinations to generate combined ranking scores for the corresponding model in a plurality of models 207.

[0129] refer to Figure 5 , Figure 5 yes Figure 3 5. A schematic representation of how the SAMOR engine 302 generates ranking scores for data samples in FIG. 5. As will be apparent from the further description below, the SAMOR engine 302 can be used to perform a model-based sample search using energy scores 510 for ranking data samples. In some embodiments of the present technology, the SAMOR engine 302 can be used to perform a model-based sample search according to Algorithm (3).

[0130]

[0131] For example, the end user 502 may provide the pre-trained model 308 as an input to the SAMOR engine 302. The SAMOR engine 302 is operable to execute the energy function 408 to generate an energy score 510 for a corresponding data sample in the plurality of data samples 307. For example, assuming that f is the input pre-trained model 308, X: {x0, x1, ..., x N} is a plurality of data samples 307 obtained from the database 306. In this example, the SAMOR engine 302 is used to process and rank the plurality of data samples 307, and output a list of ranked samples 514 (denoted as X * ), the samples are ranked from high to low according to their corresponding energy scores in the plurality of energy scores 510 .

[0132] In some embodiments, the SAMOR engine 302 can be used to determine which samples of a plurality of data samples produce a high likelihood in the density probability function of the pre-trained model 308. The developers of the present technology have realized that because the energy score is linearly aligned with the log-likelihood function (linear relationship), the energy score is very suitable for determining which samples are most suitable for a given pre-trained model. It should be noted that the higher the energy score, the higher the likelihood, and the more samples are suitable for input into the pre-trained model. In addition, the developers of the present technology also realized that because the energy function is aligned with the probability density of the input (linear relationship), the energy score is relatively less affected by the confidence problem, while other types of scores (e.g., softmax confidence scores, etc.) have this problem.

[0133] To calculate each data sample x j The energy score of each data sample is input to the input pre-trained model f. To do this, a pair (x j ; f) is fed into the energy function 408E(.). The SAMOR engine 302 can implement the energy score calculation process similarly according to the energy score calculation algorithm shown in the above algorithm 2. It should be noted that the SAMOR engine 302 can be used to calculate real-valued logarithms by inputting data samples into the pre-trained model 308. The SAMOR engine 302 can calculate the real-valued logarithm corresponding to a pair (x according to equation (1) j ; f) The corresponding energy score.

[0134] With a list of all energy scores The SAMOR engine 302 may rank all samples X from highest to lowest total energy score: X * =Rank(X;S), etc. This column of ranked data samples X * (or at least the top-ranked data samples) can be provided to end users as search results.

[0135] In summary, the SAMOR engine 302 is configured to input the data sample 307 into the pre-trained model 308 and apply the energy function 408 to the corresponding sample-model pair. The SAMOR engine 302 can use the scores generated by the energy function 408 to generate corresponding energy scores 510. The SAMOR engine 302 is then configured to perform a sample ranking process 512 to generate a ranked list 514, in which the top sample 516 is associated with the highest energy score in the energy scores 510, and the bottom sample 518 is associated with the lowest energy score in the energy scores 510.

[0136] As mentioned above, SAMOR engine 302 can be used for further obtaining text input to provide text-based sample-model retrieval framework.In certain embodiments, SAMOR engine 302 can obtain text-based query and model-based query as input.SAMOR engine 302 can be used for text description / caption / label (tag) execution text-based search to data sample.The text information of data sample can be associated with the corresponding data sample in multiple data samples 307 and stored in sample database 306.Then, text-based search results can be combined with sample-based search results (for example, the score of these search results is averaged) to provide final search results.In certain embodiments, known technology can be used to generate text-based ranking score by comparing text description and text input.Then, SAMOR engine 302 can use these text-based scores in combination with energy score, so that the corresponding data sample for multiple models 307 generates combined ranking score.

[0137] refer to Figure 7A , Figure 7A Another embodiment of the present technology is shown. System 700 includes a SAMOR engine 702 for performing sample-based model validation. Sample-based model validation can be used to verify whether a model 706 can process a given input sample 704 (e.g., provide accurate predictions for the sample). First, an energy score corresponding to the model 706 on the given input sample 704 is calculated. If the energy score is greater than a predetermined threshold (e.g., provided by an operator), then the model 706 is "validated" to have good performance on the sample 704, thereby generating a positive indication 710.

[0138] refer to Figure 7B , Figure 7BAnother embodiment of the present technology is shown. System 750 includes a SAMOR engine 752 that performs sample-based model clone detection. Sample-based model clone detection can be used to check whether two models 756 and 758 have similar performance on a given sample 754. First, the energy score of each of the two models 756 and 758 is calculated (e.g., according to Equation 1). If the energy scores of the two models 756 and 758 are equal (or within a predetermined acceptable interval), then the two models 756 and 758 are considered to be clone models (having similar performance on the same set of samples).

[0139] refer to Figure 8 , Figure 8 is a schematic block diagram of a method 800 that may be performed by the computing environment 100, according to at least some non-limiting embodiments of the present technology. Each step of the method 800 is discussed in detail below.

[0140] Step 802: The electronic device obtains a set of data samples from the sample database

[0141] In step 802, electronic device 201 is used to obtain a set of data samples 204. In some embodiments, set of data samples 204 includes at least one of text samples, image samples, audio samples, and video samples. In other embodiments, set of data samples 204 may be a set of unlabeled data samples.

[0142] Step 804: The electronic device obtains a set of pre-trained models from the model database

[0143] In step 804, the electronic device 201 is configured to obtain a set of pre-trained models 207 from the model database 206. In some embodiments, the set of pre-trained models 207 includes at least one of a ResNet model, a DenseNet model, and a Transformer-based model.

[0144] Step 806: The electronic device generates an energy score for a given model in the set of pre-trained models using the energy function and the set of data samples.

[0145] In step 806, the electronic device 201 is configured to generate an energy score for a given model in the set of pre-trained models 207 using the energy function 408 and the set of data samples 204. For example, the electronic device 201 may be configured to generate the energy score s1 according to Equations 1 and 2.

[0146] It is contemplated that the energy score for a given model may be the average of the scores calculated for corresponding pairs consisting of the given model and a given sample in the set of data samples 204 .

[0147] Step 808: The electronic device ranks the set of pre-trained models using the energy score of the given model, thereby generating a list of ranked pre-trained models.

[0148] In step 808, the electronic device 201 is configured to rank the set of pre-trained models 207 using the energy scores of the given models, thereby generating a list of ranked pre-trained models 414. It is contemplated that the electronic device 201 may generate energy scores for the corresponding models and rank the models using the corresponding energy scores of the models.

[0149] Step 810: The electronic device generates a search result using the list of ranked pre-trained models, wherein the search result includes the given model, and the given model is the highest-ranked model in the list of ranked pre-trained models.

[0150] In step 810, electronic device 201 is configured to generate search results using the list of ranked pre-trained models 414. The search results include a given model. The given model is the highest-ranked model in the list of ranked pre-trained models 414. It should be noted that the search results may include the top N models in the list of ranked pre-trained models 414 without departing from the scope of the present technology.

[0151] refer to Figure 9 , Figure 9 is a schematic block diagram of a method 900 that may be performed by the computing environment 100 according to at least some non-limiting embodiments of the present technology. The various steps of the method 900 are discussed in detail below.

[0152] Step 902: The electronic device obtains a set of data samples from the sample database

[0153] In step 902, electronic device 301 is configured to obtain a set of data samples 307. In some embodiments, set of data samples 307 includes at least one of text samples, image samples, audio samples, and video samples. In other embodiments, set of data samples 307 may be a set of unlabeled data samples.

[0154] Step 904: The electronic device obtains a pre-trained model from the model database

[0155] In step 904, the electronic device 301 is configured to obtain a pre-trained model 308. In some embodiments, the set of pre-trained models 207 includes at least one of a ResNet model, a DenseNet model, and a Transformer-based model.

[0156] Step 906: The electronic device generates an energy score for a given sample in the set of data samples using the energy function and the pre-trained model

[0157] In step 906, the electronic device 301 is configured to generate an energy score for a given sample in the set of data samples 307 using the pre-trained model 308. For example, the electronic device 301 may generate the energy score s1 in the energy score 510 for the given sample according to Equation 1. It is contemplated that the electronic device 301 may be configured to generate the energy score 510 in a similar manner.

[0158] Step 908: The electronic device ranks the set of data samples using the energy score of the given data sample, and generates a column of ranked data samples for each person.

[0159] In step 908, the electronic device 301 is configured to rank the set of data samples 307 using the energy scores of the data samples, thereby generating a list of ranked data samples 514. It is contemplated that the electronic device 301 may generate energy scores for corresponding data samples in the data samples 307 and rank the data samples using the energy scores of the data samples 307.

[0160] Step 910: The electronic device generates a search result using the list of ranked data samples, wherein the search result includes the given data sample, and the given data sample is the highest-ranked data sample in the list of ranked data samples.

[0161] In step 910, electronic device 301 is configured to generate search results using a list of ranked data samples 514. The search results include a given data sample. The given data sample is the highest-ranked data sample in the list of ranked data samples 514. It should be noted that the search results may include the first N data samples in the list of ranked data samples 514 without departing from the scope of the present technology.

[0162] Modifications and improvements to the above-described implementation of the present technology will be apparent to those skilled in the art. The above description is illustrative and not restrictive. Therefore, the scope of the present technology is limited only by the scope of the appended claims.

Claims

1. A method for generating search results, characterized in that: The method may be performed by an electronic device that is communicatively coupled to a sample database and a model database, and includes: The electronic device obtains a set of data samples from the sample database; The electronic device obtains a set of pre-trained models from the model database; The electronic device generates an energy score for a given model in the set of pre-trained models using the energy function and the set of data samples; The electronic device ranks the set of pre-trained models using the energy score of the given model, thereby generating a list of ranked pre-trained models; The electronic device generates the search results using the list of ranked pre-trained models, wherein the search results include the given model, which is the highest-ranked model in the list of ranked pre-trained models.

2. The method according to claim 1, characterized in that The set of data samples includes at least one of text samples, image samples, audio samples, and video samples.

3. The method according to claim 1 or 2, characterized in that The set of data samples is a set of unlabeled data samples.

4. The method according to any one of claims 1 to 3, characterized in that The set of pre-trained models includes at least one of a ResNet model, a DenseNet model, and a Transformer-based model.

5. The method according to any one of claims 1 to 4, characterized in that The method also includes: the electronic device fine-tuning the target model using the set of data samples.

6. The method according to any one of claims 1 to 5, characterized in that The energy function is: Where L = f i (x j ) Among them, f i is the given model, x j is a given data sample among the plurality of data samples, l is a pair (x j ,f i ).

7. The method according to claim 6, characterized in that The energy fraction is: Among them, s i is the energy score of the given model, X is the plurality of data samples, and N is the number of data samples in the plurality of data samples.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: The electronic device obtains a text query; generating, by the electronic device, a text-based score for the given model using the text query and text information associated with the given model; The rankings include: The electronic device ranks the set of pre-trained models using the energy score and the text-based score.

9. A method for generating search results, characterized in that: The method may be performed by an electronic device that is communicatively coupled to a sample database and a model database, and includes: The electronic device obtains a set of data samples from the sample database; The electronic device obtains a pre-trained model from the model database; The electronic device generates an energy score for a given sample in the set of data samples using the energy function and the pre-trained model; The electronic device ranks the set of data samples using the energy score of the given data sample, thereby generating a list of ranked data samples; The electronic device generates the search result using the list of ranked data samples, wherein the search result includes the given data sample, and the given data sample is the highest-ranked data sample in the list of ranked data samples.

10. The method according to claim 1, characterized in that The set of data samples includes at least one of text samples, image samples, audio samples, and video samples.

11. The method according to any one of claims 9 to 10, characterized in that The set of data samples is a set of unlabeled data samples.

12. The method according to any one of claims 9 to 11, characterized in that The pre-trained model is at least one of a ResNet model, a DenseNet model, and a Transformer-based model.

13. The method according to any one of claims 9 to 12, characterized in that The method also includes: the electronic device fine-tuning the pre-training model using the given data sample.

14. The method according to any one of claims 9 to 13, characterized in that The energy function is: Where L = f i (x j ) Among them, E(x j ;f i ) is the given data sample x j The energy fraction, f i is the pre-training model, l is a pair (x j ,f i ).

15. The method according to any one of claims 9 to 14, characterized in that The method further comprises: The electronic device obtains a text query; generating, by the electronic device, a text-based score for the given data sample using the text query and text information associated with the given data sample; The rankings include: The electronic device ranks the set of data samples using the energy score and the text-based score.

16. An electronic device for generating search results, characterized in that: The electronic device is communicatively coupled to the sample database and the model database, and is configured to: Acquire a set of data samples from the sample database; Obtain a set of pre-trained models from the model database; generating an energy score for a given model in the set of pre-trained models using an energy function and the set of data samples; Ranking the set of pre-trained models using the energy score of the given model to generate a list of ranked pre-trained models; The search results are generated using the list of ranked pre-trained models, wherein the search results include the given model, which is the highest-ranked model in the list of ranked pre-trained models.

17. The electronic device according to claim 16, wherein: The set of data samples includes at least one of text samples, image samples, audio samples, and video samples.

18. The electronic device according to claim 16 or 17, characterized in that: The set of data samples is a set of unlabeled data samples.

19. The electronic device according to any one of claims 16 to 18, characterized in that: The set of pre-trained models includes at least one of a ResNet model, a DenseNet model, and a Transformer-based model.

20. The electronic device according to any one of claims 16 to 19, characterized in that: The method also includes: the electronic device fine-tuning the target model using the set of data samples.

21. The electronic device according to any one of claims 16 to 20, characterized in that: The energy function is: Where L = f i (x j ) Among them, f i is the given model, x j is a given data sample among the plurality of data samples, l is a pair (x j ,f i ).

22. The electronic device according to claim 21, wherein: The energy fraction is: Among them, s i is the energy score of the given model, X is the plurality of data samples, and N is the number of data samples in the plurality of data samples.

23. The electronic device according to any one of claims 16 to 22, characterized in that: The electronic device is also used for: Get text query; generating a text-based score for the given model using the text query and text information associated with the given model; For ranking purposes, the electronic devices were used to: The set of pre-trained models is ranked using the energy score and the text-based score.

24. An electronic device for generating search results, characterized in that The electronic device is communicatively coupled to the sample database and the model database, and is configured to: Acquire a set of data samples from the sample database; Obtain a pre-trained model from the model database; generating an energy score for a given sample in the set of data samples using an energy function and the pre-trained model; Ranking the set of data samples using the energy score of the given data sample to generate a list of ranked data samples; The search result is generated using the list of ranked data samples, wherein the search result includes the given data sample, and the given data sample is the highest-ranked data sample in the list of ranked data samples.

25. The electronic device according to claim 24, characterized in that The set of data samples includes at least one of text samples, image samples, audio samples, and video samples.

26. The electronic device according to claim 24 or 25, characterized in that: The set of data samples is a set of unlabeled data samples.

27. The electronic device according to any one of claims 24 to 26, characterized in that: The pre-trained model is at least one of a ResNet model, a DenseNet model, and a Transformer-based model.

28. The electronic device according to any one of claims 24 to 27, characterized in that: The method also includes: the electronic device fine-tuning the pre-training model using the given data sample.

29. The electronic device according to any one of claims 24 to 28, characterized in that: The energy function is: Where L = f i (x j ) Among them, E(x j ;f i ) is the given data sample x j The energy fraction, f i is the pre-training model, l is a pair (x j ,f i ).

30. The electronic device according to any one of claims 24 to 29, characterized in that: The electronic device is also used for: Get text query; generating a text-based score for the given data sample using the text query and text information associated with the given data sample; For ranking purposes, the electronic devices were used to: The set of data samples is ranked using the energy score and the text-based score.

Images