Device and method for providing benchmark result of artificial intelligence based model

Abstract

To provide benchmark results of a specific model on a specific node in an efficient manner.SOLUTION: There is provided a method to be executed in a computing device, for providing benchmark results. The method comprises the steps of: acquiring first input data containing at least one of an inference task and a data set; determining a target model to be benchmarked relating to the inference task and at least one target node on which the inference task of the target model is executed, where the determined target model corresponds to an artificial intelligence-based model to be benchmarked relating to the inference task on the at least one target node based on the data set; and providing benchmark results obtained by running the target model on the at least one target node.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to artificial intelligence technology, and more particularly to benchmarking technology for artificial intelligence-based models. [Background technology]

[0002] With the development of AI technology, various AI-based models have been developed. As the need for computing resources to process these models increases, related industries are also continuously developing hardware with new capabilities.

[0003] Demand for edge AI, which can perform calculations directly on networked devices such as personal computers, smartphones, automobiles, wearable devices, and robots, is increasing, and research is being conducted on artificial intelligence-based models that take hardware resources into account.

[0004] As edge AI technology advances, the importance of hardware in the field of artificial intelligence (AI) technology is increasing. Therefore, developing and launching AI-based solutions requires not only AI-based models but also extensive knowledge of the various hardware on which the AI-based models run. For example, even if a model has excellent performance in a specific domain, the inference performance of the model may vary depending on the hardware on which the model runs. A model with optimal performance in a specific domain may not be supported by the specific hardware used to provide the service. Therefore, determining both an AI-based model suitable for a service to be provided and the hardware suitable for the AI-based model requires extensive background knowledge of AI and hardware technologies and a significant amount of resources. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] U.S. Patent Publication No. 2022-0121927 Summary of the Invention [Problem to be solved by the invention]

[0006] The present disclosure has been devised in view of the above-mentioned background art, and aims to efficiently provide benchmark results for a specific model on a specific node.

[0007] The technical problems in the present disclosure are not limited to those described above, and a person skilled in the art can clearly understand problems other than the above-mentioned technical problems based on the following description. [Means for solving the problem]

[0008] According to one embodiment of the present disclosure, a method for providing benchmark results, executed on a computing device, is disclosed. The method may include: acquiring first input data including at least one of an inference task and a dataset; determining a target model to be benchmarked for the inference task and at least one target node on which the inference task of the target model is to be executed, the determined target model corresponding to an artificial intelligence-based model on which the benchmark for the inference task is to be executed on the at least one target node based on the dataset; and providing benchmark results obtained by executing the target model on the at least one target node.

[0009] In one embodiment, the step of determining the at least one target node may include the steps of providing a candidate node list including candidate nodes proposed for benchmarking related to the inference task, and determining the at least one target node based on second input data for selecting the at least one target node from the candidate node list.

[0010] In one embodiment, the candidate node list may include identification information for each of the candidate nodes and estimated latency information for each of the candidate nodes when the target model is executed.

[0011] In one embodiment, the order of the candidate nodes included in the candidate node list may be determined based on the magnitude of the expected latency information.

[0012] In one embodiment, if the difference in the magnitude of the predicted latency information between a first candidate node and a second candidate node among the candidate nodes is within a predetermined threshold range, the order of the first candidate node and the second candidate node can be determined based on the memory usage and CPU usage of the first candidate node and the second candidate node.

[0013] In one embodiment, the candidate node list may include power mode information indicating CPU core usage in at least some of the candidate nodes, and fan mode information indicating fan usage in at least some of the candidate nodes.

[0014] In one embodiment, the candidate node list may include information regarding at least one model that can be supported in each of the candidate nodes, library information required for installing the at least one model that can be supported in each of the candidate nodes, and power usage information indicating power usage obtained from a power measurement sensor connected to the candidate nodes.

[0015] In one embodiment, the size of the model is determined based on the size of the dataset of the first input data, and the candidate node list may include candidate nodes that have free memory capacity that exceeds the determined size of the model.

[0016] In one embodiment, the step of providing the candidate node list may include a step of including, as the candidate nodes, a plurality of first nodes having an execution environment that supports a first operator included in the input source model in the candidate node list.

[0017] In one embodiment, the step of providing the candidate node list may include a step of including, as the candidate nodes, a plurality of second nodes in the candidate node list, the second nodes having an execution environment that does not support a first operator included in the input source model but that supports a second operator different from the first operator that can substitute for the first operator.

[0018] In one embodiment, the benchmark results may include preprocessing time information, which is information related to the time required for preprocessing inference of the target model in the at least one target node; inference time information, which is information related to the time required for inference on the target model in the at least one target node; preprocessing memory usage information, which is information related to the memory usage required for preprocessing inference of the target model in the at least one target node; and inference memory usage information, which is information related to the memory usage required for inference on the target model in the at least one target node.

[0019] In one embodiment, the benchmark results may include quantitative information regarding inference time, which is obtained by repeatedly inferring the target model a predetermined number of times in the at least one target node, and quantitative information regarding memory usage in each of the NPU, CPU, and GPU, which is obtained by inferring the target model in the at least one target node.

[0020] In one embodiment, the step of providing the benchmark results obtained by executing the target model on the at least one target node may include a step of distinguishing between nodes capable of wireless communication and nodes incapable of wireless communication on the at least one target node, and a step of connecting the nodes capable of wireless communication to the nodes incapable of wireless communication using serial communication, and sending a benchmark task to the nodes capable of wireless communication corresponding to the nodes incapable of wireless communication to obtain benchmark results for the nodes incapable of wireless communication.

[0021] In one embodiment, if the at least one target node includes a third node that cannot be externally confirmed, the step of providing the benchmark result may include the steps of sending a first acknowledgment message to the third node in response to receiving a first low-power wireless signal from the third node, the first acknowledgment message including a benchmark task for benchmarking the target model at the third node, and receiving a second low-power wireless signal from the third node including the benchmark result generated by the third node.

[0022] In one embodiment, if the at least one target node includes a third node that cannot be confirmed externally, the step of providing the benchmark result may further include determining that the benchmark task in the third node has failed and setting the third node to an inactive state if the second low-power wireless signal is not received within a time corresponding to a predetermined waiting time threshold, and setting the state of the third node to an active state in response to receiving a third low-power wireless signal from the third node set to the inactive state.

[0023] In one embodiment, the first low-power radio signal may include information regarding whether the third node is performing a benchmark, information regarding memory usage of the third node, and hardware identification information of the third node.

[0024] In one embodiment, the benchmark task included in the first response message may include target model information that the third node can download and node configuration information used to convert the target model downloaded by the third node, and the benchmark results generated by the third node may include results obtained by executing the target model in the execution environment of the third node based on the node configuration information and the target model information.

[0025] In one embodiment, based on the second input data, information of a target model supported by at least one target node included in the second input data can be automatically provided.

[0026] In one embodiment, the information on the target model may include identification information corresponding to the target model supported by the at least one target node, and software version information of the target model supported by the at least one target node.

[0027] In one embodiment, a computer program stored on a computer-readable storage medium is disclosed. The computer program, when executed by a computing device, causes the computing device to perform the following operations to provide benchmark results, the operations may include: acquiring first input data including at least one of an inference task and a dataset; determining a target model to be benchmarked for the inference task and at least one target node on which the inference task of the target model is executed, the determined target model corresponding to an artificial intelligence-based model on which the benchmark for the inference task is performed on the at least one target node based on the dataset; and providing benchmark results obtained by executing the target model on the at least one target node.

[0028] In one embodiment, a computing device for generating benchmark results is disclosed. The computing device may include at least one processor and a memory. The at least one processor receives first input data including at least one of an inference task and a dataset, determines a target model to be benchmarked for the inference task and at least one target node on which the inference task of the target model is executed, the determined target model corresponds to an artificial intelligence-based model on which the benchmark for the inference task is performed on the at least one target node based on the dataset, and provides benchmark results for the target model obtained by executing the target model on the at least one target node. [Effects of the Invention]

[0029] The method according to one embodiment of the present disclosure can provide benchmark results for a specific model on a specific node in an efficient manner. [Brief explanation of the drawings]

[0030] [Figure 1] FIG. 1 illustrates a schematic block diagram of a computing device in accordance with one embodiment of the present disclosure. [Figure 2] FIG. 2 illustrates an exemplary structure of an artificial intelligence-based model in one embodiment of the present disclosure. [Figure 3] FIG. 3 illustrates an exemplary schematic diagram of a system for providing benchmark results in accordance with one embodiment of the present disclosure. [Figure 4] FIG. 4 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure. [Figure 5] FIG. 5 illustrates an exemplary tabular data structure used to generate a candidate node list in one embodiment of the present disclosure. [Figure 6] FIG. 6 illustrates an exemplary tabular data structure used to generate a candidate node list in one embodiment of the present disclosure. [Figure 7] FIG. 7 illustrates an exemplary tabular data structure used to generate a candidate node list in one embodiment of the present disclosure. [Figure 8] FIG. 8 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure. [Figure 9] FIG. 9 illustrates an exemplary method for providing benchmark results in one embodiment of the present disclosure. [Figure 10] FIG. 10 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure. [Figure 11] FIG. 11 illustrates an exemplary method for providing benchmark results for nodes that cannot be externally verified in accordance with one embodiment of the present disclosure. [Figure 12] FIG. 12 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure. [Figure 13] FIG. 13 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure. [Figure 14] FIG. 14 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure. [Figure 15] FIG. 15 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure. [Figure 16] FIG. 16 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure. [Figure 17] FIG. 17 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure. [Figure 18] FIG. 18 is a schematic diagram illustrating a computing environment in one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0031] Various examples will be described below with reference to the drawings. Various descriptions are provided herein to facilitate understanding of the present disclosure. In the specific description for implementing the present disclosure, configurations that are not directly related to the technical gist of the present disclosure are omitted to the extent that they do not obscure the technical gist of the present invention. Furthermore, the terms and phrases used in the present specification and claims should be interpreted in a way that is consistent with the technical concept of the present invention, based on the principle that the inventor can appropriately define the concept of the term in order to best describe his or her invention.

[0032] As used herein, terms such as "module," "system," and the like refer to computer-related entities, hardware, firmware, software, a combination of software and hardware, or software implementations, and may be used interchangeably. For example, a module may be, but is not limited to, a procedure running on a processor, a processor, an object, a thread of execution, a program, an application, and / or a computing device. One or more modules may reside within a processor and / or thread of execution. A module may be localized within one computer. A module may be distributed across two or more computers. Such modules may also execute from various computer-readable media having various data structures stored therein. Modules may communicate via local and / or remote processes, for example, using signals carrying one or more data packets (e.g., data from one component interacting with other components in a local or distributed system and / or signals, data transmitted over a network such as the Internet, to other systems, etc.).

[0033] Furthermore, the term "or" is intended to mean an inclusive "or," not an exclusive "or." In other words, unless otherwise specified or clear from the context, "X utilizes A or B" refers to one of the natural inclusive permutations. That is, if X utilizes A; X utilizes B; or X utilizes both A and B, "X utilizes A or B" can refer to any of these. Furthermore, the terms "and / or" and "at least" herein refer to and include all possible combinations of one or more items among the multiple related items listed. The same interpretation applies to "and / or." For example, the terms "at least one of A or B," "at least one of A and B," "at least one of A and B," and "at least one of A and B" are intended to mean "including only A," "including only B," and "including both A and B."

[0034] Additionally, the predicate "comprises" and / or the modifier "comprises" should be understood to mean the presence of the feature and / or component in question. However, the predicate "comprises" and / or the modifier "comprises" should be understood not to exclude the presence or addition of one or more other further features, components and / or groups thereof. Additionally, unless a specific number is specified or the context is clear that a singular form is indicated, the singular form in this specification and claims should generally be construed to mean "one or more."

[0035] Those skilled in the art will further recognize that the various illustrative logical components, blocks, modules, circuits, means, logic, and algorithms described in accordance with the embodiments disclosed herein can be embodied using electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the various illustrative components, blocks, means, logic, modules, circuits, and steps have been described above in general terms in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design limitations of the overall computing device.

[0036] The description of the embodiments set forth herein is provided to enable one of ordinary skill in the art to utilize or practice the present disclosure. Various modifications to these embodiments will be apparent to those of ordinary skill in the art. The generic principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present invention is not limited to the embodiments set forth herein. The present invention is to be accorded the broadest scope consistent with the principles and novel features disclosed herein.

[0037] In this disclosure, terms such as "1st," "2nd," "3rd," and "Nth" are used to distinguish at least one entity. For example, an entity referred to as "1st" and an entity referred to as "2nd" may be the same or different. Terms such as "1-1st," "1-2nd," "1-Nth," etc. may also be used to distinguish between these entities.

[0038] In this disclosure, the term "benchmark" may refer to the act of running or testing a model on a node or measuring the performance of a model on a node. Benchmark results or benchmark result information in this disclosure may include information obtained by a benchmark or information that is processed from information obtained by a benchmark.

[0039] In this disclosure, the term "artificial intelligence-based model" may be used interchangeably with artificial intelligence model, computational model, neural network, network function, neural network, and model. In this disclosure, model may be used to collectively refer to a model file and / or model type information. In one embodiment, model type information may refer to information for identifying the execution environment, framework, or type of the model. For example, model type information may include TensorRT, Tflite, and Onnxruntime.

[0040] In this disclosure, the term "node" may correspond to hardware information that is the target of benchmarking for a model. Such hardware information may be used to encompass physical hardware, virtual hardware, hardware that is not accessible externally via a network, hardware that is not visible externally, and / or hardware that is visible within the cloud. For example, a node in this disclosure may include various types of hardware, such as Raspberry Pi, Coral, Jetson Nano, AVH Raspberry Pi, Mobile, etc.

[0041] In the present disclosure, a node in an artificial intelligence-based model may be used to mean a component that constitutes a neural network; for example, a node in a neural network may correspond to a neuron.

[0042] FIG. 1 illustrates a schematic block diagram of a computing device (100) in accordance with one embodiment of the present disclosure.

[0043] In one embodiment of the present disclosure, a computing device (100) may include a processor (110) and a memory (130).

[0044] The configuration of the computing device (100) shown in Figure 1 is merely a simplified example. In one embodiment of the present disclosure, the computing device (100) may include other components for implementing the computing environment of the computing device (100), and the computing device (100) may be configured with only some of the disclosed components.

[0045] In this disclosure, the term "computing device 100" may be used interchangeably with "computing apparatus," and may encompass any type of server and any type of terminal.

[0046] The computing device (100) in this disclosure may refer to any form of component that constitutes a system for implementing embodiments of the present disclosure.

[0047] The computing device 100 may refer to any type of user terminal or any type of server. The components of the computing device 100 described above are merely examples, and some may be omitted or other components may be added. For example, if the computing device 100 includes a user terminal, an output unit (not shown) and an input unit (not shown) may be included in the computing device 100.

[0048] In one embodiment, the computing device 100 may refer to a device that communicates with multiple nodes and manages and / or executes benchmarking of a specific AI-based model against the multiple nodes. For example, the computing device 100 may be referred to as a device farm. In one embodiment, the computing device 100 may refer to a device that interacts with a user to generate a training model, generate a compressed model, and generate download data for model deployment. In one embodiment, the computing device 100 may refer to a device that manages and / or executes benchmarking of an AI-based model against multiple nodes, interacts with a user to generate a training model, generate a compressed model, and generate download data for model deployment.

[0049] In one embodiment, the computing device 100 may refer to a device that generates a learning model through modeling of an input dataset, compresses the input model to generate a lightweight model, and / or generates download data for deploying the input model to a specific node. In this disclosure, "deployment" may refer to any type of activity that makes software (e.g., a model) available. For example, "deployment" may refer to the general process of customizing a model or node to meet specific requirements or characteristics. Examples of such deployment include release, installation, activation, deactivation, uninstallation, update, built-in update, modification, and / or version tracking.

[0050] The computing device (100) in the present disclosure is capable of implementing technical features based on each embodiment in the present disclosure described below.

[0051] For example, the computing device (100) can determine whether to convert the AI-based model based on model type information of the AI-based model input for benchmarking and target type information for identifying the model type to be benchmarked, determine a candidate node list including multiple candidate nodes determined based on the target type information, determine at least one target node based on input data for selecting at least one target node from the candidate node list, and provide benchmark results obtained by executing the target model obtained depending on whether the AI-based model is converted on the at least one target node.

[0052] For example, the computing device (100) may receive first input data including an inference task and a dataset, determine a target model to be benchmarked for the inference task and at least one target node on which the inference task of the target model is executed, and provide benchmark results obtained by executing the target model on the at least one target node.

[0053] For example, the computing device 100 may receive module identification information from another computing device that includes multiple modules performing different operations related to an AI-based model, indicating which of the multiple modules in the other computing device triggers the benchmark operation of the computing device 100, and provide benchmark results to the other computing device based on the module identification information. In this case, the benchmark results provided to the other computing device may differ depending on the module identification information.

[0054] In other embodiments of the present disclosure, the computing device 100 may obtain the results of the benchmark from another computing device or an external entity. In other embodiments of the present disclosure, the computing device 100 may obtain the results of the conversion from another computing device or an external entity (e.g., a conversion device).

[0055] In one embodiment, the processor (110) may be configured with at least one core and may refer to a processor for data analysis and / or processing, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU) of the computing device (100).

[0056] The processor 110 is capable of reading a computer program stored in the memory 130 to provide benchmark results according to one embodiment of the present disclosure.

[0057] According to one embodiment of the present disclosure, the processor (110) is capable of performing calculations for neural network training. The processor (110) is capable of performing calculations for neural network training, such as processing input data for training, extracting features from the input data, calculating errors, and updating neural network weights using backpropagation, in deep learning (DL). At least one of the CPU, GPGPU, and TPU of the processor (110) can process network function training. For example, the CPU and the GPGPU can both train the network function or classify data using the network function. Note that, in one embodiment of the present disclosure, processors of multiple computing devices can be used together to train the network function or classify data using the network function. Furthermore, in one embodiment of the present disclosure, the computer program executed in the computing device (100) can be a program executable by the CPU, GPGPU, or TPU.

[0058] Furthermore, the processor 110 can generally process the overall operation of the computing device 100. For example, the processor 110 can process data, information, signals, etc. input or output through components included in the computing device 100, or execute application programs stored in the storage unit to provide appropriate information or functions to a user.

[0059] In one embodiment of the present disclosure, the memory 130 can store any type of information generated or determined by the processor 110 and any type of information received by the computing device 100. In one embodiment of the present disclosure, the memory 130 can be a storage medium that stores computer software that causes the processor 110 to perform operations in one embodiment of the present disclosure. Therefore, the memory 130 can refer to a computer-readable medium for storing software code necessary to perform multiple embodiments of the present disclosure, data on which the code is executed, and results of the execution of the code.

[0060] In one embodiment of the present disclosure, the memory 130 may refer to any type of storage medium. For example, the memory 130 may include at least one of the following types of storage media: flash memory, hard disk, micro multimedia card, card-type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk. The computing device 100 may also operate in conjunction with web storage that performs the storage function of the memory 130 over the Internet. The above memory descriptions are for illustrative purposes only, and the memory 130 used in the present disclosure is not limited to the above examples.

[0061] The communication unit (not shown) in the present disclosure can be configured regardless of the communication mode, such as wired or wireless, and can be configured with various communication networks, such as a short-range communication network (PAN: Personal Area Network), a short-range communication network (WAN: Wide Area Network), etc. The network unit (150) can also operate based on the well-known World Wide Web (WWW), and can also use wireless transmission technologies used for short-range communication, such as infrared data association (IrDA) or Bluetooth (registered trademark).

[0062] In the present disclosure, the computing device 100 may include any type of user terminal and / or any type of server, and therefore, embodiments of the present disclosure may be implemented using a server and / or a user terminal.

[0063] In one embodiment, a user terminal may include any type of terminal capable of interacting with a server or other computing device, including, for example, a mobile phone, a smartphone, a laptop computer, a personal digital assistant (PDA), a slate PC, a tablet PC, or an ultrabook.

[0064] In one embodiment, a server may include any type of computing system or device, such as, for example, a microprocessor, a mainframe computer, a digital processor, a handheld device, a device controller, and the like.

[0065] In one embodiment, the server may store and manage benchmark results, a candidate node list, node performance information, latency information between nodes and models, and / or conversion result information. The server may include a storage unit (not shown) for storing benchmark results, a candidate node list, node performance information, latency information between nodes and models, and / or conversion result information. Such a storage unit may be included in the server or may exist under the control of the server. As another example, the storage unit may exist outside the server and be embodied in a form capable of communicating with the server. In this case, the storage unit may be managed and controlled by a separate external server different from the server.

[0066] FIG. 2 illustrates an exemplary structure of an artificial intelligence-based model in one embodiment of the present disclosure.

[0067] In the present disclosure, the terms model, artificial intelligence model, artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably.

[0068] The artificial intelligence-based models in the present disclosure can include models that can be used in a variety of domains, such as models for image processing, such as object segmentation, object detection, and / or object classification, and models for text processing, such as data prediction, text semantic inference, and / or data classification.

[0069] A neural network can be composed of a collection of interconnected computational units, generally called nodes. Such nodes can also be called neurons. A neural network is composed of at least one or more nodes. The nodes (or neurons) that make up a neural network can be interconnected by one or more links.

[0070] In an artificial intelligence-based model, a node may be used to mean a component that constitutes a neural network; for example, a node in a neural network may correspond to a neuron.

[0071] In a neural network, one or more nodes connected via links can have a relative relationship of an input node and an output node. The concepts of input node and output node are relative, and any node that is an output node for one node can be an input node for another node, and vice versa. As mentioned above, the relationship between input node and output node can be established around links. One or more output nodes can be connected to one input node via links, and vice versa.

[0072] In a relationship between an input node and an output node connected via a link, the value of the data in the output node can be determined based on the data input to the input node. Here, the link interconnecting the input node and the output node can have a weight. The weight can be variable and can be changed by a user or an algorithm to perform the function required by the neural network. For example, when one or more input nodes are interconnected to one output node by respective links, the output node can determine the value of the output node based on the value input to the input node connected to the output node and the weight set for the link corresponding to each input node.

[0073] As mentioned above, a neural network has one or more nodes interconnected via one or more links, forming a relationship between an input node and an output node within the neural network. In a neural network, the characteristics of the neural network can be determined by the number of nodes and links, the correlation between the nodes and links, and the weight value assigned to each link. For example, if there are two neural networks that have the same number of nodes and links but different link weight values, the two neural networks can be recognized as different.

[0074] A neural network can be composed of a set of one or more nodes. A subset of the nodes in a neural network can form a layer. Some of the nodes in a neural network can form a layer based on their distance from a first input node. For example, a set of nodes that are n distances from a first input node can form an n-th layer. The distance from a first input node can be defined based on the minimum number of links that must be traversed to reach that node from the first input node. However, this definition of a layer is arbitrary for illustrative purposes, and the position of a layer in a neural network can be defined differently from the above description. For example, the layer of a node can be defined based on its distance from the final output node.

[0075] In one embodiment of the present disclosure, a collection of neurons or nodes can be defined in terms of a "layer" or "layer."

[0076] A first input node can refer to one or more nodes in a neural network that directly receive data without passing through a link with other nodes. Alternatively, a first input node can refer to a node in a neural network that does not have any other input nodes connected via links in a link-based node-to-node relationship. Similarly, a final output node can refer to one or more nodes in a neural network that do not have any output nodes with respect to other nodes. Furthermore, a hidden node can refer to a node that is neither a first input node nor a final output node and that constitutes a neural network.

[0077] In one embodiment of the present disclosure, the neural network may be a neural network in which the number of nodes in the input layer is the same as the number of nodes in the output layer, and the number of nodes decreases once and then increases again as the network progresses from the input layer to the hidden layer. In another embodiment of the present disclosure, the neural network may be a neural network in which the number of nodes in the input layer is fewer than the number of nodes in the output layer, and the number of nodes decreases as the network progresses from the input layer to the hidden layer. In another embodiment of the present disclosure, the neural network may be a neural network in which the number of nodes in the input layer is greater than the number of nodes in the output layer, and the number of nodes increases as the network progresses from the input layer to the hidden layer. In another embodiment of the present disclosure, the neural network may be a neural network that combines the above-mentioned neural networks.

[0078] A deep neural network (DNN) can refer to a neural network that includes multiple hidden layers in addition to an input layer and an output layer. Deep neural networks can be used to understand the latent structures of data, such as photos, text, video, audio, protein sequence structures, gene sequence structures, peptide sequence structures, and music (e.g., what objects are in the photo, what is the content and emotion of the text, what is the content and emotion of the audio, etc.), and / or the binding affinity between peptides and MHCs. Deep neural networks can include convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders, generative adversarial networks (GANs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), Q-networks, U-networks, Siamese networks, generative adversarial networks (GANs), etc. The aforementioned deep neural networks are merely examples, and the present disclosure is not limited thereto.

[0079] The artificial intelligence-based model in the present disclosure can be expressed as any of the network structures described above, including an input layer, a hidden layer, and an output layer.

[0080] Neural networks that can be used in the clustering models of the present disclosure can be trained using supervised, unsupervised, semisupervised, or reinforcement learning techniques. Training a neural network can be the process of applying knowledge to the neural network to enable it to perform a specific task.

[0081] Neural networks can be trained to minimize output errors. Training involves repeatedly inputting training data into the neural network, calculating the error between the neural network's output and the target for the training data, and backpropagating the neural network's error from the output layer to the input layer of the neural network to update the weights of each node in the neural network in a way that reduces the error. In supervised learning, training data in which the correct answer is labeled is used (i.e., labeled training data). In unsupervised learning, the correct answer may not be labeled. For example, in supervised learning for data classification, the training data may be data in which each training data is labeled with a category. Labeled training data is input into the neural network, and the error can be calculated by comparing the neural network's output (category) with the label of the training data. As another example, in unsupervised learning for data classification, the error can be calculated by comparing the input training data with the neural network's output. The calculated error is backpropagated in the reverse direction in the neural network (i.e., from the output layer to the input layer), and the connection weights of each node at each layer of the neural network can be updated through backpropagation. The amount of change in the connection weights of each node to be updated can be determined by the learning rate. The neural network calculation for input data and backpropagation of the error can constitute a learning cycle (epoch). The application method of the learning rate can change depending on the number of iterations of the neural network learning cycle. For example, in the early stages of neural network learning, the learning rate can be increased to allow the neural network to quickly achieve a certain level of performance, thereby improving efficiency, and in the later stages of learning, the learning rate can be decreased to improve accuracy.

[0082] In neural network training, the training data is typically a subset of the actual data (i.e., the data to be processed using the trained neural network). Therefore, there may be a learning cycle in which the error associated with the training data decreases while the error associated with the actual data increases. Overfitting is a phenomenon in which the error associated with the actual data increases due to excessive learning of the training data. For example, a neural network that has learned to recognize a yellow cat may be unable to recognize a cat in a different color than yellow, which can be an example of overfitting. Overfitting can increase the error of machine learning algorithms. Various optimization methods can be applied to prevent overfitting. Methods that can be used include increasing the amount of training data, regularization, dropout (which deactivates some nodes in the network during the training process), and batch normalization.

[0083] In one embodiment of the present disclosure, a computer-readable medium is disclosed having stored thereon a data structure including benchmark results and / or artificial intelligence-based models, wherein the data structure can be stored in a storage unit (not shown), executed by a processor (110), and transmitted and received via a communication unit (not shown) in the present disclosure.

[0084] A data structure can refer to the organization, management, and storage of data that allows efficient access and modification. A data structure can refer to the organization of data to solve a specific problem (e.g., data retrieval, data storage, or data modification in the shortest time). A data structure can also be defined as a physical or logical relationship between multiple data elements designed to support a specific data processing function. A logical relationship between multiple data elements can include a connection relationship between multiple user-defined data elements. A physical relationship between multiple data elements can include an actual relationship between multiple data elements physically stored on a computer-readable storage medium (e.g., a permanent storage device). A data structure can specifically include a collection of data, the relationships between the data, and functions or instructions that can be applied to the data. By utilizing effectively designed data structures, a computing device can perform operations while minimizing the use of computing device resources. Specifically, a computing device can increase the efficiency of operations, reading, retrieving, deleting, comparing, exchanging, and searching through effectively designed data structures.

[0085] Data structures can be divided into linear and nonlinear data structures depending on their form. Linear data structures can be structures in which one piece of data is followed by only one other piece of data. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a collection of data that has an internal order. Lists can include linked lists. A linked list can be a data structure in which data is linked in a linear fashion, with each piece of data having a pointer. In a linked list, the pointer can contain information about the connection to the next piece of data or the previous piece of data. Depending on the form, linked lists can be expressed as singly linked lists, doubly linked lists, or circularly linked lists. A stack can be a data array structure with limited access to data. A stack can be a linear data structure in which data can only be manipulated (e.g., inserted or deleted) at one end of the data structure. Data stored in a stack can be a last-in, first-out (LIFO) data structure. A queue is also a data structure that limits access to data, but unlike a stack, it can be a first-in, first-out (FIFO) data structure. A deck can be a data structure that allows data to be processed at both ends of the data structure.

[0086] A nonlinear data structure can be a structure in which multiple pieces of data are connected after one piece of data. A nonlinear data structure can include a graph data structure. A graph data structure can be defined by vertices and edges, and an edge can include a line connecting two different vertices. A graph data structure can include a tree data structure. A tree data structure can be a data structure in which there is only one path connecting two different vertices among the multiple vertices included in the tree. In other words, a graph data structure can be a data structure that does not form a loop.

[0087] The data structure may include a neural network. The data structure including the neural network may be stored on a computer-readable medium. The data structure including the neural network may include data preprocessed for processing by the neural network, data input to the neural network, neural network weights, neural network hyperparameters, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for neural network training. The data structure including the neural network may include any of the components disclosed above. That is, when configuring the data structure including the neural network, it may include all of the data preprocessed for processing by the neural network, data input to the neural network, neural network weights, neural network hyperparameters, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for neural network training, or any combination thereof. In addition to the above configurations, the data structure including the neural network may include any other information that determines the characteristics of the neural network. Furthermore, the data structure may include any form of data used or generated in the computational process of a neural network, and is not limited to the foregoing. The computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may be composed of a collection of interconnected computational units generally called nodes. Such nodes may also be referred to as neurons. A neural network is composed of at least one or more nodes.

[0088] The data structure may include data to be input to the neural network. The data structure including the data to be input to the neural network may be stored on a computer-readable medium. The data to be input to the neural network may include training data input during the training process of the neural network and / or input data to the neural network after training has been completed. The data to be input to the neural network may include data that has undergone pre-processing and / or data that is the target of pre-processing. Pre-processing may include a data processing process for inputting data to the neural network. Thus, the data structure may include data that is the target of pre-processing and data resulting from pre-processing. The above-described data structures are merely exemplary, and the present disclosure is not limited thereto.

[0089] The data structure may include neural network weights. (In this specification, weights and parameters may be considered to have the same meaning.) The data structure including the neural network weights may be stored on a computer-readable medium. The neural network may include multiple weights. The weights may be variable and may be changed by a user or an algorithm to enable the neural network to perform a desired function. For example, if one or more input nodes are interconnected to an output node by respective links, the output node may determine the value of the data to be output from the output data based on multiple values ​​input to the input nodes connected to the output node and weights set for the links corresponding to each input node. The above data structures are merely examples, and the present disclosure is not limited thereto.

[0090] By way of example and not limitation, the weights may include variable weights during the training process of a neural network and / or weights at which the neural network has completed training. Variable weights during training of a neural network may include weights at the start of a training cycle and / or weights that change during a training cycle. Trained weights of a neural network may include weights at which the training cycle has completed. Thus, a data structure including neural network weights may include a data structure including weights that change during the training process of a neural network and / or weights at which the neural network has completed training. Thus, the above weights and / or combinations of each weight are intended to be included in a data structure including neural network weights. The aforementioned data structures are merely exemplary, and the present disclosure is not limited thereto.

[0091] The data structure including the neural network weights can be stored in a computer-readable storage medium (e.g., memory, hard disk) after undergoing a serialization process. Serialization can be a process of converting the data structure into a form that can be stored on the same or a different computing device and later reconstructed for use. A computing device can serialize the data structure and transmit or receive the data over a network. The serialized data structure including the neural network weights can be reconstructed on the same computing device or another computing device through deserialization. The data structure including the neural network weights is not limited to serialization. Furthermore, the data structure including the neural network weights can include data structures that increase computational efficiency while minimizing the use of computing device resources (e.g., nonlinear data structures such as B-tree, R-tree, Trie, m-way search tree, AVL tree, and Red-Black tree). The foregoing are merely examples, and the present disclosure is not limited thereto.

[0092] The data structure may include hyper-parameters of the neural network. The data structure including the hyper-parameters of the neural network may be stored on a computer-readable medium. The hyper-parameters may be variables that are changed by a user. The hyper-parameters may include, for example, a learning rate, a cost function, a number of learning cycle iterations, weight initialization (e.g., setting a range of weight values ​​for weight initialization), and the number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layers). The aforementioned data structures are merely examples, and the present disclosure is not limited thereto.

[0093] FIG. 3 illustrates an exemplary schematic diagram of a system (300) for providing benchmark results, according to one embodiment of the present disclosure.

[0094] In one embodiment, the second computing device (320) may include multiple modules performing different operations related to an AI-based model. For example, the second computing device (320) may include a first module (330), a second module (340), and a third module (350). In one embodiment, the first module (330) may generate a learning model based on an input dataset. The second module (340) may compress the input model to generate a lightweight model. The third module (350) may generate download data for deploying the input model to at least one target node. While three modules are illustrated in the example of FIG. 3, those skilled in the art will readily understand that the second computing device (320) may include a variety of modules depending on the implementation.

[0095] In one embodiment, the first computing device (310) and the second computing device (320) can interact to provide benchmark results to a user. For example, the first computing device (310) can provide the second computing device (320) with benchmark results necessary for the operation of the second computing device (320) in response to a request from the second computing device (320).

[0096] In one embodiment, FIG. 3 depicts the first computing device (310) as a separate entity existing outside of the second computing device (320), although in some implementations the first computing device (310) may operate as a module included in the second computing device (320).

[0097] In one embodiment, the first computing device 310 may receive a benchmark request from an entity other than the second computing device 320 and provide benchmark results in response to the request. For example, the first computing device 310 may provide results for benchmarking a user-prepared artificial intelligence-based model (e.g., a user-created learning model or a compressed model).

[0098] In one embodiment, the first computing device (310) receives module identification information indicating which module of a plurality of modules of the second computing device (320) triggers a benchmark operation of the first computing device (310), and can provide benchmark results to the second computing device (320) based on the module identification information. The benchmark results provided to the second computing device (320) can vary depending on the module identification information. For example, if the module identification information indicates a first module (330), the first computing device (310) can provide performance information for the entire input model to the second computing device (320). If the module identification information indicates a second module (340), the first computing device (310) can provide performance information for the entire input model to the second computing device (320) and can provide performance information for each block of the input model. As another example, if the module identification information indicates the first module (330), the first computing device (310) can provide the second computing device (320) with benchmark results for determining a target node on which to execute a learning model or converted learning model corresponding to the input dataset, and if the module identification information indicates the second module (340), the first computing device (310) can provide the second computing device (320) with benchmark results including compressed configuration data used to generate a lightweight model corresponding to the input model.

[0099] In one embodiment, the first computing device (310) may correspond to an entity that manages multiple nodes. The first computing device (310) may execute benchmarking on nodes included in a node list that includes a first node (360), a second node (370), ..., an Nth node (380). While FIG. 3 illustrates an example in which the first node (360), the second node (370), ..., and the Nth node (380) are included in the first computing device (310), depending on the implementation, the first node (360), the second node (370), ..., and the Nth node (380) may exist outside the first computing device (310) and interact with it through communication with the first computing device (310).

[0100] In one embodiment, the first computing device (310) can generate benchmark results for multiple nodes in response to a user request and / or a request from the second computing device (320). In one embodiment, the first computing device (310) can interact with the conversion device (390) to generate benchmark results for multiple nodes in response to a user request and / or a request from the second computing device (320).

[0101] In one embodiment, the conversion device (390) is a device for converting a first model to a second model. As shown in Figure 3, the conversion device (390) can exist as a separate entity from the first computing device (310) and the second computing device (320), or can operate as part of the first computing device (310) and / or the second computing device (320).

[0102] In one embodiment, the benchmark results may include results of executing (e.g., inferencing) the artificial intelligence model on the target node. As one example, the benchmark results may include performance measurements when the artificial intelligence model is executed on the target node. As another example, the benchmark results may include performance measurements when the converted artificial intelligence model is executed on the target node.

[0103] In one embodiment, the benchmark results can be used in a variety of ways for a variety of purposes. For example, the benchmark results can be used to determine a target node on which to run the prepared model. For example, the benchmark results can be used to generate a candidate node list corresponding to an input model. For example, the benchmark results can be used to optimize or compress the prepared model. For example, the benchmark results can be used to deploy the prepared model on a target node.

[0104] FIG. 4 illustrates an exemplary flowchart for providing benchmark results in accordance with one embodiment of the present disclosure.

[0105] In one embodiment, the method illustrated in Figure 4 may be performed on a computing device (100). As an example, the method illustrated in Figure 4 may be performed by a first computing device (310). As another example, the method illustrated in Figure 4 may be performed by a computing device (100) that includes both the first computing device (310) and the second computing device (320).

[0106] The following is a specific example in which the steps in Fig. 4 are executed by a computing device 100. It will be readily apparent to those skilled in the art that some of the steps shown in Fig. 4 may be omitted or additional steps may be added depending on the implementation mode.

[0107] In one embodiment, the computing device (100) can obtain (410) model type information of the artificial intelligence-based model input for benchmarking and target type information for identifying the model type to be benchmarked.

[0108] The computing device 100 may also receive input data including instructions to benchmark a particular model and information related to the model to be benchmarked. For example, the input data may include information related to a model file on which the modeling was performed and a model type to be benchmarked. In another example, the input data may include a model file on which the modeling was performed and model type information corresponding to the model file. In another example, the input data may include a model file on which the modeling was performed, model type information corresponding to the model file, and information related to a target type to be benchmarked. In response to the input data, the computing device 100 may provide information for selecting nodes on which the model will be benchmarked.

[0109] In one embodiment, the model type information may include any type of information for identifying the input artificial intelligence-based model. For example, the model type information may include information representing the model's execution environment, such as Tflite, Onnxruntime, or Tensorrt. For example, the model type information may include library information and software version information related to the model's execution environment. In the above example, the model type information may be expressed as Tflite's Python 3.7.3 and Pillow 5.4.1.

[0110] In one embodiment, the target type information for benchmarking may include any form of information for identifying an artificial intelligence-based model for benchmarking.

[0111] In another embodiment, the computing device 100 can extract corresponding model type information from an input AI model. The computing device 100 can parse the input AI model (e.g., a model file) to obtain the model's execution environment and / or library information. The computing device 100 can compare the extracted model type information with the input target type information to determine whether to convert the model. In the above embodiment, the model type information of the input AI-based model can be determined based on the input AI-based model without user input defining the model type information.

[0112] In one embodiment, the target type information may differ from the model type information of the input AI-based model. In this case, the computing device 100 may obtain a conversion result in which the input AI-based model is converted to have the target type information. Differences between the target type information and the model type information of the input model may mean that information related to the model's execution environment and / or library information related to the execution environment differ. For example, the conversion may include replacing operators included in the input model with other operators to correspond to the target type information. For example, the conversion may include changing library information or software version of the input model to correspond to the target type information. For example, the conversion may include changing the execution environment of the input model to an execution environment corresponding to the target type information.

[0113] As another example, if the model type information of the input artificial intelligence-based model and the target type information are identical, the computing device (100) can generate benchmark results for the input artificial intelligence-based model without performing a conversion operation.

[0114] In one embodiment, the computing device 100 may determine 420 whether to convert the artificial intelligence-based model based on the model type information and the target type information.

[0115] The computing device (100) can determine whether to convert the AI-based model by comparing the model type information of the input model with the target type information. The computing device (100) can determine whether the model type information of the input AI-based model matches the input target type information. If the model type information of the input AI-based model does not match the input target type information, the computing device (100) can generate benchmark results based on the conversion results. If the model type information of the input AI-based model matches the input target type information, the computing device (100) can generate benchmark results by executing the input AI-based model on the target node.

[0116] In one embodiment, if the model type information and the target type information are different, the computing device (100) decides to convert the artificial intelligence-based model to correspond to the target type information, but if the model type information and the target type information correspond to each other, the computing device (100) may decide not to convert the artificial intelligence-based model and use the artificial intelligence-based model as a target model to be benchmarked.

[0117] In one embodiment, the decision on whether to convert may be made in response to input of model type information and target type information corresponding to the model. Thus, when generating or acquiring a candidate node list, predicted performance information for the converted model (or operator) and each of the plurality of candidate nodes may be generated in accordance with the decision on whether to convert. In one embodiment, the predicted performance information may refer to predicted information related to performance determined based on previously measured performance information for each of the plurality of candidate nodes, per model or operator. As an example, the predicted performance information may include predicted latency information.

[0118] In another embodiment, the computing device 100 may determine whether to perform conversion on the model based on information related to the model (e.g., a model file, model type information corresponding to the model file, and / or target type information) and information about the selected node. For example, the computing device 100 may determine whether to perform conversion based on whether the selected node supports the determined model or whether the selected node supports operators included in the determined model. In this case, the selected node may not support the determined model. In such a case, the computing device 100 may determine whether to perform conversion on the determined model or whether to perform conversion on at least some of the operators included in the determined model. As another example, if the selected node does not support the determined model, the first computing device 1000c may determine that conversion is necessary for the determined model or may determine to replace the selected node with another node.

[0119] In one embodiment, the computing device (100) may provide a candidate node list (430) that includes a plurality of candidate nodes determined based on the target type information.

[0120] In one embodiment, the candidate nodes can be used to determine a target node from among a plurality of nodes to benchmark the model, and the target node to be benchmarked can be determined from among the plurality of candidate nodes included in the candidate node list based on the input data.

[0121] In one embodiment of the present disclosure, the candidate nodes can be determined in various ways. In one embodiment, the candidate nodes can be determined based on the result of the decision on whether to perform conversion, the AI-based model (or model type information), and the target type information.

[0122] For example, among the plurality of nodes under the management of the computing device 100, a node that supports the input artificial intelligence-based model can be determined as a candidate node. For example, among the plurality of nodes under the management of the computing device 100, a node that supports an execution environment corresponding to the target type information can be determined as a candidate node.

[0123] For example, among nodes supporting an execution environment corresponding to the target type information, a plurality of first nodes having an execution environment supporting a first operator included in an AI-based model may be determined as the candidate nodes. As an example, the computing device 100 may extract a plurality of operators included in the input AI-based model. Among nodes having a runtime matching the target type information, if the runtime matches but the version of the runtime supported by the node does not support the extracted operator, the node on which the version of the runtime is installed may be excluded from the candidate nodes.

[0124] For example, among the nodes having an execution environment corresponding to the target type information, a plurality of second nodes having an execution environment that does not support a first operator included in the artificial intelligence-based model but supports a second operator different from the first operator that can replace the first operator can be determined as the candidate nodes. As an example, if there is another operator that can replace the unsupported operator, the computing device (100) can request the user to replace or change the operator. If the user requests to replace the operator, the computing device (100) can include the node in the candidate nodes, and if not, can remove the node from the candidate nodes.

[0125] For example, a plurality of third nodes having free memory capacity exceeding the size of the artificial intelligence-based model can be determined as candidate nodes.

[0126] The computing device (100) can generate the candidate node list by combining multiple candidate node determination methods described in the various examples above.

[0127] In one embodiment, the computing device 100 can transmit the candidate node list to the computing device that requested the benchmark, and can determine the target node for benchmarking based on the user's selection based on the candidate node list.

[0128] In one embodiment, the candidate node list may include identification information for each of the candidate nodes and expected latency information at each of the candidate nodes when the target model is executed.

[0129] In one embodiment, the expected latency information may include an expected inference time for each model of each node. A smaller value of the expected latency information may indicate a shorter inference time. Therefore, since the expected latency value may be interpreted as a performance indicator of a combination of an AI-based model and a node, the computing device 100 may provide a candidate node list sorted in order based on the magnitude of the expected latency information. In the above example, a candidate node list sorted in order of the magnitude of the latency information may be provided.

[0130] In one embodiment, the identification information of the candidate node list may include hardware information corresponding to the candidate node, such as the product name corresponding to the hardware, installed execution environment information, library information related to the execution environment, power mode information, fan mode information, current board temperature information, and / or current board power usage information.

[0131] In one embodiment, the power mode information may be determined based on CPU usage. For example, if all CPU cores are being used, the power mode information may be determined as MAX, or may be determined in a manner that quantitatively expresses usage, such as 30W, 20W, 15W, 10W, etc. For example, the larger the quantitative amount of the power mode information, the lower the latency. As another example, if the power mode is MAX, the latency may be lower than that of other nodes that are not using the power mode.

[0132] In one embodiment, the fan mode information can be expressed in the form of information indicating the strength of the fan, such as Null, Quite, Cool, etc. For example, when the fan mode is Quiet, the board temperature can be lowered more than when the fan mode is Null, so latency is likely to be lower. For example, when the fan mode is Cool, the board temperature can be lowered more than when the fan mode is Cool, so latency is likely to be lower.

[0133] In one embodiment, the library information may refer to library information required to install the execution environment (e.g., runtime) information installed on a particular node. Depending on the characteristics of the node, it may include multiple execution environments, which may make the library information compatible across multiple execution environments.

[0134] In one embodiment, the current power usage of a board may refer to the power usage obtained from power measurement sensors connected to multiple nodes. It may be interpreted that the smaller the value of the current power usage of a board, the higher the possibility of using the node.

[0135] In one embodiment, the order of the candidate nodes included in the candidate node list can be determined based on the magnitude of the expected latency information. The computing device 100 can provide a candidate node list sorted based on the magnitude of the expected latency information. In the example described above, the candidate node list can be provided in ascending order of the magnitude of the latency information.

[0136] In another embodiment, the ordering of multiple candidate nodes can be determined based on factors such as memory usage and CPU usage. For example, the ordering of multiple candidate nodes can be determined based on factors other than expected latency information alone, such as memory usage and CPU usage. In the example described above, if the difference in the magnitude of expected latency information between a first candidate node and a second candidate node is within a predetermined threshold range, the ordering of the first candidate node and the second candidate node can be determined based on the memory usage and CPU usage of the first candidate node and the second candidate node. As an example, if the expected latencies are the same, the ordering can be based on the current memory (e.g., RAM) usage and CPU usage.

[0137] In another embodiment, the computing device 100 may sort specific nodes, such as the Jetson series, by adding additional factors to the sort order. For example, for specific types of nodes, such as the Jetson series, additional sorting may be performed for the specific nodes. As another example, the computing device 100 may sort specific types of nodes relative to other types based on expected latency, but if there is not much difference in expected latency between multiple nodes of that type, the computing device 100 may add considerations such as the power field and / or fan field to the sort order. For example, the computing device 100 may further consider factors corresponding to the power field and sort the nodes in descending order of power field value. For example, for specific nodes, such as the Jetson series, if the power field values ​​are the same or the difference is within a predetermined threshold range, the computing device 100 may further sort the nodes in descending order of fan operation based on the magnitude or strength of fan operation.

[0138] As described above, for multiple nodes with similar predicted latency information, it is possible to determine the order of candidate nodes by adding additional factors to be considered. In this way, when providing a list of candidate nodes, the candidate nodes are arranged in a way that allows the user to intuitively check the predicted performance, allowing the user to more easily and efficiently check the predicted performance of the nodes on the candidate node list and more efficiently determine the target node.

[0139] In one embodiment, the computing device 100 can coordinate the conversion operations in the process of generating the candidate node list.

[0140] For example, the computing device 100 may acquire a target model in which an AI-based model has been converted to correspond to target type information. In another example, the computing device 100 may perform conversion internally or acquire a target model converted to the target type from an external conversion device. The computing device 100 may acquire sublatency information corresponding to each of multiple operators included in the target model. For example, one operator may correspond to one sublatency. In this case, sublatency information may be calculated for each of the candidate nodes. There may be multiple operators in the model, and the computing device 100 may measure or determine the sublatency that occurs when each operator is executed at a specific target node. The computing device 100 may generate predicted latency information for the target model at each of the candidate nodes based on the sublatency information of the multiple operators. For example, the computing device 100 may obtain expected latency information corresponding to the model by summing sublatencies corresponding to each of the operators included in the model, and may provide a candidate node list including the expected latency information and identification information of the candidate nodes.

[0141] In one embodiment, the AI-based model may include multiple operators. The operators may correspond to the operations of the AI-based model. Different operations in the model may have different operators. For example, the model may include an operator corresponding to Conv2D, which refers to a convolution operation on a 2D image.

[0142] In one embodiment, the computing device (100) may generate the candidate node list using a latency table that matches a plurality of operators for each model with a plurality of nodes.

[0143] In one embodiment, if a latency table exists, the computing device (100) can obtain predicted performance information for each of the plurality of candidate nodes using the latency table without measuring the performance of each of the plurality of candidate nodes in the process of generating the candidate node list. If a latency table does not exist, the computing device (100) can measure the performance of each of the plurality of candidate nodes in the process of generating the candidate node list and generate a list of the plurality of candidate nodes including the measured performance.

[0144] As an example, one latency table can be generated for one model. In one embodiment, the latency table can include sublatency information obtained by executing each of a plurality of pre-stored operators on a pre-stored node. One of the rows and columns of the latency table contains information related to the operators, and the other contains information related to the nodes. The value of the element can be expressed as a sublatency. After the latency table is initially generated by measurements on the computing device (100), the predicted performance of a candidate node list can be quickly obtained using the pre-generated latency table without additional measurements or executions.

[0145] As described above, the computing device (100) can use a pre-generated latency table to obtain sub-latency information corresponding to each of a plurality of operators included in the target model, and generate predicted latency information for the target model for each of a plurality of candidate nodes based on the sub-latency information for the plurality of operators. Through the above-described method, a candidate node list including predicted latency information and candidate node identification information can be provided. The computing device (100) can determine a latency table corresponding to the input model, and generate predicted latency information for the model for each candidate node based on the node-specific sub-latency information for the plurality of operators included in the determined latency table. The predicted latency information for the model can be generated by summing the sub-latency information for each of the plurality of operators.

[0146] In one embodiment, the computing device 100 may use a convert matching table to generate the candidate node list.

[0147] In one embodiment, the computing device 100 can determine which type (e.g., Type A) of model to convert to which type (e.g., Type B) based on the input information. If there is a history of conversion from Type A to Type B, a conversion matching table corresponding to conversion from Type A to Type B may also exist. In such a case, the computing device 100 can extract operators that cause latency by simply analyzing the model corresponding to Type A, and calculate predicted performance information based on the latency when the extracted operators are converted to Type B.

[0148] In one embodiment, if there is no history of conversion from Type A to Type B, the computing device (100) can convert the model (or operator) corresponding to Type A to Type B, analyze the converted model to extract operators, calculate or measure latency for the extracted operators, and calculate predicted performance information.

[0149] In one embodiment, the computing device 100 may pre-generate the aforementioned conversion matching tables for a plurality of predetermined conversion types and / or for a plurality of conversion types (or models) whose conversion times exceed a predetermined threshold time.

[0150] In one embodiment, when an operator extracted from an AI-based model corresponding to model type information is converted to correspond to target type information, the convert matching table may include sublatency information corresponding to the converted operator. For example, one of the rows and columns of the convert matching table may include an operator corresponding to the model type information before conversion, and the other may include an operator corresponding to the target type information after conversion. Elements in the convert matching table may include sublatency information corresponding to the converted operator. In one embodiment, the convert matching table may be generated for each node. In the above example, one node may correspond to one convert matching table. In one embodiment, the convert matching table may be generated for each node and for each combination of multiple models. In the above example, sublatency information for each operator at the first node of the second model converted from the first model may be represented as a single table.

[0151] In another embodiment, the computing device 100 may use a conversion matching table containing information on the conversion between operators, i.e., a specific operator in a specific model is converted to another operator in another model. In the above example, the conversion matching table may contain information indicating how operators are converted within a model when a conversion between models occurs. For example, one of the rows and columns in the conversion matching table may contain operators in the model before conversion, and the other may contain operators in the model after conversion. The aforementioned conversion matching table may be used to determine how operators are changed by conversion. In such an embodiment, the computing device 100 may use the conversion matching table to identify operators in the converted model, and may use the latency table described above to obtain node-specific sublatency information for each of the multiple operators in the converted model. By summing the sublatencies, it may be possible to obtain node-specific expected performance information for the converted model.

[0152] In one embodiment, when it is determined to convert an AI-based model, the computing device (100) determines whether a conversion matching table exists for matching model type information with target type information, and if a conversion matching table exists, generates expected latency information of the target model for each of the candidate nodes based on operators included in the AI-based model and the conversion matching table. The computing device (100) can provide the candidate node list including the expected latency information and identification information of the candidate nodes.

[0153] In one embodiment, the computing device (100) can combine the latency table and the convert matching table to determine the sub-latency for each of the multiple operators in the converted model, and sum the determined sub-latencies to generate expected latency information for the converted model.

[0154] In another embodiment, the computing device (100) may generate predicted latency information for a target model at each of a plurality of candidate nodes by combining a latency table containing node-specific latency measurement results on a per-model basis rather than on a per-operator basis and / or a conversion matching table containing node-specific and conversion-based latency measurement results on a per-model basis rather than on a per-operator basis.

[0155] In one embodiment of the present disclosure, the computing device 100 can determine whether to perform conversion based on a comparison between model type information corresponding to the input model and input target type information. If it is determined that conversion is to be performed, the computing device 100 can extract multiple operators included in the input model and determine candidate nodes to include in a candidate node list from multiple nodes having execution environments that match the target type information.

[0156] In one embodiment, if a particular node has an execution environment that matches the target type information but the execution environment does not support an operator included in the input model, the computing device 100 may determine not to include the particular node in which the execution environment is installed in the candidate node list. For example, if the runtime supported by the node matches an operator included in the input model but the version of the runtime of the node does not support the operator included in the input model, the node in which the version of the runtime is installed may be excluded from the candidate nodes.

[0157] In one embodiment, the computing device (100) can send a request to determine whether to replace an operator when a specific node does not support an operator included in an input model in its execution environment but an operator that can replace the operator exists. If the computing device (100) receives input from a user indicating that the operator should be replaced, the computing device (100) can include the specific node in a candidate node list. However, if the computing device (100) receives input from a user indicating that the operator should not be replaced, the computing device (100) can exclude the specific node from the candidate node list. For example, if the runtime supported by the node matches an operator included in the input model but the version of the node's runtime does not support the operator included in the input model, and if the version of the node's runtime supports an operator that can replace the operator, the computing device (100) can determine the node as a candidate node by requesting that the operator be replaced. If the computing device (100) receives input indicating that the operator should be replaced, the computing device (100) can determine the node as a candidate node; otherwise, the computing device (100) can exclude the node from the candidate node list.

[0158] In one embodiment, the candidate node list may include a document structure, for example in the form of a table.

[0159] In one embodiment, the computing device (100) may determine (440) at least one target node based on the input data to select at least one target node from the list of candidate nodes.

[0160] In one embodiment, the computing device 100 can receive user input data selecting a particular node from the list of candidate nodes, and the computing device 100 can determine the selected node included in the user input data as the target node.

[0161] In one embodiment, the computing device (100) can receive user input data selecting a single target node from the list of candidate nodes. In another embodiment, the computing device (100) can receive user input data selecting multiple target nodes from the list of candidate nodes. In another embodiment, the computing device (100) can automatically select the node from the list of candidate nodes that has the highest performance based on a particular factor (e.g., latency) as the target node without receiving user input.

[0162] In one embodiment, the computing device (100) can provide benchmark results (450) obtained by executing a target model on at least one target node, the target model being obtained depending on whether the artificial intelligence-based model is converted.

[0163] In one embodiment, the computing device 100 is capable of generating benchmark results that include results of inferring the target model at the target node.

[0164] In one embodiment, if one node is determined as the target node, benchmark request information can be sent to the node. In one embodiment, if multiple nodes are determined as target nodes, benchmark request information can be sent to each of the multiple nodes. The benchmark request information can include information related to the target model to be benchmarked. The information related to the target model can include, for example, files or links related to the model and / or target type information for the model.

[0165] In one embodiment, the benchmark results may be generated by the computing device (100) or may be performed by another server (e.g., a server including multiple nodes) under the control of the computing device (100).

[0166] In one embodiment, the benchmark results may include performance information for the target node of the target model.

[0167] In one embodiment, the benchmark results may vary depending on which module of the other computing device triggered or requested the benchmark operation of the computing device 100. In another embodiment, the benchmark operation may vary depending on which module of the other computing device triggered or requested the benchmark operation of the computing device 100. For example, if a first module triggered the benchmark operation of the computing device 100, the computing device 100 may provide performance information for the entire input model, and if a second module triggered the benchmark operation of the computing device 100, the computing device 100 may provide performance information for the entire input model and may also provide performance information for each block of the input model. As another example, if the module that triggered the benchmark operation of the computing device (100) is a first module, the computing device (100) can provide benchmark results for determining a target node on which to run a learning model or a converted learning model corresponding to the input dataset, and if the module that triggered the benchmark operation of the computing device (100) is a second module, the computing device (100) can provide benchmark results including compressed configuration data used to generate a lightweight model corresponding to the input model.

[0168] By way of example and not limitation, the multiple modules that trigger benchmarking operations of the computing device (100) may include a first module that generates a learning model based on an input dataset, a second module that compresses the input model to generate a lightweight model, and a third module that generates download data for deploying the input model to at least one target node.

[0169] In one embodiment, the benchmark request information may include information regarding conversion of the model. Based on the information regarding conversion included in the benchmark request information, the input model may be converted.

[0170] In one embodiment, when the computing device 100 determines to convert an AI-based model to correspond to target type information, the computing device 100 can determine a specific converter from among a plurality of converters using information related to the input model. For example, the computing device 100 can determine converter identification information corresponding to a combination of model type information and target type information of the input model. Once a converter corresponding to the determined conversion identification information is determined, the determined converter can perform the conversion operation. For example, the computing device 100 can obtain a target model into which the AI-based model has been converted using a model file corresponding to the AI-based model and converter identification information corresponding to a combination of model type information and target type information. As described above, when a conversion is determined, the conversion can be performed using the model file to be converted and information for identifying the converter that will perform the conversion (e.g., an identifier for identifying the combination of the model type before conversion and the model type after conversion). As an example, the conversion can be performed by a conversion device (e.g., a conversion server).

[0171] As another example, depending on the embodiment, the above conversion may be performed by the computing device 100. In the above example, if the model file resulting from the project is not for an execution environment corresponding to a specific type (e.g., Tensorrt) but for an execution environment corresponding to another type (e.g., Onnxruntime), the computing device 100 may use a conversion function to convert the model of the other type (e.g., Onnxruntime) into a model of the specific type (e.g., Tensorrt) and provide the model file resulting from the conversion.

[0172] In one embodiment, an AI-based model can be converted to a target model in a virtualized operating system using a converter Docker image corresponding to converter identification information. For example, an entity performing the conversion (e.g., a conversion server) can obtain a Docker image corresponding to the input converter identification information and execute the converter's sh file within Docker to perform the model conversion. In this case, the sh file can refer to a file containing instructions to be executed within Docker.

[0173] In one embodiment, the benchmark results may include a first type of quantitative information related to time and a second type of quantitative information related to memory usage.

[0174] In one embodiment, the benchmark results obtained by executing the target model on at least one target node may include preprocessing time information, which is information related to the time required for preprocessing of inference of the target model on at least one target node; inference time information, which is information related to the time required for inference of the target model on at least one target node; preprocessing memory usage information, which is information related to the memory usage used for preprocessing of inference of the target model on at least one target node; inference memory usage information, which is information related to the memory usage used for inference of the target model on at least one target node; quantitative information related to the inference time obtained by repeatedly inferring the target model a predetermined number of times on at least one target node; and / or quantitative information related to memory usage in each of the NPU, CPU, and GPU obtained by inferring the target model on at least one target node.

[0175] In one embodiment, the preprocessing time information may include, for example, information about the time required for preprocessing before performing an inference operation, such as calling a model. In addition, the preprocessing time information may include quantitative information (for example, the minimum, maximum, and / or average value of the time required for pre-inference) related to the time required for pre-inference when the pre-inference is repeated a predetermined number of times to activate a GPU or the like prior to measuring the inference value.

[0176] In one embodiment, the inference time information is information related to the time required for the inference process, and may include, for example, time information related to the time required for the first inference operation for a model and / or inference time information obtained when inference is repeated a predetermined number of times, such as minimum time information, maximum time information, average time information, and / or median time information. Furthermore, for example, in a situation where the CPU takes over and processes a calculation that the NPU cannot process, the NPU becomes idle, and the inference time information may include a first cycle value when the NPU becomes idle. Furthermore, the inference time information may include a second cycle value when inference is performed by the NPU, and / or a third cycle value obtained by adding the first and second cycle values.

[0177] In one embodiment, the benchmark result information may further include the preprocessing memory usage information and total time information obtained by adding up the quantitative information relating to the inference time.

[0178] In one embodiment, the benchmark result information may further include a quantitative value relating to RAM usage, ROM usage, total memory usage, and / or SRAM area used by the NPU.

[0179] In one embodiment, when multiple benchmark results are generated due to multiple nodes being selected as target nodes, the computing device (100) can sort the multiple benchmark results using latency as a sorting criterion. For example, the benchmark results can be sorted and output in order of lowest latency. In another example, when the latencies are within a predetermined close range, or when benchmark results corresponding to each of the same multiple nodes exist, the benchmark results can be sorted in order by adding memory usage and / or CPU usage as sorting criteria. The sorting of the benchmark results can include features related to the sorting of the candidate node list.

[0180] In one embodiment, the benchmark results may include a document structure, for example in the form of a table.

[0181] In one embodiment, the computing device 100 may change the method of sending benchmark request information depending on whether wireless communication is available among the connected nodes. Wireless communication may include, for example, HTTP communication. For example, when transmitting a benchmark request to an independent node capable of wireless communication, the computing device 100 may send the benchmark request information to the node or a server associated with the node. For example, when transmitting a benchmark request to a node that does not have wireless communication, the computing device 100 may send the benchmark request information to a node (e.g., an Rpi4) that is capable of wireless communication and is connected to the node that does not have wireless communication via USB / GPIO. The node (e.g., a node capable of HTTP communication) that receives the benchmark request information may execute a program on the connected node to be benchmarked (i.e., a node that does not have HTTP communication) using serial communication via a USB / GPIO connection to obtain benchmark results. The expected memory usage may be measured by executing an execution environment corresponding to the target model type to be executed on an independent node capable of wireless communication. Using the measured expected memory usage and benchmark request information, it is possible to create and compile a benchmark program that will be executed in the node that is the target of the benchmark and that does not have wireless communication. The created and compiled program can be flashed via serial communication to the node that is the target of the benchmark and that does not have wireless communication. In this way, it is possible to obtain benchmark results for the node that does not have wireless communication.

[0182] In one embodiment, when the target node includes a node that cannot be externally confirmed (e.g., a virtual node), the computing device (100) can obtain benchmark results using the following method.

[0183] In response to receiving a first low-power wireless signal from an externally unobservable node, the computing device (100) may send a first acknowledgment message to the node, the first acknowledgment message including a benchmark task for benchmarking a target model at the node. For example, the first low-power wireless signal may include a beacon signal. For example, the first low-power wireless signal may include information regarding whether the node is performing a benchmark, information regarding memory usage of the node, and hardware identification information of the node.

[0184] The computing device 100 may receive a second low-power wireless signal (e.g., a callback signal) from the node, the second low-power wireless signal including the benchmark result generated by the node. In another embodiment, if the computing device 100 fails to receive the callback signal within a time corresponding to a predetermined waiting time threshold, the computing device 100 may determine that the benchmark task at the node has failed and set the node to an inactive state. The computing device 100 may set the node to an active state in response to receiving a third low-power wireless signal (e.g., a beacon signal) from the node set to an inactive state. In this case, the benchmark task included in the first response message may include target model information that the node can download and node configuration information used to convert the target model downloaded by the node. Furthermore, the benchmark result generated by the node may include a result obtained by executing the target model in the execution environment of the node based on the node configuration information and the target model information.

[0185] FIG. 5 illustrates an exemplary tabular resource structure (500) used to generate a candidate node list in accordance with one embodiment of the present disclosure.

[0186] The data structure (500) in FIG. 5 is shown for illustrative purposes only, and other data structures with varying numbers of nodes, operators, types of performance information, and / or models may be considered within the scope of this disclosure, depending on the implementation.

[0187] The document structure (500) shown in Figure 5 allows matching of multiple operators for each model with multiple nodes. Rows (or columns) (510) in the document structure (500) indicate identification information for the nodes, and columns (or rows) (520) indicate identification information for the operators included in a particular model.

[0188] In one embodiment, the values ​​of elements included in the data structure 500 may indicate performance information when a particular operator of a particular model is executed (e.g., inferred) at a particular node. For example, as illustrated in FIG. 5, the performance information may include information related to latency. In this case, the performance information may, for example, refer to performance information for each operator. For example, in this case, the performance information may refer to sublatency in FIG. 4.

[0189] 5, it can be interpreted that the smaller the element value (i.e., the smaller the latency value), the better the performance. For example, the first operator may have better performance than the third node, the first node, the second node, and the fourth node, in that order.

[0190] In one embodiment, the computing device (100) can use one or more data structures (500) to generate predicted performance information for the model-node combinations required to generate a candidate node list. The data structure (500) can further include a value that sums the performance information of multiple operators for each node. In such an embodiment, a first node can have predicted performance information of 3 ms + 4 ms + 5 ms + 7 ms = 19 ms for a particular model, a second node can have predicted performance information of 34 ms, a third node can have predicted performance information of 22 ms, and a fourth node can have predicted performance information of 32 ms. For example, the candidate node list can sort the nodes in descending order of performance. In such an example, the candidate nodes can be sorted in the following order: first node, third node, fourth node, second node.

[0191] In one embodiment, the data structure 500 can be generated for each model. In such an embodiment, one model can correspond to one data structure 500, but depending on the implementation, one data structure 500 can be generated to correspond to multiple models.

[0192] In one embodiment, the resource structure 500 may correspond to a latency table that matches a plurality of operators with a plurality of nodes for each model, and the computing device 100 may use the latency table to generate a candidate node list.

[0193] In one embodiment, the latency table may include sublatency information that can be obtained by executing each of a plurality of pre-stored operators on a pre-stored node. One of the rows and columns of the latency table may include information related to the operators, and the other may include information related to the nodes. The value of the element may be represented as a sublatency. Once the latency table is generated by measurements on the computing device (100), the pre-generated latency table may be used in response to a user input to quickly generate or obtain a candidate node list without additional measurements or executions.

[0194] As described above, the computing device (100) can use a pre-generated latency table to obtain sub-latency information corresponding to each of a plurality of operators included in the target model, and generate predicted latency information for the target model at each of a plurality of candidate nodes based on the sub-latency information for the plurality of operators. The computing device (100) can determine a latency table corresponding to the input model, and generate predicted latency information for the model for each of the candidate nodes based on the node-specific sub-latency information for the plurality of operators included in the determined latency table. The predicted latency information for the model can be generated by summing the sub-latency information for each of the plurality of operators.

[0195] In one embodiment, the resource structure 500 can be updated by the computing device 100 in response to change events related to model updates, new node additions, and / or new model additions. In the absence of change events, the resource structure 500 can be utilized to generate a candidate node list in response to user input without the computing device 100 having to perform any additional execution or measurement.

[0196] FIG. 6 illustrates an exemplary tabular resource structure (600) used to generate a candidate node list in accordance with one embodiment of the present disclosure.

[0197] 6 is a table-like data structure (600) that indicates, when a model is converted, the source operators (620) of the pre-conversion model (e.g., source model) and the target operators (610) of the converted model (e.g., target model). The values ​​of the elements of data structure (600) can indicate the expected performance (e.g., sublatency) of a particular node in the target operator (610) when the source operator (620) is converted to the target operator (610).

[0198] In one embodiment, a data structure (600) can be generated for each combination of multiple models and for each node. For example, one data structure (600) can cover the case where a first model is converted to a second model and the converted second model is executed on a first target node. In this example, the case where the second model is converted back to the first model can be covered by another data structure. Depending on the implementation, the scope covered by one data structure (600) can vary. For example, one data structure (600) can include the sublatencies of converted operators in multiple nodes.

[0199] In another example, the data structure 600 may be generated on a node-by-node basis. That is, when converting from a source model to a target model, it is possible to determine which source operators in the source model are changed to which target operators in the target model. In such a case, the computing device 100 may refer to the data structure 600 to obtain sublatency information between the source operators and the target operators involved in the conversion, and accordingly obtain expected performance information for the converted target model or the converted target operators.

[0200] In one embodiment, the data structure 600 may correspond to a conversion matching table. When the computing device 100 determines to convert an AI-based model, it determines whether a conversion matching table exists. If a conversion matching table exists, it can generate expected latency information for the target model for each candidate node based on the operators included in the AI-based model and the conversion matching table. The computing device 100 can provide the candidate node list, which includes expected latency information and identification information of the candidate nodes.

[0201] In one embodiment, the conversion matching table may include sublatency information corresponding to an operator extracted from an AI-based model corresponding to model type information when the operator is converted to correspond to target type information. The conversion matching table is a table-type data structure (600) that organizes the performance of the converted operator at a specific node when conversion is performed between multiple operators. For example, one of the rows and columns (620) of the conversion matching table may include the operator corresponding to the model type information before conversion, and the other (610) may include the operator corresponding to the target type information after conversion. Elements in the conversion matching table may include sublatency information corresponding to the converted operator after conversion.

[0202] In one embodiment, a convert matching table can be generated for each node. In the above example, one convert matching table can correspond to one node. In one embodiment, a convert matching table can be generated for each node and for each combination of multiple models (or multiple operators). In this example, sublatency information for each operator in the first node of the second model converted from the first model can be expressed as a single table. In this example, the source operator (620) can include operators that are changed during the conversion to the target model among all operators included in the source model.

[0203] In one embodiment, the resource structure 600 can be updated by the computing device 100 in response to change events such as model updates, new node additions, and / or new model additions. In the absence of change events, the resource structure 600 can be utilized to generate a candidate node list in response to user input without any additional execution or measurement by the computing device 100.

[0204] FIG. 7 illustrates an exemplary tabular data structure used to generate a candidate node list in one embodiment of the present disclosure.

[0205] In one embodiment, the data structure (700) can indicate the conversion relationship between operators in the pre-conversion model and operators in the post-conversion model. Columns (720) in Figure 7 indicate operators to be converted in the pre-conversion model and the corresponding first model. Rows (710) in Figure 7 indicate converted operators included in each of the post-conversion models.

[0206] For example, in the material structure (700), the 1-1 operator of the model before conversion is not changed to the 1-1 operator when converted to the second model, is changed to the 1-3 operator when converted to the third model, is changed to the 1-3 operator when converted to the fourth model, and is not changed to the 1-1 operator when converted to the fifth model.

[0207] In one embodiment, the resource structure 700 can be used together with the resource structure 500 and / or the resource structure 600 when generating a candidate node list. For example, the computing device 100 can determine whether to convert the input model by comparing the model type information of the input model with the input target type information. If it is determined that conversion is necessary, the computing device 100 can identify the pre-conversion model and the post-conversion model and determine the resource structure 700 corresponding to the identified model. Based on the determined resource structure 700, the computing device 100 can determine which operators in the pre-conversion model will be changed to which operators in the post-conversion model. The computing device 100 can use the resource structure 500 and / or the resource structure 600 to determine the performance (e.g., latency) of the changed operators for a particular node. In this manner, the computing device 100 can generate a candidate node list.

[0208] In one embodiment, the computing device 100 may utilize a conversion matching table containing information on the conversion between operators, i.e., how a specific operator in a specific model is converted to another operator in another model. The data structure 700 may correspond to the conversion matching table. In the example described above, the conversion matching table may contain information indicating how operators are converted within a model when a conversion between models occurs. For example, one row and column in the conversion matching table (720) may contain operators in the model before conversion, and the other row and column (710) may contain operators in the model after conversion. The aforementioned conversion matching table may be used to determine how operators are changed by conversion. In such an embodiment, the computing device 100 may use the conversion matching table to identify operators in the converted model and the latency table described above to obtain node-specific sublatency information for each of the multiple operators in the converted model. By summing the sublatencies, expected performance information for each node in the converted model may be obtained.

[0209] FIG. 8 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure.

[0210] In one embodiment, the method illustrated in Figure 8 may be performed on a computing device (100). As an example, the method illustrated in Figure 8 may be performed by a first computing device (310). As another example, the method illustrated in Figure 8 may be performed by a computing device (100) that includes both the first computing device (310) and the second computing device (320).

[0211] The following is a specific example of how the steps in Fig. 8 are executed by a computing device 100. It will be readily apparent to those skilled in the art that some of the steps shown in Fig. 8 may be omitted or additional steps may be added depending on the implementation.

[0212] In one embodiment, the computing device (100) may obtain (810) input data including an inference task and a dataset.

[0213] In one embodiment, the computing device 100 may receive input data including an inference task, which includes an inference type and / or goal of the AI-based model, and a dataset for training, validation, and / or testing of the AI-based model. For example, the inference task may include any type of goal to be achieved through inference of the AI ​​model, such as object classification, object detection, object segmentation, clustering, sequence prediction, sequence determination, anomaly detection, and / or natural language processing. For example, the dataset may include training data used to train the AI-based model, validation data for evaluating the learning performance during the AI-based model training process, and / or test data for evaluating the performance of the trained AI-based model.

[0214] In another embodiment, the computing device (100) may receive information (e.g., model files) related to a trained artificial intelligence-based model.

[0215] In another embodiment, the computing device 100 may receive information about an AI-based model and target type information, which is a model type to be benchmarked. In this case, the information about the AI-based model may include a model file, a link for downloading the model file, and / or model type information corresponding to the model.

[0216] In other embodiments, the computing device 100 can generate a dataset if one is not provided. For example, the computing device 100 can randomly generate a dataset. For example, the computing device 100 can generate a dataset based on task-related information entered by a user. For example, the computing device 100 can generate a dataset for modeling using a generative model such as a GAN.

[0217] In one embodiment, the computing device (100) may determine (820) a target model to be benchmarked against an inference task and at least one target node on which the inference task of the target model will be executed.

[0218] In one embodiment, the computing device (100) is capable of determining a target node on which an inference task will be performed by providing a candidate node list including candidate nodes proposed for benchmarking against the inference task and receiving input data for selecting at least one target node from the candidate node list.

[0219] In one embodiment, the candidate node may be used to collectively refer to a node that is ready to run a benchmark and / or a node that can support a target model. In one embodiment, the candidate node list may include, for example, a document structure in the form of a table.

[0220] In one embodiment, the candidate nodes can be used to determine a target node from among a plurality of nodes for benchmarking a model. From among the plurality of candidate nodes included in the candidate node list, a target node for benchmarking can be determined based on input data. In such an embodiment, the candidate nodes can correspond to nodes that can support the target model, for example.

[0221] In one embodiment, the candidate node list can be used to determine, from among a plurality of nodes, a target node to be benchmarked and a target model to be benchmarked. For example, the candidate node list can include, from among a plurality of nodes, candidate nodes that are ready to execute a benchmark task (i.e., in a standby state). In such an embodiment, the candidate node can correspond to a node that is ready to perform a benchmark. In such a candidate node list, the target node to be benchmarked can be determined in response to a user's input to select a node to be benchmarked. In the above example, when determining a target node based on input data in the candidate node list, information regarding a target model having an execution environment (e.g., runtime, etc.) that the determined target node can support can be provided. In this case, the information regarding the target model can include identification information regarding multiple models that the determined target node can support. As an example, when determining a target node in the candidate node list in response to a user's input, suggested information regarding target models that the target node can support can be generated. The suggested information can include information regarding one or more target models. The target model can be determined in response to the user's input in the suggested information. In this case, the target model may include first information of the target model indicating a framework of the target model and second information of the target model indicating a software version of the target model. In one embodiment, information of the target model that can be supported by at least one target node included in the input data in the candidate node list may be automatically provided without a separate input from the user. In one embodiment, the user may select a framework of the desired target model and a software version of the desired target model in the proposal information related to the target model.As described above, the computing device (100) automatically provides suggested information on target models that the target node can support in response to a selection input for the target node, thereby enabling a user to easily select a node to be benchmarked and a corresponding model, even if the user does not have extensive knowledge in the field of artificial intelligence.

[0222] In one embodiment, the computing device 100 may provide a candidate node list including candidate nodes proposed for benchmarking an inference task. For example, the candidate node list may include identification information for each of a plurality of candidate nodes and information about the expected latency of each of the candidate nodes when the target model is executed. As an example, the order of the candidate nodes included in the candidate node list may be determined based on the magnitude of the expected latency information. Because a small expected latency value may indicate that the time required for inference at that node is relatively short, the computing device 100 may provide the candidate node list in order of smallest expected latency value (i.e., in order of best performance). This allows a user to intuitively check the expected performance of each of the plurality of candidate nodes and select a target node from the candidate node list in a more efficient manner.

[0223] In one embodiment, the ordering of the candidate node list may be determined by considering a variety of factors.

[0224] In one embodiment, the candidate node list may be sorted based on the magnitude of expected latency. For example, the list may be sorted in ascending order of expected latency.

[0225] In one embodiment, the order of the candidate nodes in the candidate node list can be determined based on CPU utilization. As an example, the candidate nodes can be arranged in ascending order of CPU utilization. If the current CPU utilization is high, there is a possibility that hardware resources will be limited when performing an inference benchmark. Therefore, the candidate nodes in the candidate node list can be arranged in descending order of CPU utilization.

[0226] In one embodiment, the candidate node list may be sorted based on memory usage. For example, the candidate nodes may be sorted in ascending order of memory usage. If the current memory usage is high, there may be limitations on the hardware resources when performing an inference benchmark. Therefore, the candidate nodes in the candidate node list may be sorted in descending order of memory usage.

[0227] In one embodiment, the order of candidate nodes in the candidate node list can be determined based on the priorities of multiple factors. For example, the priority of expected latency can be higher than the priority of CPU usage and / or memory usage. In the above example, if the difference in the magnitude of expected latency information between a first candidate node and a second candidate node among the candidate nodes is within a predetermined threshold range, or if the magnitude of expected latency information is the same, the order of the first candidate node and the second candidate node can be determined based on the memory usage and / or CPU usage of the first candidate node and the second candidate node. In the above example, multiple candidate nodes whose expected latency magnitudes are within a small range can be sorted in order of lowest memory usage and / or CPU usage.

[0228] In one embodiment, the candidate node list may include not only identification information of each of the multiple candidate nodes but also performance information that is useful for a user in determining a target node in the candidate node list. For example, the candidate node list may include power mode information indicating CPU core usage in at least some of the candidate nodes and / or fan mode information indicating fan usage in at least some of the candidate nodes. For example, the candidate node list may include information regarding at least one model that can be supported in each of the candidate nodes, library information required to install the at least one model that can be supported in each of the candidate nodes, and / or power usage information indicating power usage obtained from a power measurement sensor connected to the candidate node.

[0229] In one embodiment, the identification information of the candidate node list may include hardware information that can identify the candidate node. For example, the identification information may include not only the product name corresponding to the hardware, but also information about the installed execution environment, library information related to the execution environment, power mode information, fan mode information, current board temperature information, and / or current board power usage information.

[0230] In one embodiment, the power mode information may be determined based on how many CPU cores are being used. For example, if all CPU cores are being used, the power mode information may be determined as MAX, or may be determined in a manner that quantitatively expresses the amount of power usage, such as 40W, 30W, 20W, 10W, etc. For example, the larger the quantitative amount of the power mode information, the lower the latency. As another example, if the power mode is MAX, the latency may be lower than that of other nodes that are not using the power mode.

[0231] In one embodiment, the fan mode information can be expressed in the form of information indicating the fan strength, such as Null, Quite, Cool, and / or MAX. For example, when the fan mode is Quite, the board temperature can be lowered more than when the fan mode is Null, so latency is likely to be lower. For example, when the fan mode is Cool, the board temperature can be lowered more than when the fan mode is other modes, so latency is likely to be lower.

[0232] In one embodiment, the library information may refer to library information required to install the execution environment (e.g., runtime) information installed on a particular node. Depending on the characteristics of the node, it may include multiple execution environments, which may make the library information compatible across multiple execution environments.

[0233] In one embodiment, the current power usage of a board may refer to the power usage obtained from power measurement sensors connected to multiple nodes. It may be interpreted that the smaller the value of the current power usage of a board, the higher the possibility of using the node.

[0234] In one embodiment, there may be various methodologies for determining candidate nodes to be included in the candidate node list. In one embodiment of the present disclosure, the method for determining candidate nodes may be implemented through any combination of the following various methodologies:

[0235] For example, the computing device 100 may determine a list of nodes that are not currently being benchmarked or that are currently ready to be benchmarked as candidate nodes.

[0236] For example, the computing device 100 may determine, as a candidate node, a node that can support the determined target model (or determined target model information). Of the multiple nodes under the management of the computing device 100, a node that supports the input artificial intelligence-based model may be determined as a candidate node.

[0237] For example, the computing device (100) may determine, as candidate nodes, a plurality of first nodes that have an execution environment that supports a first operator included in the artificial intelligence-based model, among the nodes that support an execution environment corresponding to the target type information.

[0238] For example, the computing device (100) can determine, as candidate nodes, a plurality of second nodes that have an execution environment corresponding to the target type information and that do not support a first operator included in the artificial intelligence-based model but have an execution environment that supports a second operator different from the first operator that can replace the first operator.

[0239] For example, the computing device (100) may determine, as candidate nodes, a plurality of nodes that have free memory capacity that exceeds the size of the artificial intelligence-based model.

[0240] In one embodiment, when a specific node (e.g., Jetson-series hardware) is included in the candidate node list, the computing device 100 may add additional factors to consider when determining the sort order. For example, for nodes of a specific type, such as the Jetson series, a separate sorting process may be performed for the node, distinguishing it from other types of nodes. As another example, the computing device 100 may determine the sort order of a specific type of node relative to other types of nodes based on expected latency. However, if there is not much difference in the expected latency values ​​between multiple nodes of the same type, the computing device 100 may add a power field and / or a fan field to the sort order. For example, the computing device 100 may further consider factors corresponding to the power field and sort the nodes in descending order of power field value. For example, for specific nodes, such as the Jetson series, if the power field values ​​are the same or the difference is within a predetermined threshold range, the computing device 100 may further sort the nodes in descending order of fan operation based on the magnitude or strength of fan operation.

[0241] In one embodiment, the computing device 100 can receive user input data selecting a particular node from the list of candidate nodes, and the computing device 100 can determine the selected node included in the user input data as the target node.

[0242] In one embodiment, the computing device (100) can receive user input data selecting a single target node from the list of candidate nodes. In another embodiment, the computing device (100) can receive user input data selecting multiple target nodes from the list of candidate nodes. In another embodiment, the computing device (100) can automatically select the node from the list of candidate nodes that has the highest performance based on a particular factor (e.g., latency) as the target node without receiving user input.

[0243] In one embodiment, the computing device (100) may provide (830) benchmark results obtained by running the target model on the at least one target node.

[0244] In one embodiment, the computing device 100 is capable of generating benchmark results that include results of inferring the target model at the target node.

[0245] In one embodiment, if one node is determined as the target node, benchmark request information can be sent to the node. In one embodiment, if multiple nodes are determined as target nodes, benchmark request information can be sent to each of the multiple nodes. The benchmark request information can include information related to the target model to be benchmarked. The information related to the target model can include, for example, files or links related to the model and / or target type information for the model.

[0246] In one embodiment, the benchmark results may be generated by the computing device (100) or may be performed by another server (e.g., a server including multiple nodes) under the control of the computing device (100).

[0247] In one embodiment, the benchmark results may include performance information for the target node of the target model.

[0248] In one embodiment, the benchmark results may vary depending on which module of the other computing device triggered or requested the benchmark operation of the computing device 100. In another embodiment, the benchmark operation may vary depending on which module of the other computing device triggered or requested the benchmark operation of the computing device 100. For example, if a first module triggered the benchmark operation of the computing device 100, the computing device 100 may provide performance information for the entire input model, and if a second module triggered the benchmark operation of the computing device 100, the computing device 100 may provide performance information for the entire input model and may also provide performance information for each block of the input model. As another example, if the module that triggered the benchmark operation of the computing device (100) is a first module, the computing device (100) can provide benchmark results for determining a target node on which to run a learning model or a converted learning model corresponding to the input dataset, and if the module that triggered the benchmark operation of the computing device (100) is a second module, the computing device (100) can provide benchmark results including compressed configuration data used to generate a lightweight model corresponding to the input model.

[0249] By way of example and not limitation, the multiple modules that trigger benchmarking operations of the computing device (100) may include a first module that generates a learning model based on an input dataset, a second module that compresses the input model to generate a lightweight model, and a third module that generates download data for deploying the input model to at least one target node.

[0250] In one embodiment, the benchmark request information may include information regarding conversion of the model. Based on the information regarding conversion included in the benchmark request information, the input model may be converted.

[0251] In one embodiment, when the computing device 100 determines to convert an AI-based model to correspond to target type information, the computing device 100 can determine a specific converter from among a plurality of converters using information related to the input model. For example, the computing device 100 can determine converter identification information corresponding to a combination of model type information and target type information of the input model. Once a converter corresponding to the determined conversion identification information is determined, the determined converter can perform the conversion operation. For example, the computing device 100 can obtain a target model into which the AI-based model has been converted using a model file corresponding to the AI-based model and converter identification information corresponding to a combination of model type information and target type information. As described above, when a conversion is determined, the conversion can be performed using the model file to be converted and information for identifying the converter that will perform the conversion (e.g., an identifier for identifying the combination of the model type before conversion and the model type after conversion). As an example, the conversion can be performed by a conversion device (e.g., a conversion server).

[0252] As another example, depending on the implementation, the above conversion may be performed by the computing device 100. In the above example, if the model file resulting from the project is not for a specific type of runtime environment (e.g., Tensorrt) but for another type of runtime environment (e.g., Onnxruntime), the computing device 100 may use a conversion function to convert the model of the other type (e.g., Onnxruntime) into a model of the specific type (e.g., Tensorrt) and provide the model file resulting from the conversion.

[0253] In one embodiment, an AI-based model can be converted to a target model in a virtualized operating system using a converter Docker image corresponding to converter identification information. For example, an entity performing the conversion (e.g., a conversion server) can obtain a Docker image corresponding to the input converter identification information and execute the converter's sh file within Docker to perform the model conversion. In this case, the sh file can refer to a file containing instructions to be executed within Docker.

[0254] In one embodiment, the benchmark results may include a first type of quantitative information related to time and a second type of quantitative information related to memory usage.

[0255] In one embodiment, the benchmark results obtained by executing the target model on at least one target node may include preprocessing time information, which is information related to the time required for preprocessing of inference of the target model on at least one target node; inference time information, which is information related to the time required for inference of the target model on at least one target node; preprocessing memory usage information, which is information related to the memory usage used for preprocessing of inference of the target model on at least one target node; inference memory usage information, which is information related to the memory usage used for inference of the target model on at least one target node; quantitative information related to the inference time obtained by repeatedly inferring the target model a predetermined number of times on at least one target node; and / or quantitative information related to memory usage in each of the NPU, CPU, and GPU obtained by inferring the target model on at least one target node.

[0256] In one embodiment, the preprocessing time information may include, for example, information about the time required for preprocessing before performing an inference operation, such as calling a model. The preprocessing time information may also include quantitative information (for example, the minimum, maximum, and / or average value of the time required for pre-inference) related to the time required for pre-inference when pre-inference is repeated a predetermined number of times to activate a GPU or the like prior to measuring the inference value.

[0257] In one embodiment, the inference time information is information related to the time required for the inference process, and may include, for example, time information related to the time required for the first inference operation for a model and / or inference time information obtained when inference is repeated a predetermined number of times, such as minimum time information, maximum time information, average time information, and / or median time information. Furthermore, for example, when the CPU takes over and processes a calculation that cannot be processed by the NPU, the NPU becomes idle, and the inference time information may include a first cycle value when the NPU becomes idle. Furthermore, the inference time information may include a second cycle value when inference is performed by the NPU, and / or a third cycle value obtained by adding the first and second cycle values.

[0258] In one embodiment, the benchmark result information may further include the preprocessing memory usage information and total time information obtained by adding up the quantitative information relating to the inference time.

[0259] In one embodiment, the benchmark result information may further include a quantitative value relating to RAM usage, ROM usage, total memory usage, and / or SRAM area used by the NPU.

[0260] In one embodiment, when multiple benchmark results are generated due to multiple nodes being selected as target nodes, the computing device (100) can sort the multiple benchmark results using latency as a sorting criterion. For example, the benchmark results can be sorted and output in order of lowest latency. In another example, when the latencies are within a predetermined close range, or when benchmark results corresponding to each of the same multiple nodes exist, the benchmark results can be sorted in order by adding memory usage and / or CPU usage as sorting criteria. The sorting of the benchmark results can include features related to the sorting of the candidate node list.

[0261] In one embodiment, the benchmark results may include a document structure, for example in the form of a table.

[0262] In one embodiment, the computing device 100 may change the method of sending benchmark request information depending on whether wireless communication is available among the connected nodes. Wireless communication may include, for example, HTTP communication. For example, when transmitting a benchmark request to an independent node capable of wireless communication, the computing device 100 may send the benchmark request information to the node or a server associated with the node. For example, when transmitting a benchmark request to a node that does not have wireless communication, the computing device 100 may send the benchmark request information to a node (e.g., an Rpi4) that is capable of wireless communication and is connected to the node that does not have wireless communication via USB / GPIO. The node (e.g., a node capable of HTTP communication) that receives the benchmark request information may execute a program on the connected node to be benchmarked (i.e., a node that does not have HTTP communication) using serial communication via a USB / GPIO connection to obtain benchmark results. The expected memory usage may be measured by executing an execution environment corresponding to the target model type to be executed on an independent node capable of wireless communication. Using the measured expected memory usage and benchmark request information, it is possible to create and compile a benchmark program that will be executed in the node that is the target of the benchmark and that does not have wireless communication. The created and compiled program can be flashed via serial communication to the node that is the target of the benchmark and that does not have wireless communication. In this way, it is possible to obtain benchmark results for the node that does not have wireless communication.

[0263] In one embodiment, when the target node includes a node that cannot be externally confirmed (e.g., a virtual node), the computing device (100) can obtain benchmark results using the following method.

[0264] In response to receiving a first low-power wireless signal from an externally unobservable node, the computing device (100) may send a first acknowledgment message to the node, the first acknowledgment message including a benchmark task for benchmarking a target model at the node. For example, the first low-power wireless signal may include a beacon signal. For example, the first low-power wireless signal may include information regarding whether the node is performing a benchmark, information regarding memory usage of the node, and hardware identification information of the node.

[0265] The computing device 100 may receive a second low-power wireless signal (e.g., a callback signal) from the node, the second low-power wireless signal including the benchmark result generated by the node. In another embodiment, if the computing device 100 fails to receive the callback signal within a time corresponding to a predetermined waiting time threshold, the computing device 100 may determine that the benchmark task at the node has failed and set the node to an inactive state. The computing device 100 may set the node to an active state in response to receiving a third low-power wireless signal (e.g., a beacon signal) from the node set to an inactive state. In this case, the benchmark task included in the first response message may include target model information that the node can download and node configuration information used to convert the target model downloaded by the node. Furthermore, the benchmark result generated by the node may include a result obtained by executing the target model in the execution environment of the node based on the node configuration information and the target model information.

[0266] FIG. 9 illustrates an exemplary method for providing benchmark results in one embodiment of the present disclosure.

[0267] The method illustrated in Figure 9 may be performed by the first computing device (310), the second computing device (320), and / or the computing device (100). For ease of explanation, the following example illustrates the method of Figure 9 performed by the computing device (100).

[0268] In one embodiment, the computing device 100 may verify user access 905. The computing device 100 may provide a project, platform, web page, and / or mobile page, etc., that the user may access, and the computing device 100 may generate a training model based on a dataset entered by the user, compress the training model to generate a lightweight model, and / or generate download data for the model to be deployed to a node.

[0269] In one embodiment, the computing device 100 may provide 910 a model list.

[0270] In one embodiment, the computing device 100 can receive input from a user requesting a model list and, in response to such input, can generate a model list to be provided to the user.

[0271] In one embodiment, the model list may include, for example, a list of models that can be benchmarked, a list of models that can be trained on a dataset, a list of models that can be compressed, a list of models suitable for a particular artificial intelligence task, a list of models to suggest to the user, and / or a list of models that require user input. As an example, the model list may include the size of input data that the model accepts and / or latency information for each model.

[0272] In one embodiment, the computing device 100 may request uploading of a model for which benchmark results are to be obtained 915. For example, the computing device 100 may request input or upload of a model file, a dataset, an inference task, model type information corresponding to the model file, and / or target type information to be benchmarked.

[0273] In one embodiment, the computing device 100 may determine 920 whether conversion is necessary for benchmarking (e.g., for generating a candidate node list and / or for obtaining benchmark results). For example, if there is a difference between the type information of the uploaded model and the target type information of the benchmark, the computing device 100 may determine that conversion is necessary. As another example, if there is user input requesting conversion, the computing device 100 may determine that conversion is necessary.

[0274] In one embodiment, if it is determined that conversion is not necessary, the computing device 100 may request that detailed information about the model be entered along with the upload of the model 925. For example, the computing device 100 may request detailed information such as the inference task, the format of the AI-based model to be used, the location where the uploaded dataset will be stored (e.g., local storage or cloud storage), the training method, the purpose of the dataset (e.g., training, validation, test), the target latency, and / or the name of the project.

[0275] In one embodiment, if it is determined that conversion is necessary, the user may be prompted to enter model details along with the model upload and may be provided with conversion options 930. For example, the conversion options may include informing the user that the uploaded model will be converted to another type of model, prompting the user to select a specific model from among multiple models to be converted, providing performance information for each of multiple models to be converted, or requesting the user to enter a confirmation that they wish to proceed with the converter.

[0276] In one embodiment, the computing device 100 may receive 935 a user model selection input. The user model selection input may include information regarding a user selection of a model from among multiple proposed models, a user selection of a model to convert, and / or a user selection of a model to benchmark. As another example, the model selection input may include input for selecting a target model and a target node.

[0277] In one embodiment, the computing device 100 may receive a benchmark request 940. For example, the benchmark request may include an input to perform a benchmark against a determined target model.

[0278] In one embodiment, the computing device 100 may provide a candidate node list for selecting a target node in response to the benchmark request 945. The candidate node list may be generated by the computing device 100 or may be generated by another computing device connected to the computing device 100 and stored on the computing device 100. For a detailed description of the candidate node list, please refer to the descriptions of Figures 4 and 8 above.

[0279] In one embodiment, the computing device 100 may receive 950 user input for selecting a target node from the candidate node list and input related to benchmark configuration. For example, the computing device 100 may allow selection of at least one target node from the candidate node list. For example, the benchmark configuration input may include information included in the benchmark results, a batch size for the inference process, identification information for the target model, software version information for the target model, hardware identification information for the target device, an output data type for the target model (e.g., FP32, FP16, INT8, INT4, etc.), a target latency for the model during the training process, an image size for the model during the training process, and / or a training epoch.

[0280] In one embodiment, the computing device 100 may receive a request to run a benchmark 955. In response to the request to run a benchmark, the computing device 100 may run the target model on the target device in accordance with the benchmark settings to generate benchmark result information.

[0281] In one embodiment, the computing device 100 may provide 960 a list of benchmark tasks executed using the selected model. In one embodiment, step 960 may be executed in response to step 935 or in response to step 955. For example, the list of benchmark tasks in step 960 may include benchmark result information for benchmark tasks executed using a particular model. As an example, the list of benchmark tasks in step 960 may correspond to a list of benchmark result information for benchmarks performed on a particular model at multiple different nodes and / or multiple different time points.

[0282] In one embodiment, the computing device 100 may provide 965 a list of benchmark tasks executed using the selected node. For example, the list of benchmark tasks may include benchmark result information for benchmark tasks executed using a particular node in step 965. As an example, a list of benchmark result information for benchmarks performed on a particular node at multiple different nodes and / or at different times may correspond to the list of benchmark tasks in step 965.

[0283] In one embodiment, the computing device 100 can provide benchmark result information 970. A detailed description of the benchmark result information is provided in the detailed description of FIGS.

[0284] FIG. 10 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure.

[0285] The multiple entities (1000a, 1000b, 1000c, 1000d, and 1000e) shown in FIG. 10 are examples for illustrative purposes. Depending on the implementation, other entities not shown may be added to implement the method for providing benchmark results, or some of the multiple entities may be removed or integrated. For example, at least two of the second computing device (1000b), the first computing device (1000c), the converting device (1000d), and the node (1000e) may be integrated into a single entity. As another example, at least one of the second computing device (1000b) and the first computing device (1000c) may be divided into multiple computing devices and operate.

[0286] In one embodiment, the user (1000a) may include an entity that accesses the second computing device (1000b) through a computing device such as a terminal capable of wired or wireless communication.

[0287] In one embodiment, the second computing device 1000b may include an entity capable of interacting with the user 1000a. For example, the second computing device 1000b may provide a user interface that allows the user 1000a to select and use projects related to at least one of the second computing device 1000b, the first computing device 1000c, the conversion device 1000d, and the node 1000e. For example, the second computing device 1000b may receive various inputs from the user 1000a, such as model selection, node selection, benchmark settings, compression method settings, and / or learning method selection, through the user interface. In one embodiment, the second computing device 1000b may provide the user 1000a with various projects, such as a modeling project related to an AI-based model, a compression project, a project for generating download data for model deployment, and / or a benchmark project.

[0288] In one embodiment, the second computing device (1000b) can be configured to trigger an operation of the first computing device (1000c), thereby enabling the first computing device (1000c) to perform a benchmark operation in response to a signal from the second computing device (1000b). In another embodiment, the second computing device (1000b) can interact with the conversion device (1000d) and the node (1000e). In one embodiment, the second computing device (1000b) can correspond to the second computing device (320) in FIG. 3.

[0289] In one embodiment, the first computing device (1000c) can provide benchmark results to the second computing device (1000b) and / or the user (1000a). The first computing device (1000c) can obtain benchmark results for a model and node specified by the user (1000a) through interaction with the conversion device (1000d) and / or the node (1000e). In one embodiment, the first computing device (1000c) can obtain corresponding benchmark results based on selected node information and selected model information. In one embodiment, the first computing device (1000c) can correspond to the first computing device (310) in FIG. 3.

[0290] In one embodiment, the conversion device (1000d) can perform a conversion operation on the model. Depending on the implementation, the conversion device (1000d) can be separate or integrated with at least one of the second computing device (1000b) and / or the first computing device (1000c). In one embodiment, the conversion device (1000d) can correspond to the conversion device (390) in FIG. 3.

[0291] In one embodiment, the node 1000e may include one or more nodes. The node 1000e may refer to a target against which a selected model is benchmarked. In one embodiment, the node 1000e may be under the management of the first computing device 1000c and / or the second computing device 1000b. In this case, in response to a benchmark request from the first computing device 1000c and / or the second computing device 1000b, a benchmark is performed on the corresponding node, and the node 1000e may send benchmark result information to at least one of the first computing device 1000c and / or the second computing device 1000b. In one embodiment, the node 1000e may operate as part of the first computing device 1000c and / or the second computing device 1000b.

[0292] In one embodiment, the user (1000a) can access the second computing device (1000b) and upload the model (1005) through an interface provided by the second computing device (1000b). For example, uploading the model can include uploading a dataset or providing a link from which the dataset can be downloaded. For example, uploading the model can include providing a modeled model file or a link from which the modeled model file can be downloaded. For example, uploading the model can also include providing inference task information, target model information to be benchmarked, and / or model type information corresponding to the uploaded model.

[0293] In one embodiment, the second computing device (1000b) may receive a model list request (1010a) from the user (1000a). The model list may include information regarding a plurality of models capable of modeling or benchmarking. As an example, the information regarding the models included in the model list may be determined based on a benchmark task and / or dataset entered by the user (1000a).

[0294] In one embodiment, the second conversion device (1000b) can provide a model list (1010b) including information about a plurality of models that can be modeled or benchmarked. For example, the information about the plurality of models can include a model identifier, the size of data that can be input to the model, the inference method of the model, the learning method of the model, the structure of the neural network of the model, the model characteristics, quantitative information about the performance of the model, and / or node information suitable for the model.

[0295] In one embodiment, the second computing device (1000b) can receive user input from the user (1000a) selecting at least one model from the list of models (1015).

[0296] In one embodiment, the second computing device (1000b) can store selected model information (1020) in response to a user input selecting a model. For example, the selected model information (1020) can include a model type to benchmark, a model type to train, a model type to compress, a model type to deploy, a model file, a model identifier, and / or a model type corresponding to a dataset.

[0297] In one embodiment, the second computing device (1000b) may provide a candidate node list corresponding to the selected model information (1020) in response to a user input selecting a model. In one embodiment, the second computing device (1000b) may communicate the selected model information (1020) to the first computing device (1000c) in response to a user input selecting a model.

[0298] In one embodiment, the second computing device (1000b) may receive a node list request (1025a) from the user (1000a). The second computing device (1000b) may forward the node list request (1025b) to the first computing device (1000c). The first computing device (1000c) may provide the node list (1025c) to the second computing device (1000b). The second computing device (1000b) may provide the node list (1025d) to the user (1000a). In one embodiment, the node list may include a list of nodes among the plurality of nodes that are ready to run the benchmark. In one embodiment, the node list may include a list of candidate nodes among the plurality of nodes that correspond to the selected model information (1020).

[0299] In one embodiment, the second computing device (1000b) may receive node selection input from the user (1000a) (1030). The node selection input may refer to an input selecting at least one node (e.g., at least one target node) from the provided list of nodes.

[0300] In one embodiment, the second computing device (1000b) can store the selected node information (1035) in response to the user's input to select a node. In one embodiment, the second computing device (1000b) can also communicate the selected node information (1035) to the first computing device (1000c) in response to the user's input to select a node.

[0301] In one embodiment, the target node that the user (1000a) wants to select may not be present in the node list, or the user (1000a) may want to newly register a specific node, or the user (1000a) may want to add a node other than the selected node information (1035) for benchmarking. The second computing device (1000b) may receive a node registration request from the user (1000a) (1040a). The second computing device (1000b) may forward the node registration request to the first computing device (1000c) (1040b). In response to the node registration request, the first computing device (1000c) may register or store information about the corresponding node in a device-related DB. The first computing device (1000c) may provide a node list including the registered or stored nodes to the second computing device (1000b). The second computing device (1000b) may provide the received node list to the user (1000a). In one embodiment, the node list may include a list of nodes among the plurality of nodes that are ready to run the benchmark. In one embodiment, the node list may include a list of candidate nodes among the plurality of nodes that correspond to the selected model information (1020).

[0302] In one embodiment, the second computing device (1000b) may receive node selection input (1045) from the user (1000a). The node selection input may refer to an input selecting at least one node (e.g., at least one target node) from the provided node list. In one embodiment, the second computing device (1000b) may store selected node information (1050) in response to the user's node selection input. In one embodiment, the second computing device (1000b) may also communicate the selected node information (1050) to the first computing device (1000c) in response to the user's node selection input.

[0303] In one embodiment, the second computing device (1000b) may receive a request from the user (1000a) to proceed with benchmarking selected information (1055). In one embodiment, the selected information may include selected model information (1020), selected node information (1035), and / or selected node information (1050). For example, the benchmarking request in step 1055 may include a request to benchmark one selected model (1020) at multiple selected nodes (1035 and 1050).

[0304] In one embodiment, the second computing device (1000b) may send the selected model to the first computing device (1000c) (1056). In one embodiment, the first computing device (1000c) may store the received model (1057). In one embodiment, the second computing device (1000b) may send a benchmark request for the selected node to the first computing device (1000c).

[0305] In one embodiment, the first computing device (1000c) may determine whether to perform conversion on the received model (1059). The first computing device (1000c) may determine whether to perform conversion on the model based on the received model information and the selected node information. For example, the first computing device (1000c) may determine whether to perform conversion on the received model based on whether the selected node supports the received model or whether the selected node supports the operators included in the received model. For example, the selected node may not support the received model. In such a case, the first computing device (1000c) may determine whether to perform conversion on the received model or whether to perform conversion on at least some of the operators included in the received model. In another example, if the selected node does not support the received model, the first computing device (1000c) may determine that conversion is necessary for the received model or may decide to replace the selected node with another node.

[0306] In one embodiment, if it is determined that conversion is necessary, the first computing device (1000c) may send a request to the conversion device (1000d) to convert the model (1060). For example, the request to convert the model may include conversion identification information that indicates a combination of the type of the model to be converted and the type of the model after conversion.

[0307] In one embodiment, the conversion device 1000d may convert the model 1065 in response to a request to convert the model. For example, the conversion device 1000d may determine a converter from among multiple converts that corresponds to a convert identification included in the request to convert the model, and perform the conversion on the model using the determined convert. The conversion device 1000d may generate a converted model.

[0308] In one embodiment, the conversion device (1000d) can send the converted model to a first computing device (1000c). The first computing device (1000c) can determine, or clarify or instantiate, a target node from the converted model and check the availability of the target node (1080). For example, if the determined target node is currently executing another benchmarking task or if the available memory or available CPU of the target node is less than a predetermined capacity, the first computing device (1000c) can queue the benchmark request and issue the benchmark request when the target node becomes available.

[0309] In one embodiment, if the determined target node is available, the first computing device 1000c can send a benchmark request to the node 1000e. For example, the benchmark request can include information about the target model and the target node.

[0310] In one embodiment, the node (1000e) may execute a benchmark on the target model based on the information included in the benchmark request and generate benchmark results. For a detailed description of the benchmark results, please refer to the detailed descriptions of FIGS. 4 and 8. The node (1000e) may provide the benchmark results to the first computing device (1000c) (1085). The first computing device (1000c) may provide the benchmark results to the second computing device (1000b) (1090). The second computing device (1000b) may provide the benchmark results to the user (1000a).

[0311] FIG. 11 illustrates an exemplary method for providing benchmark results for nodes that cannot be externally verified in accordance with one embodiment of the present disclosure.

[0312] The example shown in Figure 11 illustrates interactions between a first computing device (1110), a second computing device (1130), and a node (1120) that cannot be externally confirmed, such as a virtual node, when the node is to run a benchmark as a target node.

[0313] 11, the contents that overlap with those in FIGS. 3 to 10 will not be repeated and will be omitted and substituted with the above-mentioned explanation.

[0314] In one embodiment, the system for providing benchmark results (1100) may include a first computing device (1100) and a node (1120). In another example, the system for providing benchmark results (1100) may include a second computing device (1130), the first computing device (1100), and the node (1120). In another example, the system for providing benchmark results (1100) may include the first computing device (1100), but the second computing device (1130) and the node (1120) may be external to the system (1100).

[0315] In one embodiment, the first computing device (1110) can determine, based on the identification information of the target node included in the received benchmark request, whether the target node is an externally unidentifiable node, whether the target node is an externally identifiable node, or whether the target node is a node capable of wireless communication (e.g., HTTP communication, etc.). If it is determined that the target node is an externally unidentifiable node, a technique in one embodiment of the present disclosure can be implemented by the exemplary method in FIG. 11.

[0316] Here, "external" can mean, for example, outside the network to which the node (1120) is connected. For example, the node (1120) that cannot be identified from the outside can be used to encompass a node that cannot connect to a device inside the network from outside the network, a node that cannot access the network from the outside, a node that cannot be identified from the outside but can be identified through a beacon signal, a node that cannot be identified from the outside but can be identified through a beacon signal transmitted by a node that cannot access the network from the outside, or a virtual node.

[0317] In one embodiment, the node 1120 may transmit a low-power signal, such as a beacon signal, to an external device (e.g., the first computing device 1110) at regular intervals. By way of example and not limitation, the transmission interval of the beacon signal transmitted by the node 1120 may be various values, such as 3 minutes, 2 minutes, 1 minute, 30 seconds, and / or 20 seconds. The first computing device 1110 may interact with the node 1120 by transmitting and receiving such low-power signals to and from the node 1120. Benchmarking operations with the node 1120 may be performed by including specific data (e.g., benchmark result information, information for benchmark assignment, information indicating whether or not a benchmark can be performed, node registration information, etc.) in the low-power signals.

[0318] The multiple entities (1110, 1120, 1130) shown in Figure 11 are examples for explanatory purposes. Depending on the implementation, other entities not shown may be added to implement a method for providing benchmark results for nodes that cannot be externally confirmed, or some of the multiple entities may be removed or integrated. For example, at least two of the second computing device (1130), the first computing device (1120), and the node (1120) may be integrated into a single entity. As another example, at least one of the second computing device (1120) and the first computing device (1110) may be divided into multiple computing devices and operate.

[0319] In one embodiment, the second computing device 1130 may include an entity capable of interacting with a user and the first computing device 1110. For example, the second computing device 1130 may provide a user interface to allow a user to select and use a project. For example, the second computing device 1130 may receive various inputs from the user through the user interface, such as model selection, node selection, benchmark settings, compression method settings, and / or learning method selection. In one embodiment, the second computing device 1130 may provide the user with various projects, such as a modeling project for an AI-based model, a compression project, a project for generating download data for model deployment, and / or a benchmark project.

[0320] In one embodiment, the second computing device (1130) can be configured to trigger an action on the first computing device (1100), such that the first computing device (1110) can perform a benchmark action in response to a signal from the second computing device (1130). In another embodiment, the second computing device (1130) can interact with the node (1120). In one embodiment, the second computing device (1130) can correspond to the second computing device (320) in FIG. 3 and / or the second computing device (1000b) in FIG. 10.

[0321] In one embodiment, the first computing device (1110) can provide benchmark results to the second computing device (1130) and / or a user. The first computing device (1110) can interact with the node (1120) to obtain benchmark results for the model and node received from the second computing device (1130). In one embodiment, the first computing device (1110) can obtain corresponding benchmark results based on the selected node information and the selected model information. In one embodiment, the first computing device (1110) can correspond to the first computing device (310) in FIG. 3 and / or the first computing device (1000c) in FIG. 10.

[0322] In one embodiment, the node 1120 may include one or more nodes. The node 1120 may be operable within the first computing device 1110. The node 1120 may indicate the target against which a selected model is benchmarked. In one embodiment, the node 1120 may be under the control of the first computing device 1110 and / or the second computing device 1130. In this case, in response to a benchmark request from the first computing device 1110 and / or the second computing device 1130, a benchmark is performed on the corresponding node, and the node 1120 may send benchmark result information to at least one of the first computing device 1110 and / or the second computing device 1130.

[0323] In one embodiment, the first computing device 1110 may receive a low-power signal from the node 1120. For example, the low-power signal may include a beacon signal. In this case, the low-power signal may include identification information of the node 1120. In another example, the low-power signal may include a registration request to register the node with the first computing device 1110. The first computing device 1110 may save or register 1141 the node 1120 in response to the low-power signal received from the node 1120. For example, the first computing device 1110 may determine whether the identification information of the node 1120 included in the low-power signal received from the node 1120 is registered in its DB. If the identification information of the node 1120 is not registered in its DB, the computing device 1110 may save or register the node 1120 in its DB.

[0324] In one embodiment, the first computing device 1110 may transmit a response message to the node 1120 in response to the low power signal (1142). The first computing device 1110 may set the node corresponding to the low power signal to an active state. For example, the active state may mean a state in which a benchmark operation is possible. For example, the active state may mean a state in which communication for a benchmark operation is possible. For example, the active state may mean a state in which a benchmark task can currently be executed, taking into account CPU usage and / or memory usage, etc.

[0325] In one embodiment, the second computing device (1130) may send (1143) a first benchmark request to the first computing device (1100). For example, the first benchmark request may include information regarding the model and node to be benchmarked. In one embodiment, the first computing device (1110) may determine which of the multiple nodes to send the first benchmark request to based on the node information included in the first benchmark request. For example, if the node included in the first benchmark request is a node (1120) that cannot be externally confirmed, the first computing device (1110) may determine to interact with the node (1120) regarding the first benchmark request in the manner shown in FIG. 11 .

[0326] In one embodiment, the first computing device (1110) may analyze the first benchmark request to determine whether to convert the model included in the first benchmark request, and if it is determined that conversion is necessary, the first computing device (1110) may obtain a conversion result of the model included in the first benchmark request.

[0327] In one embodiment, the first computing device (1110) may interact with the node (1120) in response to the first benchmark request (1143) to obtain benchmark results corresponding to the first benchmark request (1143). In another embodiment, the first computing device (1110) may interact with the node (1120) in response to the first benchmark request (1143) to obtain a candidate node list corresponding to the first benchmark request (1143).

[0328] In one embodiment, the first computing device 1110 may suspend and wait 1144 a benchmark operation corresponding to the first benchmark request 1143 until it receives a low-power signal from the node 1120. As described above, because the node 1120 cannot be externally identified and can communicate, for example, via a beacon signal, the first computing device 1110 may suspend and wait 1145 a subsequent low-power signal from the node 1120 before performing additional operations on the benchmark.

[0329] In one embodiment, the first computing device 1110 may receive 1145 a subsequent low power signal from the node 1120. The low power signal received in 1145 may include, for example, a beacon signal transmitted periodically or repeatedly by the node 1120. Such a beacon signal may include information about whether the node 1120 is currently running a benchmark, the node's memory usage, the node's CPU usage, and / or the node's identification information.

[0330] In one embodiment, the first computing device 1110 may generate 1146 a response message in response to the low power signal to allocate a benchmark task corresponding to the pending first benchmark request. The response message may be sent from the first computing device 1110 to the node 1120 based on the node's 1120 identification information included in the low power signal. For example, the benchmark task included in the response message may include target model information that the node 1120 can download and node configuration information used to convert the target model downloaded by the node 1120.

[0331] In one embodiment, the node 1120 may execute 1147 a first benchmark corresponding to the first benchmark request. The node 1120 may send 1150 a first benchmark result obtained by executing the first benchmark to the first computing device 1110 via a low-power signal (e.g., a callback signal). The first benchmark result may include a result obtained by executing the target model in the node 1120's execution environment based on the node configuration information and the target model information. For example, the callback signal may also have the form of a beacon signal. The callback signal (low-power signal) may include a benchmark result performed by the node 1120. The low-power signal may include the result of the benchmark. To avoid duplication, a detailed description of the benchmark result information is substituted for the description of FIGS. 4 and 8. The first benchmark result may be sent 1152 from the first computing device 1110 to the second computing device 1130.

[0332] In one embodiment, the second computing device (1130) may send a second benchmark request (1148) to the first computing device (1100). For example, the second computing device (1130) may sequentially send benchmark requests from multiple users to the first computing device (1100). In such an example, the first benchmark request (1143) may be a benchmark request corresponding to a first user, and the second benchmark request (1148) may be a benchmark request corresponding to a second user. For example, the second computing device (1130) may sequentially send multiple benchmark requests from a single user to the first computing device (1100). In the example of FIG. 11, the second benchmark request (1148) may be sent to the first computing device (1110) before the benchmark result information corresponding to the first benchmark request (1143) is transmitted to the second computing device (1130). The first computing device 1110 may suspend and wait 1149 its benchmarking operation until it receives a low-power signal in response to the second benchmarking request 1148. In one embodiment, the first computing device 1110 may send 1151 a response message to the node 1120 in response to the low-power signal 1150 received from the node 1120 to assign a benchmarking task corresponding to the second benchmarking request to the node 1120. As an example, the response message to assign the second benchmarking request may be transmitted to the node 1120 before the first benchmarking results 1152 are transmitted to the second computing device 1130. In such an example, the second benchmarking request may be transmitted 1156 to the node 1120 via the response message corresponding to the low-power signal 1150, which includes the first benchmarking results corresponding to the first benchmarking request.As described in the previous example, a new benchmark task can be assigned to the node 1120 through a response message to a low-power signal (e.g., a beacon signal) that includes the first benchmark result. The first computing device 1110 can transmit a second benchmark request to the node 1120 while obtaining the first benchmark result. This can improve the efficiency of resource usage for communication of the low-power signal and the response message between the first computing device 1110 and the node 1120.

[0333] In one embodiment, the node (1120) may execute (1153) a second benchmark in response to the second benchmark request. As a result of executing the second benchmark, the node (1120) may send (1154) a low-power signal (e.g., a callback signal) including the second benchmark results to the first computing device (1110). The first computing device (1110) may send (1155) the second benchmark results to the second computing device (1130).

[0334] In one embodiment, the second computing device (1130) may send a third benchmark request to the first computing device (1100) (1157). The first computing device (1110) may suspend benchmarking operations and wait until it receives a low-power signal from the corresponding node, and may determine that the benchmark corresponding to the third benchmark request has failed if it fails to receive a low-power signal from the corresponding node (1158). The first computing device (1110) may communicate information to the second computing device (1130) that the third benchmark corresponding to the third benchmark request has failed (1159). In another embodiment, the first computing device (1110) may determine that the benchmark task at the node has failed and set the node to an inactive state if it fails to receive a low-power wireless signal within a time corresponding to a predetermined waiting time threshold. In one embodiment, the first computing device (1110) may set a particular node to an inactive state, and then, if it receives a low-power wireless signal from the node, set the state of the node to an active state in response to the signal. For example, if it later receives a benchmark request specifying a node that has been set to an inactive state, the first computing device (1110) may transmit to the second computing device (1130) a result that the node cannot currently be benchmarked and / or a candidate node list with the node removed from the candidate nodes.

[0335] In another embodiment, the first computing device 1110 may transmit a third benchmark request to the node 1120, and if it fails to receive a low-power wireless signal (e.g., a callback signal including the benchmark result) from the node 1120 within a predetermined waiting time threshold, it may determine that the benchmark task from the node has failed and set the node to an inactive state. The first computing device 1110 may communicate to the second computing device 1130 information that the third benchmark corresponding to the third benchmark request has failed.

[0336] The techniques in several embodiments of the present disclosure enable nodes that cannot be externally confirmed to be taken into account when generating a candidate node list and / or benchmark results, thereby achieving the effect of expanding the range of nodes that can be benchmarked.

[0337] FIG. 12 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure.

[0338] In one implementation, the method illustrated in Figure 12 may be performed by the first computing device (310). As another example, the method illustrated in Figure 12 may be performed by a computing device (100), which encompasses the first computing device (310) and the second computing device (320).

[0339] The following is a specific example in which the steps in Fig. 12 are executed by a first computing device. It will be readily apparent to those skilled in the art that some of the steps shown in Fig. 12 may be omitted or additional steps may be added depending on the implementation.

[0340] In one embodiment, the first computing device in Figure 12 may correspond to the first computing device (310) in Figure 3, and the second computing device may correspond to the second computing device (320) in Figure 3. In one embodiment, the first computing device in Figure 12 may correspond to the first computing device in Figures 4-11, and the second computing device may correspond to the second computing device in Figures 4-11.

[0341] In one embodiment, a first computing device may receive module identification information from a second computing device that includes multiple modules performing different operations related to an artificial intelligence-based model, the module identification information indicating which of the multiple modules of the second computing device is to trigger a benchmark operation of the first computing device (1210).

[0342] In one embodiment, a first computing device can provide different benchmark results depending on the entity that sent the benchmark request to the first computing device. For example, a second computing device can send a benchmark request to the first computing device that includes a node and a model. The first computing device can identify the sender of the benchmark request and generate different benchmark results depending on the identified sender. In another example, the second computing device can send a benchmark request to the first computing device that includes a benchmark task to be included in the benchmark result.

[0343] In one embodiment, a first computing device may interact with each of a plurality of modules of a second computing device, and when interacting with each of the plurality of modules, may perform a benchmark in a different manner and / or obtain different benchmark results depending on which module is interacted with.

[0344] In one embodiment, the module identification information may include information for identifying a particular module(s) among a plurality of modules included in the second computing device. For example, the plurality of modules may refer to a plurality of modules that perform different operations in the second computing device. As another example, the plurality of modules may reside in at least one of the second computing device and the first computing device and be configured to perform different operations. As another example, the plurality of modules may reside separately outside the second computing device and be configured to perform different operations.

[0345] In one embodiment, the module identification information may include information for identifying a specific module among multiple modules included in the second computing device and benchmark task information. The benchmark task information may include information related to the node and / or model to be benchmarked. The benchmark task information may include performance target information for executing the benchmark (e.g., target latency information, etc.). The benchmark task information may include whether conversion is required during the benchmark execution process and / or conversion identification information. The benchmark task information may describe information to be included in the benchmark result. The benchmark task information may include a benchmarking method to be executed.

[0346] In one embodiment, multiple modules may utilize the benchmark results in different ways to generate their respective outputs.

[0347] For example, the first module may generate a learning model based on an input dataset. The first module may use the benchmark results to determine a target node on which to benchmark the learning model. The first module may use the benchmark results to confirm performance of the learning model when executed on the target node. The first module may use the benchmark results to generate a learning model or a retraining model. The first module may use the benchmark results to determine the type of learning model or retraining model corresponding to the dataset. The benchmark results may be used to evaluate performance of the learning model output from the first module. The performance of the learning model output from the first module may include memory footprint, latency, power consumption, and / or node information (e.g., the node's execution environment, processor and / or RAM size).

[0348] For example, the second module may compress the input model to generate a lightweight model. The second module may use benchmark results to determine compression setting data for the input model.

[0349] For example, the third module may generate download data for deploying the input model to at least one target node. The third module may use the benchmark result to generate the download data or to convert data into a data type supported by the target node. The third module may use the benchmark result to check how well the input model can perform on a node that has specifications closest to those of the node desired by the user.

[0350] In one embodiment, the first computing device may provide 1220 benchmark results to the second computing device based on the module identification information.

[0351] In one embodiment, a first computing device may provide different benchmark results to a second computing device depending on the module identification information. For example, the first computing device may run a benchmark in the same manner and provide different benchmark results to the second computing device when the module identification information is different. For another example, the first computing device may run a benchmark in a different manner and provide different benchmark results to the second computing device when the module identification information is different.

[0352] In one embodiment, if the module identification information indicates a first module, the first computing device can provide a benchmark result to the second computing device to determine a target node for executing a learning model or a converted learning model corresponding to the input dataset, and if the module identification information indicates a second module, the first computing device can provide a benchmark result to the second computing device, the benchmark result including compressed configuration data used to generate a lightweight model corresponding to the input model.

[0353] In one embodiment, if the module identification information indicates a first module, the first computing device provides performance information for the entire input model, and if the module identification information indicates a second module, the first computing device can provide performance information for the entire input model and / or performance information for each block of the input model.

[0354] In one embodiment, a first computing device may perform benchmarking in different manners depending on the module identification information. The benchmark results may be provided to a second computing device. If the module identification information indicates a first module, the first computing device may provide to the second computing device first benchmark results generated based on running a predetermined target model on at least one predetermined target model in a first benchmarking manner. If the module identification information indicates a second module different from the first module, the first computing device may provide to the second computing device second benchmark results generated based on running the target model on the at least one target node in the first benchmarking manner. In this case, the first benchmark results and the second benchmark results may be different.

[0355] By way of example and not limitation, the benchmarking scheme may include a first benchmarking scheme that measures performance information for an entire input model when the input model is executed on a target node. The benchmarking scheme may include a second benchmarking scheme that measures performance information for each operator of the input model when the input model is executed on a target node. The benchmarking scheme may include a third benchmarking scheme that measures performance information for each block of the input model when the input model is executed on a target node. In one embodiment, the first computing device may generate benchmark results by combining multiple benchmarking schemes.

[0356] In one embodiment, if the module identification information indicates a module that performs an operation of generating a learning model, the first computing device may provide, to the second computing device, benchmark results including first benchmark performance information obtained by executing each of the plurality of AI-based models on each of the plurality of nodes. For example, the first benchmark performance information may correspond to a table-type data structure representing latencies resulting from matching the plurality of AI-based models with the plurality of nodes. By way of example and not limitation, the data structure (500) shown in FIG. 5 may correspond to a table-type data structure representing latencies.

[0357] In one embodiment, the first benchmark performance information may include not only latency but also any other form of performance information when the model is executed on the node. For example, the first benchmark performance information may include power mode information, fan mode information, current board temperature information, and / or current board power consumption information. The power mode information may be determined based on how many CPU cores are being used. For example, if all CPU cores are being used, the power mode information may be determined as MAX, or may be determined in a manner that quantitatively expresses power consumption, such as 30W, 20W, 15W, 10W, etc. For example, the larger the quantitative value of the power mode information, the lower the latency. As another example, if the power mode is MAX, the latency may be lower than that of other nodes that do not use the power mode. The fan mode information may be expressed in the form of information indicating the fan strength, such as Null, Quite, Cool, and / or MAX. For example, if the fan mode is Quite, the board temperature may be lower than if the fan mode is Null, and therefore the latency is likely to be lower. For example, when the Fan mode is Cool mode, the board temperature can be lowered more than in other modes, so latency is likely to be lower. The current power usage of a board can refer to the power usage obtained from power measurement sensors connected to multiple nodes. It can be interpreted that the smaller the value of the current power usage of a board, the more likely it is that the node in question will be used.

[0358] In one embodiment, the first benchmark performance information can be used by the second computing device to generate performance prediction information for candidate nodes that can support the execution environment of the learning model. For example, the second computing device, which interacts with a user, can provide a candidate node list through a user interface, including candidate nodes that can support the model obtained from the user's input or that are ready to run the benchmark. When generating the candidate node list, a pre-prepared performance table can be used without interacting with the first computing device and / or the node that runs the benchmark. The performance table is a data structure that stores performance information obtained by running each of multiple models on each of multiple nodes. Thus, the second computing device can use the pre-prepared performance table to more quickly and efficiently provide the user with information on candidate nodes associated with the input dataset or input model, and performance information for each of the multiple candidate nodes.

[0359] In one embodiment, the first benchmark information may be updated by the first computing device when a new AI-based model is added, a new node is added, and / or the execution environment that the node can support is updated. In such situations, the first computing device may perform performance measurements on the new model, the new node, and / or the model and / or node corresponding to the updated execution environment, and update the performance table based on the performed performance measurements. As a result, because the first computing device intervenes in the process of providing the candidate node list only in certain situations, the second computing device may provide the user with information related to the candidate node list and performance prediction information for each of the multiple candidate nodes in a manner that improves resource efficiency.

[0360] In one embodiment, when the module identification information indicates a module that generates a learning model, the first computing device generates a candidate node list including a plurality of first nodes that support an execution environment that supports a first operator included in the learning model from a plurality of nodes that support an execution environment corresponding to the target type information based on the learning model and target type information to be benchmarked acquired from the second computing device, and provides benchmark results including the candidate node list to the second computing device. The first computing device can provide the user with more accurate information regarding the nodes to be benchmarked by determining, as candidate nodes, a plurality of nodes that can support the operators included in the learning model from a plurality of nodes appropriate for the target type information to be benchmarked. For example, if the target type information to be benchmarked by the user differs from the model type information associated with the learning model (or dataset) entered by the user, an operation must be performed to convert the entered learning model into a target model corresponding to the target type information. When the candidate nodes are converted from the entered learning model to a target model before benchmarking is performed, some of the previously presented candidate nodes may not support the converted target model. As a result, among the nodes that support the execution environment corresponding to the target type information, a plurality of first nodes having an execution environment that supports the first operator included in the learning model can be determined as the candidate nodes. As an example, the first computing device can extract operators included in the input learning model. Among the nodes that have a runtime that matches the target type information, if the runtime matches but the version of the runtime supported by the node does not support the extracted operator, the node on which the version of the runtime is installed can be excluded from the candidate nodes.

[0361] In addition, among the nodes having an execution environment corresponding to the target type information, a plurality of second nodes having an execution environment that does not support the first operator included in the learning model but supports a second operator different from the first operator that can substitute for the first operator can be determined as candidate nodes. As an example, if there is an operator that can substitute for the unsupported operator, the first computing device can request the user to replace or change the operator. If the user requests to replace the operator, the computing device (100) can include the node in the candidate nodes, and if not, can remove the node from the candidate nodes.

[0362] In one embodiment, if the module identification information indicates a module that generates a learning model, the first computing device can execute the learning model on at least one target node obtained from the second computing device to generate second benchmark performance information on the at least one target node of the learning model, and provide benchmark results including the second benchmark performance information to the second computing device.

[0363] In one embodiment, if the module identification information indicates a module that generates a learning model, the first computing device determines whether to convert the learning model based on model type information corresponding to the learning model and target type information to be benchmarked, which are acquired from the second computing device. If it is determined that the learning model should be converted, the first computing device can obtain a target model by converting the learning model to correspond to the target type information using the model type information and target type information corresponding to the learning model. The first computing device can execute the target model in at least one target node acquired from the second computing device to generate second benchmark performance information for the at least one target node of the target model. For example, the first computing device can determine the model type corresponding to the input learning model, compare the input target type information with the determined model type information, and determine that conversion is necessary if the two pieces of information are different. The first computing device can perform benchmarking by executing the converted model in the target node. As described in such an example, when a benchmark is triggered from a module that generates a learning model, the first computing device can determine whether to convert or not, and execute the benchmark in a manner that uses the model depending on the result of whether to convert or not. As described in such an example, when a benchmark is triggered from a module that generates a learning model, the first computing device can determine whether to convert or not, and provide a benchmark result of the model depending on the result of whether to convert or not.

[0364] In one embodiment, the first computing device may provide benchmark results including second benchmark performance information to the second computing device.

[0365] In one embodiment, the second benchmark performance information may include a first type of quantitative information related to time and a second type of quantitative information related to memory usage.

[0366] In one embodiment, the second benchmark performance information obtained by executing the target model on at least one target node may include preprocessing time information, which is information related to the time required for preprocessing of inference of the target model on at least one target node; inference time information, which is information related to the time required for inference of the target model on at least one target node; preprocessing memory usage information, which is information related to the memory usage used for preprocessing of inference of the target model on at least one target node; inference memory usage information, which is information related to the memory usage used for inference of the target model on at least one target node; quantitative information related to the inference time obtained by repeatedly inferring the target model a predetermined number of times on at least one target node; and / or quantitative information related to memory usage in each of the NPU, CPU, and GPU obtained by inferring the target model on at least one target node.

[0367] In one embodiment, the preprocessing time information may include, for example, information about the time required for preprocessing before performing an inference operation, such as calling a model. In addition, the preprocessing time information may include quantitative information (for example, the minimum, maximum, and / or average value of the time required for pre-inference) related to the time required for pre-inference when the pre-inference is repeated a predetermined number of times to activate a GPU or the like prior to measuring the inference value.

[0368] In one embodiment, the inference time information is information related to the time required for the inference process, and can be used to encompass, for example, time information related to the time required for the first inference operation for a model and / or inference time information when inference is repeated a predetermined number of times, such as minimum time information, maximum time information, average time information, and / or intermediate time information. Furthermore, for example, in a situation where the CPU takes over and processes a calculation that the NPU cannot process, the NPU becomes idle, and the inference time information can include a first cycle value when the NPU becomes idle. Furthermore, the inference time information can also include a second cycle value when inference is performed by the NPU, and / or a third cycle value obtained by adding the first and second cycle values.

[0369] In one embodiment, the second benchmark performance information may further include total time information obtained by adding together the preprocessing memory usage information and quantitative information related to the inference time.

[0370] In one embodiment, the second benchmark performance information may further include a quantitative value relating to RAM usage, ROM usage, total memory usage, and / or SRAM area used by the NPU.

[0371] In one embodiment, the second benchmark performance information may include, for example, a data structure in a table format.

[0372] In one embodiment, if the module identification information indicates a module that generates a lightweight model, the first computing device may provide the second computing device with benchmark results including compression setting data used to generate a lightweight model corresponding to the input model. As described above, if the module identification information corresponds to a module that generates a lightweight model, the first computing device may provide benchmark results that are different from benchmark results for other modules. The benchmark results may include benchmark results related to the compression of the model. The benchmark results may be obtained by benchmarking on a model-by-model basis and / or by benchmarking on a block-by-block basis. Here, the compression setting data may include at least one of a compression mode, a compression algorithm, a compression target, and a compression ratio.

[0373] For example, the compression mode may include a first compression mode in which compression is performed on the entire input model and / or a second compression mode in which compression is performed on a block-by-block basis included in the input model. In this specification, a block may refer to a component constituting a model. For example, a block may correspond to a convolution layer, an activation function, a normalization function, and / or an arithmetic operation. For example, a block may correspond to at least one layer in a neural network.

[0374] For example, the compression algorithm may include various known compression algorithms such as Layer-Adaptive Sparsity for Magnitude-based Pruning (LAMP) and / or Variational Bayesian Matrix Factorization (VBMF), etc. In other examples, the compression algorithm may include a weight pruning algorithm that changes the structure of a model, a weight pruning algorithm that separates channels to reduce the amount of calculations and the number of variables, a weight pruning algorithm that sets parameters other than those that affect the results to 0, etc., a weight pruning algorithm that uses a quantization method to reduce parameters expressed as floating-point numbers to a specific number of bits, and / or a weight pruning algorithm that binarizes parameters, etc.

[0375] For example, the compression target is used to indicate a block to be compressed among multiple blocks included in a model.

[0376] For example, the compression ratio can refer to quantitative information relating to the compression ratio in model compression or block compression.

[0377] In one embodiment, if the module identification information indicates a module representing a lightweight model, the first computing device may execute the model input from at least one target node acquired from the second computing device on a block-by-block basis to generate third benchmark performance information representing the performance of the input model on a block-by-block basis. The first computing device may provide the benchmark result including the third benchmark performance information to the second computing device. In one embodiment, the third benchmark performance information may include latency on a block-by-block basis of the model. In one embodiment, the third benchmark performance information may include first type quantitative information related to time on a block-by-block basis of the model and second type quantitative information related to memory usage. Specific details of the first type quantitative information and the second type quantitative information are substituted for the description of the second benchmark performance information described above.

[0378] As used herein, a lightweight model may refer to a model that has undergone compression.

[0379] As described above, when triggered by the model performing compression, the first computing device can provide benchmark results for efficient compression by using a benchmark execution method that obtains benchmark results for the entire model and / or benchmark results on a block-by-block basis.

[0380] In one embodiment of the present disclosure, a technique can utilize the benchmark results to ensure that an input model (e.g., a training model generated by a first module) is compressed in an efficient and accurate manner, so that a second module that compresses the model can utilize the benchmark results to obtain a lightweight model in a more efficient and accurate manner.

[0381] In one embodiment, the third module may change the data type of the input model to a data type supported by the target device (e.g., an 8-bit integer type) that is different from the data type of the input model (e.g., a 32-bit real number type). In one embodiment, the third module may adjust a quantization interval and perform quantization based on the adjusted quantization interval. Quantizing the input model may change the parameter values ​​(e.g., weights) of the model. The third module may provide download data that a user can install on a node. The download data may include a download file, a link to the download file, and / or a download package. Installing the download file on the target node may install an optimized artificial intelligence-based model on the target node.

[0382] In one embodiment, if the module identification information indicates a third module, the first computing device may provide the benchmark results to the second computing device for generating download data that enables the input model to be deployed to the target node.

[0383] In one embodiment, if the module identification information indicates a third module, the first computing device may provide a benchmark result to the second computing device for converting a data type of the input model into a data type supported by the target node.

[0384] In one embodiment, when the module identification information indicates a third module, the first computing device may provide benchmark results to the second computing device for adjusting the quantization interval of the input model. For example, quantizing the model may include reducing the number of bits used to represent weights and / or activation outputs, thereby reducing the size of the model. Quantizing the training model may shorten the inference time of the model. For example, the quantization interval may be determined in bits, such as 16 bits, 8 bits, 4 bits, 2 bits, 1 bit, etc.

[0385] 13, 14, 15, 16, and 17 illustrate various embodiments of the present disclosure for providing benchmark results. For example, computing device 100 may obtain benchmark results in response to a benchmark request from an external device. As another example, computing device 100 may operate in a variety of ways, such as standalone operation, operation dependent on another computing device for user interaction, direct conversion, acquisition of conversion results from another conversion device, direct generation of benchmark results, and / or reception of benchmark results from an external computing device.

[0386] In order to avoid duplication of explanation, the specific contents of FIGS. 13, 14, 15, 16, and 17 will be substituted with the corresponding parts in the explanation of FIGS. 3 to 12.

[0387] FIG. 13 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure.

[0388] The computing device 100 may receive 1310 target model information and target node information. In one embodiment, the target model information may include information related to the model to be benchmarked. For example, the target model information may include a dataset, a model file, a link to the model file, a model file and model type, and / or model file and target type information. In one embodiment, the target node information may include any form of information for identifying the target node. For example, the target node information may include an identifier of the target node, whether the target node supports wireless communication, whether the target node is externally identifiable, and / or the number of target nodes.

[0389] The computing device (100) may generate benchmark results (1320) by running the target model on the target node.

[0390] FIG. 14 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure.

[0391] The computing device 100 may receive 1410 target model information. The target model information may include information related to the model to be benchmarked. For example, the target model information may include a dataset, a model file, a link to the model file, a model file and model type, and / or a model file and target type information.

[0392] The computing device (100) may obtain (1420) a candidate node list. The candidate node list may be determined based on the target model information. The candidate node list may include a candidate node list associated with the target model information among a plurality of nodes. At least one target node to be benchmarked may be determined along with a selection input based on the candidate node list.

[0393] The computing device 100 may receive 1430 target node information. In one embodiment, the target node information may include any form of information for identifying the target node. For example, the target node information may include an identifier of the target node, whether the target node is capable of wireless communication, whether the target node is externally visible, and / or the number of target nodes. The target node information may be determined based on a selection input based on a candidate node list.

[0394] The computing device (100) may generate benchmark results (1440) by running the target model on the target node.

[0395] FIG. 15 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure.

[0396] The computing device 100 may receive 1510 an input model and target model information. In one embodiment, the input model may include any type of information related to the model. For example, the input model may include a dataset, a training model, a lightweight model, information related to model training, information related to model compression, information related to operators included in the model, and / or model type information corresponding to the model. The target model information may include any type of information related to the model to be benchmarked. For example, the target model information may include target type information and / or information for identifying the model to be converted.

[0397] The computing device 100 may obtain 1520 a candidate node list corresponding to the target model information. In one embodiment, the candidate node list may include, among a plurality of nodes, candidate nodes related to the target model information and / or the input model. For example, the candidate nodes may include nodes supporting an execution environment corresponding to the target model information, nodes supporting operators of the input model, and / or nodes supporting a target model into which the input model is converted according to the target model information.

[0398] The computing device 100 may receive 1530 target node information. In one embodiment, at least one target node to be benchmarked may be determined along with a selection input based on the candidate node list. In one embodiment, the candidate node in the candidate node list having the best performance based on the performance information may be automatically determined as the target node.

[0399] The computing device 100 may convert the input model to correspond to the target model information (1540). In one embodiment, the computing device 100 may determine a conversion method or conversion identification information by combining first information for identifying the input model and second information corresponding to the target model information. The computing device 100 may convert the operator of the input model to correspond to the target model information to generate a converted target model.

[0400] The computing device (100) may generate benchmark results by running the converted target model on the target node (1550).

[0401] FIG. 16 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure.

[0402] The computing device 100 may receive 1610 an input model and target model information. In one embodiment, the input model may include any type of information related to the model. For example, the input model may include a dataset, a training model, a lightweight model, information related to model training, information related to model compression, information related to operators included in the model, and / or model type information corresponding to the model. The target model information may include any type of information related to the model to be benchmarked. For example, the target model information may include target type information and / or information for identifying the model to be converted.

[0403] The computing device 100 may obtain 1620 a candidate node list corresponding to the target model information. In one embodiment, the candidate node list may include, among a plurality of nodes, candidate nodes related to the target model information and / or the input model. For example, the candidate nodes may include nodes supporting an execution environment corresponding to the target model information, nodes supporting operators of the input model, and / or nodes supporting a target model into which the input model is converted according to the target model information.

[0404] The computing device 100 may receive 1630 target node information. In one embodiment, at least one target node to be benchmarked may be determined along with a selection input based on the candidate node list. In one embodiment, the candidate node in the candidate node list with the best performance based on the performance information may be automatically determined as the target node.

[0405] The computing device (100) may send a conversion request (1640) to convert the input model to correspond to the target model information. In one embodiment, the conversion request may be sent to a conversion device located outside the computing device (100). For example, the first computing device may send a conversion request including the model file to be converted and the UUID of the converter to the conversion device. Here, the UUID is an identifier identified by a combination of the type of the model before conversion and the type of the model after conversion. The conversion device may convert the input model by obtaining a Docker image of the converter corresponding to the UUID and running the corresponding converter sh file in Docker.

[0406] The computing device 100 may receive 1650 the converted target model. In one embodiment, the computing device 100 may receive 1650 a conversion result from the conversion device, where the conversion result may correspond to the converted target model.

[0407] The computing device (100) may generate benchmark results by running the converted target model on the target node (1660).

[0408] FIG. 17 illustrates an exemplary method for providing benchmark results in accordance with one embodiment of the present disclosure.

[0409] The computing device 100 may receive 1710 input data including an inference task and a dataset. In one embodiment, the inference task may include a goal or result to be achieved through inference of an AI-based model, such as image classification, object detection, semantic segmentation, text prediction, and / or clustering. In one embodiment, the dataset may include any form of data used in an AI-based model. For example, a dataset may refer to a collection of data for which preprocessing has been completed. For example, a dataset may refer to a collection of data for which labeling has been completed in supervised learning. For example, a dataset may be used to train an AI-based model, to evaluate performance during training, and / or to evaluate performance after training is complete.

[0410] In response to receiving the input data, the computing device 100 may obtain 1720 a list of nodes that are ready to perform a benchmark. In one embodiment, the computing device 100 may obtain a list of nodes that are ready to perform a benchmark, such as nodes that are not currently performing a benchmark, nodes that have available memory capacity to perform benchmark tasks, or nodes that have a CPU that can perform benchmark tasks. The nodes included in the list may be determined during a process of determining the current node status through communication between nodes.

[0411] The computing device (100) may determine 1730 a target model and a target node based on input data selecting at least one node from the list of nodes. In one embodiment, when a specific node from the list of nodes is selected as a target node, a list of models that the selected node can support may be output. In response to input selecting a specific model from the list of models, a target model to be benchmarked may be determined. For example, the list of models may indicate the framework and software version associated with each of the models. For example, the list of models may include identification information associated with each of the models. For example, the list of models may include identification information associated with each of the models and performance information for each of the models when executed on the target node. In one embodiment, the order of the models in the list may be determined based on model performance or on past user selection information. In one embodiment, when a specific node from the list of nodes is selected as a target node, models that the selected node can support may be automatically output without receiving user input.

[0412] The computing device (100) may generate benchmark results (1740) by running the target model on the target node.

[0413] FIG. 18 is a schematic diagram illustrating a computing environment for a computing device (100) in accordance with one embodiment of the present disclosure.

[0414] In this disclosure, a component, module, or unit includes a routine, procedure, program, component, data structure, etc. that performs a particular task or implements a particular abstract data type. Additionally, those skilled in the art will appreciate that the methods described in this disclosure can be practiced with other computer system configurations, including computing devices having single or multiple processors, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which can operate in conjunction with one or more associated devices.

[0415] The embodiments described herein may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0416] A computing device typically includes a variety of computer-readable media. Any medium that can be accessed by a computer can be a computer-readable medium, including volatile and non-volatile media, transitory and non-transitory media, and mobile and non-mobile media. By way of example, and not limitation, computer-readable media can include computer-readable storage media and computer-readable transmission media.

[0417] Computer-readable storage media include volatile and non-volatile, transitory and non-transitory, removable and non-removable media implemented by any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data, including, but not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be accessed by a computer and used to store information.

[0418] Computer-readable transmission media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes all information delivery media. The term modulated data signal means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer-readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above media should also be included within the scope of computer-readable transmission media.

[0419] An exemplary environment 2000 for implementing various aspects of the present invention is shown, including a computer 2002, which includes a processing unit 2004, a system memory 2006, and a system bus 2008. The computer 2002 may be used interchangeably herein with the computing device 100. The system bus 2008 couples system components, including but not limited to the system memory 2006, to the processing unit 2004. The processing unit 2004 may be any of a variety of commercially available processors. Dual processors and other multi-processor architectures may also be utilized as the processing unit 2004.

[0420] The system bus (2008) can be any of several types of bus structures, including memory buses, peripheral buses, and additional interconnected local buses using any of a variety of commercially available bus architectures. The system memory (2006) includes read-only memory (ROM) (2010) and random-access memory (RAM) (2012). The basic input / output system (BIOS) is stored in non-volatile memory (2010), such as ROM, EPROM, or EEPROM, and contains the basic routines that support the transfer of information between the various components of the computer (2002), such as during startup. The RAM (2012) can also include high-speed RAM, such as static RAM, for caching data.

[0421] The computer 2002 also includes an internal hard disk drive (HDD) 1114 (e.g., EIDE, SATA), a magnetic floppy disk drive (FDD) 2016 (e.g., for reading from and writing to a removable diskette 2018), an SSD, and an optical disk drive 2020 (e.g., for reading from a CD-ROM disk 2022 or for reading from and writing to other high-capacity optical media such as a DVD). The hard disk drive 2014, magnetic disk drive 2016, and optical disk drive 2020 can be connected to the system bus 2008 by a hard disk drive interface 2024, a magnetic disk drive interface 2026, and an optical drive interface 2028, respectively. The interface (2024) for implementing an external drive includes, for example, at least one or both of USB (Universal Serial Bus) and IEEE1394 interface technologies.

[0422] These drives and their associated computer-readable media provide non-volatile storage for data, data structures, computer-executable instructions, and so forth. In the case of a computer (2002), the drives and media accommodate storing any data in a suitable digital format. While the foregoing description of computer-readable storage media refers to hard disk drives, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art will readily recognize that other types of computer-readable storage media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and many others, can also be used in the exemplary operating environment, and further, any such media can contain computer-executable instructions for performing the methods of the present invention.

[0423] A number of program modules, including an operating system (2030), one or more application programs (2032), other program modules (2034), and program data (2036), may be stored on the drive and in RAM (2012). All or portions of the operating system, applications, modules, and / or data may also be cached in RAM (2012). It will be apparent that the present invention may be implemented with various commercially available operating systems or combinations of operating systems.

[0424] A user can enter commands and information into the computer (2002) through one or more wired or wireless input devices, such as a keyboard (2038) and a pointing device such as a mouse (2040). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a gamepad, a stylus pen, a touch screen, and the like. These and other input devices are often connected to the processing unit (2004) through an input device interface (2042) connected to the system bus (2008), but can also be connected through a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, or various other interfaces.

[0425] A monitor 2044 or other type of display device is also connected to the system bus 2008 through an interface, such as a video adapter 2046. In addition to the monitor 2044, computers typically include speakers, printers, and various other peripheral output devices (not shown).

[0426] The computer 2002 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 2048, via wired and / or wireless communications. The remote computer(s) 2048 may refer to workstations, server computers, routers, personal computers, handheld computers, microprocessor-based entertainment devices, peer devices, or other conventional network nodes, and may generally include many or all of the components described for the computer 2002, although for simplicity, only a memory storage device 2050 is shown. The logical connections shown include wired and wireless connections to a local area network (LAN) 2052 and / or larger networks, e.g., a long-distance network (WAN) 2054. Such LAN and WAN networking environments are commonplace in offices and companies and facilitate the implementation and use of enterprise-wide computer networks, such as intranets, all of which may be connected to a global computer network, e.g., the Internet.

[0427] When used in a LAN networking environment, the computer 2002 connects to the local network 2052 through a wired and / or wireless communication network interface or adapter 2056. The adapter 2056 can facilitate wired or wireless communication to the LAN 2052, which may include a wireless access point configured to communicate with the wireless adapter 2056. When used in a WAN networking environment, the computer 2002 can include a modem 2058 or other means for establishing communications over the WAN 2054, such as connecting to a communications server on the WAN 2054 or through the Internet. The modem 2058, which can be internal or external and implemented as a wired or wireless device, connects to the system bus 2008 through a serial port interface 2042. In a networked environment, program modules described relative to the computer 2002, or portions thereof, may be stored in the remote memory / storage device 2050. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between computers may be used.

[0428] The computer 1602 operates to communicate with any wireless device or object that operates via wireless communication, such as a printer, a scanner, a desktop and / or handheld computer, a portable data assistant (PDA), a communications satellite, any equipment or location associated with a radio-detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth® wireless technologies. Thus, communication can be in a predefined structure, such as a traditional network, or simply ad hoc communication between at least two devices.

[0429] It should be understood that the specific order or hierarchy of steps in the processes presented herein is an example of an exemplary approach. Based on design priorities, it should be understood that the specific order or hierarchy of steps in the processes can be rearranged within the scope of this disclosure. The claims in this disclosure present elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

Claims

1. 1. A method for providing benchmark results, executed by a computing device, comprising: receiving first input data including information related to an artificial intelligence-based model; extracting model type information corresponding to the artificial intelligence-based model based on the first input data; receiving second input data including target type information to which the AI-based model is converted; converting the AI-based model based on the first input data and the second input data so that the AI-based model corresponds to the target type information; receiving user input data by the computing device for selecting a target node to be benchmarked against the converted artificial intelligence-based model, and determining the target node based on the received user input data; providing benchmark results obtained by executing the converted AI-based model on the target node; Including, method.

2. In claim 1, The target type information includes a framework corresponding to the converted artificial intelligence-based model; method.

3. In claim 1, The information related to the artificial intelligence-based model includes at least one of a dataset, a model file, and a link to the model file; method.

4. In claim 1, The step of extracting model type information corresponding to the AI-based model includes identifying a framework corresponding to the AI-based model from information related to the AI-based model. method.

5. In claim 1, determining the target node is performed based on the user input data selecting the target node on a candidate node list; and the candidate node list includes identification information of each candidate node capable of supporting a framework corresponding to the target type information or the result of the conversion; method.

6. In claim 1, The benchmark results are as follows: time information including preprocessing time information required for preprocessing of inference of the converted AI-based model in the target node, or inference time information required for inferring the converted AI-based model in the target node; memory usage information including preprocessing memory usage information used for preprocessing of inference of the converted AI-based model in the target node, or inference memory usage information used to infer the converted AI-based model in the target node; Including, method.

7. In claim 1, The benchmark results are Memory footprint information required to execute the converted AI-based model on the target node; and Latency information required to execute the converted AI-based model on the target node; and Power consumption information required to execute the converted AI-based model on the target node; and information about the target node; Including, method.

8. In claim 7, The information about the target node includes: an execution environment of the target node; a processor of the target node; and the RAM size of the target node; Including, method.

9. In claim 1, The benchmark results are as follows: GPU usage information expected when executing the converted AI-based model on the target node; and CPU usage information expected when the converted AI-based model is executed on the target node; and Latency information expected when executing the converted artificial intelligence-based model in the target node; and Information about expected power consumption when the converted artificial intelligence-based model is executed in the target node; and Including, method.

10. In claim 1, generating download data corresponding to the converted AI-based model based on the benchmark result, for deploying the converted AI-based model to the target node; method.

11. 1. A computer program stored on a computer-readable storage medium, the computer program, when executed by a computing device, causing the computing device to perform the following operations to provide benchmark results, the operations comprising: receiving first input data including information related to an artificial intelligence-based model; extracting model type information corresponding to the AI-based model based on the first input data; receiving second input data including target type information to which the AI-based model is converted; converting the AI-based model based on the first input data and the second input data so that the AI-based model corresponds to the target type information; receiving user input data by the computing device to select a target node to be benchmarked against the converted AI-based model, and determining the target node based on the received user input data; providing benchmark results obtained by executing the converted AI-based model on the target node; Including, A computer program stored on a computer-readable storage medium.

12. 1. A computing device for generating benchmark results, comprising: at least one processor; Memory and Including, The at least one processor receiving first input data including information related to an artificial intelligence-based model; extracting model type information corresponding to the AI-based model based on the first input data; receiving second input data including target type information to which the AI-based model is converted; converting the AI-based model based on the first input data and the second input data so that the AI-based model corresponds to the target type information; receiving user input data by the computing device to select a target node to be benchmarked against the converted artificial intelligence-based model, and determining the target node based on the received user input data; providing benchmark results obtained by executing the converted AI-based model on the target node; To execute Computing devices.

Images