Method, device, and system for processing prediction-related information about plurality of terminals

Abstract

The present invention relates to a method, device, and system for processing prediction-related information about a plurality of terminals. The method according to an embodiment of the present invention is performed by a combination processing device in a cluster including a plurality of terminals and the combination processing device. The method comprises the steps of: storing and managing prediction-related information according to prediction performance transmitted from each of the terminals, which predicts input data by using a mounted model; and performing analysis by using the prediction-related information.

Description

Method, device and system for processing prediction-related information of multiple terminals

[0001] The present invention relates to a technology for processing prediction-related information of multiple terminals, and more specifically, to a technology for receiving and processing prediction-related information, which is information related to performing prediction using a machine learning model, from multiple terminals.

[0002] Recently, technologies utilizing machine learning models (hereinafter referred to as "models") trained using machine learning techniques are being utilized in various fields. Processing related to these models can be divided into training, which trains the model using training data, and inference, which utilizes the trained model to perform actual tasks or services. In other words, through inference of the trained model, results such as predictions required in various fields can be obtained. However, this inference can be used interchangeably with the term "prediction," and will be referred to as "prediction" below.

[0003] Meanwhile, technologies utilizing previously trained models (hereinafter referred to as "prior technologies") perform predictions on the model using a single terminal. However, these prior technologies suffer from the following problems.

[0004] (1) Problem 1: Prediction performance limitations based on a single terminal

[0005] First, conventional techniques utilize prediction results from a single terminal, making it difficult to improve prediction performance based on that prediction. Specifically, in conventional techniques, a terminal is equipped with a model, which then predicts information (e.g., images, video, audio, text, etc.) input to the terminal. However, these predictions are only used within the terminal and are not shared with other devices. Furthermore, single-terminal inference makes it difficult to improve prediction performance.

[0006] Here, prediction performance refers to performance according to a certain metric that evaluates how well the ground truth set and the prediction set match, such as accuracy, precision, recall, and f1-score.

[0007] Meanwhile, models are typically trained to operate uniformly across multiple devices, rather than being tailored to a specific device. For example, a learning method is used that evenly distributes the class distribution of the training data to avoid bias toward specific classes. However, in real-world prediction environments, it's common to process data whose distribution differs from the data generated during the training process.

[0008] Typically, the larger the model's size and complexity, the higher its accuracy. Therefore, when hardware performance is limited on a device, a compromise is made: a less complex model with slightly lower predictive performance is installed to allow the model to run on that device.

[0009] (2) Second problem: Difficulty in obtaining forecast-related information

[0010] Furthermore, prior art has the problem of making it difficult to grasp information related to model prediction performance (i.e., prediction-related information). Prediction-related information refers to information related to the performance of predictions, such as predictions using a model. Specifically, prediction-related information refers to information about the time required to perform predictions for a given model on a given terminal, the resources required (CPU, memory, etc.), and the execution complexity. Of course, this prediction-related information could also be referred to as "terminal performance information."

[0011] For example, when inputting a specific image with Model A mounted on a random terminal, it is very important to understand (i.e., understand prediction-related information) how long it takes and how much CPU and memory resources are consumed when inputting a specific image (i.e., understand prediction-related information) when considering service response time prediction, load balancing, system stability, etc.

[0012] Accordingly, in the past, during the service development stage, predictions for a specific model were performed on the terminal, and after experimentally verifying information such as required time and memory usage, the number of terminals for parallel processing was determined based on this, or a process such as replacing the model with a model of different complexity or lightly converting the model size was performed.

[0013] This prediction-related information is typically discontinued or discarded once the target service successfully performs the service based on the prediction on the designated terminal. This is because once a model is deployed, it typically remains unchanged, and the format and size of the input data are typically fixed to fit the model.

[0014] However, models running on terminals need to be continuously updated with the latest models. Furthermore, terminals participating in a cluster can be dynamically added or removed, and the hardware specifications and software environments of these terminal devices can vary widely.

[0015] In summary, as models change dynamically, the types of terminals that make up a cluster become more diverse, and the dynamic joining and leaving of heterogeneous terminals within a cluster becomes more frequent, understanding prediction-related information when executing a given model becomes a highly cumbersome and challenging task. Furthermore, considering that the type of information input to a model can vary across terminals in a cluster environment, the difficulty of understanding prediction-related information when executing a given model on each terminal increases even further.

[0016] (3) Third issue: Personal information management and security issues

[0017] Meanwhile, with the advancement of machine learning-based artificial intelligence technology, personal information management and data security issues are becoming major concerns. Accordingly, methods that transmit input data from specific devices to cloud servers for processing should be avoided, as they can create privacy and security issues. However, most existing technologies still adhere to this method, leaving them completely unprepared for these privacy and security issues.

[0018] However, the above-described content merely provides background information on the present invention and does not correspond to previously disclosed technology.

[0019] In order to solve the problems of the prior art as described above, the purpose of the present invention is to provide a technology for receiving and processing prediction-related information from multiple terminals.

[0020] That is, the purpose of the present invention is to provide a technology in which a combined processing device manages, analyzes, and processes prediction-related information provided by each terminal when each terminal performs prediction in a cluster composed of a combined processing device and a plurality of terminals.

[0021] In addition, another purpose of the present invention is to provide a technology that can predict the execution complexity when a specific model is executed on a specific terminal based on analysis and processing of prediction-related information, or to improve the prediction performance by analyzing prediction execution results of multiple terminals together.

[0022] In addition, another purpose of the present invention is to provide a technology implemented so that a combined processing unit does not directly check input data or models processed by each terminal within a cluster when managing, analyzing, and processing prediction-related information, taking into account personal information protection and security issues.

[0023] However, the problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0024] A method according to one embodiment of the present invention for solving the above-described problem is a method performed by a combined processing device in a cluster including a plurality of terminals and a combined processing device, comprising: a step of storing and managing prediction-related information according to the prediction performance transmitted from each terminal that performs prediction on input data using a loaded model; and a step of performing analysis using the prediction-related information.

[0025] The above prediction-related information may include prediction result information, which is a result of prediction execution, prediction process collection information, which is information on resources and status at the time of prediction execution, input data recognition information, which is information for distinguishing input data used at the time of prediction execution, and model recognition information, which is information for distinguishing a model used at the time of prediction execution.

[0026] In the step of performing the above analysis, when a prediction is performed by loading a specific model on a specific terminal, the expected computational processing performance on the specific terminal can be analyzed using the prediction-related information.

[0027] In the step of performing the above analysis, the expected operation processing performance of each specific terminal that performs prediction by loading a specific model is derived using the prediction-related information, and the derived operation processing performance is compared to analyze the relative performance between each specific terminal.

[0028] In the step of performing the above analysis, if a specific terminal must complete the prediction within a target time, a recommendation model can be recommended to the specific terminal by considering the analyzed operation processing performance.

[0029] In the step of performing the above analysis, prediction-related information received from each terminal is analyzed to identify the relationship between each model, thereby calculating the complexity of each model, and the recommendation can be performed using the complexity of each model and the analyzed operation processing performance.

[0030] In the step of performing the above analysis, when multiple terminals collaborate to perform prediction, information for distributing prediction tasks to each terminal can be provided by considering the analyzed operation processing performance.

[0031] In the step of performing the above analysis, the prediction result information according to the prediction performed by multiple terminals on the same input data can be fused.

[0032] Intentional noise may be inserted into the same input data, and the input data recognition information may be composed of hash information for the same input data before noise insertion.

[0033] The above input data recognition information is composed of hash information for the same input data before noise insertion, and each prediction result information according to the prediction performed for the same input data may have intentional noise inserted.

[0034] In the step of performing the above analysis, when multiple terminals use the same model, the input data recognition information composed of embedded information and the characteristics of the input data used by each terminal can be compared to recommend a recommendation model to a specific terminal.

[0035] In the step of performing the above analysis, when a new terminal participates in the cluster, sample data and a sample model are transmitted to the new terminal, first prediction process collection information according to prediction performance using the sample data and sample model in the new terminal is received, and second prediction process collection information according to prediction performance using the sample data and sample model in another terminal previously belonging to the cluster is compared with the first prediction process collection information, thereby determining the relative computational processing performance of the new terminal with respect to the other terminal.

[0036] A device according to one embodiment of the present invention is a device that performs combined processing on a plurality of terminals in a cluster including a plurality of terminals, the device comprising: a memory storing prediction-related information according to the performance of the prediction transmitted from each terminal that performs prediction on input data using a loaded model; and a control unit that controls the performance of analysis using the prediction-related information stored in the memory.

[0037] The above prediction-related information may include prediction result information, which is a result of prediction execution, prediction process collection information, which is information on resources and status at the time of prediction execution, input data recognition information, which is information for distinguishing input data used at the time of prediction execution, and model recognition information, which is information for distinguishing a model used at the time of prediction execution.

[0038] The above control unit can control to analyze the expected operation processing performance of a specific terminal using the prediction-related information when a prediction is performed by loading a specific model on a specific terminal.

[0039] A system according to one embodiment of the present invention comprises: a plurality of terminals that perform predictions on input data using a loaded model; and a combined processing device that implements a cluster together with the plurality of terminals, stores and manages prediction-related information according to the corresponding prediction performance transmitted from each terminal, and performs analysis using the prediction-related information.

[0040] The above prediction-related information may include prediction result information, which is a result of prediction execution, prediction process collection information, which is information on resources and status at the time of prediction execution, input data recognition information, which is information for distinguishing input data used at the time of prediction execution, and model recognition information, which is information for distinguishing a model used at the time of prediction execution.

[0041] The above-mentioned combined processing device can analyze the expected computational processing performance of a specific terminal using the prediction-related information when a prediction is performed by loading a specific model on a specific terminal.

[0042] The present invention, configured as described above, has the advantage that when each terminal performs a prediction in a cluster composed of a combined processing device and a plurality of terminals, the combined processing device can manage, analyze, and process prediction-related information provided by each terminal.

[0043] In addition, the present invention has the advantage of being able to predict the execution complexity when a specific model is executed on a specific terminal based on analysis and processing of prediction-related information, or to improve the prediction performance by analyzing prediction execution results of multiple terminals together.

[0044] In addition, the present invention has the advantage of strengthening personal information and security by being implemented so that the combined processing unit does not directly check the input data or model processed by each terminal within the cluster when managing, analyzing, and processing prediction-related information, taking into account personal information protection and security issues.

[0045] In addition, the present invention has an advantage in that, when a prediction is performed by loading a specific model on a specific terminal, the expected operation time or the operation processing performance for CPU and memory resources, etc., on the specific terminal can be calculated for an unknown single input data or a large amount of input data having an arbitrary statistical distribution.

[0046] In addition, the present invention has an advantage in that, when a specific terminal must complete prediction within a target time, a model capable of completing the execution within the target time can be recommended by taking into account the computational processing performance of the specific terminal.

[0047] In addition, the present invention has the advantage of being able to provide information helpful in distributing prediction tasks when a plurality of terminals collaborate to provide a predetermined collaborative prediction service.

[0048] In addition, the present invention has an advantage in that prediction performance can be improved based on prediction-related information such as past prediction result information stored and managed in a combined processing device when a specific terminal performs prediction using a specific model.

[0049] In addition, the present invention has an advantage in that, when a plurality of terminals perform predictions using various models for the same input data, the combined processing device can improve prediction performance by utilizing the prediction results performed on each of the plurality of terminals.

[0050] In addition, the present invention has the advantage of being able to compare the characteristics of data sources (i.e., input data) input to each terminal when multiple terminals use the same model, thereby recommending a model with better performance to the corresponding terminal.

[0051] In addition, the present invention has an advantage in that even when a new terminal joins a cluster, the characteristics of the new terminal can be easily identified through a predetermined procedure for the new terminal.

[0052] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention belongs from the description below.

[0053] Figure 1 shows a block diagram of a system (10) according to one embodiment of the present invention.

[0054] FIG. 2 illustrates a conceptual diagram in which each terminal (100) shares and processes multiple data sources and models in a system (10) according to one embodiment of the present invention.

[0055] Figure 3 shows the information transfer process between the terminal (100) and the combined processing device (200) related to model recognition information.

[0056] Figure 4 shows a block diagram of a combination processing device (200) according to one embodiment of the present invention.

[0057] Figure 5 shows a flowchart of a method according to one embodiment of the present invention.

[0058] Figure 6 shows an example of a case where p terminals (100) (where p is a natural number greater than or equal to 2) are equipped with a single model (i.e., the same model) and receive input data from a single data source (the same data source) to perform prediction.

[0059] Figure 7 shows an example of a case where p terminals (100) are equipped with different models capable of ensembling and receive input data from a single data source (the same data source) to perform prediction.

[0060] Figure 8 shows an example of visualizing a large number of input data by embedding them and displaying them in a high-dimensional vector space.

[0061] Figure 9 shows the information transfer process between a new terminal (100') and a combination processing device (200) when a new terminal (100') participates in a cluster.

[0062] The above-described objects, means, and resulting effects of the present invention will become more apparent through the following detailed description, taken in conjunction with the accompanying drawings. Accordingly, those skilled in the art will be able to readily implement the technical concepts of the present invention. Furthermore, in describing the present invention, if a detailed description of known technology related to the present invention is deemed to unnecessarily obscure the gist of the invention, such detailed description will be omitted.

[0063] The terminology used herein is for the purpose of describing embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms, unless specifically stated otherwise. In this specification, terms such as "include," "provide," "provide," or "have" do not exclude the presence or addition of one or more other components other than the mentioned components.

[0064] In this specification, terms such as "or", "at least one", etc. may refer to one of the words listed together, or to a combination of two or more. For example, "A or B", "at least one of A and B" may include only one of A or B, or may include both A and B.

[0065] In this specification, descriptions using the phrase "for example" or the like should not be construed as limiting the embodiments of the invention in terms of the effects of variations such as tolerances, measurement errors, limitations of measurement accuracy, and other commonly known factors, including the information presented, such as cited characteristics, variables, or values, may not be exact matches.

[0066] In this specification, when a component is described as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is described as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0067] In this specification, when a component is described as being "on" or "in contact with" another component, it should be understood that it may be directly on or connected to the other component, but there may be another component in between. Conversely, when a component is described as being "directly on" or "in direct contact with" another component, it should be understood that there is no other component in between. Other expressions that describe the relationship between components, such as "between" and "directly between", can be interpreted similarly.

[0068] In this specification, terms such as "first" and "second" may be used to describe various components, but the components should not be limited by these terms. Furthermore, these terms should not be construed to limit the order of each component, but rather may be used to distinguish one component from another. For example, a "first component" may be referred to as a "second component," and similarly, a "second component" may also be referred to as a "first component."

[0069] Unless otherwise defined, all terms used herein may be used in their common sense by those of ordinary skill in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0070]

[0071] Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the attached drawings.

[0072] FIG. 1 shows a block diagram of a system (10) according to one embodiment of the present invention, and FIG. 2 shows a conceptual diagram in which each terminal (100) shares and processes multiple data sources and models in the system (10) according to one embodiment of the present invention.

[0073] Referring to FIG. 1, a system (hereinafter referred to as “the system”) (10) according to one embodiment of the present invention is a system that constitutes a cluster including n terminals (100) (where n is a natural number greater than or equal to 1) and c combined processing devices (200) (where c is a natural number greater than or equal to 1). At this time, the terminal (100) is connected to the combined processing device (200) to perform wired or wireless communication.

[0074] That is, a cluster refers to a configuration unit in which a terminal (100) and a combined processing unit (200) are capable of communicating with each other. Such a cluster can interconnect not only devices physically located on the same network, but also devices existing on heterogeneous networks via a communication network such as the Internet. Accordingly, a terminal (100) physically located in a remote location can also participate in the cluster via a communication network such as the Internet and communicate with the combined processing unit (200).

[0075] However, for convenience of explanation, the following description assumes that n is 2 or more and c is 1, and that the present system is implemented with multiple terminals (100) and one combined processing device (200). Of course, this description can be appropriately modified and applied even when two or more combined processing devices (200) are provided.

[0076] For example, when there are two or more combined processing devices (200), at least one combined processing device (200) may operate as a main device, and at least one other combined processing device (200) may operate as a backup device in case of a failure or error of the main device. In addition, each combined processing device (200) may perform parallel processing on prediction-related information.

[0077] Accordingly, when two or more combined processing devices (200) are provided, the safety of the system (10) can be improved in situations of failure and error, etc., compared to when one combined processing device (200) is provided, and it can be expanded to perform parallel processing, etc.

[0078] The terminal (100) performs prediction using a pre-trained machine learning model (hereinafter referred to as "model"). Prediction-related information, which is information related to the prediction performance, is transmitted to the combined processing unit (200).

[0079] Here, the term "prediction" can be used interchangeably with the term "inference." Specifically, prediction encompasses machine learning tasks such as classification or regression, and typically refers to the process of generating output for input data using parameters related to the weights and biases of a model (i.e., an artificial neural network) learned through training. Typically, the output of a model based on prediction may include an embedding vector (latent vector, enriched information), a probability value, or an index value for a classification class. Hereinafter, the term "model" refers to a pre-trained model.

[0080] The term terminal (100) may be used interchangeably with terms such as terminal device (client), edge, or edge device. That is, the terminal (100) refers to a device that participates in a cluster to perform prediction using a pre-learned model.

[0081] The terminal (100) is an electronic device capable of computing for performing predictions using a loaded model, etc., and may be a general-purpose computing device, a dedicated embedded system, an IoT device, a home appliance, an industrial device, a wearable device, etc., but is not limited thereto. For example, the general-purpose computing device may be a desktop personal computer, a laptop personal computer, a tablet personal computer, a netbook computer, a workstation, a personal digital assistant (PDA), a smartphone, a smartpad, or a mobile phone, but is not limited thereto.

[0082] Each terminal (100) can perform prediction using a data source as input data for a model. At this time, as illustrated in FIG. 2, various k data sources (where k is a natural number greater than or equal to 2) and various m models (where m is a natural number greater than or equal to 2) may be stored in a database or provided from a separate device. At this time, each terminal (100) can access the database or a separate device to utilize the data source as needed. At this time, each model provided from the database or a separate device may be a different model trained to have different weights and biases. That is, each terminal (100) can be connected to the database or a separate device via wired or wireless communication to receive and use the necessary data source and model.

[0083] Of course, the database storing the data sources and the database storing the models may be physically different devices. Such databases may be included in a cluster according to the present system (10). However, since the data sources stored in such a database may contain information that may cause personal information management and security issues, it may be desirable for the system (10) to be implemented so that the combined processing unit (200) cannot directly access the database. Accordingly, the combined processing unit (200) cannot access the database and directly check the contents of the data sources stored in the database.

[0084] A data source can include data in any form, such as a photo, video, audio, or text. For example, data stored on any storage device can be a data source, video from a network camera or similar device can be a data source, or audio from an audio device such as a microphone can be a data source.

[0085] A model is a model trained using machine learning techniques using training data. The machine learning techniques may include supervised learning, unsupervised learning, or reinforcement learning. Such a model may include an input layer receiving input data, an output layer receiving output data, and a hidden layer between the input and output layers. There may be multiple hidden layers. Nodes within each layer may be connected using weight and bias parameters.

[0086] For example, machine learning techniques may include, but are not limited to, Artificial neural network, Boosting, Bayesian statistics, Decision tree, Gaussian process regression, Nearest neighbor algorithm, Support vector machine, Random forests, Symbolic machine learning, Ensembles of classifiers, or Deep learning.

[0087] In particular, the model may be a deep learning model trained using deep learning techniques. In this case, the model expresses the relationship between input and output data in the training data through multiple layers. These multiple representational layers are sometimes referred to as a "neural network." In other words, the model can express the relationship between input and output data in the training data through the parameters of multiple hidden layers.

[0088] For example, deep learning techniques may include, but are not limited to, Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), and Deep Q-Networks.

[0089] Each terminal (100) selectively loads one of m models and uses one of k data sources as input data for the selectively loaded model to perform prediction. Prediction-related information based on this prediction is transmitted to the combined processing unit (200).

[0090] The combination processing device (200) receives prediction-related information from each terminal (100) and performs a function (hereinafter referred to as “first function”) of managing the received prediction-related information.

[0091] Additionally, the combined processing device (200) may perform a function (hereinafter referred to as the "second function") of analyzing the prediction-related information being managed when necessary. During this second function, the combined processing device (200) may also perform processing on the prediction-related information.

[0092] In addition, considering personal information protection and security issues, the combined processing device (200) can perform a function (hereinafter referred to as the "third function") to strengthen personal information and security by not directly confirming or utilizing the input data or model processed by each terminal (100) within the cluster when managing, analyzing, and processing prediction-related information. That is, in relation to the third function, although the input data of each terminal (100) is not directly shared with the combined processing device (200), the combined processing device (200) can estimate the characteristics of the prediction environment itself between the terminals (100) and relationship information between multiple prediction environments.

[0093] Referring to FIG. 2, in the process of transmitting prediction-related information between each terminal (100) and the combination processing device (200), x s t (where s is a natural number greater than or equal to 1) refers to the input data of the data source input to the s terminal (100_s) at time t. Of course, it is assumed that this input data is part of the k data sources. w s refers to a model installed and used in the terminal (100_s). Of course, it is assumed that this model is part of m models.

[0094] y s t means the output for the prediction performance of the s terminal (100_s). That is, y s t is x s t The input data is output data that is output as the model loaded on the s terminal (100_s) is input. Of course, this output data can be included in the prediction-related information of the s terminal (100_s).

[0095] g(w s , x s t) is a function corresponding to the prediction (i.e., artificial neural network processing) performed by the model installed in the s terminal (100_s). In order to effectively support data-based learning, g(w s , x s t ) may be preferably a nonlinear function. For example, the model may be an artificial neural network model that outputs an embedding vector (latent vector, latent information, representation information, etc.) that provides concentrated information about the input data.

[0096] In general, g(w) according to prediction performance in the s terminal (100_s) s , x s t ) and y s t The relationship between y s t = g(w s , x s t ) can be expressed as g(w). However, s , x s t ) may be subject to additional operations such as multiplying or adding an additional noise signal, or reducing the dimension. These additional operations can be expressed as z(). Accordingly, when the additional operations are performed, y s t = z(g(w s , x s t )) can be expressed as. Of course, the addition of noise signal can also be performed by first applying it to the original data source, in which case y s t = g(w s , z(x s t )) can be expressed as

[0097] In particular, the prediction-related information transmitted by the terminal (100) to the combined processing device (200) after performing the prediction may include at least one of terminal device information, prediction result information, prediction process collection information, input data recognition information, and model recognition information.

[0098] At this time, the terminal device information corresponds to information about the hardware and operating system of the terminal (100). That is, information about the hardware, such as the CPU model name, memory model name, system temperature, CPU temperature, GPU temperature, CPU clock speed, total memory amount, or total disk capacity, and information about the operating system, such as the operating system version, may be included in the terminal device information. Since such terminal device information has a characteristic that it does not change easily in the terminal (100), it may not be transmitted again until it is changed after being transmitted to the combined processing device (200). However, such terminal device information does not necessarily have to be included in the prediction-related information and transmitted to the combined processing device (200). That is, if the combined processing device (200) can utilize the terminal device information, it is easier to understand the performance of each terminal (100), but even without such terminal device information, it is possible to have an effect equivalent to the effect of utilizing the terminal device information by using other prediction-related information described below.

[0099] The prediction result information is the result of the prediction performed by the terminal (100) using the model. That is, the s terminal (100_s) is y s t can be transmitted to the combined processing device (200) as prediction result information. Of course, y s t As described above, g(w s , x s t ), z(g(w s , x s t )) or g(w s , z(x s t )) may be the result.

[0100] The prediction process collection information corresponds to information about resources and status when the terminal (100) performs prediction on the model using input data. That is, the prediction process collection information may include information about the computational resources of the terminal (100) used when performing prediction or information about the system status of the terminal (100) when performing prediction. For example, the computational resource information or system status information may include work time (i.e., prediction execution time), CPU usage (CPU load), GPU usage (GPU load), memory usage, swap memory amount, network usage (network traffic), disk usage, disk read size / write size, or cache memory usage.

[0101] The prediction process collection information of the terminal (100_s) can be transmitted as array-type information. For example, w s The model and b input data (x) (where b is a natural number greater than or equal to 2) s t , x s t+1 , … x s t+b-1 ) is used for the prediction process. The collected information is r(w s , [x s t , x s t+1 , … x s t+b-1 ]) can be expressed as array-type information. In the field of machine learning, batch processing is generally used to process multiple input data at once, considering parallel processing in parallel processors or GP-GPU (General Purpose Graphic Processing Unit). Accordingly, in the present invention, in order to correspond to this batch processing method, multiple input data (x) are input. s t , x s t+1 , … x s t+b-1) can be transmitted after configuring the information in the form of an array. In the formula, b elements are expressed as being entered sequentially in chronological order, but this is to simplify the formula expression, so it is also possible to extract b arbitrary elements from the data source and process them.

[0102] Input data recognition information is information about the data source used as input data when performing prediction, and is index information for distinguishing the corresponding input data. In other words, input data recognition information is information used to uniquely distinguish and recognize each input data. This input data recognition information may be information obtained by applying a hash function (i.e., hash information) or information obtained by applying an embedding function (i.e., embedding information). When this input data recognition information is received by the combined processing device (200), the combined processing device (200) can use the received input data recognition information to identify the index of the input data used by the terminal (100), although it cannot directly observe the input data. This input data recognition information can be broadly divided into data that has passed through a hash function and the result of passing through an embedding function. This can be expressed in a formula as follows.

[0103] h(x s t ) is the result of processing the single input data used by the s terminal (100_s) with a hash function. The s terminal (100_s) sends the corresponding h(x) to the combination processing device (200). s t ) value is passed, the combination processing unit (200) compares the hash values ​​and h(x) according to the characteristics of the hash function. s t) is unknown, but can be filtered and analyzed only for the results of predictions made using the original input data. In this case, the hash function is a one-way function, and can be a function that applies a widely known hash algorithm such as CRC, MD5, SHA256, or SHA512, but is not limited thereto.

[0104] e(x s t ) is the result of embedding processing using an embedding function for a single input data used by the s terminal (100_s). At this time, the embedding function can output vector space representation information (embedding vector, embedding information) for the input data. That is, the embedding processing corresponds to the process of extracting concentrated information using a pre-learned artificial neural network structure such as a transformer, a vision transformer, an autoencoder, a variational autoencoder, a Resnet, a VGG Net, etc. For example, in the case of an embedding processing technology based on a transformer structure that applies a self-attention technique, the information implication ability is excellent even when the input data is large, and it is also good for performing transfer learning, so it can be used in the present invention, but the present invention is not limited to this specific embedding technology.

[0105] Embedding functions can be compared to hash functions. A hash function is a one-way function that transforms large input data into smaller data. This hash function significantly reduces the original information of the input data. Therefore, the hash function result cannot be used to reverse engineer the original input data, and attempting to calculate the similarity between the original input data using the hash function result is undesirable. Of course, if data with even a single bit difference from the original input data is input to the hash function, a completely different hash function result is output. This technique can be used to determine whether the original input data has not been modified, or if the hash value is identical, the original input data is likely identical. On the other hand, an embedding function reflects the high-level features or context contained in the input data and outputs a vector of a specific dimension, making it possible to calculate the similarity between the original input data.

[0106] Furthermore, embedding functions can be thought of as weak one-way encoding functions that generate vector space representation information. A one-way encoding function, like a hash function, is a function that makes it easy to obtain an output value for an input, but makes it extremely difficult to infer the input value in reverse using the output value. However, what differentiates these embedding functions from hash functions is that even when input values ​​pass through the embedding module and output an embedding vector, they can maintain the correlation between input data, and even hidden patterns can emerge. This can be interpreted in terms of manifold theory in high-dimensional vector spaces, and this characteristic enables the comparison of characteristics of prediction environments and between prediction environments using embedding vectors (vector space representation information).

[0107] However, the reason why the embedding function is expressed as a weak one-way function is because, although it is very difficult, it is possible in principle to design a decoder corresponding to the encoder that performed the embedding, and use it to decode the vector space representation information to estimate the original input value. Nevertheless, it is very difficult to inversely estimate the original input data using only the vector space representation information output through the embedding module. However, in order to cope with even this rare possibility, the present invention proposes a method of generating an embedding vector by passing the input data through a function that adds a predetermined amount of noise before performing the embedding function in the terminal (100). Through this, the effect of making it difficult to inversely estimate the input data used in prediction can be strengthened.

[0108] [h(x s t ), h(x s t+1 ), … h(x s t+b-1 )] is b input data (where b is a natural number greater than or equal to 2) (x s t , x s t+1 , … x s t+b-1 ) is expressed as array-type information for the result processed by the hash function. That is, the s terminal (100_s) can transmit input data recognition information of array-type information having multiple hash data as elements to the combination processing device (200). At this time, the reason for expressing it as array-type information is as explained above.

[0109] [e(x s t ), e(x s t+1 ), … e(x s t+b-1 )] is b input data (where b is a natural number greater than or equal to 2) (x s t , x s t+1 , … xs t+b-1 ) is expressed as array-type information after performing embedding processing with an embedding function. That is, the s terminal (100_s) can transmit input data recognition information in the form of array-type information having multiple such embedding data as elements to the combination processing device (200). At this time, the reason for expressing it as array-type information is as explained above.

[0110] z(e(x s t )) is the result of the terminal (100_s) performing embedding processing by passing a predetermined noise function z() on a single input data. Typically, when embedding processing is performed on input data, the characteristics of the input data can be implicitly expressed with much less data even when the data size is large, and it is not easy to perfectly restore the original data in reverse using the embedding data. Of course, it is not impossible to attempt to restore the original data using various attack techniques. Therefore, in the present invention, the terminal (100) may take this possibility into account and apply a noise function that adds a predetermined noise to the input data, and then perform embedding processing and transmit it to the combined processing device (200).

[0111] [z(e(x s t )), z(e(x s t+1 )), … z(e(x s t+b-1 ))] is b input data (x, where b is a natural number greater than or equal to 2) s t , x s t+1 , … x s t+b-1 ) is expressed as array-type information after applying a noise function and performing embedding processing. The explanation of the necessity of applying noise and the expression as array-type information is omitted as it overlaps with the previously described content.

[0112] That is, with respect to the third function, the combined processing device (200) cannot directly confirm the content of the data source of the input data used by the terminal (10). Instead, by utilizing the input data recognition information, the combined processing device (200) can only distinguish and recognize the type of data source used by the terminal (10). Accordingly, personal information management and security issues that may arise when directly confirming the content of the data source can be prevented in advance.

[0113] Model recognition information is information about the model used when performing prediction on the terminal (100), and is index information for distinguishing the corresponding model. In other words, model recognition information is information used to uniquely distinguish and recognize each model.

[0114] The terminal (100) avoids control or restrictions by external management devices as much as possible. However, if the terminal (100) belongs to a cluster and provides certain information to the combined processing unit (200), the terminal (100) can receive the service proposed by the present invention. Similarly, it is assumed that the terminal (100) can download the model it needs from any database or load it from a place that has control over the terminal (100) without being controlled or restricted. Due to this characteristic, the combined processing unit (200) needs index information about the model used when performing prediction on the terminal (100). This is because the purpose of the present invention is to determine how complex the computational processing is performed on a specific model on a specific terminal (100) and to improve the performance of the results according to the corresponding model.

[0115] Figure 3 shows the information transfer process between the terminal (100) and the combined processing device (200) related to model recognition information.

[0116] Meanwhile, the combined processing device (200) cannot access the model stored in the database, and can distinguish and recognize the type of model used by the terminal (100) through model recognition information. To use this model recognition information, referring to FIG. 3, the terminal (100) can provide the combined processing device (200) with information about the loaded model (i.e., model information) before using the model when loading the model. At this time, the model information can include information about the structure of the model used (i.e., the structure of each layer) and the weight and bias of the model. Accordingly, the combined processing device (200) can perform a function (hereinafter referred to as the “fourth function”) of registering the corresponding model according to the received model information, and then generating model recognition information corresponding to the registered model and providing it to the terminal (100). At this time, the combined processing device (200) can manage the registered model through the file system of the memory (240) or a separate database, etc. By performing this fourth function, the terminal (100) can then perform prediction using the model and transmit the prediction-related information, including model recognition information for the model, to the combined processing device (200).

[0117] Meanwhile, when multiple different models are used through various terminals (100), model recognition information for each model can be generated and used by performing the second function according to FIG. 3, and in this manner, model recognition information for all models can be generated and used. Of course, when performing the second function, the combination processing device (200) can generate the same model recognition information for the same model information among the received multiple model information.

[0118] Meanwhile, the combined processing device (200) can operate as a server for each terminal (100) in the cluster according to the present system (10). That is, it can transmit various information related to the first to fourth functions and result information according to the performance of the first to fourth functions to the terminal (100) that requires it.

[0119] This combined processing device (200) is an electronic device capable of computing for performing the first to fourth functions. For example, the combined processing device (200) may be a general-purpose computing device such as a desktop personal computer, a laptop personal computer, a tablet personal computer, a netbook computer, a workstation, a personal digital assistant (PDA), a smartphone, a smartpad, or a mobile phone, or a dedicated embedded system, but is not limited thereto.

[0120] Figure 4 shows a block diagram of a combination processing device (200) according to one embodiment of the present invention.

[0121] Specifically, the combination processing device (200) may include an input unit (210), a communication unit (220), a display (230), a memory (240), and a control unit (250), as illustrated in FIG. 4.

[0122] The input unit (210) generates input data in response to various user inputs and may include various input means. For example, the input unit (210) may include, but is not limited to, a keyboard, a keypad, a dome switch, a touch panel, a touch key, a touch pad, a mouse, a menu button, etc.

[0123] The communication unit (220) is a component that performs communication with other devices. For example, the communication unit (220) may perform wireless communication such as 5G (5th generation communication), LTE-A (long term evolution-advanced), LTE (long term evolution), Bluetooth, BLE (Bluetooth low energy), NFC (near field communication), and WiFi communication, or may perform wired communication such as cable communication, but is not limited thereto. For example, the communication unit (220) may receive model information, prediction-related information, and the like from the terminal (100). In addition, the communication unit (220) may transmit information related to the execution of the method described below to another device or the terminal (100).

[0124] The display (230) is a configuration that displays various image data on the screen. For example, the display (230) may be configured as a non-luminous panel or a luminous panel. For example, the display (230) may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a micro electro mechanical systems (MEMS) display, or an electronic paper display. For example, the display (230) may display, on the screen, the execution process or result of the method described below. In addition, the display (230) may be implemented as a touch screen or the like by being coupled with the input unit (210).

[0125] The memory (240) stores various information necessary for the operation of the combined processing device (200). For example, the stored information may include, but is not limited to, information received from the terminal (100), model information, prediction-related information, or program information related to a method to be described later. For example, the memory (240) may include, but is not limited to, a hard disk type, a magnetic media type, a compact disc read-only memory (CD-ROM), an optical media type, a magneto-optical media type, a multimedia card micro type, a flash memory type, a read-only memory type, or a random access memory type, depending on its type. In addition, the memory (240) may be, but is not limited to, a cache, a buffer, a main memory, an auxiliary memory, or a separately provided storage system, depending on its use / location.

[0126] The control unit (250) can perform various control operations of the combined processing device (200). That is, the control unit (250) can control the execution of the method described below, and can control the operations of the remaining components of the combined processing device (200), such as the input unit (210), the communication unit (220), the display (230), the memory (240), etc. For example, the control unit (250) may include, but is not limited to, a hardware processor or a software process executed on the processor.

[0127] Hereinafter, the method according to the present invention will be described in more detail.

[0128] Figure 5 shows a flowchart of a method according to one embodiment of the present invention.

[0129] A method according to one embodiment of the present invention (hereinafter referred to as "the present method") may be performed under the control of a control unit (250). This present method may include S610 and S620, as illustrated in FIG. 5.

[0130] First, in S610, the control unit (250) controls the performance of the first function.

[0131] That is, the control unit (250) receives prediction-related information according to the prediction performance from the terminal (100) that performed the prediction on the input data of the arbitrary data source using the arbitrary model through the communication unit (220), and can control the received prediction-related information to be stored in the memory (240) or a separate database, etc., and comprehensively managed. At this time, the control unit (250) can manage the prediction-related information according to various methods such as a file, an in-memory database, a relational database, a time series database, an unstructured database, etc.

[0132] In particular, the control unit (250) can control to manage and store prediction-related information for each terminal (100) within the cluster. Accordingly, when a specific terminal (100) requests prediction-related information related to itself (particularly, prediction result information, etc.), the control unit (250) can control to transmit the prediction-related information to the specific terminal (100). As a result, the terminal (100) does not need to directly manage prediction result information and prediction process collection information according to its own prediction, and can easily use the prediction result information and prediction process collection information, etc. when necessary.

[0133] Next, in S620, the control unit (250) controls the performance of the second function.

[0134] That is, the control unit (250) controls various analyses to be performed using the prediction-related information being managed. In this second function, the control unit (250) may also control processing of the prediction-related information.

[0135] At this time, the control unit (250) can control to perform analysis to improve prediction performance using prediction-related information (e.g., prediction process collection information, prediction result information, etc.). To this end, the control unit (250) can use the prediction-related information to predict the expected execution complexity when a specific model is executed on a specific terminal (100), or can analyze the prediction execution results of multiple terminals (100) together.

[0136] Of course, with respect to the second function, the control unit (250) may also operate as a server that provides information based on analysis performed using prediction-related information (i.e., information on the performance results of the analysis) to a terminal (100) within the cluster that requires it or to another device outside the cluster.

[0137] Specifically, the control unit (250) can analyze performance similarity between terminals (100) using prediction-related information.

[0138] At this time, among the prediction-related information, the terminal device information is information indicating the characteristics of the hardware and operating system of the terminal (100). Therefore, the control unit (250) can control to analyze the performance similarity between the terminals (100) using this terminal device information. That is, using the terminal device information, the presence or absence of specification consistency between the terminals (100) or the similarity of the hardware and operating system can be compared. For example, if the CPU model name, CPU clock speed, and total memory amount among the terminal device information are the same, the control unit (250) can determine that the corresponding terminals (100) are similar devices.

[0139] Of course, the control unit (250) can analyze the expected computational processing performance (computation time or CPU and memory resources) for each model in a specific terminal (100) using other prediction-related information without terminal device information, or can control the analysis of performance similarity between terminals (100) in terms of the analyzed computational processing performance.

[0140] Figure 6 shows an example of a case where p terminals (100) (where p is a natural number greater than or equal to 2) are equipped with a single model (i.e., the same model) and receive input data from a single data source (the same data source) to perform prediction.

[0141] Referring to Fig. 6, each terminal (100) within the cluster is equipped with the same model (w) and the same input data (x t ), the prediction values ​​of each terminal (100) are the same. However, since each terminal (100) may have different hardware and operating systems, the operation resource information (i.e., prediction process collection information), such as the work time, CPU usage, memory usage, network usage, and cache memory usage required to perform prediction using the same model for each terminal (100), may be different.

[0142] Considering these characteristics, when multiple terminals (100) perform prediction using the same model for the same input data, the control unit (250) can control the terminals (100) to calculate the expected computational processing performance when performing prediction for the corresponding model by using the computational resource information (i.e., prediction process collection information) among the prediction-related information transmitted by the terminals (100). At this time, whether the same model is used can be determined by using the model recognition information among the prediction-related information of the terminals (100). In addition, whether the same input data is used can be determined by using the input data recognition information among the prediction-related information.

[0143] Expanding on this, the control unit (250) can control to calculate the expected operation processing performance of the specific terminal (100) when performing prediction on a single unknown input data or a large amount of input data having an arbitrary statistical distribution when a specific model is loaded and operated on the specific terminal (100). At this time, the input data can be identified using the input data recognition information, and the specific model can be identified using the model recognition information. In addition, the operation processing performance for the operation time or CPU and memory resources can be calculated using the prediction process collection information.

[0144] An example of a method for analyzing the computational processing performance of a specific terminal (100) when performing prediction using a specific model by using input data recognition information, model recognition information, and prediction process collection information is described below.

[0145] Serial number terminal ID input data recognition information (hash information) model recognition information prediction process collection information 1 ID 1 H 1 M 1 R 1 2 ID 1 H 2 M 1 R 2 3 ID 1 H 3 M 2 R 3 4 ID 1 H 4 M 2 R 4 5 ID 2 H 1 M 1 R 5 6 ID 2 H 2 M 1 R 6 7 ID 2 H 3 M 2 R 7 8 ID 2 H 4 M 2 R 8 9 ID 3 H 1 M 1 R 9 10 ID 3 H 2 M 1 R 10 11 ID 3 H 3 M 2 R 11 12 ID 3 H 4 M 2 R 12

[0146] First, we assume that data, such as those in Table 1, were collected based on prediction-related information. However, for simplicity, we assume that the prediction process collection information is the time required to perform predictions on input data (i.e., the required work time), that the input data recognition information is expressed as hash information, and that each model can be clearly distinguished using the model recognition information. Furthermore, although the input data corresponding to hash information such as H1 in Table 1 corresponds to the actual input data, for simplicity, we will express hash information such as H1 as input data. Furthermore, although the model corresponding to M1 corresponds to the actual model, for simplicity, we will express M1 as a model.

[0147] Serial number terminal ID input data recognition information (hash information) model recognition information prediction process collection information 1 ID 1 H 1 M 1 R 1 5 ID 2 H 1 M 1 R 5 9 ID 3 H 1 M 1 R 9

[0148] If we filter for the case where model M1 is used for input data H1 in Table 1, we get the result in Table 2. That is, Table 2 shows the same input data x as shown in Fig. 6. t This corresponds to the case where the same model w is loaded. At this time, by comparing R1, R5 and R9, which are the prediction process collection information of terminals ID1, ID2 and ID3, with each other, the computational processing performance such as the computation time required by each terminal (100) of ID1, ID2 and ID3 when performing prediction for the same model M1 can be calculated. Of course, by comparing the computational processing performance for each terminal (100), the relative performance between the terminals (100) can also be calculated. That is, the control unit (250) calculates the expected computational processing performance of each specific terminal (100) when a specific model (same model) is loaded and operated in a plurality of specific terminals (100), and by comparing the calculated computational processing performance, the relative performance between each specific terminal (100) can be determined.

[0149] Of course, in the case of Table 2, it is an example of filtering for H1 and M1, but it is of course possible to filter in various combinations such as {H1, M2}, {H2, M1}, {H2, M2}, etc. and compare the prediction process collection information for each terminal.

[0150] Serial number terminal ID input data recognition information (hash information) model recognition information prediction process collection information 1 ID 1 H 1 M 1 R 1 2 ID 1 H 2 M 1 R 2 5 ID 2 H 1 M 1 R 5 6 ID 2 H 2 M 1 R 6 9 ID 3 H 1 M 1 R 9 10 ID 3 H 2 M 1 R 10

[0151] For example, if we filter the case where model M1 is used for input data {H1, H2} in Table 1, we obtain results as in Table 3. In Table 3, by comparing the average values ​​(R1+R2) / 2, (R5+R6) / 2 and (R9+R10) / 2 of the prediction process collection information corresponding to terminals ID1, ID2 and ID3, we can calculate the computational processing performance, such as the computation time at each terminal, when performing prediction for model M1. Of course, the relative performance between terminals (100) can also be calculated.

[0152] Serial number Terminal ID Input data recognition information (hash information) Input data recognition information (embedding information) Model recognition information Prediction result information Prediction process Collection information 1 ID 1 H 1 E 1 M 1 Y 1 R 2 ID 1 H 2 E 2 M 1 Y 2 R 3 ID 1 H 3 E 3 M 1 Y 3 R 3 4 ID 1 H 4 E 4 M 1 Y 4 R 4 5 ID 2 H 6 E 5 M 1 Y 5 R 5 6 ID 2 H 7 E 6 M 1 Y 6 R 6 7 ID 2 H 8 E 7 M 1 Y 7 R 7 8 ID 2 H 9 E 8 M 1 Y 8 R 8

[0153] Meanwhile, in order to calculate the execution complexity when performing prediction for a specific model in a terminal (100) or to perform a relative comparison between each terminal (100), the characteristics of the input data must be taken into account, and in some applications, the input data must be strictly controlled for comparison. Table 4 shows an example of such a case. That is, although terminals ID1 and ID2 use the same model M1, there is no input data shared with each other when looking at the input data recognition information (hash information). In this case, the prediction process collection information can be processed for each terminal (100) using the following three processing methods. (1) First processing method: In order to predict the operation time for a specific model of a single terminal (100), the prediction process collection information is processed by extracting a field corresponding to a specific model according to the terminal ID.

[0154] (2) Second processing method: In order to compare the operation time for a specific model among multiple terminals (100), the prediction result information for a specific model is used together according to the relative terminal ID to process the prediction process collection information.

[0155] (3) Third processing method: In order to compare the operation time for a specific model among multiple terminals (100), the prediction process collection information is processed by using input data recognition information (embedding information) corresponding to a specific model together with the relative terminal ID.

[0156] Regarding the first processing method, as long as the terminals (100) in Table 4 use the same model, the prediction process collection information can be effectively utilized even if the input data is different. For example, considering only the rows for model M1 in ID1, although different input data {H1, H2, H3, H4} were provided, since the same model was used, it can be expected that the required time {R1, R2, R3, R4} will be almost the same. In other words, it is not too much of a stretch to consider the computation time required for arbitrary input data in ID1 as (R1+R2+R3+R4) / 4.

[0157] With regard to the second processing method, if the terminals (100) in Table 4 use the same model and can use the prediction result information, the prediction process collection information can be processed from another aspect. For example, ID1 used model M1, but the prediction result information is {Y1, Y2, Y3, Y4}, and the prediction result information of ID2 is {Y5, Y6, Y7, Y8}. At this time, if the prediction result information Y1 and Y5 are the same or similar to each other, and the prediction result information Y2 and Y6 are the same or similar to each other, it is appropriate to compare {R1, R2} and {R5, R6} to compare the performance of ID1 and ID2.

[0158] With regard to the third processing method, if the terminals (100) in Table 4 use the same model and can use the embedding information, the prediction process collection information can be processed from another aspect. For example, the embedding information of ID1 is {E1, E2, E3, E4}, and the prediction result information of ID2 is {E5, E6, E7, E8}. At this time, if E1 and E5 are similar, E2 and E6 are similar, E3 and E7 are similar, and E4 and E8 are similar to each other, it is appropriate to compare {R1, R2, R3, R4} and {R5, R6, R7, R8} to compare the performance of ID1 and ID2.

[0159] The {use of the same model, use of prediction result information} of the second processing method and the {use of the same model, use of embedding information} of the third processing method can also be used to predict the operation time at the terminal (100) for unknown single input data or a large amount of data having an arbitrary statistical distribution.

[0160] For example, if a particular model classifies {cat, lion, tiger}, the prediction process data collected for the {cat: 60%, lion: 40%} distribution can be selected and processed to predict how a particular model will utilize CPU and memory resources given a specific data distribution. For example, rows that achieve the desired data distribution can be extracted and the corresponding computation time measured.

[0161] In addition, the control unit (250) can control to recommend a model (hereinafter referred to as a “recommended model”) that can complete prediction within the target time when a specific terminal (100) must complete prediction within the target time and transmit the recommended model to the specific terminal (100). At this time, the control unit (250) can recommend the recommended model by considering the previously calculated operation processing performance (i.e., operation time or CPU and memory resources, etc.) of the specific terminal (100).

[0162] In particular, the control unit (250) can analyze prediction-related information received from each terminal (100) to determine the relationship between each model and calculate the complexity of each model. Accordingly, by utilizing the complexity of each model and the previously calculated computational processing performance, a recommended model capable of performing prediction within the target time on a specific terminal (100) can be selected.

[0163] For example, a relatively lightweight recommendation model may be recommended for a terminal (100) whose computational processing performance is below a certain level (i.e., relatively low-performance). On the other hand, a recommendation model with excellent prediction performance and relatively complex performance may be recommended for a terminal (100) whose computational processing performance is above a certain level (i.e., relatively high-performance).

[0164] In addition, the control unit (250) can control to provide information (i.e., help information) that is helpful in distributing prediction tasks when providing a service in which multiple terminals (100) within a cluster collaborate to perform predictions (hereinafter referred to as a "collaborative prediction service"). At this time, the control unit (250) can provide the help information by considering the computational processing performance (i.e., computation time or CPU and memory resources, etc.) of each terminal (100) selected in advance.

[0165] For example, let's assume that there are terminal A with a workload of 10 per hour (i.e., a value inversely proportional to the operation time) and terminal B with a workload of 1 per hour. In this case, if the total amount of work to be performed is 11, if a workload of 10 is assigned to terminal A and a workload of 1 is assigned to terminal B, the two terminals A and B can process their assigned tasks in the same amount of time. Accordingly, in terms of overall work processing, the work can be completed in the shortest amount of time. This can be expanded to multiple terminals (100) with different operation processing capabilities to quickly process the total workload.

[0166] Meanwhile, when performing predictions using a specific model, the terminal (100) can improve prediction performance by utilizing past prediction result information. This past prediction result information is stored and managed in the combined processing device (200) and can be transmitted to the terminal (100) at the terminal's request.

[0167] For example, the terminal (100) may be equipped with a module that performs embedding processing on input data and transmit data (i.e., embedding information) resulting from the embedding processing on the input data to the combined processing device (200). By collecting and analyzing such past embedding information, it can be utilized to improve the prediction performance of the equipped model.

[0168] Serial number terminal ID input data recognition information (hash information) input data recognition information (embedding information) model recognition information prediction result information 1 ID 1 H 1 E 1 M 1 Y 1 2 ID 1 H 2 E 2 M 1 Y 2 3 ID 1 H 3 E 3 M 1 Y 3 4 ID 1 H 4 E 4 M 1 Y 4

[0169] For example, when prediction-related information such as Table 5 is collected in the combined processing device (200), the prediction result information corresponding to the input data {H1, H2, H3, H4} based on hash information is {Y1, Y2, Y3, Y4}, and the input data recognition information based on embedding is {E1, E2, E3, E4}. Assume that {E1, E2, E3, E4} have almost similar values. In addition, assume that {Y1, Y2, Y3} correspond to the same prediction result, and Y4 corresponds to a different prediction result. In this case, the prediction error of Y4 can be suspected. This can be the basis for correcting the prediction of Y4 in terminal ID1 using previously collected information. As an example of a correction method, the prediction result with the maximum value can be obtained by comparing the similarity between vectors of the embedding information and using the prediction result information with a similarity within a threshold value, or performing an ensemble. For example, it can be calculated as S1 = similarity(E4, E1), S2 = similarity(E4, E2), S3 = similarity(E4, E3), and if the similarity S3 does not exceed the threshold, and the similarities S1 and S2 exceed the threshold, Y1 and Y2 can be candidates. At this time, the terminal (100) can perform an ensemble technique such as using the value with the highest similarity in the candidate group as the prediction of Y4, averaging the probability values ​​in the prediction result information Y1 and Y2 if there are probability values, or taking the index with the highest number from the output index.

[0170] Specific ensemble methods can be very diverse, but the key is that by using input data recognition information generated through an embedding function, prediction result information can be improved in the terminal (100).

[0171] In addition, the control unit (250) can control to improve prediction performance by using each prediction result performed in the plurality of terminals (100) when the plurality of terminals (100) perform prediction using various models for the same input data.

[0172] For example, a conventionally well-known ensemble prediction technique can be used to improve the prediction results of terminals (100) constituting a cluster. However, while conventional ensemble prediction could be performed in an environment defined in advance by a predetermined method, the present invention differs in that it supports ensemble prediction in a general and complex environment.

[0173] That is, in conventional ensemble prediction, the same input data passes through multiple models to obtain multiple prediction results, and the accuracy of the prediction is increased by applying the majority rule or calculating the average of the output probabilities. On the other hand, since the terminals (100) belonging to the cluster of the present invention each use their own model, different values ​​of input data may be input, and the combined processing device (200) cannot directly check the input data input to each terminal (100), it is difficult to solve the problem using conventional methods.

[0174] Accordingly, in the present invention, when a plurality of terminals (100) perform predictions using various models, a terminal (100) equipped with a model capable of fusing the prediction results is selected, and prediction performance can be improved by applying a technique such as conventional ensemble prediction to the prediction results provided by the selected terminal (100).

[0175] Figure 7 shows an example of a case where p terminals (100) are equipped with different models capable of ensembling and receive input data from a single data source (the same data source) to perform prediction.

[0176] That is, referring to Fig. 7, p terminals (100) are models {w1, w2, … w p} are respectively loaded, and the input data x of the same data source t The prediction result {y 1 t , y 2 t …y p t} are output respectively. At this time, by applying ensemble prediction to each prediction result, the prediction performance, such as accuracy, for the prediction results of different terminals (100) can be improved.

[0177] Specific ensemble prediction methods may be as follows. However, (2) and (3) below correspond to techniques that enhance privacy and security.

[0178] (1) First ensemble prediction method: The control unit (250) performs fusion prediction on the prediction result information for each terminal (100) using a model capable of fusion prediction (i.e., ensemble prediction).

[0179] (2) Second ensemble prediction method: An appropriate amount of noise (i.e., intentional noise) is inserted into the input data of each terminal (100), and the control unit (250) receives hash-type input data recognition information for the input data before noise insertion, and performs fusion prediction using a model capable of fusion prediction (i.e., ensemble prediction) on the prediction result information of each terminal (100).

[0180] (3) Third ensemble prediction method: The control unit (250) receives input data recognition information in hash format and performs fusion prediction using a model capable of fusion prediction (i.e., ensemble prediction) on the prediction result information of each terminal (100) with an appropriate amount of noise (i.e., intentional noise) inserted.

[0181] According to the first ensemble prediction method, the prediction result information {y 1 t , y 2t …y p t} can be used to improve prediction performance using conventional ensemble methods. Ensemble methods will be described later. To improve prediction accuracy, the key is that while the models that obtain the prediction result information perform the same prediction task, the learning process for each model is diverse, and each model has unique and distinctive characteristics. Table 6 provides an example of this.

[0182] Serial number terminal ID input data recognition information (hash information) model recognition information prediction result information 1 ID 1 H 1 M 1 Y 1 2 ID 2 H 1 M 2 Y 2 3 ID 3 H 1 M 3 Y 3 4 ID 4 H 1 M 4 Y 4

[0183] That is, different terminals {ID1, ID2, ID3, ID4} inject the same input data corresponding to H1 into different models {M1, M2, M3, M4} and output the prediction result information {Y1, Y2, Y3, Y4}. At this time, if the models {M1, M2, M3, M4} perform the same task and can be fused (i.e., if ensemble prediction is possible), the prediction result information {Y1, Y2, Y3, Y4} can be used to perform fusion prediction. A method for determining whether fusion prediction is possible for different models {M1, M2, M3, M4} will be described later. Ensemble prediction, a representative method of fusion prediction, is a method to improve performance by collecting the outputs of various models for a specific prediction task and applying methods such as majority voting or taking the average of neural network outputs (e.g., probability values). Predictive models (i.e., predictors) for majority voting often outperform the best individual models in the ensemble. This increased accuracy stems from the ability to interpret binomial distribution problems using the law of large numbers in probability theory.

[0184] Ensemble methods perform best when individual models are as independent as possible, so creating an ensemble combined prediction model is essentially creating a highly diverse set of individual models. A simple way to obtain multiple independent models is to train them using different algorithms or by varying the training data. This process creates multiple models with distinct characteristics, each performing its own unique inference, resulting in a highly random discrepancy between the predicted and unknown values. This random discrepancy can be interpreted as a random variable with a mean of zero. Therefore, if the prediction results of individual models are averaged or combined using a majority vote, the error can be expected to converge to zero.

[0185] The second ensemble prediction method is related to aspects of improving personal information and security. For example, it is assumed that the terminals (100) have gone through the process of analyzing a fusion-capable model, which will be described later, and are thus determined to be ensemble-capable models. Furthermore, it is assumed that the prediction result information can be used after an additional process of inserting an appropriate amount of noise into the input data. In this case, the transmitted prediction result information {y 1 t , y 2 t …y p t} can be used to use the ensemble prediction method. This does not seem to be much different from the first ensemble prediction method, but since the first ensemble prediction method did not add noise to the input data, the pth prediction result information is y p t = g(w p , x p t ) is expressed as y, but in the method of adding noise, p t = g(w p , z(x p t)) is expressed differently. The specific procedure or method for inserting noise applied to the terminal (100) is characterized in that it operates even if the combined processing device (200) does not know about it. Of course, if the terminal (100) and the combined processing device (200) share the noise insertion method, it may be more advantageous in ensemble restoration.

[0186] Even if noise is included in the original input data in this way, when ensemble prediction is performed, a kind of noise removal effect occurs, and the combined processing unit (200) can output a result with the noise removed. On the other hand, if an arbitrary attacker exists between the terminal (100) and the combined processing unit (200) and attempts to steal the prediction result of a specific terminal (100) and use it to inversely estimate the original input data, the attack becomes more difficult due to the inserted noise.

[0187] Input data recognition information (hash information) without added noise, model recognition information, prediction result information (noise added to input data) 1 ID 1 H 1 M 1 Y 1 2 ID 2 H 1 M 2 Y 2 3 ID 3 H 1 M 3 Y 3 4 ID 4 H 1 M 4 Y 4

[0188] Table 7 is an example of the second ensemble prediction method. Table 7 looks similar to Table 6. However, the difference here is that it has input data recognition information before noise insertion, and the prediction result information is the result of performing the prediction in a state where noise is added to the input data. In addition, the combined processing device (200) can perform ensemble combined prediction in the same way as when noise is not inserted in the terminal (100). Of course, if the noise insertion method or the parameters therefor are shared between the terminal (100) and the combined processing device (200) as described above, a modified prediction is also possible. The third ensemble prediction method is also related to the aspect of improving personal information and security capabilities. That is, for terminals (100) that commonly use a model capable of fusion prediction (ensemble prediction), it is a method of receiving input data recognition information in a hash format and performing fusion prediction using the prediction result information with an appropriate amount of noise inserted. This is similar to the second ensemble prediction method, but the difference is that the prediction result information {y 1 t , y 2 t …y p t}, the method is to add noise to the data that passed through the model (i.e., the prediction result) rather than the original input data. In other words, the pth prediction result information is y p t = z(g(w p , x p t )) can be expressed as

[0189] Here, there are various ways to add noise to the prediction result information. For example, if the output (i.e., the prediction result) is a class-specific probability value, the probability value can be additionally passed through a noise function with a mean of 0. If it is an embedding vector, dimensionality reduction can be performed, or Gaussian noise can be inserted into the elements constituting the vector.

[0190] Sequential input data recognition information (hash information) Model recognition information Prediction result information 1H1M1Y12H2M1Y23H3M1Y34H4M1Y45H1M2Y56H2M2Y67H3M2Y78H4M2Y8

[0191] In addition, the control unit (250) can select a model capable of fusion prediction, such as an ensemble, through fusion-capable model analysis, and Table 8 is an example thereof. That is, Table 8 shows a process of determining a fusion-capable model using input data recognition information (hash information), model recognition information, and prediction result information. For two different models {M1, M2}, input data {H1, H2, H3, H4} are input, respectively. As prediction result information, {Y1, Y2, Y3, Y4} is output for model M1, and {Y5, Y6, Y7, Y8} is output for model M2. If the difference between {Y1, Y5} is calculated, and similarly the differences between {Y2, Y6}, {Y3, Y7}, and {Y4, Y8} are calculated and combined, if the differences between the two prediction sets for M1 and M2 are similar within a threshold value, it can be determined that models M1 and M2 perform similar inferences. If we repeat this process for other models, we can select models that can be fused together to perform ensemble prediction.

[0192] The necessity of this process is further explained as follows. Ensemble techniques can be used to improve the predicted values ​​of terminals (100) that constitute a cluster. Conventionally, models for ensemble prediction were designed and trained in advance. However, the present invention differs in that it supports ensemble prediction even in the absence of such assumptions.

[0193] In conventional ensemble prediction, the same input data is passed through multiple predictors to obtain multiple prediction results, and the majority rule is applied or the average of the output probabilities is calculated to increase the prediction accuracy. On the other hand, since the terminals (100) belonging to the cluster of the present invention each use their own model, different values ​​of input data may be input, and the combined processing device (200) cannot directly check the input data input to each terminal (100), it is difficult to solve the problem using conventional methods. In order to solve this problem, in the present invention, when the terminal (100) registers a model to the combined processing device (200) according to FIG. 3, the combined processing device (200) selects a fusion-capable model through the fusion-capable model analysis described above.

[0194] Figure 8 shows an example of visualizing a large number of input data by embedding them and displaying them in a high-dimensional vector space.

[0195] In addition, when multiple terminals (100) use the same model, the control unit (250) can control the model (100) to be recommended to the terminal (100) by comparing the characteristics of the data source (input data) input to each terminal (100).

[0196] For example, in the combined processing device (200), the embedding information [e(x)] for the N input data used in the nth terminal (100_n) n 1), e(x n 2), … e(x n N)] can be collected. This embedding information corresponds to information expressed in a high-dimensional vector space by embedding the input data, as illustrated in Fig. 8. Accordingly, the combined processing device (200) can identify the characteristics of the data source using the embedding information of the collected input data recognition information, and can also compare the similarity (or difference) between different data sources among multiple terminals (100). Of course, for the embedding information, [z(e(x n 1)), z(e(x n 2)), … z(e(x n N ))] Noise may be added.

[0197] Figure 8 shows the information transfer process between a new terminal (100') and a combination processing device (200) when a new terminal (100') participates in a cluster.

[0198] In addition, when a new terminal (100') joins a cluster, the control unit (250) can control a predetermined procedure to be performed to identify the characteristics of the new terminal (100'), as illustrated in FIG. 8. At this time, the predetermined procedure corresponds to a procedure for exchanging predetermined information with the new terminal (100').

[0199] Specifically, the prescribed procedure is as follows. First, the control unit (250) controls {sample data, sample model} to be transmitted to the new terminal (100'). The new terminal (100') uses the received sample data as input data and uses the received sample model as a model for prediction. Accordingly, the new terminal (100) performs prediction and transmits prediction-related information of the prediction process collection information (hereinafter referred to as "first prediction process collection information") according to the prediction performance to the combination processing device (200).

[0200] At this time, {sample data, sample model} is data previously stored in the combined processing device (200), and the terminals (100) belonging to the cluster have already performed prediction using the {sample data, sample model} in the past, and prediction-related information of the prediction process collection information (hereinafter referred to as “second prediction process collection information”) according to the prediction performance is previously stored in the combined processing device (200). Accordingly, the control unit (250) can determine the relative operation processing performance of the new terminal (100') with respect to other terminals (100) by comparing the first and second prediction process collection information for {sample data, sample model}.

[0201] Additionally, in S620, the control unit (250) may implement a third function in addition to performing the second function described above. However, a detailed description of the enhancement of personal information and security according to this third function has already been described above, and thus will be omitted.

[0202] Additionally, in S620, the control unit (250) can control to perform the fourth function described above. However, since a detailed description of the fourth function regarding the process of model registration and model recognition information generation has already been described above, it will be omitted below.

[0203] The present invention, configured as described above, has the advantage that when each terminal performs a prediction in a cluster composed of a combined processing unit and a plurality of terminals, the combined processing unit can manage, analyze, and process prediction-related information provided by each terminal. In addition, the present invention has the advantage that, based on the analysis and processing of the prediction-related information, the combined processing unit can predict the execution complexity when a specific model is executed on a specific terminal, or can enhance the prediction performance by analyzing the prediction execution results of multiple terminals together. In addition, the present invention has the advantage of enhancing personal information and security by being implemented so that the combined processing unit does not directly check the input data or models processed by each terminal in the cluster when managing, analyzing, and processing prediction-related information, in consideration of personal information protection and security issues. In addition, the present invention has the advantage of being able to estimate the expected computation time or computational processing performance in terms of CPU and memory resources, etc., of a specific terminal for an unknown single input data or a large amount of input data having an arbitrary statistical distribution when a specific model is loaded on a specific terminal to perform a prediction. In addition, the present invention has an advantage in that, when a specific terminal must complete prediction within a target time, a model capable of completing the prediction within the target time can be recommended by taking into account the computational processing performance of the specific terminal. In addition, the present invention has an advantage in that it can provide information helpful in distributing prediction tasks when a plurality of terminals collaborate to provide a predetermined collaborative prediction service. In addition, the present invention has an advantage in that, when a specific terminal performs prediction using a specific model, the prediction performance can be improved based on prediction-related information such as past prediction result information stored and managed in a combined processing unit. In addition, the present invention has an advantage in that, when a plurality of terminals perform prediction using various models on the same input data, the combined processing unit can improve the prediction performance by utilizing the respective prediction results performed on the plurality of terminals.Furthermore, the present invention has the advantage of being able to compare the characteristics of the data sources (i.e., input data) input to each terminal when multiple terminals use the same model, thereby recommending a model with superior performance to that terminal. Furthermore, the present invention has the advantage of being able to easily identify the characteristics of a new terminal through a predefined procedure when a new terminal joins a cluster.

[0204] While the detailed description of the present invention has described specific embodiments, it should be understood that various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention is not limited to the described embodiments, but should be determined by the claims and their equivalents.

[0205] [National Research and Development Project Supporting This Invention]

[0206] [Project ID]1711193285

[0207] [Assignment Number] 2021-0-00907-003

[0208] [Ministry Name] Ministry of Science and ICT

[0209] [Name of Project Management (Specialist) Institution] Information and Communications Technology Planning and Evaluation Institute

[0210] [Research Project Name] Information and Communication Broadcasting Technology Development

[0211] [Research Project Title] Development of Adaptive, Lightweight Edge-Linked Analysis Technology Capable of Active, Immediate Response and Rapid Learning

[0212] [Name of the project performing organization] Electronics and Telecommunications Research Institute

[0213] Research Period: April 1, 2021 - December 31, 2024

[0214] The present invention relates to a technology for processing prediction-related information of multiple terminals, and provides a method, device, and system for receiving and processing prediction-related information, which is information related to prediction performance using a machine learning model, from multiple terminals, and thus has industrial applicability.

Claims

1. A method performed by a combined processing unit in a cluster including a plurality of terminals and combined processing units, A step for storing and managing prediction-related information according to the prediction performance transmitted from each terminal that performs prediction on input data using the loaded model; and A step of performing analysis using the above prediction-related information; including; The above prediction-related information includes prediction result information, which is a result of prediction performance, prediction process collection information, which is information on resources and status at the time of prediction performance, input data recognition information, which is information for distinguishing input data used at the time of prediction performance, and model recognition information, which is information for distinguishing models used at the time of prediction performance. A method of analyzing the expected computational processing performance of a specific terminal by using the prediction-related information when a prediction is performed by loading a specific model on a specific terminal in the step of performing the above analysis.

2. In paragraph 1, A method of analyzing the relative performance between specific terminals by deriving the expected operation processing performance of each specific terminal in which a specific model is installed and prediction is performed using the prediction-related information, and comparing the derived operation processing performance of each specific terminal in the step of performing the above analysis.

3. In paragraph 1, A method of recommending a recommendation model to a specific terminal by considering the analyzed operation processing performance in a case where a specific terminal must complete prediction within a target time in the step of performing the above analysis.

4. In paragraph 3, A method of performing the above analysis by analyzing prediction-related information received from each terminal, identifying relationships between each model, calculating complexity for each model, and performing the above recommendation by using the complexity for each model and the analyzed operation processing performance.

5. In paragraph 1, In the step of performing the above analysis, when a plurality of terminals collaborate to perform prediction, a method of providing information for distributing prediction work to each terminal by considering the analyzed operation processing performance.

6. In paragraph 1, A method of integrating prediction result information according to predictions made by multiple terminals on the same input data in the step of performing the above analysis.

7. In paragraph 6, A method in which intentional noise is inserted into the same input data, and the input data recognition information is composed of hash information for the same input data before noise insertion.

8. In paragraph 6, The above input data recognition information is composed of hash information for the same input data before noise insertion, and each prediction result information according to the prediction performed on the same input data is a method in which intentional noise is inserted.

9. In paragraph 1, In the step of performing the above analysis, when a plurality of terminals use the same model, a method of recommending a recommendation model to a specific terminal by comparing the input data recognition information composed of embedded information with the characteristics of the input data used in each terminal.

10. In paragraph 1, A method for determining the relative computational performance of the new terminal relative to the other terminals by, in the step of performing the above analysis, transmitting sample data and a sample model to the new terminal when a new terminal participates in the cluster, receiving first prediction process collection information according to prediction performance using the sample data and the sample model of the new terminal, and comparing second prediction process collection information according to prediction performance using the sample data and the sample model of another terminal previously belonging to the cluster with the first prediction process collection information.

11. A device that performs binding processing for a plurality of terminals in a cluster including a plurality of terminals, A memory storing prediction-related information according to the prediction performance transmitted from each terminal that performs prediction on input data using the loaded model; and A control unit that controls the performance of analysis using the prediction-related information stored in the above memory; The above prediction-related information includes prediction result information, which is a result of prediction performance, prediction process collection information, which is information on resources and status at the time of prediction performance, input data recognition information, which is information for distinguishing input data used at the time of prediction performance, and model recognition information, which is information for distinguishing models used at the time of prediction performance. The above control unit is a device that controls the expected computational processing performance of a specific terminal to be analyzed using the prediction-related information when a prediction is performed by loading a specific model on a specific terminal.

12. A plurality of terminals that perform predictions on input data using the installed model; and A combined processing device that implements a cluster with the above-mentioned multiple terminals, stores and manages prediction-related information according to the corresponding prediction performance transmitted from each terminal, and performs analysis using the prediction-related information; The above prediction-related information includes prediction result information, which is a result of prediction performance, prediction process collection information, which is information on resources and status at the time of prediction performance, input data recognition information, which is information for distinguishing input data used at the time of prediction performance, and model recognition information, which is information for distinguishing models used at the time of prediction performance. The above-mentioned combined processing device is a system that analyzes the expected computational processing performance of a specific terminal using the prediction-related information when a prediction is performed by loading a specific model on a specific terminal.

Images