METHOD FOR OPTIMIZING LEARNING MODEL FOR TARGET DEVICE AND SYSTEM THEREFOR

Abstract

The method and system for optimizing a learning model for a target device may include a step in which a learning model generation device generates a learning model by training a network function based on a predetermined framework using learning data, a step in which the learning model generation device encodes the learning model via a general-purpose encoding module so that it can be applied to a plurality of inference engines corresponding to a plurality of different operating environments or frameworks, a step in which the learning model generation device passes on the encoded learning model to the target device, a step in which the target device acquires site data related to the applied process, and a step in which the target device uses the site data to perform inference based on the learning model via an inference engine among the inference engines that is suitable for the operating environment of the target device.

Description

[Technical field]

[0001] The technical idea of ​​this disclosure relates to a method and system for optimizing a learning model for a target device. [Background technology]

[0002] Recently, there has been a demand to add artificial intelligence (AI)-based functions to computing equipment, particularly embedded devices, which are used in fields such as security, transportation, manufacturing, medical care, autonomous driving, and smart homes. However, there has not yet been sufficient effort to develop development tools such as application programs for efficiently installing deep learning models formed using AI in such embedded devices.

[0003] The efficiency of developing AI-based programs for embedded devices remains low, and is considered to be the biggest bottleneck impeding the development of related fields. In particular, existing AI development tools are primarily designed for the development of servers or cloud platforms, and there are currently insufficient AI development tools suited to embedded devices such as Internet of Things (IoT) devices.

[0004] In addition, in order to effectively install deep learning models in embedded devices with various operating environments, various optimization processes are required, but the technologies that have been introduced so far have limitations in that they cannot guarantee optimal operation for deep learning models. Summary of the Invention [Problem to be solved by the invention]

[0005] The technical problem that the method and system for optimizing a learning model for a target device based on the technical idea of ​​this disclosure aims to solve is to provide a method and system that can optimize and install a learning model to suit various operating environments of the target device.

[0006] The technical problems that the method and system according to the technical ideas of this disclosure attempt to solve are in no way limited to the above-mentioned technical problems, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0007] According to one aspect of the technical idea of ​​this disclosure, a method for optimizing a learning model for a target device may include a step in which a learning model generation device generates a learning model by training a network function based on a predetermined framework using learning data, a step in which the learning model generation device encodes the learning model via a general-purpose encoding module so that it is applicable to a plurality of inference engines corresponding to a plurality of different operating environments or frameworks, a step in which the learning model generation device passes on the encoded learning model to the target device, a step in which the target device acquires site data related to the applied process, and a step in which the target device uses the site data to perform inference based on the learning model via an inference engine among the inference engines that is suitable for the operating environment of the target device.

[0008] According to an exemplary embodiment, the operating environments may be differentiated based on the operating system and / or hardware of the target device.

[0009] According to an exemplary embodiment, the operating environment differentiated by the hardware may include at least one of a general-purpose central processing unit (CPU) environment, a general-purpose graphics processing unit (GPU) environment, and an embedded environment.

[0010] According to this exemplary embodiment, the field data input to the learning model and the output data of the learning model may be modified into a predetermined material structure via at least one buffer.

[0011] According to an exemplary embodiment, the method for optimizing a learning model for the target device may further include a step in which the learning model generation device retrains the learning model based on the on-site data and the output data.

[0012] According to one aspect of the technical idea of ​​this disclosure, a learning model optimization system for a target device may include a learning model generation device that generates a learning model by training a network function based on a predetermined framework using learning data, the learning model generation device encoding the learning model via a general-purpose encoding module so that the learning model is applicable to a plurality of inference engines corresponding to a plurality of different operating environments or frameworks, and the learning model generation device passing the encoded learning model to the target device, and the target device that acquires site data related to a process and uses the site data to perform inference based on the learning model via an inference engine among the inference engines that is suitable for the operating environment. Effect of the Invention

[0013] According to an embodiment of the technical concept of this disclosure, a learning model generated in a learning model generation device (server) can be converted and operated so as to be suitable for various operating environments of a target device.

[0014] The effects obtained by the method and system therefor based on the technical ideas of this disclosure are in no way limited to the effects described above, and other effects not mentioned will be clearly understood by a person having ordinary skill in the technical field to which this disclosure pertains from the following description.

[0015] In order to more fully understand the drawings referred to in this disclosure, a brief description of each of the drawings is provided. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a simplified block diagram of a system for optimizing a learning model for a target device according to an embodiment of the present disclosure. [Diagram 2] 1 is a flowchart illustrating a method for optimizing a learning model for a target device according to an embodiment of the present disclosure. [Diagram 3] A block diagram conceptually showing the configuration of a learning model generation module of a learning model generation device and an inference module of a target device according to an embodiment of the present disclosure. [Figure 4] 1 is a flowchart illustrating a method for optimizing a learning model for a target device according to an embodiment of the present disclosure. [Diagram 5] 1 is a block diagram showing a simplified configuration of a learning model generation device according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a block diagram illustrating a simplified configuration of a target device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Since the technical idea of ​​this disclosure can be modified in various ways and can have various embodiments, a specific embodiment will be illustrated in the drawings and described in detail. However, this is not intended to limit the technical idea of ​​this disclosure to the specific embodiment, and includes all modifications, equivalents, and alternatives included in the scope of the technical idea of ​​this disclosure.

[0018] In explaining the technical ideas of this disclosure, if it is recognized that a specific explanation of a known technology related to the present invention may make the gist of this disclosure unclear, the detailed explanation will be omitted. Note that numbers used in the explanation of this disclosure (e.g., "first", "second", etc.) are merely identification codes for distinguishing a certain component from other components.

[0019] In addition, in this disclosure, when a component is referred to as being "coupled" or "connected" to another component, the component may be directly coupled or connected to the other component, but unless otherwise specified in this specification or clearly contradicted by the context, there may be other components between them and the components may be coupled or connected via the other components.

[0020] Furthermore, the terms "unit," "device," "child," "module," and the like used in this disclosure refer to a unit that processes at least one function or operation, which may be realized by hardware or software, or a combination of hardware and software, such as a processor, microprocessor, microcontroller, central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), etc.

[0021] It should be made clear that the distinction between components in this disclosure is merely made according to the main function that each component is responsible for. In other words, two or more components described below may be combined into one component, or one component may be divided into two or more components for each further subdivided function. It goes without saying that each component described below may perform part or all of the functions that are assigned to other components in addition to the main function that it is responsible for, or part of the main function that each component is responsible for may be exclusively performed by other components.

[0022] The methods according to the embodiments of the present disclosure may be performed in a computing device such as a personal computer, a workstation, or a server with computing capabilities, or may be performed in a separate device for this purpose.

[0023] The method may be performed in one or more computing devices. For example, at least one or more steps of the method according to the embodiment of the present disclosure may be performed in a client device, and other steps may be performed in a server device. In such a case, the client device and the server device may be connected via a network to transmit and receive the computation results. Alternatively, the method may be performed using a distributed computing technique.

[0024] Also, throughout this specification, network function may be used synonymously with neural network and / or neural network. Here, a neural network may generally be composed of a set of interconnected computational units that may be referred to as nodes, and such nodes may also be referred to as neurons. A neural network is generally composed of a plurality of nodes. The nodes that make up a neural network may be connected to each other by one or more links.

[0025] Some of the nodes that make up the neural network may form a layer based on their distance from the first input node. For example, a set of nodes that are a distance n from the first input node may form an n-th layer.

[0026] The neural networks described herein may comprise deep neural networks (DNNs) that include multiple hidden layers in addition to input and output layers.

[0027] The embodiments of this disclosure will be described in detail below.

[0028] FIG. 1 is a simplified block diagram of a system for optimizing a learning model for a target device according to an embodiment of the present disclosure.

[0029] The system may include a learning model generating device 110 and a target device 120 .

[0030] In an embodiment, the learning model generation device 110 may be a cloud server.

[0031] The learning model generation device 110 may generate a learning model and pass it to the target device 120. To this end, the learning model generation device 110 may include a learning model generation module. The learning model generation module may be a software module.

[0032] Specifically, the learning model generation module may generate a network function based on a predetermined framework (e.g., TensorFlow), and if learning data is input from the outside, may learn the network function based on the input data, and generate a learning model as a result of the learning. In addition, the learning model generation module may encode the generated learning model via a general-purpose encoding module into a format that can be run in multiple operating environments and / or frameworks. The learning model may be passed to the target device 120 in the form of an encoded file.

[0033] In addition, in an embodiment, the learning model generation device 110 may retrain the learning model using the on-site data passed from the target device 120 and the output data output by the learning model based on the on-site data.

[0034] The target device 120 indicates a target device on which a learning model is to be realized, and covers all kinds of computing devices such as a server, a personal computer (PC), a smart phone, a smart pad, a tablet PC, etc. In the embodiment, the target device 120 may further include an IoT terminal, an edge device, an embedded board, etc. that operates in an embedded environment.

[0035] The target device 120 may receive at least one execution code, program, library, etc. for generating an encoded learning model and an inference module from the learning model generation device 110, and drive the learning model through the generated inference module. In this case, the inference module may be a software module.

[0036] The inference module may include multiple inference engines and at least one buffer (input / output buffer) corresponding to different operating environments. The inference module may run the learning model through an inference engine suitable for the operating environment, such as the operating system and / or hardware of the target device 120, and perform inference using field data acquired by the target device 120.

[0037] At this time, the field data input as the learning model and the output data of the learning model may be converted into a predetermined data structure via a buffer.

[0038] FIG. 2 is a flowchart for explaining a method for optimizing a learning model for a target device according to an embodiment of the present disclosure, and FIG. 3 is a block diagram conceptually showing the configuration of a learning model generation module of a learning model generation device according to an embodiment of the present disclosure and an inference module of a target device.

[0039] In step S210, the learning model generation device 110 may generate a learning model 312 by inputting learning data to the network function 311 of the learning model generation module 310 and performing learning.

[0040] In this case, the network function 311 may be generated based on a predetermined framework. The framework indicates a kind of package configured to allow a developer to easily use a plurality of verified libraries and modules, algorithms that have completed pre-learning, etc., and may include, for example, Tensorflow, Keras, Pytorch, Cafe, MXnet, etc., but is not limited thereto, and a wide variety of frameworks can be used to generate the network function 311.

[0041] In this embodiment, the learning model generation module 310 may generate the learning model 312 using a network function based on tensorflow.

[0042] In an embodiment, the training data may be a plurality of images. For example, the training data may be process images acquired (or photographed) during any process such as production, manufacturing, processing, etc. of a product, and may include information about a label to be detected or read.

[0043] In step S220, the learning model generation device 110 may encode the learning model 312 via a generic encoding module 313 of the learning model generation module 310. Through step S220, the learning model 312 can be converted into a format that can be run in a number of different operating environments and / or frameworks.

[0044] In an embodiment, the encoding module 313 may convert the learning model 312 into a neural network data format such as Neural Network Exchange Format (NNEF), Open Neural Network Exchange (ONNX), etc., but is not limited thereto.

[0045] In step S230, the learning model generation device 110 may pass the encoded learning model 312 to the target device 120.

[0046] Here, the target device 120 is provided to control a process or measure or acquire specific data within the process environment and perform diagnosis or prediction related to the process based on the data, and may be provided with an inference module 320 for driving a learning model to perform inference.

[0047] In step S240, the target device 120 may acquire on-site data related to the process. For example, the target device 120 may be provided as a single component, or may acquire the on-site data from a sensing device (at least one sensor, camera, etc.) connected to the target device 120 via wired or wireless communication.

[0048] In an embodiment, the in-situ data may be at least one image (eg, a process image).

[0049] In step S250, the target device 120 may perform inference using the in-situ data via the inference module 320.

[0050] In the S250 embodiment, the inference module 320 may be provided with multiple inference engines 321, each suitable for a different operating environment, and may perform inference based on on-site data by driving the learning model 312 through an inference engine 321 suitable for the operating environment of the target device 120 among the multiple inference engines 321 installed.

[0051] Here, the operating environments may be distinguished based on at least one of the operation system and the hardware (or hardware architecture) of the target device 120.

[0052] In an embodiment, the operating environment distinguished by the operation system may include at least one of a Linux environment and a window environment, however, this is merely exemplary and different operating environments may be defined according to various operating systems such as IOS, Android, Mac OS, etc.

[0053] In addition, in an embodiment, the operating environment distinguished by hardware may include at least one of a general-purpose central processing unit (CPU) environment, a general-purpose graphics processing unit (GPU) environment, and an embedded environment. However, this is merely exemplary, and different operating environments may be defined according to a wide variety of hardware architectures, such as Nvidia, Jetson, Raspberry, ARM Cortex-A Family, Qcam Family, Hisilicon Family, X86-CPU, and X86-GPU.

[0054] In an embodiment, the inference engine 321 may include a first inference engine suitable for a general-purpose GPU environment and / or embedded environment, and a second inference engine suitable for a general-purpose CPU environment.

[0055] In an embodiment, the input data and output data of the learning model 312 driven via the inference engine 321 may be transformed into a predetermined material structure via at least one buffer 322 .

[0056] For example, the buffer 322 may include an input buffer and an output buffer, and the target device 120 may assign the on-site data to the input buffer and convert it into a material structure suitable for driving the learning model 312, and may assign the output data generated based on the inference result of the learning model 312 to the output buffer and convert it into a specified material structure. The material structure may be a structure corresponding to a program language that the target device 120 can parse.

[0057] FIG. 4 is a flowchart illustrating a method for optimizing a learning model for a target device according to an embodiment of the present disclosure.

[0058] Steps S410 to S450 in the method 400 are similar to steps S210 to S250 in the method 200 described above with reference to FIG. 2, and therefore a duplicated description will be omitted.

[0059] The method 400 may further include retraining the learning model 313 .

[0060] Specifically, in step S460, the target device 120 may input on-site data and on-site data and pass on output data obtained via the inference engine 321 and the learning model 313 to the learning model generation device 110, and the learning model generation device 110 may update the learning model 313 by re-learning the learning model 313 based on the received on-site data and output data.

[0061] The updated learning model 313 may be encoded and transmitted to the target device 120 via a generic encoding module 313 .

[0062] In an embodiment, step S460 may be performed periodically at regular time intervals.

[0063] FIG. 5 is a block diagram showing a simplified configuration of a learning model generating device according to an embodiment of the present disclosure.

[0064] The communication unit 510 may transmit and receive data to and from the outside. The communication unit 510 may include a wired / wireless communication unit. When the communication unit 510 includes a wired communication unit, the communication unit 510 may include one or more components that perform communication via a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, a satellite communication network, and combinations thereof. When the communication unit 510 includes a wireless communication unit, the communication unit 510 may transmit and receive data or signals wirelessly using cellular communication, wireless LAN (e.g., Wi-Fi), and the like. In an embodiment, the communication unit may transmit and receive data or signals to and from an external device or an external server under the control of the processor 540.

[0065] The input unit 520 may receive a wide variety of user commands through an external operation. To this end, the input unit 520 may include one or more input devices or may be connected thereto. For example, the input unit 520 may be connected to an interface for a wide variety of inputs, such as a keypad or a mouse, to receive user commands. To this end, the input unit 520 may include an interface such as a Thunderbolt as well as a USB port. In addition, the input unit 520 may include a wide variety of input devices, such as a touch screen or a button, or may be coupled to these to receive external user commands.

[0066] The memory 530 may store programs and / or program commands for the operation of the processor 540, and may temporarily or permanently store input / output data. The memory 530 may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0067] Memory 530 may also store various network functions, algorithms, libraries, etc., and may store a wide variety of data, programs (one or more instructions), applications, software, instructions, code, etc. for driving and controlling device 500.

[0068] The processor 540 may execute at least one program and / or instructions to control the overall operation of the device 500. The processor 540 may execute one or more programs stored in the memory 530. The processor 540 may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which the methods according to the concepts of the present disclosure are performed.

[0069] In an embodiment, the processor 540 may generate the learning model generation module 310 based on at least one program and / or instruction. The learning model generation module 310 may be a software module.

[0070] In an embodiment, the processor 540 may generate a learning model by training a network function based on a predetermined framework using training data, and the learning model generation device may encode the learning model via a general-purpose encoding module so that the learning model can be applied to multiple inference engines corresponding to multiple different operating environments or frameworks, and pass the encoded learning model to the target device.

[0071] In an embodiment, the processor 540 may retrain the learning model based on the in-situ data received from the target device and the output data of the inference engine and the learning model.

[0072] FIG. 6 is a block diagram showing a simplified configuration of a target device according to an embodiment of the present disclosure.

[0073] The communication unit 610 may receive data from an external source. The communication unit 610 may include a wired / wireless communication unit. When the communication unit 610 includes a wired communication unit, the communication unit 610 may include one or more components that perform communication via a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, a satellite communication network, or a combination thereof. When the communication unit 610 includes a wireless communication unit, the communication unit 610 may transmit and receive data or signals wirelessly using cellular communication, wireless LAN (e.g., Wi-Fi), or the like. In an embodiment, the communication unit may transmit and receive data or signals to and from an external device or an external server under the control of the processor 640.

[0074] The input unit 620 may receive a wide variety of user commands through an external operation. To this end, the input unit 620 may include one or more input devices or may be connected thereto. For example, the input unit 620 may be connected to an interface for a wide variety of inputs, such as a keypad or a mouse, to receive user commands. To this end, the input unit 620 may include an interface such as a Thunderbolt as well as a USB port. In addition, the input unit 620 may include a wide variety of input devices, such as a touch screen or a button, or may be coupled to these to receive external user commands.

[0075] The memory 630 may store programs and / or program commands for the operation of the processor 640, and may temporarily or permanently store input / output data. The memory 630 may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0076] Memory 630 may also store various network functions, algorithms, libraries, etc., and may store a wide variety of data, programs (one or more instructions), applications, software, instructions, code, etc. for driving and controlling device 600.

[0077] The processor 640 may execute at least one program and / or instructions to control the overall operation of the device 600. The processor 640 may execute one or more programs stored in the memory 630. The processor 640 may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which the methods according to the concepts of the present disclosure are performed.

[0078] In this embodiment, the processor 640 may generate the inference module 320 based on the learning model, at least one program and / or instructions received from the learning model generating device. The inference module 320 may be a software module.

[0079] In an embodiment, the processor 540 may acquire field data associated with the process and use the field data to make inferences based on the learning model via an inference engine that is appropriate for the operating environment of the device 600 among the inference engines.

[0080] In an embodiment, the processor 540 may retrain the learning model based on the in-situ data received from the target device and the output data of the inference engine and the learning model.

[0081] On the other hand, although not shown, the device 600 may further include a sensing unit and / or an image capturing unit for acquiring on-site data. In this case, the sensing unit may be configured with at least one sensing module, and the image capturing unit may be configured with a camera module.

[0082] Methods according to embodiments of the present disclosure may be embodied in a type of program commands that may be executed by a wide variety of computer means and recorded on a computer readable medium. The computer readable medium may include, alone or in combination, program commands, data files, data structures, and the like. The program commands recorded on the medium may be those specially designed and constructed for this disclosure, or may be those known and available to those skilled in the art of computer software. Examples of computer readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD), magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as Read Only Memory (ROM), Random Access Memory (RAM), and flash memory. Examples of program commands include not only machine code, such as produced by a compiler, but also higher level language code executable by a computer using, for example, an interpreter.

[0083] Methods according to the disclosed embodiments may also be provided in a computer program product, which may be traded as a commodity between sellers and buyers.

[0084] The computer program product may include a S / W program and a computer-readable storage medium on which the S / W program is stored. For example, the computer program product may include a product in the form of a S / W program (e.g., a downloadable application) that is electronically distributed by an electronic device manufacturer or via an online market (e.g., Google Play Store, Application Store). For electronic distribution, at least a part of the S / W program may be stored in the storage medium or may be temporarily generated. In this case, the storage medium may be a storage medium of a manufacturer's server, an online market server, or a relay server that temporarily stores the SW program.

[0085] In a system including a server and a client device, the computer program product may include a storage medium of the server or a storage medium of the client device. Alternatively, when a third device (e.g., a smartphone) is connected to the server or the client device by communication, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S / W program itself that is transmitted from the server to the client device or the third device, or transmitted from the third device to the client device.

[0086] In this case, one of the server, the client device, and the third device may run the computer program product to perform the method according to the disclosed embodiment, or two or more of the server, the client device, and the third device may run the computer program product to perform the method according to the disclosed embodiment in a distributed manner.

[0087] For example, a server (e.g., a cloud server or an artificial intelligence server) may launch a computer program product stored on the server to control client devices communicatively connected to the server to perform methods according to the disclosed embodiments.

[0088] Although the embodiments have been described in detail above, the scope of the disclosure is in no way limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of this disclosure as defined in the appended claims also fall within the scope of the disclosure.

Claims

1. 1. A method for optimizing a learning model for a target device, comprising: A learning model generating device generates a learning model by training a network function based on a predetermined framework using training data; The learning model generation device encodes the learning model via a general-purpose encoding module so that the learning model can be applied to a plurality of inference engines corresponding to a plurality of different operating environments or frameworks; The learning model generation device transmits the encoded learning model to the target device; the target device acquiring in situ data associated with the applied process; The target device performs inference based on the learning model using the on-site data via an inference engine that is suitable for the operating environment of the target device among the inference engines. A method comprising:

2. The operating environments are differentiated based on at least one of the operating system and the hardware of the target device. The method of claim 1.

3. The operating environment distinguished by the hardware includes at least one of a general-purpose central processing unit (CPU) environment, a general-purpose graphics processing unit (GPU) environment, and an embedded environment. The method of claim 2.

4. The field data input to the learning model and the output data of the learning model are converted into a predetermined data structure through at least one buffer. The method of claim 1.

5. The learning model generating device further includes a step of re-learning the learning model based on the on-site data and the output data. The method according to claim 4.

6. In a system for optimizing a learning model for a target device, a learning model generation device that generates a learning model by training a network function based on a predetermined framework using training data, encodes the learning model via a general-purpose encoding module so that the learning model can be applied to a plurality of inference engines corresponding to a plurality of different operating environments or frameworks, and passes the encoded learning model to the target device; the target device acquires on-site data related to the process, and performs inference based on the learning model using the on-site data via an inference engine that is suitable for a driving environment among the inference engines. A system characterized by:

Images