Apparatus and method for using machine learning models shared among multiple applications

Abstract

The electronic device can map a target application to a machine learning model that matches a request of the target application among a plurality of machine learning models, can generate an inference result for sensing data corresponding to the machine learning model based on sensing data sensed by at least one sensor, and can transmit the generated inference result to at least one of the target application and another application.

Description

TECHNICAL FIELD

[0001] The disclosure relates to a technology using a machine learning model shared by a plurality of applications. BACKGROUND

[0002] Research is being conducted to apply a human efficient pattern recognition method to an actual computer. The research includes research on an artificial neural network (ANN) obtained by modeling characteristics of human biological neurons through mathematical expressions. To solve a problem of classifying an input pattern into a predetermined group, the ANN employs an algorithm based on a human learning ability. Through the algorithm, the ANN can generate a mapping between an input pattern and an output pattern, and has a generalization ability to generate a relatively correct output for an input pattern not used for training. SUMMARY

[0003] According to an example embodiment, an electronic device includes a processor, a memory electrically connected to the processor and configured to store a plurality of machine learning models, and at least one sensor configured to sense sensing data, wherein the processor is configured to map a target application to a machine learning model of the plurality of machine learning models that matches a request of the target application, generate an inference result by applying the mapped machine learning model to the sensing data in response to the at least one sensor sensing the sensing data corresponding to the machine learning model, and transmit the generated inference result to at least one of the target application and another application mapped to the machine learning model.

[0004] The processor can be further configured to provide the generated inference result to the target application configured to operate in one of a foreground process and a background process and the other application configured to operate in one of the foreground process and the background process.

[0005] The processor can be further configured to generate an inference result using the machine learning model before a corresponding application is activated based on the application mapped to the machine learning model being deactivated, and activate the corresponding application based on the inference result.

[0006] The processor can be further configured to activate the deactivated application based on at least one of an estimated label and an output value included in the inference result.

[0007] The processor can be further configured to activate the deactivated application and transmit the inference result to the target application in response to the output value included in the inference result being outside a threshold range, and maintain a state of the deactivated application in response to the output value included in the inference result being within the threshold range.

[0008] The processor can be further configured to activate the deactivated application in response to an estimated label included in the inference result matching a reference label registered in the deactivated application.

[0009] The processor can be further configured to provide another inference result to the corresponding application whenever the other inference result is generated based on the corresponding application being activated.

[0010] The processor can be further configured to determine a machine learning model matching the target application from among a plurality of machine learning models based on at least one or a combination of two or more of a type of input data required in an inference operation of the target application, a format of the input data, an input source, a type of output data, a label list of the output data, and a format of the output data.

[0011] The processor can be further configured to generate an inference result by implementing a machine learning model on sensed data using an accelerator. The accelerator can include at least one of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and a neural processing unit (NPU), or a combination of two or more thereof.

[0012] The plurality of machine learning models can be stored in a storage space different from a storage space in which the target application is stored.

[0013] According to an example embodiment, a method implemented by a processor includes mapping a target application to a machine learning model matching a request of the target application from among a plurality of machine learning models, generating an inference result by applying the mapped machine learning model to sensed data in response to at least one sensor sensing the sensed data corresponding to the machine learning model, and transferring the generated inference result to at least one of the target application and another application mapped to the machine learning model. BRIEF DESCRIPTION OF DRAWINGS

[0014] The above and other aspects, features and advantages of certain embodiments of the disclosure will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0015] Figure 1 is a block diagram illustrating an example electronic device in a network environment according to various embodiments;

[0016] Figure 2 is a block diagram illustrating an example configuration of a program according to various embodiments;

[0017] Figure 3 is a flowchart illustrating an example method of using a shared machine learning model according to various embodiments;

[0018] Figure 4is a block diagram illustrating an example configuration of a machine learning model sharer according to various embodiments;

[0019] Figure 5 is a signal flow diagram illustrating an example operation of mapping and registering an application and a machine learning model according to various embodiments;

[0020] Figure 6 is a block diagram illustrating an example of a machine learning model sharer according to various embodiments; Figure 5 is a diagram illustrating an example of a mapping result according to various embodiments;

[0021] Figure 7a and Figure 7b is a flow diagram illustrating an example of an inference operation using a shared machine learning model according to various embodiments; and

[0022] Figure 8 is a diagram illustrating an example of transmitting an inference result of a shared machine learning model to a first application and a second application according to various embodiments. DETAILED DESCRIPTION

[0023] Hereinafter, various example embodiments will be described in greater detail with reference to the accompanying drawings. When example embodiments are described with reference to the drawings, the same elements are designated with like reference numerals, and repetitive description thereof will not be provided.

[0024] Figure 1 is a block diagram illustrating an example electronic device 101 in a network environment 100 according to various example embodiments. Referring to Figure 1 , the electronic device 101 in the network environment 100 can communicate with an electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or at least one of an electronic device 104 or a server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an example embodiment, the electronic device 101 can communicate with the electronic device 104 via the server 108. According to an example embodiment, the electronic device 101 can include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, and a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a subscriber identification module (SIM) 196, or an antenna module 197. In various example embodiments, at least one of the components (e.g., the connection terminal 178) can be omitted from the electronic device 101, or one or more other components can be added in the electronic device 101. In various example embodiments, some of the components (e.g., the sensor module 176, the camera module 180, or the antenna module 197) can be configured as a single component (e.g., the display module 160).

[0025] The processor 120 can execute, for example, software (e.g., a program 140) to control at least one other component (e.g., a hardware or software component) of the electronic device 101 connected to the processor 120 and can perform various data processing or computation. According to example embodiments, as at least a part of the data processing or computation, the processor 120 can store a command or data received from another component (e.g., the sensor module 176 or the communication module 190) in the volatile memory 132, process the command or the data stored in the volatile memory 132, and store result data in the non-volatile memory 134. According to example embodiments, the processor 120 can include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) or a subsidiary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that can operate independently of the main processor 121 or in conjunction with the main processor 121. For example, when the electronic device 101 includes the main processor 121 and the subsidiary processor 123, the subsidiary processor 123 can be adapted to consume less power than the main processor 121 or to be specialized in a specified function. The subsidiary processor 123 can be implemented as a separate entity from or as a part of the main processor 121.

[0026] The auxiliary processor 123 can control at least some of the functions or states related to at least one of the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190), instead of the main processor 121 when the main processor 121 is in an inactive (e.g., sleep) state, or together with the main processor 121 when the main processor 121 is in an active state (e.g., executing an application). According to an example embodiment, the auxiliary processor 123 (e.g., an ISP or a CP) can be implemented as a part of another component (e.g., the camera module 180 or the communication module 190) functionally related to the auxiliary processor 123. According to an example embodiment, the auxiliary processor 123 (e.g., an NPU) can include a hardware structure designated for processing of a machine learning model. The machine learning model can be generated by machine learning, and can also be referred to as an artificial intelligence model. Such learning can be performed by, for example, the electronic device 101 in which artificial intelligence is executed, or via a separate server (e.g., the server 108). The learning algorithm can include, but is not limited to, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The AI model can include multiple artificial neural network layers. The artificial neural network can include, for example, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), and a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more thereof, but is not limited thereto. The AI model can additionally or alternatively include a software structure other than the hardware structure.

[0027] The memory 130 can store various data used by at least one component (e.g., the processor 120 or the sensor module 176) of the electronic device 101. The various data can include, for example, software (e.g., the program 140) and input data or output data about a command related thereto. The memory 130 can include the volatile memory 132 or the non-volatile memory 134.

[0028] The program 140 can be stored in the memory 130 as software, and can include, for example, an operating system (OS) 142, middleware 144, or an application 146.

[0029] The input module 150 can receive a command or data to be used by another component (e.g., the processor 120) of the electronic device 101, from the outside (e.g., a user) of the electronic device 101. The input module 150 can include a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0030] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as playing multimedia or playing record. The receiver can be used to receive an incoming call. According to an example embodiment, the receiver can be implemented as part of the speaker, or can be implemented separately from the speaker.

[0031] The display module 160 can visually provide information to the outside (e.g., a user) of the electronic device 101. The display module 160 can include, for example, a control circuit for controlling a display, a hologram device or a projector, and a control circuit for controlling a corresponding one of the display, the hologram device, and the projector. According to an example embodiment, the display module 160 can include a touch sensor adapted to detect a touch, or a pressure sensor adapted to measure the intensity of force incurred by the touch.

[0032] The audio module 170 can convert a sound into an electrical signal, and vice versa. According to an example embodiment, the audio module 170 can obtain sound via the input module 150, or output sound via the sound output module 155 or an external electronic device (e.g., an electronic device 102 such as a speaker or a headphone) directly or wirelessly connected thereto.

[0033] The sensor module 176 can detect an operational state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a state of a user) external to the electronic device 101, and generate an electrical signal or data value corresponding to the detected state. According to an example embodiment, the sensor module 176 can include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0034] The interface 177 can support one or more designated protocols for coupling the electronic device 101 directly (e.g., wiredly) or wirelessly to an external electronic device (e.g., an electronic device 102). According to an example embodiment, the interface 177 can include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0035] The connection terminal 178 can include a connector through which the electronic device 101 can be physically connected to the external electronic device (e.g., the electronic device 102). According to an example embodiment, the connection terminal 178 can include, for example, a HDMI connector, a USB connector, a SD card connector, or an audio connector (e.g., a headphone connector).

[0036] The haptic module 179 can convert electrical signal into a mechanical stimulus (e.g., vibration or movement) or electrical stimulus that a user can feel through his or her tactile sensation. According to an example embodiment, the haptic module 179 can include at least one of a motor, a piezoelectric, or an electric stimulator.

[0037] The camera module 180 can capture still images and moving images. According to an example embodiment, the camera module 180 can include one or more lenses, image sensors, image signal processors, or flashes.

[0038] The power management module 188 can manage power supplied to the electronic device 101. According to an example embodiment, the power management module 188 can be implemented as at least a part of, for example, a power management integrated circuit (PMIC). The power management module 188 can include a power meter that can measure usage state information of the battery 189 (e.g., capacity, charging or discharging times, voltage, or temperature of the battery 189).

[0039] The battery 189 can supply power to at least one component of the electronic device 101. According to an example embodiment, the battery 189 can include, for example, a primary cell which is not rechargeable, a secondary cell which is rechargeable, or a fuel cell.

[0040] The communication module 190 can support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., the electronic device 102, the electronic device 104, or the server 108) and performing communication via the established communication channel. The communication module 190 can include one or more communication processors that are operable independently of the processor 120 (e.g., an AP) and support direct (e.g., wired) communication or wireless communication. According to an example embodiment, the communication module 190 can include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with the external electronic device 104 via a first network 198 (e.g., a short-range communication network such as BluetoothTM, wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or wide area network (WAN)). These different types of communication modules can be implemented as a single component (e.g., a single chip) or can be implemented as separate components (e.g., separate chips) from each other. The wireless communication module 192 can identify and authenticate the electronic device 101 in a communication network, such as the first network 198 or the second network 199, using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the SIM 196.

[0041] The wireless communication module 192 can support a 5G network after a 4G network, and next-generation communication technologies, such as new radio (NR) access technology. The NR access technology can support enhanced mobile broadband (eMBB), massive machine-type communications (mMTC), or ultra-reliable low-latency communications (URLLC). The wireless communication module 192 can support a high frequency band (e.g., a millimeter wave band) to achieve, for example, high data transmission rates. The wireless communication module 192 can support various technologies for securing performance on a high frequency band, such as, for example, beamforming, massive multiple-input multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module 192 can support various requirements designated in the electronic device 101, an external electronic device (e.g., an electronic device 104), or a network system (e.g., a second network 199). According to an example embodiment, the wireless communication module 192 can support a peak data rate for implementing eMBB (e.g., 20 Gbps or more), a loss coverage for implementing mMTC (e.g., 164 dB or less), or a U-plane latency for implementing URLLC (e.g., 0.5 ms or less in each of downlink (DL) and uplink (UL), or 1 ms or less for a round trip).

[0042] The antenna module 197 can transmit or receive a signal or power to or from an outside (e.g., an external electronic device) of the electronic device 101. According to an example embodiment, the antenna module 197 can include an antenna including a radiating element including a conductive material or a conductive pattern formed in or on a base (e.g., a printed circuit board (PCB)). According to an example embodiment, the antenna module 197 can include a plurality of antennas (e.g., array antennas). In this case, for example, the communication module 190 can select at least one antenna suitable for a communication scheme suitable for use in a communication network, such as the first network 198 or the second network 199, from among the plurality of antennas. A signal or power can be transmitted or received between the communication module 190 and the external electronic device via at least one selected antenna. According to an example embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiating element can additionally be formed as part of the antenna module 197.

[0043] According to various example embodiments, the antenna module 197 can form a millimeter wave (mmWave) antenna module. According to example embodiments, the mmWave antenna module can include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., a bottom surface) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., a top surface or a side surface) of the printed circuit board and capable of transmitting or receiving a signal of the designated high frequency band.

[0044] At least some of the above-described components can be coupled mutually via an inter-peripheral communication scheme (e.g., a bus, a general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicate signals (e.g., commands or data) therebetween.

[0045] According to example embodiments, commands or data can be transmitted or received between the electronic device 101 and the external electronic device 104 via the server 108 coupled with the second network 199. Each of the external electronic devices 102 or 104 can be a device of a same type as or different from the electronic device 101. According to example embodiments, all or some of the operations to be executed by the electronic device 101 can be executed at one or more external electronic devices 102 and 104 and the server 108. For example, if the electronic device 101 needs to perform a function or a service automatically, or in response to a request from a user or another device, the electronic device 101, instead of, or in addition to, executing the function or the service, can request one or more external electronic devices to perform at least a part of the function or the service. The one or more external electronic devices receiving the request can execute at least a part of the requested function or service, or an additional function or an additional service related to the request, and can transmit a result of the execution to the electronic device 101. The electronic device 101 can provide the result with or without further processing of the result, as at least a part of a reply to the request. To this end, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing techniques can be used. The electronic device 101 can use, for example, distributed computing or mobile edge computing to provide an ultra-low latency service. In example embodiments, the external electronic device 104 can include an Internet of Things (IoT) device. The server 108 can be an intelligent server using machine learning and / or a neural network. According to example embodiments, the external electronic device 104 or the server 108 can be included in the second network 199. The electronic device 101 can be applied to intelligent services (e.g., smart home, smart city, smart car, or health care) based on 5G communication technology or Internet of Things-related technology.

[0046] Figure 2is a block diagram 200 illustrating an example configuration of the program 140 according to various embodiments. According to an example embodiment, the program 140 can include an OS 142 to control one or more resources of the electronic device 101, the middleware 144, or the applications 146 executable in the OS 142. The OS 142 can include, for example, AndroidTM, iOS TM, WindowsTM, SymbianTM, TizenTM, or BadaTM. At least a portion of the program 140 can be preloaded on the electronic device 101 during a manufacturing process, or can be downloaded or updated from an external electronic device (e.g., the electronic devices 102 or 104, or the server 108) during a user's use, for example.

[0047] The OS 142 can control management (e.g., allocation or de-allocation) of one or more system resources (e.g., processes, memory, or power) of the electronic device 101. The OS 142 can additionally or alternatively include other one or more drivers to drive other hardware devices of the electronic device 101, such as the input module 150, the sound output module 155, the display module 160, the audio module 170, the sensor module 176, the interface 177, the haptic module 179, the camera module 180, the power management module 188, the battery 189, the communication module 190, the SIM 196, or the antenna module 197.

[0048] The middleware 144 can provide various functions to the applications 146 so that the applications 146 can use functions or information provided from one or more resources of the electronic device 101. The middleware 144 can include, for example, an application manager 201, a window manager 203, a multimedia manager 205, a resource manager 207, a power manager 209, a database manager 211, a package manager 213, a connectivity manager 215, a notification manager 217, a location manager 219, a graphic manager 221, a security manager 223, a telephony manager 225, or a voice recognition manager 227.

[0049] The application manager 201, for example, can manage a life cycle of the applications 146. The window manager 203, for example, can manage one or more graphical user interface (GUI) resources used on a screen. The multimedia manager 205, for example, can identify one or more formats for playing media files, and can encode or decode a corresponding one of the media files using a codec suitable for a corresponding format selected from the one or more formats. The resource manager 207, for example, can manage source code or a memory space of the memory 130 of the applications 146. The power manager 209, for example, can manage a capacity, a temperature, or power of the battery 189, and can determine or provide related information for an operation of the electronic device 101 based at least in part on corresponding information of the capacity, the temperature, or the power of the battery 189. According to an example embodiment, the power manager 209 can interwork with a basic input / output system (BIOS) (not shown) of the electronic device 101.

[0050] The database manager 211, for example, can generate, search, or change a database to be used by the applications 146. The package manager 213, for example, can manage installation or update of an application distributed in the form of a package file. The connectivity manager 215, for example, can manage wireless connectivity or direct connectivity between the electronic device 101 and an external electronic device. The notification manager 217, for example, can provide a function of notifying a user of occurrence of a specific event (e.g., an incoming call, a message, or an alarm). The location manager 219, for example, can manage location information on the electronic device 101. The graphic manager 221, for example, can manage one or more graphic effects to be provided to a user or a user interface related to the one or more graphic effects.

[0051] The security manager 223, for example, can provide system security or user authentication. The telephony manager 225, for example, can manage a voice call function or an image call function provided by the electronic device 101. The voice recognition manager 227, for example, can transmit voice data of a user to the server 108, and can receive a command corresponding to a function to be executed on the electronic device 101 based at least in part on text data converted based at least in part on the voice data from the server 108. According to an example embodiment, the middleware 144 can dynamically delete some existing components or add new components. According to an example embodiment, at least a portion of the middleware 144 can be included as a part of the OS 142, or can be implemented as another software separate from the OS 142.

[0052] The applications 146 can include, for example, a home 251, a dialer 253, a short message service (SMS) / multimedia message service (MMS) 255, an instant message (IM) 257, a browser 259, a camera 261, an alarm 263, a contact 265, voice recognition 267, an e-mail 269, a calendar 271, a media player 273, a album 275, a watch 277, a health 279 (e.g., for measuring an exercise level or bio information such as blood sugar), or an environment information 281 (e.g., for measuring atmospheric pressure, humidity, or temperature information) application. According to an example embodiment, the applications 146 can further include an information exchange application (not shown) capable of supporting information exchange between the electronic device 101 and an external electronic device. For example, the information exchange application can include a notification relay application adapted to transmit specified information (e.g., a call, a message, or an alarm) to an external electronic device, or a device management application adapted to manage the external electronic device. The notification relay application can transmit notification information corresponding to the occurrence of a specified event (e.g., the reception of an e-mail) at another application (e.g., the e-mail application 269) of the electronic device 101 to the external electronic device. Additionally or alternatively, the notification relay application can receive notification information from the external electronic device and provide the notification information to the user of the electronic device 101.

[0053] The device management application can control a power supply (e.g., turning on or off) or a function (e.g., adjustment of brightness, resolution, or focus) of the external electronic device in communication with the electronic device 101 or a portion of its components (e.g., a display module or a camera module of the external electronic device). The device management application can additionally or alternatively support installation, deletion, or update of an application running on the external electronic device.

[0054] The electronic device according to various example embodiments can be one of various types of electronic devices. The electronic devices can include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. According to example embodiments of the present disclosure, the electronic devices are not limited to those described above.

[0055] It should be understood that various example embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to particular embodiments but include various changes, equivalents, or replacements for a corresponding technical feature. The similar reference numerals can be used to refer to a similar or related element throughout the accompanying drawings. It is to be understood that a singular form of a noun corresponding to an item consists of one or more of the things, unless the relevant context clearly dictates otherwise. As used herein, each of "A or B," "at least one of A and B," "at least one of A or B," "at least one of A, B, or C," and "A, B, or C" can include all possible combinations of the items enumerated together in a corresponding one of the phrases. Terms such as "first" and "second" can be used to simply distinguish a relevant component from another, and does not limit the components relative to each other (e.g., a first component and a second component can be replaced with a third component). It is to be understood that if an element (for example, a first element) is referred to as being "coupled to" or "connected to" another element (for example, a second element), it can be directly coupled to or connected to the other element or be coupled to or connected to the other element via another element (for example, a third element).

[0056] As used in connection with various example embodiments of the present disclosure, the term "module" can include a unit implemented in hardware, software, or firmware, or a combination thereof, and can interchangeably be used with other terms, for example, "logic", "logic block", "part", or "circuitry". The module can be a single integral component, or a minimum unit or part thereof, adapted to perform one or more functions. For example, according to an example embodiment, the module can be implemented in a form of an application-specific integrated circuit (ASIC).

[0057] Various example embodiments set forth herein can be implemented as software (e.g., program 140) including one or more instructions that are stored in a storage medium (e.g., internal memory 136 or external memory 138) that is readable by a machine (e.g., electronic device 101). For example, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) can invoke at least one of the one or more instructions stored in the storage medium, and execute it. This allows the machine to be operated to perform at least one function according to the at least one instruction invoked. The one or more instructions can include a code generated by a complier or an executable code made by an interpreter. The machine-readable storage medium can be provided in the form of a non-transitory storage medium. Here, the "non-transitory" storage medium is a tangible device and can not include a signal (e.g., an electromagnetic wave), but the term does not differentiate between where data is semi-permanently stored in the storage medium and where the data is temporarily stored in the storage medium.

[0058] According to an embodiment, a method according to various example embodiments of the disclosure can be included and provided in a computer program product. The computer program product can be traded as a product between a seller and a buyer. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or be distributed online via an application store (e.g., Google Play Store, Apple App Store, or the like). If the computer program product is distributed online, at least part of the computer program product can be temporarily stored or temporarily generated in a machine-readable storage medium such as a memory of a manufacturer's server, a server of an application store, or a relay server. TM ) online (e.g., downloaded or uploaded), or distributed directly between two user devices (e.g., smart phones). If distributed online, at least part of the computer program product can be temporarily generated or at least temporarily stored in a machine-readable storage medium such as a memory of a manufacturer's server, a server of an application store, or a relay server.

[0059] According to various example embodiments, each of the above-described components (e.g., a module or a program) can include a single entity or a plurality of entities, and some of the plurality of entities can be separately configured in different components. According to various example embodiments, one or more of the above-described components can be omitted, or one or more other components can be added. Alternatively or additionally, a plurality of components (e.g., modules or programs) can be integrated into a single component. In this case, according to various example embodiments, the integrated component can still perform one or more functions of each of the plurality of components in the same or similar manner as before the integration. According to various example embodiments, operations performed by the module, the program, or another component can be carried out sequentially, in parallel, repeatedly, or heuristically, or one or more operations can be executed in a different order or omitted, or one or more other operations can be added.

[0060] Figure 3 is a flowchart illustrating an example method of using a shared machine learning model according to various embodiments.

[0061] In operation 310, a processor of an electronic device (e.g., the electronic device 101) according to example embodiments can map a target application and a machine learning model. For example, the electronic device can map the target application to a machine learning model that matches a request of the target application among a plurality of machine learning models. The machine learning model can be a pre-trained artificial intelligence model, and can be a model designed to output data in response to input data. The plurality of machine learning models can have various input data formats and output data formats. The data formats of the plurality of machine learning models will be described in greater detail below with reference to FIGS. 4A and 4B. Figure 1 Figure 4 The data formats of the plurality of machine learning models will be described in greater detail below with reference to FIGS. 4A and 4B.

[0062] ​The target application can indicate an application of a plurality of applications installed in the electronic device that needs an inference operation using a machine learning model. The target application can perform an inference operation of the target application using an internal machine learning model included in the target application, or can delegate the inference operation to a machine learning model outside the target application. For example, the electronic device can map the target application to a machine learning model to which an inference operation of the target application is to be delegated. After the machine learning model is mapped to the target application, the inference operation of the target application can be performed using the mapped machine learning model. In the following description, the machine learning model can be a machine learning model other than the target application.

[0063] The request of the target application can indicate a request to delegate an inference operation of the target application. For example, the request of the target application can include an identifier (e.g., a model name) of the target application among a plurality of machine learning models. The target application can transmit a request to select a machine learning model matching an input / output data format (e.g., a format and / or a type of input / output data) required for an inference operation of the target application and to map the target application to the selected machine learning model.

[0064] In operation 320, the electronic device can generate an inference result by applying the mapped machine learning model to the sensing data. For example, when one or more sensors sense sensing data corresponding to the machine learning model, the electronic device can generate an inference result by applying the mapped machine learning model to the sensing data. By inputting the sensing data to the machine learning model, the electronic device can generate output data as an inference result according to an inference operation. For example, the electronic device can apply an operation (e.g., a convolution operation, a pooling operation, and an operation to propagate input data to a plurality of artificial neural network layers) according to the machine learning model to the sensing data to generate an inference result.

[0065] The inference result can be a result obtained by inferring, estimating, predicting, identifying, verifying, and / or recognizing one task based on the sensing data. For example, the task can include object recognition, speech recognition, and estimation of a device danger level. The inference result of one task can include an estimated label and / or an output value of the task. In an example, the label included in the inference result of one task can indicate at least one of a plurality of classes according to a classification of the task. In another example, the output value included in the inference result of one task can be a probability value of each class and / or a regression value of a target of the task.

[0066] For example, when the sensing data is image data, the inference result can be an object detection result, an object classification result, and / or a facial expression recognition result. The object detection result can be a result obtained by detecting, extracting, and / or dividing a region corresponding to an object part of the image data. The object classification result can be a result obtained by classifying an object detected in the image data into an object label, and the object label can indicate an attribute of the object. In an example, the object classification result can be a result obtained by classifying the detected object into one of a cat, a dog, a person, and a vehicle. In another example, the object classification result can be a result obtained by classifying the detected object into one of a hand, an arm, a face, a leg, and other body parts of a person. The facial expression recognition result can be a result obtained by recognizing a human facial expression detected in the image data as a label corresponding to one of a plurality of emotions (e.g., anger, happiness, sadness, and neutral).

[0067] For example, when the sensing data is sound data, the inference result can be a dangerous sound detection result, a sound recognition result, and / or a speech recognition result. The dangerous sound detection result can be a result obtained by detecting a sound associated with a danger approaching the user (e.g., a sound of an approaching vehicle). The sound recognition result can be a result obtained by classifying a sound included in the sensed sound data into a label, and the sound label can represent an attribute of the sound. The sound recognition result can be, for example, a result obtained by classifying a detected sound source into one of a vehicle, a person, and an animal. The speech recognition result can be a result obtained by recognizing a human utterance and / or a conversation from the sound data, and can be, for example, a text extraction result and / or a voice command corresponding to the human utterance.

[0068] For example, when the sensing data is system data, the inference result can include a system danger detection result. The system data can include, for example, temperature data and current data. In an example, the system danger detection result can be a result indicating whether the electronic device is in danger (e.g., overcurrent). In another example, the system danger detection result can be a danger level of the electronic device (e.g., a possibility of a battery explosion).

[0069] However, the above-described inference results are merely examples provided to help understanding, and various embodiments are not limited thereto. Depending on the design for each of a multi-class classification task, a multi-label classification task, and / or a regression task, inference results of different formats can be generated.

[0070] At operation 330, the electronic device can deliver the inference result to the application. For example, the electronic device can transmit the generated inference result to at least one of the target application and another application mapped to the machine learning model. When a plurality of applications are mapped to the machine learning model, the electronic device can transmit the inference result to the plurality of applications whenever the inference result using the corresponding machine learning model is generated. However, the example embodiments are not limited thereto. When the inference result using the corresponding machine learning model is generated, the electronic device can select at least one application from among the plurality of applications, and can transmit the inference result to the at least one selected application. The machine learning model to which a plurality of applications are mapped can be a single model, but is not limited thereto, and can be a model in which a plurality of artificial intelligence models are combined. The same combination of machine learning models can be equally mapped to a plurality of applications.

[0071] The electronic device can share the machine learning model by executing various applications requiring the machine learning model to perform data collection and inference operations, thereby reducing resources. For example, the electronic device can detect sensing data in a state in which the application is deactivated, and can calculate an inference result by applying the machine learning model to the sensed sensing data. The electronic device can prevent and / or reduce unnecessary operations and use of random access memory (RAM) by performing the inference operation in a state in which the application is deactivated.

[0072] In the present disclosure, the application in the deactivated state and the deactivated application can be an application in a state in which a process of the corresponding application is stopped or not started, and can be, for example, an application in which CPU scheduling for the process of the corresponding application is not present or in which the process is removed from the memory (e.g., RAM). For example, the deactivated application can be an application in which, even though the process is started after booting of the electronic device, the process is not started or is stopped, and the process can be in a stopped state or a killed state. The application in the activated state and the activated application can be an application in which the process of the corresponding application is being executed in the memory (e.g., RAM), and can be, for example, an application operating in a foreground process and / or a background process.

[0073] Further, by mapping the same machine learning model to a plurality of applications and using the machine learning model, the electronic device can minimize and / or reduce a space occupied by the machine learning model in the memory. For example, the electronic device can prevent and / or reduce redundant processing, use the same machine learning model for two or more applications having the same input data and output data required for inference, and can minimize and / or reduce the use of RAM to prevent and / or reduce a lagging problem. By preventing and / or reducing the redundant processing, an increase in a consumption current and an increase in a temperature of the electronic device can be suppressed. Because the competing use of the accelerator by the plurality of applications is prevented and / or reduced, the computing resources can be equally allocated to the plurality of applications.

[0074] The electronic device can share the machine learning model for a plurality of applications requiring the same inference requirement to save device processing of a CPU, a GPU, and / or a digital signal processor (DSP), and to improve an operation speed. Further, even if it is not possible to execute the applications in the foreground and / or the background due to a lack of a remaining capacity of the RAM, the electronic device can perform an inference operation required for the corresponding applications regardless of execution of all the applications.

[0075] Figure 4 is a block diagram illustrating an example configuration of a machine learning model sharer according to various embodiments.

[0076] The machine learning model sharer 410 can include various processing circuitry and / or executable program instructions, and manage sharing of a plurality of machine learning models that can be mapped to a plurality of applications (e.g., a first application 401, a second application 402, and an n-th application 409, where "n" is an integer greater than or equal to "2") requiring an inference operation. The machine learning model sharer 410 can be implemented as a software module, a hardware module, or a combination of the software module and the hardware module. The machine learning model sharer 410 can include a model register 411, a data collector 412, a model selector 413, an inference delegator 414, an application controller 415, and a model storage 420. Each module included in the machine learning model sharer 410 can be implemented as a software module, a hardware module, or a combination of the software module and the hardware module. At least some or all operations of the machine learning model sharer 410 and each module included in the machine learning model sharer 410 to be described below can be performed by a processor (e.g., the processor 120 of FIG. 1). Figure 1

[0077] The model register 411 can register a plurality of machine learning models installed in the electronic device (e.g., the electronic device 100 of FIG. 1) in the model storage 420. The model register 411 can register the plurality of machine learning models in the model storage 420 in response to a request of the data collector 412. The model register 411 can register the plurality of machine learning models in the model storage 420 in response to a request of the model selector 413. The model register 411 can register the plurality of machine learning models in the model storage 420 in response to a request of the inference delegator 414. The model register 411 can register the plurality of machine learning models in the model storage 420 in response to a request of the application controller 415. Figure 1 ​In the electronic device 101), the target application mapping for the inference operation in the first application 401 to the nth application 409 is mapped and registered to one of multiple machine learning models (e.g., the first machine learning model 421, the second machine learning model 422, and the mth machine learning model 429, where m is an integer greater than or equal to "1"). References will follow below. Figure 5 Describe in more detail the registration of the model delegated to the application.

[0078] Data collector 412 can collect data for inference operations. For example, data collector 412 can collect data sensed by sensor 440, data generated by electronic devices, and / or data generated by external devices. According to an example embodiment, data collector 412 can acquire data from camera 441 (e.g., ...). Figure 1 Image data captured by camera module 180. Data collector 412 can acquire image data captured by sound sensor 442 (e.g., ...). Figure 1 The input module 150 (microphone) captures sound data. The data collector 412 can acquire sound data captured by the temperature sensor 443 (e.g., Figure 1 The temperature data sensed by the sensor module 176. The data collector 412 can acquire the temperature data sensed by the current sensor (e.g., current sensor module 176). Figure 1 The sensor module 176 senses the current data. The data collector 412 can collect the sensed data through the hardware abstraction layer (HAL) 430. According to the example embodiment, the data collector 412 can also collect image data pre-stored in the electronic device, user-written files (e.g., documents), and / or content received from external devices.

[0079] Data collector 412 can collect the data required for applications mapped to multiple machine learning models. For example, data collector 412 can selectively extract and collect the sensing data required for the machine learning model to which the application is mapped from sensing data monitored by sensor 440. Furthermore, data collector 412 can collect data according to the data type and collection period specified for the machine learning model included in model memory 420. For example, when "k" applications are mapped to the same machine learning model, the processing resources required for data collection using the inference operation of the machine learning model can be reduced by 100 / k%. In this example, "k" can be an integer greater than or equal to "2".

[0080] The model selector 413 can process data transmission between the sensor 440, the model storage 420, and the application. In an example, the model selector 413 can select a machine learning model corresponding to the sensing data sensed by the sensor 440 from among a plurality of machine learning models based on the sensing data. Also, the model selector 413 can select an application to which an inference result based on the corresponding machine learning model is to be transmitted. In another example, the model selector 413 can select a machine learning model to be activated from among deactivated machine learning models based on the sensing data.

[0081] The inference delegator 414 can delegate an inference operation to a machine learning model registered in a target application. For example, the inference delegator 414 can generate an inference result for data collected by the data collector 412 using a machine learning model selected from among a plurality of machine learning models. The inference delegator 414 can perform an operation according to the selected machine learning model using a predefined accelerator 450 at compile time.

[0082] The application controller 415 can transmit an inference result to an application. In an example, the application controller 415 can transmit a generated inference result based on a selected machine learning model to a target application mapped to the selected machine learning model each time the inference result is generated. In another example, the application controller 415 can transmit an inference result to a target application mapped to a selected machine learning model when an output value included in the inference result based on the selected machine learning model exceeds a threshold range. In another example, the application controller 415 can activate a target application mapped to a machine learning model when the target application is deactivated.

[0083] The model storage 420 can include a plurality of applications, for example, a first machine learning model 421, a second machine learning model 422, and an m-th machine learning model 429. The plurality of machine learning models can be stored in a storage space different from a storage space in which a plurality of applications (for example, target applications) are stored. The model storage 420 can include a model describer 425 including information about an input / output data format defined for an inference operation of each machine learning model. The plurality of machine learning models can be stored in a memory space of an electronic device or a memory space of an external device (for example, a server).

[0084] The model describer 425 can include, for example, a model name of each machine learning model, an input type, an input source, an input format, an output format, whether to wake up an application, an output type, and an output label list. The model describer 425 of each machine learning model can include, for example, an input type, an input source, an input format, an output type, a label list, and an output format, as shown in Table 1 below.

[0085] [Table 1]

[0086]

[0087] In Table 1, the input type can be a data type used as an input, and can include, for example, an image, a sound, a temperature, and a current. The input source can be a data source, and can indicate a sensor module that senses data or an application that provides data. For example, "self" can be data pre-processed, processed, and / or provided in an application, and can indicate an input source of, for example, an image pre-stored in a photo album. Also, "camera" can indicate an input source of image data captured by the camera 441, and "microphone" can indicate an input source of sound data captured by a microphone. Also, "temperature" can indicate an input source of temperature data sensed by a temperature sensor 443, and can describe a surface temperature predicted based on a sensed temperature of different parts of the electronic device. Also, "current" can represent an input source of current data sensed by a current sensor, and can be measured by, for example, a power meter. "Illuminance" can represent an input source of illuminance data sensed by an illuminance sensor, and "screen touch" can represent an input source of touch data sensed by a touch screen. Also, "grabbing sensor" can indicate an input source of grabbing data sensed by a grabbing sensor. The input format can represent a vector dimension of input data. For example, (1, 224, 224, 3) can indicate a 1x224x224x3 dimensional input format. A single image captured by the camera 441 can be a vector dimension of input data, which indicates a width of "224" pixels, a height of "224" pixels, and a pixel value in each of three color channels (for example, red, green, and blue channels). The output type can be a type of output data, and can indicate whether an inference result is object detection, classification, and other recognition. The label list can be a label list classified by a task. For example, in the case of object classification, the label list can include a cat, a dog, a vehicle, and other object labels. In the case of facial expression recognition, the label list can include happy, sad, angry, and other emotion labels. Similar to the input format, the output format can indicate a vector dimension of output data. However, Table 1 is merely an example provided to help understanding, and the format of input / output data is not limited thereto.

[0088] In an example, in a descriptor for a machine learning model for image classification, the model name can be "img_classifier", the input type can be "image", the input source can be "self", the input format can be "(1, 224, 224, 3)", the output format can be "(1, 1000)", whether to wake up the application can be "no", the output type can be "classification", and the output label list can be "dog, cat, bear". In another example, in a descriptor for a machine learning model for checking temperature, the model name can be "temp_checker", the input type can be "etc", the input source can be "temp", the input format can be "(1, 5)", the output format can be "(1, 3)", whether to wake up the application can be "yes", the output type can be "warning", and the label list can be "hot, cold". In another example, in a descriptor for a machine learning model for checking spoken words, the model name can be "word_checker", the input type can be "sound", the input source can be "mic", the input format can be "(10, 300)", the output format can be "(1, 3)", whether to wake up the application can be "yes", the output type can be "warning", and the label list can be "hi, hello".

[0089] The plurality of applications can perform a subsequent operation based on the received inference result. The subsequent operation can include data processing based on the inference result. The target application can request fast sampling for fast data processing. For example, according to the request of the application, the data collection period can be reduced for fast sampling.

[0090] The accelerator 450 can be set for an inference operation of a machine learning model, and can include at least one of a CPU 451, a GPU 452, a DSP 453, and an NPU 454, or a combination of two or more thereof. The electronic device can generate an inference result by implementing a machine learning model on sensed data using the accelerator 450.

[0091] In the present disclosure, for convenience of description, an example in which the first application 401 and the second application 402 are mapped to the first machine learning model 421 is described, however, example embodiments are not limited thereto. The mapping between the application and the machine learning model can vary according to the application setting and / or the user setting.

[0092] Figure 5is a signal flow diagram illustrating an example operation of mapping and registering an application and a machine learning model according to various embodiments.

[0093] When the target application 501 (e.g., a first application 401 of FIG. 4) is installed and / or when the target application 501 is executed, the electronic device (e.g., the electronic device 101 of FIG. 1) can perform an operation of mapping and registering a machine learning model in response to a user input. Figure 4 Figure 1

[0094] At operation 511, the target application 501 can transmit a model list request. The model selector 502 (e.g., the model selector 413 of FIG. 4) can transfer the model list request of the target application 501 to the model storage 503 (e.g., the model storage 420 of FIG. 4). Figure 4 Figure 4

[0095] At operation 512, the target application 501 can receive a model list from the model storage 503 via the model selector 502. The model list can include a model descriptor (e.g., the model descriptor 425 of FIG. 4). The model descriptor can include a definition of an input / output data format of each machine learning model. Figure 4

[0096] At operation 513, the target application 501 can transmit a model mapping request. For example, the target application 501 can determine a machine learning model matching the target application 501 based on at least one or a combination of two or more of a type of input data required in an inference operation of the target application, a format of the input data, an input source, a type of output data, a label list of the output data, and a format of the output data, among a plurality of machine learning models. For example, the target application 501 can search for a machine learning model matching the input / output data format of the target application 501, and can determine the found machine learning model. The target application 501 can extract a machine learning model matching the input type required by the target application 501 from the model list. The target application 501 can extract a machine learning model matching the input format from the extracted machine learning model. The target application 501 can sequentially determine a machine learning model satisfying the requirements of the target application 501 based on the input type, the input format, the input source, the output type, the label list, and the output format. When a plurality of machine learning models satisfy the requirements of the target application 501, the target application 501 can select a machine learning model to be mapped to the target application 501 from the plurality of machine learning models in response to a user input. The target application 501 can transmit a model mapping request for mapping to the determined machine learning model.

[0097] ​​​​​However, example embodiments are not limited thereto. For example, when information (e.g., a model name) predefined to map to a machine learning model of the target application 501 is provided, the target application 501 can skip the model search and recognition process and can immediately transmit a request for matching with the predefined machine learning model.

[0098] At operation 514, the model selector 502 can transmit a model confirmation request for the determined machine learning model to the model storage 503.

[0099] At operation 515, the target application 501 can receive a mapping confirmation from the model storage 503 that the target application 501 is mapped to the determined machine learning model.

[0100] At operation 516, the target application 501 can transmit a wake-up setting request to the model selector 502. The wake-up can be an operation of activating a deactivated application. The target application 501 can set a wake-up condition of the target application 501. For example, the target application 501 can transmit a request for setting a reference tag and / or a threshold range as a wake-up criterion. As a reference, for a plurality of applications mapped to the same machine learning model, different threshold ranges and / or different reference tags can be set. According to circumstances, based on whether the threshold range is satisfied and / or whether the reference tag is satisfied, a plurality of applications can be simultaneously operated, or only some of the plurality of applications can be operated, which will be described below with reference to FIG. 6. Figure 7a

[0101] At operation 517, the target application 501 can receive a confirmation of the completion of the wake-up setting.

[0102] Through the wake-up setting, the electronic device can wake up the application in response to a defined situation and perform a designated task, rather than allowing the target application 501 to reside in the memory or continuously use the CPU. The electronic device can reduce the use of the RAM and the use of the CPU.

[0103] An inference period and a data sampling frequency can be additionally set. Additional settings including whether to wake up the application and the sampling frequency are described below with reference to Table 2.

[0104] [Table 2]

[0105] Item Given value Description Selection model 4 Number representing a machine learning model Sampling 5 Sampling / s (sampling frequency per second) Background True Whether to execute the application when it is applied in the background Wake up True Whether to activate the deactivated application Threshold Hot Threshold / reference tag as a wake-up criterion

[0106] As a reference, although data is exchanged between the model storage 503 and the target application 501 by the model selector 502 as shown in FIG. 5, example embodiments are not limited thereto. The processor of the electronic device can also perform the above-described operations. Figure 5

[0107] Figure 6 is a diagram illustrating a model storage 503 according to various embodiments. Figure 5 ​​A diagram showing an example of the mapping results.

[0108] According to an example embodiment, model register 611 (e.g., Figure 4 The model register 411) can store the model memory 620 (e.g., Figure 4 The machine learning model (model memory 420) is mapped and registered to multiple applications, for example, the first application 601 (e.g., Figure 4 First application 401), second application 602 (e.g., Figure 4 The second application 402) and the nth application 609 (e.g., Figure 4 The nth application 402). Model register 611 can hold the model selector (e.g., Figure 4 The model selector 413 maps the machine learning model selected by the model descriptor 625 to the target application, as shown in the reference above. Figure 5 As stated above.

[0109] exist Figure 6 In the example, model registrar 611 can map the first application 601 and the second application 602 together to the first machine learning model 621 (e.g., Figure 4 The first machine learning model 421). Model register 611 can map the nth application 609 to the mth machine learning model 629 (e.g., Figure 4 The m-th machine learning model 429). The second machine learning model 622 (e.g., Figure 4 The second machine learning model (422) can also be mapped to another application. However, the example embodiments are not limited to this, and a portion of the machine learning model may not be mapped to an application.

[0110] The following text will describe in more detail examples of the operations of the first application 601 and the second application 602 mapped to the first machine learning model 621.

[0111] Figure 7a and Figure 7b This is a flowchart illustrating examples of inference operations using a shared machine learning model according to various embodiments.

[0112] Figure 7a The diagram illustrates the operation when at least one of multiple applications is set to wake up.

[0113] In operation 701a, electronic devices (e.g., Figure 1 The electronic device 101 can determine whether at least one of a plurality of applications is set to wake up. When no application is set to wake up among the plurality of applications mapped to the machine learning model, the electronic device can terminate wake-up-related operations. When an application is set to wake up, the electronic device can continue performing wake-up-related operations.Figure 7a In an embodiment, an example in which a first application (e.g., a first application 401 of FIG. 4A) and a second application (e.g., a second application 402 of FIG. 4A) are set to be woken up is shown. Figure 4 Figure 4

[0114] At operation 710, the electronic device can collect sensing data from the sensor 704 (e.g., the sensor 440 of FIG. 4A). Figure 4

[0115] When sensing data corresponding to a machine learning model is sensed, at operation 720, the electronic device can generate an inference result by applying the mapped machine learning model to the sensing data. For example, the electronic device can select a machine learning model corresponding to the sensing data from among a plurality of machine learning models included in the model storage 702 (e.g., the model storage 420 of FIG. 4A), and can implement the selected machine learning model using the accelerator 705 (e.g., the accelerator 450 of FIG. 4A). When the application mapped to the machine learning model is deactivated, the electronic device can generate an inference result using the machine learning model before the application is activated. Figure 4 Figure 4

[0116] At operation 730, the electronic device can determine whether an application set to be woken up is activated. For example, the electronic device can determine whether the application mapped to the machine learning model is in an activated state. When the application is activated, the electronic device can transmit the inference result to the application without waking up the application, for example, in operations 752 and 762. When the application is deactivated, the electronic device can determine whether to wake up the application based on the inference result determined in operation 740, which will be described below.

[0117] Referring to Table 2, an example in which a first application and a second application are mapped to a machine learning model is described. When the first application is activated, at operation 752, the electronic device can determine that the first application does not need to be woken up, and can deliver the inference result to the first application. Similarly, when the second application is activated, at operation 762, the electronic device can determine that the second application does not need to be woken up, and can transmit the inference result to the second application. The electronic device can provide the generated inference result to a target application running in one of a foreground process and a background process and to another application running in one of the foreground process and the background process. However, example embodiments are not limited thereto. As described in Table 2, when inference in a background process is disabled, the electronic device can restrict delivery of the inference result to an application running in the background process. Figure 7a

[0118] ​​​​​​At operation 740, the electronic device can determine an inference result to determine whether to activate the deactivated application. The electronic device can activate the corresponding application based on the inference result. According to an example embodiment, the electronic device can activate the deactivated application based on at least one of the estimated label and the output value included in the inference result. In Figure 7a In an example, the wake-up condition of the first application can be set to a threshold range of the output value, and the wake-up condition of the second application can be set to a reference label of the estimated label of the inference result. When the inference result does not satisfy the wake-up condition, at operation 710, the electronic device can continue to collect data.

[0119] At operation 751, the electronic device can wake up the first application based on the inference result. In an example, when the output value included in the inference result is outside the threshold range, the electronic device can activate the deactivated application. In another example, when the output value included in the inference result is within the threshold range, the electronic device can maintain the state of the deactivated application. At operation 752, the electronic device can transmit the inference result to the first application. For example, the sensing data can include temperature data and current data, and the inference result can include a possibility of a hazard occurrence and a hazard degree. When the hazard degree is outside the threshold range, the electronic device can perform an operation of managing temperature and current to reduce the hazard degree by executing the first application for power management.

[0120] At operation 761, the electronic device can wake up the second application based on the inference result. For example, when the estimated label included in the inference result matches the reference label registered in the deactivated application, the electronic device can activate the deactivated application. At operation 762, the electronic device can transmit the inference result to the activated second application. For example, the inference result can include a hazard degree and a possibility of a hazard occurrence estimated based on temperature and current. When the estimated label included in the inference result indicates a dangerous situation (e.g., a battery is about to explode), the electronic device can provide a warning related to temperature and current to the user by executing the second application for an emergency warning.

[0121] According to an example embodiment, after the deactivated application is activated, each time another inference result is generated, the electronic device can provide the other inference result to the corresponding application. The other inference result can be a subsequent inference result generated after the inference result for wake-up.

[0122] The electronic device can wake up either or both of the first and second applications, or can transmit the inference result to either or both of the first and second applications. The electronic device can perform the same data collection and inference operation on the first and second applications that require the same input data, while performing different functions, and thus can improve resource use efficiency.

[0123] Figure 7bThis shows an example of what happens when the application is not set to wake up.

[0124] In Operation 701b, the electronic device can determine whether at least one of multiple applications is activated. When all multiple applications mapped to the machine learning model are deactivated, the electronic device can terminate the inference-related operations. When at least one application is activated, the electronic device can continue performing the inference-related operations. Figure 7b The image shows an example of activating the first and second applications.

[0125] Figure 7b Operations 710, 720, 752, and 762 are the same as those mentioned above. Figure 7a The operations described are the same or similar, so their detailed description will not be repeated here. However, with Figure 7a The examples are different, in Figure 7b In this context, electronic devices can skip additional operations (e.g., Figure 7a Operations 730, 740, 751, and 761 can be performed, and the inference results generated in operation 720 can be transmitted to the first application and the second application in operations 752 and 762.

[0126] The above reference Figure 7a and Figure 7b The operations described are not limited to those described above, and may be performed sequentially, in parallel, repeatedly or heuristically, or one or more of the operations may be performed in a different order or omitted, or one or more other operations may be added.

[0127] Figure 8 This is a diagram illustrating examples of transmitting the inference results of a shared machine learning model to a first application and a second application, according to various embodiments.

[0128] Electronic devices (e.g., Figure 1 The electronic device 101 can be derived from the sensor 840 (e.g., Figure 4 The sensor 440 collects sensing data. The collected sensing data can be transmitted via HAL 830 (e.g., ...). Figure 4 The HAL 430 is transmitted to the model selector 811 (e.g., Figure 4 The model selector 413) and / or the inference delegate 814 (e.g., Figure 4 The inference delegate 414). The model selector 811 can select a first machine learning model 821 from multiple machine learning models based on sensed data (e.g., Figure 4 The first machine learning model 421), the plurality of machine learning models, such as the first machine learning model 821, the second machine learning model 822 (e.g., Figure 4a second machine learning model 422) and an m-th machine learning model 829 (e.g., Figure 4 a model storage 820 (e.g., Figure 4 a model storage 420) of the electronic device 800. The inference delegator 814 can generate an inference result of the sensing data by implementing the selected first machine learning model 821 using an accelerator 850 (e.g., Figure 4 an accelerator 450) of the electronic device 800. The accelerator 850 can include at least one of a CPU 851, a GPU 852, a DSP 853, and an NPU 854, or a combination of two or more thereof. The application controller 815 (e.g., Figure 4 an application controller 415) of the electronic device 800 can transmit the inference result to the first application 801 (e.g., Figure 4 a first application 401) and the second application 802 (e.g., Figure 4 a second application 402). Hereinafter, an operation example of each of a sound recognition task, a heat recognition task, and an image recognition task will be described.

[0129] According to an example embodiment, the electronic device can sense sound data obtained by capturing a sound generated outside the electronic device using a sound sensor 842 (e.g., Figure 4 a sound sensor 442) of a sensor 840 (e.g.,

[0130] In an example, a popular word (e.g., "high, AAA") can be set as a reference tag of the first application 801, and the first machine learning model 821 can be mapped to the first application 801. When sound data is collected, the model selector 811 can select the first machine learning model 821, and the inference delegator 814 can generate an inference result of the sound data using the selected first machine learning model 821. When the inference result of the sound data matches the popular word set for the first application 801, the application controller 815 can activate the first application 801. Also, when another popular word (e.g., "high, BBB") is set for the second application 802, and when the inference result matches the corresponding popular word, the application controller 815 can activate the second application 802.

[0131] In another example, a first attribute of a sound (e.g., an attribute indicating a music sound) can be set as a reference tag of a first application 801, and a second attribute (e.g., an attribute indicating a sound of an approaching vehicle or a conversation sound) can be set as a reference tag of a second application 802. For example, the first application 801 can be a music playback application, and the second application 802 can be an application for warning of an object or a person approaching. In an example, when a music sound is identified as an inference result of sound data, the application controller 815 can activate the first application 801. In another example, when a dangerous sound or a conversation sound is identified as an inference result of sound data, the application controller 815 can activate the second application 802. In another example, when an inference result generated while the first application 801 is being executed is a second attribute, the application controller 815 can activate the second application 802.

[0132] In another example, the first application 801 can be an application for translating a first language into a second language, and the second application 802 can be an application for recording a speech and / or a conversation as text. When activated, the first application 801 and the second application 802 can operate in a foreground process and / or a background process. In addition, the application controller 815 can simultaneously transmit content of a user speech inferred from sound data based on the first machine learning model 821 to the first application 801 and the second application 802.

[0133] According to an example embodiment, the electronic device can sense temperature data and / or current data of the electronic device as sensing data through a temperature sensor 843 (e.g., the temperature sensor 443 of FIG. 4) and / or a current sensor of the sensor 840. When a dangerous recognition result inferred from the temperature data and / or the current data based on the machine learning model indicates a danger, the electronic device can activate a target application. Figure 4

[0134] In an example, a normal range can be set as a threshold range of the first application 801 and the second application 802. The first machine learning model 821 can output a danger level estimated from the temperature data and / or the current data. For example, when the danger level of the output value as an inference result exceeds the normal range, the application controller 815 can execute the first application 801 that controls a current of the electronic device and the second application 802 that warns of heating of the electronic device. Subsequently, the application controller 815 can transmit the inferred danger level to the first application 801 and / or the second application 802.

[0135] ​In another example, the first machine learning model 821 can be set to output a possibility of a hazard occurrence based on temperature data and / or current data. In this example, when the inference result matches a 'hazard' label, the application controller 815 can activate the first application 801 and / or the second application 802, and / or can transmit the inference result to the first application 801 and / or the second application 802.

[0136] According to an example embodiment, the electronic device can sense image data obtained by capturing a scene outside the electronic device using a camera 841 (e.g., a camera 441) of the sensor 840 as sensing data. The electronic device can activate a target application based on an image recognition result inferred from the image data based on a machine learning model. When the image recognition result inferred from the image data based on the machine learning model matches a predetermined object and / or a predetermined action, the electronic device can transmit the image recognition result to the target application. The action can be, for example, movement and action of an object or a person appearing in the image data. Figure 4 Figure 4

[0137] In an example, the first application 801 can be a food recognition application, and the second application 802 can be a shopping support application. The camera application can be executed separately from the first application 801 and the second application 802, and the electronic device can generate an inference result from image data of the camera application using the first machine learning model 821 of the model storage 820. The camera application can operate in a background process, however, the example embodiments are not limited thereto. When a food label (e.g., a hamburger) and a tableware label (e.g., a fork) are recognized from the image data based on the first machine learning model 821, the application controller 815 can deliver the food label by activating the first application 801, and can deliver the tableware label by activating the second application 802. The first application 801 can record food calories ingested by a user based on the food label, and the second application 802 can recommend product purchase for the user based on the tableware label.

[0138] In another example, the first application 801 can be an application for guiding a driving route of a vehicle, and can use the camera 841, and the second application 802 can be an application for providing a collision warning. When an object having a collision danger is inferred from collected image data based on the first machine learning model 821 while the first application 801 is operating, the application controller 815 can activate the second application 802, and can provide the same inference result to the first application 801 and the second application 802.

[0139] ​As described above, the electronic device can reduce system resource usage by monitoring the same data of a plurality of applications without redundancy as a whole. When an application is in a deactivated state, the electronic device can perform an inference operation in advance, thereby preventing and / or reducing CPU processing unnecessarily resident in the RAM. The electronic device can minimize and / or reduce storage space by sharing a machine learning model for a plurality of applications. If a common machine learning model is used in the development of a new application, the development period can be shortened because there is no need to retrain and install the machine learning model.

[0140] While the disclosure has been illustrated and described with reference to various example embodiments, it will be understood by those skilled in the art that various changes in form and details can be made therein without departing from the true spirit and full scope of the disclosure, including the appended claims and their equivalents. It should also be understood that any embodiment described herein can be used in combination with any other embodiment described herein.

Claims

1. An electronic device, comprising: processor; The memory is configured to store multiple machine learning models and is electrically connected to the processor; as well as At least one sensor is configured to sense sensing data. The processor is configured as follows: Map the target application to a machine learning model among the plurality of machine learning models that matches the request of the target application; In response to the at least one sensor sensing sensing data corresponding to the machine learning model, an inference result is generated by applying the mapped machine learning model to the sensing data; and The generated inference results are transmitted to the target application and another application mapped to the machine learning model. The processor is further configured as follows: The inference result is generated using the machine learning model based on the deactivation of the application mapped to the machine learning model before the corresponding application is activated; and The corresponding application is activated based on the inference result. The processor is further configured as follows: Whenever another inference result is generated based on the activation of the corresponding application, the other inference result is provided to the corresponding application.

2. The electronic device according to claim 1, wherein, The processor is also configured to provide the generated inference results to a target application running in one of the foreground and background processes and another application running in one of the foreground and background processes.

3. The electronic device according to claim 1, wherein, The processor is also configured to activate the deactivated application based on at least one of the estimated labels and output values ​​included in the inference results.

4. The electronic device according to claim 1, wherein, The processor is also configured to: In response to an output value included in the inference result being outside a threshold range, the deactivated application is activated and the inference result is transmitted to the target application; and In response to the output value included in the inference result being within the threshold range, the state of the deactivated application is maintained.

5. The electronic device according to claim 1, wherein, The processor is also configured to activate the deactivated application in response to an estimated label included in the inference result matching a reference label registered in the deactivated application.

6. The electronic device according to claim 1, wherein, The processor is further configured to determine a machine learning model matching the target application from the plurality of machine learning models based on at least one or a combination of two or more of the following: the type of input data required in the inference operation of the target application, the format of the input data, the input source, the type of output data, the label list of the output data, and the format of the output data.

7. The electronic device according to claim 1, wherein: The processor is also configured to generate the inference result by implementing the machine learning model on the sensed data using an accelerator, and The accelerator includes at least one or a combination of two or more of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and a neural processing unit (NPU), and The multiple machine learning models are stored in a different storage space than the storage space where the target application is stored.

8. A method implemented by a processor, the method comprising: Map the target application to multiple machine learning models that match the requests of the target application; In response to at least one sensor sensing sensing data corresponding to a machine learning model, an inference result is generated by applying the mapped machine learning model to the sensing data; as well as The generated inference results are transmitted to the target application and another application mapped to the machine learning model. in, The generation of the inference result includes deactivating the application mapped to the machine learning model, generating the inference result using the machine learning model before the corresponding application is activated, and... The transmission of the generated inference results includes activating the corresponding application based on the inference results. The transmission of the generated inference results includes providing the other inference result to the corresponding application whenever another inference result is generated based on the activation of the corresponding application.

9. The method according to claim 8, wherein, The delivery of the generated inference results includes providing the generated inference results to a target application running in one of the foreground and background processes, as well as to another application running in one of the foreground and background processes.

10. The method according to claim 8, wherein, The transmission of the generated inference results includes: In response to an output value included in the inference result being outside a threshold range, the deactivated application is activated and the inference result is transmitted to the target application; In response to the output value included in the inference result being within the threshold range, the deactivated application state is maintained. The transmission of the generated inference results includes: activating the deactivated application in response to a match between the estimated label included in the inference results and a reference label registered in the deactivated application.

11. The method according to claim 8, wherein, The mapping from the target application to the machine learning model includes: determining a machine learning model matching the target application from a plurality of machine learning models based on at least one or a combination of two or more of the following: the type of input data required in the inference operation of the target application, the format of the input data, the input source, the type of output data, the label list of the output data, and the format of the output data. The generation of the inference result includes implementing the machine learning model on the sensed data using an accelerator to generate the inference result, and The accelerator includes at least one or a combination of two or more of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and a neural processing unit (NPU).

12. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 8-11.

Images