Edge device-based service providing method using code streaming, and system thereof

Abstract

Disclosed is a service providing method using code streaming, which is performed by an edge device-based service providing system. According to an embodiment, the service providing method may comprise the steps of: on the basis of a selection of a user with respect to a first service, identifying a motion recognition code module for providing the first service to the user; recognizing a motion of the user by executing the identified motion recognition code module on the basis of a motion image of the user input to a camera of the service providing system; requesting a service code module, which is operated by a code execution device of the service providing system, from a service providing server to provide the first service; generating real-time service data for the first service on the basis of the recognized motion of the user according to the streaming reception of the service code module; and outputting the generated service data through a service output device of the service providing system.

Description

Method and system for providing edge device-based services using code streaming

[0001] The present invention relates to a method for providing edge device-based services. More specifically, it relates to a service providing method of an edge device-based service providing system utilizing code streaming.

[0002] A variety of services are being provided through edge devices. To achieve this, specific service-related applications are typically stored entirely as code in the edge device's internal memory. The edge device then executes the code via its processor, running the application and providing the service.

[0003] However, edge devices have relatively few resources compared to cloud computing servers, limiting the expansion of services offered. Furthermore, if a service requires modification, the code must be updated and reloaded onto the edge device, creating the hassle of reconfiguring the device. To overcome this issue, a method has been proposed that migrates services to an edge computing environment, which is more advantageous for service provision and resource management when multiple service accesses occur.

[0004] However, the above-mentioned proposed service transfer method is also effective only in situations where continuous resource expansion is possible, and does not provide a solution to the fundamental problem of having to store all related codes for service provision in situations where resource expansion of the edge device itself is impossible.

[0005] The technical problem to be solved by the present invention to solve the above problem is to provide a method and system for receiving necessary codes through streaming in connection with service provision without storing all codes in an edge device.

[0006] Specifically, the technical problem to be solved by the present invention is to provide a method and system for determining whether to provide a service based on existing code loaded on an edge device or to provide a service by combining code provided in a streaming manner from a service providing server, depending on required service specifications such as the type of user-selected service and the level of service to be provided.

[0007] In addition, another technical problem that the present invention seeks to solve is to provide a method and system for combining a code module provided in a streaming manner with some code modules of the existing code.

[0008] Specifically, another technical problem that the present invention seeks to solve is to provide a method and system for providing a service by executing a code module provided based on streaming in real time.

[0009] Another technical problem that the present invention seeks to solve is to provide a system in which, in addition to the loaded code, an expandable hardware module for executing a streamed code module is applied.

[0010] The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art of the present invention from the description below.

[0011] A service providing method using code streaming, performed by an edge device-based service providing system according to one embodiment of the present invention for solving the above problem, may include a step of identifying a motion recognition code module for providing a first service to a user based on a user's selection of the first service, a step of recognizing the motion of the user by executing the identified motion recognition code module based on a motion image of the user input to a camera of the service providing system, a step of requesting a service providing server for a service code module operated in a code execution device of the service providing system to provide the first service, a step of generating real-time service data for the first service based on the recognized motion of the user upon streaming reception of the service code module, and a step of outputting the generated service data through a service output device of the service providing system.

[0012] In one embodiment, the step of requesting a service code module for providing the first service may include the step of determining whether a basic code module matching the first service, which is stored in advance in the service providing system, exists, and if the basic code module exists, the step of executing the basic code module to generate first service data for providing the first service.

[0013] In one embodiment, the step of generating the real-time service data may include the step of receiving the service code module by streaming, executing the received service code module to generate second service data for the first service, and the step of generating the real-time service data based on the first service data and the second service data.

[0014] In one embodiment, the streamed service code module includes an Application Programming Interface (API) for communicating with the identified motion recognition code module, and the step of generating the real-time service data may include a step of generating the real-time service data by executing the service code module based on the recognized motion of the user for the first service through the API.

[0015] In one embodiment, the step of identifying the motion recognition code module may include the step of determining whether the identified motion recognition code module satisfies a recognition level for providing the first service, and, if the identified motion recognition code module does not satisfy the recognition level as a result of the determination, the step of requesting an additional code module of the motion recognition code module for providing the first service from the service providing server, and the step of executing the motion recognition code module and the additional code module in conjunction with each other as the additional code module is received by streaming, and the step of recognizing the user's motion may include the step of recognizing the user's motion as the motion recognition module and the additional code module are executed in conjunction with each other.

[0016] In one embodiment, the additional code module includes a first API for communicating with the identified motion recognition code module, and the step of executing in conjunction may include a step in which the motion recognition code module and the additional code module communicate with each other and execute in response to the input motion image of the user.

[0017] In one embodiment, the additional code module may include a second API for communicating with the streamed service code module, and the step of generating the real-time service data may include the steps of transmitting the user's motion to the streamed service code module through the second API, analyzing the transmitted user's motion by executing the streamed service code module, and generating the real-time service data for the first service based on the analyzed result.

[0018] In one embodiment, the code execution device further includes a step of identifying that at least one computing device is added through a hardware expansion unit of the service providing system, and the step of linking and executing the motion recognition code module and the additional code module may include a step of executing the motion recognition code module through the code execution device and a step of executing the additional code module based on the first API using the at least one added computing device.

[0019] In one embodiment, the at least one processing device includes a first processing device and a second processing device, and the step of executing the additional code module based on the first API using the at least one added processing device may include the step of executing the additional code module using the first processing device, and the step of generating real-time service data for the first service may include the step of executing the streamed service code module using the second processing device.

[0020] In one embodiment, the step of outputting the generated service data may include the steps of identifying a UI (User Interface) output code module for outputting the generated service data, determining whether the identified UI output code module satisfies an output level for providing the first service, and, if the identified UI output code module does not satisfy the output level as a result of the determination, requesting an additional code module of the UI output code module for providing the first service from the service providing server, and, upon streaming reception of the additional code module, executing the UI output code module and the additional code module in conjunction with each other, and outputting the service data generated for the first service by executing the additional code module and the UI output code module in conjunction with each other.

[0021] In one embodiment, the method may further include a step of receiving a selection of the user for the second service, a step of identifying a motion recognition code module for providing the second service to the user, a step of recognizing a motion of the user for providing the second service by executing the identified motion recognition code module based on an image of the user input to the camera, a step of requesting a service providing server for a service code module for providing the second service, a step of removing a service code module stream-received for providing the first service as the service code module for providing the second service is stream-received, a step of generating real-time service data for the second service based on the recognized motion of the user, and a step of outputting the service data generated for the second service through a service output device of the service providing system.

[0022] In one embodiment, the step of identifying a motion recognition code module for providing the second service may include, in addition to the motion recognition code module, if there is an additional code module received for providing the first service, a step of determining whether the additional code module satisfies a recognition level for providing the second service, and a step of determining whether to delete the additional code module based on whether the additional code module satisfies the recognition level for providing the second service as a result of the determination.

[0023] In one embodiment, the step of determining whether the identified motion recognition code module satisfies a recognition level for providing the first service may include the steps of receiving an update message of the first service from the service providing server and determining whether the recognition level is satisfied based on the received update message, wherein the step of receiving the update message of the first service may include the step of generating a request message for an updated service code module in response to the update message, and the step of requesting an additional code module may include the step of transmitting a request message for the generated updated service code module together with a request message for the additional code module to the service providing server.

[0024] In one embodiment, the code execution device may further include a step of identifying that a computing device is added through a hardware expansion part of the service providing system, and a step of executing at least one of the additional code module and the updated service code module on the added computing device as the additional code module and the updated service code module are streamed.

[0025] According to an embodiment of the present invention for solving the above problem, a code streaming system includes a service providing server storing at least one service code module, and a service providing system requesting code streaming for a service selected by a user to the service providing server, and executing the service code module as the service code module for the selected service is streamed from the service providing server in response to the request, wherein the service providing system may include a camera for receiving a motion image of the user, a code execution device for recognizing the motion image of the user by executing a motion recognition code module, and generating service data by executing the service code module based on the recognized motion image, and a service output device for outputting the service data by executing a UI output code module based on the service data.

[0026] A hardware-extendable edge device-based service providing system according to one embodiment of the present invention for solving the above problem can request code streaming for a user's selected service from a service providing server storing at least one service code module, and execute the service code module as the service code module for the selected service is streamed from the service providing server in response to the request.

[0027] In one embodiment, the service providing system may include a camera that receives a motion image of the user and executes a motion recognition code module to recognize the motion image of the user, and based on the recognized motion image, executes the service code module to generate service data, and includes a code execution device that includes a hardware extension unit for parallel operation for at least one of the motion recognition code module and the service code module, and a service output device that outputs the service data by executing a UI output code module based on the service data.

[0028] In one embodiment, the constant code execution device includes a control unit that stores the motion recognition code module in advance and identifies the stored motion recognition code module, and the streamed service code module includes an API (Application Programming Interface) for communicating with the identified motion recognition code module, and the control unit can generate the service data by executing the service code module based on the recognized motion image of the user for the selected service through the API.

[0029] In one embodiment, the control unit can execute one of the motion recognition code module and the streamed service code module upon identifying that at least one computational unit is added to the hardware expansion unit, and control the execution of the remaining code modules other than the one of the computational units through the computational unit.

[0030] In one embodiment, the control unit may request an additional module from the service providing server when at least one of the recognition level of the motion recognition code module and the output level of the UI output code module does not satisfy the required level of the selected service.

[0031] In one embodiment, the additional module includes an API for communicating with at least one of the pre-stored motion recognition code module and the UI output code module, and the control unit can, when the requested additional module is streamed from the service providing server, link and execute the additional module with the at least one module to satisfy the required level of the selected service through the API.

[0032] In one embodiment, the control unit executes the at least one module and the service code module, and upon identifying that at least one processing unit is added to the hardware expansion unit, executes some of the modules among the at least one module, the motion recognition code module, the service code module, and the UI output code module, and controls the remaining code modules other than the some of the modules to be executed through the processing unit.

[0033] According to one embodiment of the present invention, by providing service codes in a streaming manner, the edge device can receive services simply by inserting the service call code into its internal memory. This allows for the freeing up of limitations on the size of the edge device's internal memory, enabling the provision of various services in real time.

[0034] According to one embodiment of the present invention, when a service modification occurs, the service call code of an edge device does not need to be replaced. Instead, the service code can be replaced by the service provider server and the replaced code can be streamed, thereby enabling application to all edge devices. This eliminates the inconvenience of building code for service modifications and then reloading the newly built code for each edge device.

[0035] In addition, by utilizing the code streaming method according to an embodiment of the present invention, when an API for the service main code is disclosed, a service code of a third party other than the service provider is streamed to an edge device through a service providing server, so that various services of the third party can be easily provided to the user.

[0036] According to another embodiment of the present invention, since hardware resources can be expanded at the edge device level, there is an advantage in that resource acquisition and operation can be optimized, delay can be minimized, and services can be provided in real time, compared to the method of expanding server-side resources.

[0037] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0038] FIG. 1 is an exemplary diagram of a code streaming system using an edge device according to one embodiment of the present invention.

[0039] FIG. 2 is a block diagram of a code execution device according to another embodiment of the present invention.

[0040] FIG. 3 is an example of code streaming execution on an edge device according to another embodiment of the present invention.

[0041] FIG. 3 is a block diagram of a code execution device according to another embodiment of the present invention.

[0042] Figure 4 is a flowchart of a service providing method according to another embodiment of the present invention.

[0043] FIGS. 5 and 6 are examples of code streaming target code modules referenced in some embodiments of the present invention.

[0044] FIG. 7 is a flowchart of a method for determining a streaming target based on required resources for each selected service, referenced in another embodiment of the present invention.

[0045] FIG. 8 is an example of an embodiment in which a streamed code unit is changed for each type of specific service, as referenced in some embodiments of the present invention.

[0046] FIG. 9 is an example of an extensible hardware module for executing a streamed code module, referenced in another embodiment of the present invention.

[0047] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings. The advantages and features of the present invention, and methods for achieving them, will become clear with reference to the embodiments described in detail below together with the attached drawings. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various different forms. These embodiments are provided only to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Like reference numerals designate like elements throughout the specification.

[0048] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in a sense commonly understood by those of ordinary skill in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise. The terminology used herein is for the purpose of describing embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms, unless specifically stated otherwise.

[0049] Hereinafter, in this specification, the edge device-based service provision system using code streaming may be abbreviated as a service provision system or system.

[0050] In this specification, descriptions such as "module," "unit," and "part" may refer to units of software and / or hardware, for example, a code module may mean a bundle of code that performs a specific function related to a service selected by a user of the present invention.

[0051] A hardware module may be a hardware resource that exists as a processor-specific unit to perform operations of a specific function related to the service, for example.

[0052] The above module may not only exist as a software module or a hardware module, but may also mean a unit in which specific software and hardware are combined.

[0053] Hereinafter, a service according to an embodiment of the present invention is a training service that analyzes and evaluates the user's motion in real time by analyzing the user's motion through an artificial intelligence-based real-time skeleton, guiding the user's motion according to the type of exercise selected or programmed by the user, and analyzing and evaluating the user's motion in real time. For example, the service may be a home workout service including PT (Personal Training), CrossFit, yoga, Pilates, gymnastics, dance, and meditation.

[0054] In addition, the above service can manage exercise courses, exercise volume, and / or mental and physical stability through analysis and evaluation of the user's motions, but is not limited to the types described above.

[0055] FIG. 1 is an exemplary diagram of a code streaming system using an edge device according to one embodiment of the present invention.

[0056] Referring to FIG. 1, the code streaming system includes a service providing system (10) and a service providing server (20).

[0057] The service provision system (10) and the service provision server (20) are computing devices that perform data communication with each other. According to an embodiment of the present invention, the service provision system (10) may provide the user's service selection information to the service provision server (20) and request a service code module. The service provision server (20) may stream the code module in response to the request from the service provision system (10).

[0058] In FIG. 1, the service providing system (10) and the service providing server (20) are represented as separate components, but according to another embodiment of the present invention, the service providing system (10) may also include the service providing server (20).

[0059] The service providing system (10) may include at least one edge device. For example, the service providing system (10) may include a code execution device (100), a camera (101), and a service output device (102). Although not shown, the service providing system (10) may further include an IoT exercise device and may additionally include an audio device.

[0060] The components of the above service provision system (10) can communicate with each other via a wired or wireless network. In particular, the code execution device (100) is an embedded edge device that provides a service according to an embodiment of the present invention and can comprehensively control the functions and operations of the components of the service provision system (10).

[0061] Additionally, although each component of the service provision system (10) is expressed separately, it may be configured as a single integrated device.

[0062] When a user selects a service, the code execution device (100) can identify a pre-stored motion recognition code module to provide the selected service based on the selected service.

[0063] For example, the types of services available for selection may be any one of PT, yoga, rehabilitation exercise, diet exercise, and dance training, and the selection of services may include not only the types of services, but also the selection of courses and levels (advanced, intermediate, and lower) for each service.

[0064] The above motion recognition code module is executed by the code execution device (100) to identify the user's motion, and in particular, the user's motion is identified based on the content of the selected service.

[0065] That is, according to one embodiment, the code execution device (100) can identify and execute different motion recognition code modules depending on the selected service.

[0066] According to another embodiment, the code execution device (100) may omit execution of at least some code of the motion recognition code module depending on the selected service.

[0067] The content of the above-described service may include motion guide displays, motion tracking scores, exercise amount information, course achievement information, sound and graphic effects, etc. that are applied differently depending on the type, course and level of the selected service.

[0068] For example, for rehabilitation movements, accurate three-dimensional movement identification is crucial. For dance training, the speed of movement identification may be crucial. For diet programs, measuring exercise volume may be a priority. In other words, depending on the selected service, the requirements for analysis results resulting from the execution of the motion recognition code module will vary.

[0069] When a user's motion image is input into the camera (101), the code execution device (100) can recognize the user's motion by executing the identified motion recognition code module based on the selected service.

[0070] For example, the code execution device (100) can recognize the user's motion by analyzing the artificial intelligence-based skeleton, as illustrated in FIG. 1.

[0071] Based on the user's motion recognized as described above, the code execution device (100) can execute the service code module to generate service content.

[0072] Although not shown, the content of the service may include, for example, tutor actions to guide users through motions appropriate for the service type, course, and level. The tutor actions may be displayed in the first area of ​​the service output device (102), depending on the service settings. The tutor may be a virtual character or an actual trainer teaching the service.

[0073] For example, if the type of service is dance training, the user can watch the tutor's movements displayed in the first area and take a dance motion in real time, and the user's dance motion can be input through the camera (101) and displayed in the second area of ​​the service output device (102).

[0074] At this time, the user's dance motion may be skeleton analyzed as the motion code module is executed by the code execution device (100), and may be displayed in combination with dance training content as the service code module is executed.

[0075] The above service output device (102) can simultaneously display content including the tutor's dance movements and the user's dance motion in the first area and the second area, respectively, and can also be displayed in a combined form with dance training content.

[0076] According to one embodiment of the present invention, in order to provide a service selected by a user, the code execution device (100) may request a service code module to be operated from the service provision server (20). That is, the code execution device (100), as an edge device, may not store all codes due to storage capacity limitations, but may receive a corresponding service code module in real time by streaming from the service provision server (20) based on the type, course, and / or grade of the selected service.

[0077] As the service code module is streamed, the code execution device (100) can generate real-time service data for the selected service based on the recognized user's motion.

[0078] The code execution device (100) can output the generated service data through the service output device (102), and in Fig. 1, the service data screen (103) of the service output device (102) is exemplified. The service data may be data in which the recognized user motion and the content of the service described above are combined.

[0079] FIG. 2 is a block diagram of a code execution device (100) according to another embodiment of the present invention.

[0080] According to one embodiment, the code execution device (100) may include an input unit (110), a communication unit (120), a hardware expansion unit (130), and a control unit (140).

[0081] For example, the input unit (110) may include at least one of various types of devices that allow a user to input data into the code execution device (100), such as a keyboard, mouse, touch screen, and dedicated remote controller.

[0082] In one embodiment, the user can use the input unit (110) to select the type of service, the service course, the service level, and / or the IoT device to be used for the service.

[0083] The communication unit (120) supports wired and wireless communication with the code execution device (100), the camera (101), the service output device (102), and / or the service provision server (20). For example, the communication unit (120) may support various communication methods, such as short-range communication and wired connection, in addition to the public communication network, the Internet. To this end, the communication unit (120) may be configured to include at least one of a communication module and a connection terminal well known in the technical field of the present invention.

[0084] The hardware expansion unit (130) provides an interface for attaching or detaching one or more computational units including a processor, and can distribute the computational power of the code execution unit (100).

[0085] For example, each computing device can operate like a computer. For example, if a hardware expansion unit (130) is equipped with a single computing device, code modules can all be executed within that single computing device.

[0086] As another example, even if it is a code module for executing a single function, for example, a motion recognition code module, some of the code may be processed in the code execution unit, and the remaining code may be processed in the computation unit.

[0087] On the other hand, if two or more computational units are mounted on the hardware expansion unit (130), one code module can be distributed and processed in parallel on two computational units.

[0088] In another example, when two or more computational units are mounted on a hardware expansion unit (130), one of the plurality of code modules may be assigned an execution path to the first computational unit and executed on the first computational unit, and the remaining code modules may be executed on the second computational unit.

[0089] According to one embodiment, the computing device may be at least one of a general-purpose computing device including a general-purpose processor and a dedicated computing device including a processor for specific software operations.

[0090] For example, a general-purpose computing device can have the same performance as edge device hardware used in existing IoT devices utilizing processors. The general-purpose computing device can execute all types of software running on the code execution device (100).

[0091] For example, a dedicated processing unit may be equipped with a hardware accelerator based on an FPGA to increase the processing speed. The dedicated processing unit may only execute a specific designated code module. Additionally, a specific code module that can be executed on the dedicated processing unit may be executed on at least one of a general-purpose processing unit and a dedicated processing unit. To this end, the control unit (140) may control the execution of a specific code module. A further description of the hardware expansion unit (130) will be provided later in the description of FIG. 9.

[0092] The control unit (140) controls the overall operation of each component of the code execution device (100) and the service provision system (10). To this end, the control unit (140) is configured to include one or more processors, and may be configured to include at least one processor of any type well known in the technical field of the present invention, for example, a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Controller Unit), an AP (Application Processor), or the like.

[0093] The control unit (140) can perform operations on at least one application or program for executing a method according to embodiments of the present invention. Although not illustrated, the control unit (140) may include memory and storage, and may load software stored in the storage onto the memory for operation.

[0094] In one embodiment, the control unit (140) may receive a user motion image from the camera. For example, the user motion image may be a video capturing the user's body movements for healthcare or exercise coaching purposes.

[0095] According to one embodiment, the control unit (140) may execute a predetermined code module on a processor embedded in at least one of the one or more processing units attached to the hardware expansion unit (130) to generate analysis data for a user motion image. At this time, the analysis data may be data analyzing the user's body, such as self, movement, and balance, and may be data based on skeleton analysis as described above.

[0096] According to one embodiment, the control unit (140) may provide a setting UI for the type, course, level, and difficulty of the service from the user through the service output device (102), and receive execution request data requesting execution of a related code module through the input unit (110).

[0097] According to one embodiment, the control unit (140) may request a predetermined software for service provision from the service provision server (20) via the communication unit (120). For example, the code execution device (100) may store a specific type of software in its own memory (not shown). However, in the case of specific software, such as large-capacity software or software with low usage frequency, the software may not be stored in the memory, but may be received and executed from the service provision server (20) whenever necessary.

[0098] According to one embodiment, the code execution device (100) stores code modules such as a motion recognition code module and a UI output code module, and when a service selection input is received from a user, the code execution device (100) can generate service data by simultaneously receiving and executing a service code module through streaming from a service provision server (20).

[0099] FIG. 3 is an example of code streaming execution on an edge device according to another embodiment of the present invention. Referring to FIG. 3, a code execution device (100) can execute software (300) for providing a service.

[0100] The software (300) may be configured to include multiple code modules of functional units. In FIG. 3, a case in which a motion recognition code module (310) for a motion recognition function, a service execution code module (320) for providing a service, and a UI output code module (330) for outputting service data including service content is illustrated as an example. Hereinafter, the service execution code module (320) may be referred to as a service code module.

[0101] As described above, the code execution device (100) stores a predetermined code module in memory, but can receive and execute a specific code module from the service provision server (20) whenever necessary.

[0102] For example, the code execution device (100) stores the motion recognition code module (310) and the UI output code module (330) in memory, but the service execution code module (320) can be streamed and executed from the service provision server (20) when the provision of the selected service is requested according to the user's service selection.

[0103] For example, depending on the user's selection of one of the service yoga (321), PT (322), and CrossFit (323) services, the execution code module for the selected service may be streamed. Accordingly, the code execution device (100) may execute the motion recognition code module (310) in real time to analyze the user motion image, and based on the analysis result, may simultaneously receive and execute the service execution code module (320) while streaming, thereby generating service data.

[0104] Figure 4 is a flowchart of a service providing method according to another embodiment of the present invention.

[0105] Each step of FIG. 4 is performed by the service providing system (10), and specifically, each step can be performed by a component of the service providing system (10). In particular, the various code modules below are executed as calculated by the control unit (140) of the code execution device (100).

[0106] Referring to FIG. 4, the code execution device (100) receives image data from a camera (101) (S10). The image data is a motion image of the user, and as described above, analysis of the motion image of the user is performed according to the execution of the motion recognition code module in the code execution device (100).

[0107] Analysis of a user's motion image could be, for example, artificial intelligence-based skeleton analysis.

[0108] In one embodiment, the code execution device (100) may request streaming of a service code module to a service providing server as a user's motion image is received.

[0109] In another embodiment, if a service code module matching a service selected by a user is stored in the internal memory of the code execution device (100), the code execution device (100) can execute the service code module stored in the internal memory to generate service data based on the skeleton analysis.

[0110] The code execution device (100) can determine whether the service code module stored in the internal memory is sufficient to execute the service selected by the user. Specifically, the service code module is matched with a service information table including at least one of the type of service, difficulty level, service course, level, service output setting, and exercise amount, such as yoga or dance, and is stored in the service providing server. The code execution device (100) scans the service code module stored in the memory and identifies the matching value of the service information table, thereby determining what type of exercise the service course module is related to, and whether the code module corresponds to the course type, level, and to what extent the exercise amount and output setting.

[0111] The amount of computation of the code execution device (100) varies depending on the type of service, type of course, grade, and output settings.

[0112] For example, if the user's selected service is dance training that requires continuous movements within a short period of time, the code execution device (100) can execute a motion recognition code module to perform rapid analysis of the continuous movements. The code execution device (100) generates service data by executing a service code module to process a number of events matching the analysis results of the continuous movements.

[0113] To this end, the service code module makes calls and responses to the motion recognition code module and / or the UI output code module for each event.

[0114] At this time, if the user's output setting for the user-selected service is at a high level, for example, if sensitive analysis of user motion is required and the required level of graphic effects of the UI output is set high, the code execution device (100) may determine that the service code module stored in the internal memory is insufficient. In this case, the motion recognition code module, the service code module, and the UI output code module must all be configured with codes corresponding to the service whose output is set at a high level.

[0115] Therefore, the code execution device (100) requests streaming of a matching service code module to the service providing server in order to satisfy the high level output settings.

[0116] In another example, if the user output setting for the user-selected service is at a low level, for example, if the requirement level for at least one of sensitive analysis of user motion and the requirement level for graphic effects of UI output is low, the code execution device (100) may determine that the service code module stored in the internal memory is sufficient.

[0117] At this time, even if the motion recognition code module is configured with code capable of responding to high-level output settings that enable sensitive analysis of user motion, the code execution device (100) may omit responses to at least some commands called from the motion recognition code module in response to low-level output settings when executing the service code module and / or the UI output code module.

[0118] In this case, the code execution device (100) does not need to request a service code module to respond to a high level output setting, so it can determine whether a service code module stored in the memory exists.

[0119] According to an embodiment of the present invention, if the stored service code module is a basic code module commonly used regardless of the type of service, the code execution device (100) determines that the basic code module exists and can execute the basic code module to generate service data (S20).

[0120] That is, if a service code module for responding to high-level output settings is unnecessary, the code execution device (100) can generate first service data including events that must be commonly provided through a service selected by a user by executing a basic code module once stored instead of a service code module streaming request. The first service data is service data generated by the code execution device (100) by executing a basic code module stored in memory.

[0121] However, if the service data generated by executing only the basic code module does not match the user-selected service and / or output settings, the code execution device (100) may request and receive streaming of the service code module even if the basic code module is stored (S30).

[0122] The code execution device (100) can execute the received service code module to generate second service data for a user-selected service (S40). The second service data is service data generated by the code execution device (100) by executing the service code module received through streaming.

[0123] The code execution device (100) can generate real-time service data based on the first service data and the second service data (S50).

[0124] According to one embodiment of the present invention, the code execution device (100) can execute the motion recognition code module, the service code module, and the UI output code module sequentially or in parallel, or at least some of the codes can be executed sequentially while other some of the codes can be executed in parallel.

[0125] To this end, the motion recognition code module, service code module, and UI output code module communicate with each other via an Application Programming Interface (API), exchanging requests and responses. Furthermore, the motion recognition code module, service code module, and UI output code module may also be internally organized into multiple code module groups.

[0126] For example, a service code module is a code that is functionally executed by a code execution device (100) to generate service data, but is internally composed of a basic code module and a streamed service code module that communicate with each other through an API and can generate service data.

[0127] As another example, a motion recognition code module, a UI output code module, and also internally may be composed of a combination of multiple groups of code modules.

[0128] Figures 5 and 6 are examples of code modules targeted for code streaming referenced in some embodiments of the present invention. In particular, for convenience of explanation in Figures 5 and 6, the software code is represented as a physical structure. Specifically, each code module is represented as a block, and in Figure 5, each code module is represented as a structure in which protrusions and grooves into which the protrusions are seated are connected to each other.

[0129] In Figures 5 and 6, the code modules are represented as structured blocks for convenience of explanation, but in reality they are bundles of software code.

[0130] Each code module includes an Application Programming Interface (API) for communicating with each other, and in FIGS. 5 and 6, communication between the APIs provided by each code module is expressed as a combination or connection between a code module block and another code module block.

[0131] Referring to FIG. 5, the code execution device (100) can execute code (500) to provide a service. The code (500) can include a motion recognition code module (510), a service code module (520), and a UI output code module (530).

[0132] It is assumed that the motion recognition code module (510) and the UI output code module (530) are code modules that can be universally applied to various services, and that only the service code module for executing them is different for each service.

[0133] The service provision system (10) can provide a predetermined service to a user according to the execution of the service code module (520) of the code execution device (100).

[0134] At this time, if the user selects a new service or changes the level of output settings, execution of a code different from code (500) is required.

[0135] The above other code may be a code in which at least one of the motion recognition code module (510), service code module (520), and UI output code module (530) constituting the code (500) has been changed.

[0136] When the user selection service is changed, for this purpose, the motion recognition code module (510) and the UI output code module (530) must be executed as is, but code module a (521) or code module b (522) must be executed instead of the service code module (520).

[0137] However, referring to FIG. 5, code module a (521) or code module b (522) cannot be structurally applied to code (500). Specifically, the lower shape of the motion recognition code module (510) and the upper shape of code module a (521) or code module b (522) have a structure that cannot be combined. In addition, the upper shape of the UI output code module and the lower shape of code module a (521) or code module b (522) also have a structure that cannot be overlapped.

[0138] This means that code module a (521) or code module b (522) does not have an API for communicating with the motion recognition code module (510) and the UI output code module (530) by replacing the service code module (520) in the code (500).

[0139] Accordingly, among the codes (500), it is impossible to replace the service code module (520) with either code module a (521) or code module b (522), and the code execution device (100) must download the entire code corresponding to the service corresponding to the code (500) and the other service from the service providing server (20) in advance and store it in memory.

[0140] That is, if the code for a service has a structure like that of Fig. 5, in situations such as a change in the type of service, modification or update of the service, all edge devices connected to the service providing server must each download a new code.

[0141] Fig. 6 proposes a code (600) according to an embodiment of the present invention that solves the structural problem of the code (500) of Fig. 5. In Fig. 6, for convenience of explanation, each code module is configured as a flat cross-section and the joint structure is expressed in a manner in which they are joined to each other.

[0142] Referring to FIG. 6, the code (600) includes a motion recognition code module (610), a service code module (620), and a UI output code module (630).

[0143] Compared to the code (500) of FIG. 5, the cross-sections of the motion recognition code module (610), the service code module (620), and the UI output code module of the code (600) are all expressed flat.

[0144] For example, the motion recognition code module (610) has an API (611) and can communicate with the service code module through the API (621), and the service code module (620) has an API (622) and can communicate with the UI output code module (630) through the API (631).

[0145] The service code module (620) includes an API for communicating with the coupling unit (621) of the motion recognition code module (610), and includes a coupling unit (621) and a coupling unit (622) for coupling with the coupling unit (611) and the coupling unit (631).

[0146] The code execution device (100) executes the code (600) in a manner of generating real-time service data by executing the service code module (620) based on the user's motion recognized for the selected service through the API (611, 621) of the motion recognition code module (610).

[0147] After executing the code (600), if the user selects a new service or changes the level of the output setting, the code execution device (100) must execute a code different from the code (600). Referring to FIG. 6, the service code module (620) on the code (600) can be replaced with the code module X (640). That is, as the code module X (640) is streamed from the service providing server (20) to the service providing system (10), the code module X (640) can be applied to the code (600) stored in the code execution device (100).

[0148] To this end, API (641) can be connected to API (611) by replacing the API (621) portion, thereby combining code module X (640) and motion recognition code module (610). In addition, API (642) can be connected to API (631) by replacing the API (622) portion, thereby combining code module X (640) and UI output code module (630).

[0149] The user motion recognition result of the motion recognition code module (610) is transmitted to the code module X (640) through the API (611) and API (641), and the code execution device (100) executes the streamed code module X (640), thereby enabling the service provision system (10) to analyze the transmitted user motion and generate real-time service data for the selected service based on the analysis.

[0150] That is, according to an embodiment of the present invention, since the API of the code (600) is disclosed, service code modules can be developed by various service developers to be linked to the code (600). Accordingly, various service code modules having APIs applicable to the code (600) can communicate with the motion recognition code module (610) and / or the UI output code module (630), thereby enabling the provision of diverse services.

[0151] According to an embodiment of the present invention, when a third party's service code module is uploaded, the service providing system (10) and / or the service providing server (20) can scan the uploaded service code module.

[0152] Based on the scanning results, the service providing system (10) and / or the service providing server (20) can determine whether the pre-stored code and the uploaded service code module can communicate with each other to provide the service according to the embodiment of the present invention through the API of the uploaded service code module.

[0153] The service provision system (10) and / or the service provision server (20) may delete the uploaded service code module if communication between the pre-stored code and the uploaded service code module is impossible.

[0154] In the above, an embodiment of replacing a service code module on a code has been mainly described, but the present invention is not limited thereto. Hereinafter, an embodiment in which at least a portion of a motion recognition code module or a UI output code module is replaced in addition to the service code module will be described.

[0155] Figure 7 is a flowchart illustrating a method for determining a streaming target based on required resources for each selected service, as referenced in another embodiment of the present invention. Figure 8 also illustrates an example of an embodiment in which the code unit to be streamed changes depending on the type of specific service, as referenced in some embodiments of the present invention.

[0156] Referring to FIG. 7, when image data is received from the camera in step (S10) of FIG. 3, the service providing system (10) can identify the motion recognition code module stored in the memory of the code execution device (100) (S701).

[0157] The service provision system (10) can determine whether the motion recognition code module stored in the memory satisfies the recognition level required for the service selected by the user (S703). For example, if the service selected by the user requires precise analysis of three-dimensional motion through skeleton analysis, and the motion recognition code module stored in the memory is for two-dimensional motion analysis, the code execution device (100) can determine that the recognition level of the stored motion recognition code module does not satisfy the required level.

[0158] If the identified motion recognition code module does not satisfy the required recognition level, the service providing system (10) may request an additional code module of the motion recognition code module for providing the selected service from the service providing server (20) (S705).

[0159] The relationship between the motion recognition code module and the additional code module is similar to the relationship between the motion recognition code module and the service code module that communicate via the API described above in the description of FIG. 6, and the relationship between the service code module and the UI output code module.

[0160] That is, the additional code module is streamed from the service providing server (20) to the code execution device (100), and the previously stored motion recognition code module can communicate with the additional code module through an API that is mutually linked (S707).

[0161] According to one embodiment, the code execution device (100) can execute a motion recognition code module and an additional code module in conjunction with each other.

[0162] In another embodiment, a code module satisfying a recognition level for providing a service selected by a user can be generated by communication through an API of a motion recognition code module and an additional module.

[0163] The code execution device (100) can recognize user motion by executing a code module that satisfies the recognition level (S709).

[0164] The embodiment of FIG. 7 described above can be applied to a UI output code module in addition to a motion recognition code module.

[0165] In step (S50) of FIG. 4, the code execution device (100) can identify a UI output code module for outputting the generated service data.

[0166] The code execution device (100) can determine whether the identified UI output code module satisfies the output level for providing a service selected by the user, and if the determination result shows that the identified UI output code module does not satisfy the output level, it can request an additional code module of the UI output code module for providing the selected service from the service providing server (20).

[0167] The code execution device (100) can execute the UI output code module and the additional code module by linking them as it receives the additional code module through streaming, thereby outputting service data that satisfies the output level of the selected service.

[0168] In FIG. 8, the code (800) includes a motion recognition code module (810), and the code execution device (100) can stream an additional code module, the motion recognition Y code module (811), if the motion recognition code module (810) does not satisfy the recognition level of the selected service. The motion recognition code module (810) and the motion recognition Y code module (811) can communicate via an API.

[0169] In this way, user motion recognition satisfying the service requirements is achieved, and the code execution device (100) can execute the service code module Y (820) to generate service data.

[0170] The code execution device (100) executes the UI output code module (830) to output service data. At this time, if the UI output code module (830) does not satisfy the output level of the selected service, the code execution device (100) may stream the UI output Y code module (831) from the service provision server (20) and output the service data.

[0171] FIG. 9 is an example of an extensible hardware module for executing a streamed code module, referenced in another embodiment of the present invention.

[0172] In Fig. 9, the hardware expansion unit (130) is illustrated as a slot structure, and a case in which the operation unit (131), the operation unit (132), and the operation unit (133) can be inserted into the slots of the hardware expansion unit (130) is illustrated.

[0173] Each of the computing devices (130, 131, 132) may store software for executing memory and code modules. In another example, at least some of the computing devices (131, 132, 133) may have their own processor.

[0174] As at least one of the operation unit (131), operation unit (132) and operation unit (133) is inserted into the hardware expansion unit (130), the code execution device (100) can identify that at least one operation unit is added through the hardware expansion unit (130) of the service providing system.

[0175] In one embodiment, when the motion recognition code module and the additional code module are linked and executed, the motion recognition code module is executed by the code execution device (100), and the additional code module can be executed through an API that is linked with the motion recognition code module using the added operation device (131). The control unit (140) may load and operate the motion recognition code module through the internal memory, and may load and operate the additional code module through the memory within the operation device. Alternatively, the control unit (140) may operate the motion recognition code module, and the additional code module may be operated by the processor within the operation device.

[0176] In another example, when a motion recognition code module and an additional code module are linked and executed, the code execution device (100) may execute the motion recognition code module, execute the additional code module using the added operation unit (131), and execute the service code module using the added operation unit (132).

[0177] According to an embodiment of the present invention, the computing device (131, 132, 133) can be utilized even when the service is updated.

[0178] The execution target service code module of the code execution device (100) may be changed according to modifications and / or updates to the service content.

[0179] The code execution device (100) can receive a service update message from the service providing server (20), and based on the received update message, can determine whether the recognition level of the motion recognition code module stored in the memory satisfies the requirements of the updated service.

[0180] The code execution device (100) can generate a request message for an updated service code module in response to an update message of the service.

[0181] At this time, the code execution device (100) may transmit a request for an additional code module for the motion recognition code module to the service providing server along with a request message for the generated updated service code module. If the recognition level of the motion recognition code module is low and an additional code module is required, this is because not only streaming of the updated service code module but also the additional code module is required.

[0182] At this time, when the code execution device (100) identifies that an operation unit has been added through the hardware expansion part of the service provision system (10), at least one of the additional code module and the updated service code module can be executed on the added operation unit as the additional code module and the updated service code module are streamed.

[0183] That is, at least one of the computational units and the code execution unit (100) can execute additional code modules and updated service code modules in parallel.

[0184] Meanwhile, according to another embodiment of the present invention, after the first service is executed by the code execution device (100), a user may input a selection for a second service. For example, after performing yoga exercises as the first service, the user may select dance training as the second service.

[0185] The service providing system (10) or code execution device (100) can identify a motion recognition code module for providing a second service.

[0186] In addition to the motion recognition code module for providing the first service, the service provision system (10) or code execution device (100) can determine whether an additional code module received for providing the first service satisfies the recognition level for providing the second service if there is an additional code module for providing the first service.

[0187] The service provision system (10) or code execution device (100) can determine whether to delete the additional code module based on the result of the judgment, whether the additional code module satisfies the recognition level for providing the second service.

[0188] The service provision system (10) or code execution device (100) can recognize the motion of the user to provide a second service by executing a motion recognition code module identified based on the user's image input to the camera.

[0189] The service provision system (10) can request a service code module for providing a second service from the service provision server (20).

[0190] The code execution device (100) can remove the service code module streamed for providing the first service as the service code module for providing the second service is streamed.

[0191] The code execution device (100) can generate real-time service data for a second service based on the recognized user's motion and output the generated service data through a service output device.

[0192] The determination and / or calculation methods of the processor and / or control unit according to the embodiments of the present invention described with reference to the attached drawings so far can be performed by executing a computer program implemented as a computer-readable code. The computer program can be transmitted from a first computing device to a second computing device via a network such as the Internet and installed in the second computing device, thereby allowing it to be used in the second computing device. The first computing device and the second computing device include all of stationary computing devices such as server devices, desktop PCs, and mobile computing devices such as laptops, smartphones, and tablet PCs.

[0193] Although embodiments of the present invention have been described with reference to the attached drawings, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering the technical concept or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. A method for providing a service using code streaming, performed by an edge device-based service providing system, A step of identifying a motion recognition code module for providing the first service to the user based on the user's selection of the first service; A step of recognizing the motion of the user by executing the identified motion recognition code module based on the motion image of the user input to the camera of the service providing system; In order to provide the first service, a step of requesting a service code module operated in a code execution device of the service providing system to a service providing server; A step of generating real-time service data for the first service based on the recognized user's motion upon receiving the service code module by streaming; and Including a step of outputting the generated service data through a service output device of the service providing system. Method of providing services using code streaming.

2. In paragraph 1, The step of requesting a service code module for providing the above first service is: A step of determining whether there is a basic code module matching the first service, which is pre-stored in the service providing system; and If the basic code module exists, a step of executing the basic code module to generate first service data for providing the first service is included. Method of providing services using code streaming.

3. In paragraph 2, The step of generating the above real-time service data is: A step of streaming the service code module and executing the received service code module to generate second service data for the first service; and A step of generating the real-time service data based on the first service data and the second service data, Method of providing services using code streaming.

4. In the first paragraph, the streaming service code module includes an API (Application Programming Interface) for communicating with the identified motion recognition code module, The step of generating the above real-time service data is: A step of generating the real-time service data by executing the service code module based on the recognized user's motion for the first service through the API, Method of providing services using code streaming.

5. In the first paragraph, the step of identifying the motion recognition code module comprises: A step of determining whether the identified motion recognition code module satisfies the recognition level for providing the first service; As a result of the above judgment, if the identified motion recognition code module does not satisfy the recognition level, a step of requesting an additional code module of the motion recognition code module for providing the first service from the service providing server; and As the additional code module is received by streaming, a step of executing the motion recognition code module and the additional code module in conjunction with each other, The step of recognizing the user's motion is as follows: A step of recognizing the user's motion, wherein the motion recognition module and the additional code module are executed in conjunction with each other. Method of providing services using code streaming.

6. In the fifth paragraph, the additional code module includes a first API for communicating with the identified motion recognition code module, The steps to be executed in conjunction with the above are: A step in which the motion recognition code module and the additional code module communicate with each other and execute in response to the input user's motion image, Method of providing services using code streaming.

7. In the 6th paragraph, the additional code module includes a second API for communicating with the streaming service code module, The step of generating the above real-time service data is: A step of transmitting the user's motion to the streaming service code module through the second API; A step of analyzing the transmitted user's motion by executing the above-described streaming service code module; and A step of generating real-time service data for the first service based on the results of the above analysis, Method of providing services using code streaming.

8. In paragraph 6, The above code execution device further includes a step of identifying that at least one computational device is added through a hardware expansion part of the service providing system, The step of executing the above motion recognition code module and the above additional code module in conjunction with each other is as follows: A step of executing the motion recognition code module through the code execution device; and Comprising a step of executing the additional code module based on the first API using the at least one additional computational unit; Method of providing services using code streaming.

9. In the 8th paragraph, the at least one operation device includes a first operation device and a second operation device, The step of executing the additional code module based on the first API using the at least one additional computational unit is: A step of executing the additional code module using the first operation unit is included, The step of generating real-time service data for the above first service is: A step of executing the streamed service code module using the second computing device, Method of providing services using code streaming.

10. In paragraph 1, The step of outputting the above generated service data is: A step of identifying a UI (User Interface) output code module for outputting the generated service data; A step of determining whether the identified UI output code module satisfies the output level for providing the first service; As a result of the above judgment, if the identified UI output code module does not satisfy the output level, a step of requesting an additional code module of the UI output code module for providing the first service from the service providing server; A step of executing the UI output code module and the additional code module in conjunction with each other as the additional code module is received by streaming; and A step of outputting service data generated for the first service by linking and executing the additional code module and the UI output code module, Method of providing services using code streaming.

11. In paragraph 1, Further comprising a step of receiving a selection for the second service of the user; A step of identifying a motion recognition code module for providing the second service to the user; A step of recognizing the motion of the user for providing the second service by executing the identified motion recognition code module based on the image of the user input to the camera; A step of requesting a service code module for providing the second service from a service providing server; A step of removing a service code module stream-received for providing the first service as the service code module for providing the second service is stream-received; A step of generating real-time service data for the second service based on the recognized user's motion; and A step of outputting service data generated for the second service through a service output device of the service providing system, Method of providing services using code streaming.

12. In paragraph 11, The step of identifying a motion recognition code module for providing the above second service is: In addition to the above motion recognition code module, if there is an additional code module received for providing the first service, a step of determining whether the additional code module satisfies the recognition level for providing the second service; and As a result of the above judgment, the additional code module includes a step of determining whether to delete the additional code module depending on whether the recognition level for providing the second service is satisfied. Method of providing services using code streaming.

13. In paragraph 5, The step of determining whether the above-identified motion recognition code module satisfies the recognition level for providing the first service is as follows: A step of receiving an update message of the first service from the service providing server; and A step of determining whether the recognition level is satisfied based on the received update message, The step of receiving an update message of the above first service is: Including a step of generating a request message for an updated service code module in response to the above update message, The step of requesting the above additional code module is: A step of transmitting a request message for the generated updated service code module to the service providing server together with a request message for the additional code module, Method of providing services using code streaming.

14. In paragraph 13, The above code execution device further includes a step of identifying that a computational device is added through a hardware expansion part of the service providing system, As the additional code module and the updated service code module are streamed, the added computing device includes a step of executing at least one of the additional code module and the updated service code module. Method of providing services using code streaming.

15. A service providing server storing at least one service code module; and A service providing system that requests the service providing server to stream code for a user's selected service, and executes the service code module as the service code module for the selected service is streamed from the service providing server in response to the request, The above service provision system, A camera that receives motion images of the user; A code execution device that recognizes the motion image of the user by executing a motion recognition code module and generates service data by executing the service code module based on the recognized motion image; and A service output device that outputs the service data by executing a UI output code module based on the service data, Code streaming system.

Images