Method for controlling plurality of electronic devices according to context of user and computing device for performing method

Abstract

A method of controlling a plurality of electronic devices according to a context of a user is provided. The method includes determining, by a computing device, a context of a user, selecting, by the computing device, at least one modality to be considered in controlling the plurality of electronic devices based on the determined context, obtaining, by the computing device, a score map for each selected modality, the score map indicates a region affected by inputs and outputs of the plurality of electronic devices, calculating, by a computing device, a score for each selected modality by using the obtained score map, and controlling, by the computing device, an operation of the plurality of electronic devices based on the calculated scores, the score may include a value indicative of a degree of optimization of the user experience in the determined context.

Description

Technical Field

[0001] The present disclosure relates to a method for controlling multiple electronic devices based on a user's context, and a computing system for performing the method. Background Art

[0002] Recently, the types of electronic devices (home appliances) used in homes and offices have diversified. In addition, the application scope of Internet of Things (IoT) technology, which controls electronic devices by using information collected from electronic devices, has also expanded.

[0003] When controlling multiple electronic devices, the IoT server can perform control based on the status and function of each electronic device. However, if multiple electronic devices are operating in a certain space, and the IoT server independently controls each electronic device without considering the overall situation, the electronic devices may not interact smoothly with each other, or the user experience quality of using the electronic devices may deteriorate.

[0004] The above information is presented as background information only to assist with an understanding of the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above might be applicable as prior art with respect to the present disclosure. Summary of the Invention

[0005] Technical Solution Aspects of the present disclosure are to address at least the above-mentioned problems and / or disadvantages and to provide at least the advantages described below. Therefore, one aspect of the present disclosure is to provide a method for controlling multiple electronic devices based on user context, and a computing system for executing the method.

[0006] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0007] According to one aspect of the present disclosure, a method for controlling multiple electronic devices based on a user's context is provided. The method includes determining the user's context by a computing device, selecting, by the computing device, at least one modality to be considered when controlling the multiple electronic devices based on the determined context, obtaining, by the computing device, a score map for each selected modality, the score map indicating areas affected by inputs and outputs of the multiple electronic devices, calculating, by the computing device, a score for each selected modality using the obtained score map, and controlling, by the computing device, the operations of the multiple electronic devices based on the calculated scores, wherein the scores include values indicating the degree to which the user experience is optimized in the determined context.

[0008] According to another aspect of the present disclosure, a computing device is provided. The computing device includes: a communication interface configured to perform communication with multiple electronic devices; a memory configured to store a program for controlling the multiple electronic devices according to a user's context; and at least one processor, wherein the at least one processor is configured to execute the program to determine the user's context, select at least one modality to be considered when controlling the multiple electronic devices based on the determined context, obtain a score map indicating areas affected by inputs and outputs of the multiple electronic devices for each selected modality, calculate a score for each selected modality by using the obtained score map, and then control the operations of the multiple electronic devices based on the calculated score, wherein the score includes a value indicating the degree of optimization of the user experience in the determined context.

[0009] According to one aspect of the present disclosure, a non-transitory computer-readable recording medium may have stored therein a program for executing at least one of the disclosed embodiments of the method on a computer.

[0010] According to one aspect of the present disclosure, a program is stored in a computer-readable recording medium so as to execute at least one of the embodiments of the disclosed method on a computer.

[0011] Other aspects, advantages, and salient features of the present disclosure will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the accompanying drawings, discloses various embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which: Figure 1 is a diagram illustrating a system environment according to an embodiment of the present disclosure; Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 and Figure 6 is a diagram illustrating a first embodiment (the context is a case of “watching TV”) according to various embodiments of the present disclosure; Figure 7 、 Figure 8 、 Figure 9 、 Figure 10 and Figure 11 is a diagram illustrating a second embodiment (the context is a case of “communication with an artificial intelligence (AI) speaker”) according to various embodiments of the present disclosure; Figure 12is a diagram illustrating a detailed configuration of a computing device (server) for executing a method of controlling a plurality of electronic devices according to a user's context according to an embodiment of the present disclosure; and Figure 13 、 Figure 14 、 Figure 15 and Figure 16 is a flowchart illustrating a method of controlling a plurality of electronic devices according to a user's context according to various embodiments of the present disclosure.

[0013] Throughout the drawings, it should be noted that like reference numbers are used to depict the same or similar elements, features, and structures. DETAILED DESCRIPTION

[0014] The following description, with reference to the accompanying drawings, is provided to facilitate a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. It includes various specific details to assist understanding, but these details are to be regarded as exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. Furthermore, descriptions of well-known functions and structures may be omitted for clarity and conciseness.

[0015] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustration purposes only and not for the purpose of limiting the present disclosure as defined by the appended claims and their equivalents.

[0016] It will be understood that singular forms include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of such surfaces.

[0017] It should be understood that the blocks in each flowchart and the combination of flowcharts may be performed by one or more computer programs comprising computer executable instructions. The entirety of one or more computer programs may be stored in a single memory, or one or more computer programs may be divided into different parts stored in different multiple memories.

[0018] Any functions or operations described herein may be processed by a processor or a combination of processors. A processor or a combination of processors is a circuit that performs processing and includes circuits such as an application processor (AP), a communication processor (CP), a graphics processing unit (GPU), a neural processing unit (NPU), a microprocessor unit (MPU), a system on a chip (SoC), an IC, and the like.

[0019] Throughout this disclosure, the expression "at least one of a, b, or c" refers to only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0020] In the following description of the present disclosure, descriptions of technologies that are well known in the art and not directly related to the present disclosure are omitted. This is to clearly convey the main purpose of the present disclosure by omitting unnecessary explanations. The terms used in the following description are defined based on the functions used in the disclosure and can be changed according to the intentions of the user or operator or the commonly used methods. Therefore, the definitions of the terms are understood based on the entire description of this specification.

[0021] For the same reason, some components may be exaggerated, omitted, or schematically shown in the drawings for the sake of clarity. In addition, the size of each component does not fully reflect the actual size. The same reference numerals are assigned to the same or corresponding components in the drawings.

[0022] With reference to the embodiments described below with reference to the accompanying drawings, the advantages and features of the present disclosure and the method for achieving them will become apparent. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The disclosed embodiments are provided to ensure that the present disclosure is complete and to fully inform those skilled in the art of the present disclosure of the scope of the present disclosure. The embodiments of the present disclosure may be defined according to the claims. Throughout the specification, the same elements are represented by the same reference numerals. In the description of the embodiments of the present disclosure, when it is determined that a detailed description of a related function or configuration may unnecessarily obscure the subject matter of the present disclosure, its detailed description will be omitted. The terms used in the specification are defined based on the functions used in the disclosure and may be changed according to the intention of the user or operator or the method commonly used. Therefore, the definition of the terms is understood based on the entire description of this specification.

[0023] In an embodiment of the present disclosure, each block of the flowchart and the combination of the flowcharts can be executed by computer program instructions. The computer program instructions can be loaded into a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, and the instructions executed by the processor of the computer or other programmable data processing device can generate a device for implementing the functions specified in (one or more) flowchart blocks. The computer program instructions can also be stored in a computer-usable or computer-readable memory, which can instruct the computer or other programmable data processing device to function in a specific manner, and the instructions stored in the computer-usable or computer-readable memory can produce an article of manufacture including instruction means for implementing the functions specified in (one or more) flowchart blocks. The computer program instructions can also be installed on a computer or other programmable data processing device.

[0024] In addition, each block of the flowchart may represent a module, segment or portion of a code comprising one or more executable instructions for performing (one or more) specified logical functions. In embodiments of the present disclosure, the functions mentioned in the blocks may not occur in order. For example, depending on the function, two blocks shown in succession may be executed substantially simultaneously, or may sometimes be executed in reverse order.

[0025] The term "unit" used in embodiments of the present disclosure refers to a software or hardware component (such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) that performs certain tasks. However, the term "unit" is not limited to software or hardware. The term "unit" can be configured in an addressable storage medium, or can be configured to reproduce one or more processors. In embodiments of the present disclosure, the term "unit" can refer to a component, such as a software component, an object-oriented software component, a class component, and a task component, and can include a process, a function, a property, a procedure, a subroutine, a program code segment, a driver, firmware, microcode, a circuit, data, a database, a data structure, a table, an array, or a variable. The functions provided by a specific element or a specific "unit" can be coupled to reduce their number or divided into additional elements. In addition, in embodiments of the present disclosure, a "unit" can include at least one processor.

[0026] Before describing specific embodiments of the present disclosure, the meanings of terms frequently used in this specification are defined.

[0027] "Context" refers to situation information determined based on the user's activities and the operating state of the electronic device in an embodiment of the present disclosure. According to an embodiment of the present disclosure, a user can act according to intention in an environment using multiple electronic devices, and a context corresponding to the user's intention can be determined based on the user's activities and the operating state of the electronic device. For example, when a user is sitting in the front area of a television (TV) and the TV is turned on, the context can be determined as "watching TV", but when the TV is turned off, the context can be determined as "idle" even when the user is sitting in the front area of the TV. In addition, various types of contexts can be determined according to the user's intention.

[0028] "Modality" may include the type of input / output used when an electronic device interacts with a user. In addition, the modality may include the feeling desired by the user when using the electronic device. According to an embodiment of the present disclosure, the modality may include a visual modality, an auditory modality, and the like. In addition, various other types of modalities may be used. For example, the visual modality of an electronic device may include data output through a display or data input through a camera, and the auditory modality of an electronic device may include data output through a speaker or data input through a microphone. The user may use the visual modality when detecting data output to the display, and may use the auditory modality when detecting data output through the speaker. In addition, the user may input the user's voice to the electronic device through a microphone, and at this time, the user's voice may be included in the auditory modality.

[0029] In the embodiments of the present disclosure, "kernel" is used as a term meaning a range in which input and output of an electronic device are affected or a range in which a user feels an operation. A kernel is defined as the size and position of an area and may include an input kernel and an output kernel.

[0030] An "input kernel" may refer to the range of an area where input from a user or electronic device is affected. For example, the range of an area where a user can hear sound may correspond to the user's input kernel for the auditory modality. Alternatively, for example, the range of an area where an electronic device can receive voice input may correspond to the electronic device's input kernel for the auditory modality.

[0031] The "output kernel" may refer to the range of an area affected by the output of a user or electronic device. For example, the range of an area where an image output from a TV can be viewed may correspond to the output kernel of the TV for a visual modality.

[0032] The input / output kernel may vary depending on the characteristics of the electronic device, and even for the same electronic device, the input / output kernel may vary depending on the context or modality. In an embodiment of the present disclosure, the input / output kernel of the electronic device is displayed on a map, and the map displaying the input / output kernel can be used when controlling the electronic device.

[0033] A "score map" refers to a map on which the input / output core of an electronic device is displayed, and can be used to calculate a score according to the context of the user. In other words, the score map may correspond to a map that shows the areas affected by the input and output of the electronic device for each modality. At this time, "score" refers to a value that quantitatively represents the degree of optimization of the user experience in a certain context. According to an embodiment of the present disclosure, the higher the score calculated in a certain context, the higher the level of experience provided to the user. The specific method of calculating the score by using the score map is described through the following specific embodiments.

[0034] The embodiments of the present disclosure provide a method for controlling multiple electronic devices according to the user's context to provide the user with the best experience. First, general principles generally applicable to the embodiments of the present disclosure are described, and then specific embodiments are described with reference to the accompanying drawings.

[0035] Figure 1 FIG is a diagram showing a system environment according to an embodiment of the present disclosure. Figure 1 , multiple electronic devices 10, 20, 30, and 40 can be used in a space where a user 1 is located, and the multiple electronic devices 10, 20, 30, and 40 can be connected to a server 100 through a wired or wireless network. The server 100 can control operations of the multiple electronic devices 10, 20, 30, and 40 according to the context of the user 1. According to an embodiment of the present disclosure, the server 100 may be an Internet of Things (IoT) server for controlling electronic devices in a home or office.

[0036] The mobile terminal 200 of the user 1 may be connected to the server 100 and used in the process of controlling the electronic devices 10, 20, 30, and 40. In addition, according to an embodiment of the present disclosure, a camera 300 capable of photographing the user 1 and the electronic devices 10, 20, 30, and 40 may be connected to the server 100 and transmit the captured image to the server 100. The types of the electronic devices 10, 20, 30, and 40 controlled by the server 100 are not limited to Figure 1 The types shown in , and can be more diverse.

[0037] 1. Description of general rules applicable to embodiments of the present disclosure In an embodiment of the present disclosure, the server 100 may determine the context of user 1 and control the electronic devices 10, 20, 30, and 40 based on the determined context. The server 100 may control the operation of the electronic devices 10, 20, 30, and 40 in a manner that provides the best experience for user 1. To this end, the server 100 may calculate a score based on a map and perform control based on the calculated score. The following describes the operation of the server 100 performing control.

[0038] (1) Determine the user’s context As described above, the context can be determined based on the intent of user 1. Therefore, server 100 can identify user 1's intent through various means and determine the context accordingly. According to an embodiment of the present disclosure, server 100 can identify at least one of user 1's activity or location and determine user 1's context based on the identification. Several embodiments of how server 100 determines user 1's context are described below.

[0039] 1) Analyze the captured video and determine the context According to an embodiment of the present disclosure, the camera 300 may transmit images of the user 1 and the electronic devices 10, 20, 30, and 40 to the server 100, and the server 100 may analyze the images captured by the camera 300 and determine the context of the user 1. The server 100 may not only recognize the positions of the user 1 and the electronic devices 10, 20, 30, and 40 from the captured images, but may also recognize the operations being performed by the user 1 and the electronic devices 10, 20, 30, and 40 from the captured images (for example, when the robot cleaner 40 moves in the image, the server 100 may determine that the robot cleaner 40 is performing a cleaning operation).

[0040] Therefore, the server 100 can determine the context of user 1 based on the location and the operations performed by the user 1 and each of the electronic devices 10, 20, 30, and 40. For example, when the user 1 is holding a book near the lamp 30, the server 100 can determine that the context is "reading." Also, for example, when the user is moving with a vacuum cleaner, the server 100 can determine that the context is "housework" or "cleaning."

[0041] 2) Determine the context based on information about the operating state received from the electronic device According to an embodiment of the present disclosure, the server 100 can receive information about the operating status of electronic devices 10, 20, 30, and 40 (e.g., whether the TV is turned on, an artificial intelligence (AI) speaker is receiving voice input, etc.) through wired and wireless communications, and determine a context based on the received information about the operating status. For example, if all electronic devices 10, 20, 30, and 40 are turned off or not operating, the server 100 may determine that the context is "sleeping." Alternatively, for example, if only the light 30 is turned on and all other electronic devices are not operating, the server 100 may determine that the context is "idle."

[0042] According to an embodiment of the present disclosure, the server 100 can determine the context by using images captured by the camera 300 and information about the operating states of the electronic devices 10, 20, 30, and 40. For example, when user 1 is within a certain distance from TV 10, the user 1 is facing toward TV 10, and TV 10 is turned on, the server 100 can determine that the context is "watching TV." Alternatively, for example, when user 1 is within a certain distance from AI speaker 20 and AI speaker 20 is receiving voice input, the server 100 can determine that the context is "communicating with the AI speaker."

[0043] 3) Determining context by using neural networks The server 100 may determine the context of the user 1 by using a neural network. According to an embodiment of the present disclosure, the neural network may be trained using data for training that labels images of the user 1 and the electronic devices 10, 20, 30, and 40 with corresponding contexts, and the server 100 may infer the context corresponding to the captured image by using the trained neural network.

[0044] 4) Identify (determine) the context based on the user's selection The context of user 1 may also be determined by the selection of user 1. According to an embodiment of the present disclosure, user 1 may access server 100 through mobile terminal 200 to select a context. For example, user 1 may select any one of various modes such as "TV viewing mode," "idle mode," and "sleep mode," and server 100 may determine the context according to the mode selected by user 1.

[0045] (2) Calculate score based on context As described above, in embodiments of the present disclosure, the "score" may be a value indicating the degree to which the user experience is optimized in any context. Thus, even when the operations of electronic devices 10, 20, 30, and 40 are identical, the impact on the score may vary depending on the context. For example, when the context is "watching TV," turning on TV 10 may contribute to an increase in the score, but when the context is "idle," turning on TV 10 may contribute to a decrease in the score. Therefore, server 100 may calculate the score based on the context of user 1.

[0046] 1) Select at least one modal based on the context According to an embodiment of the present disclosure, a score may be calculated for each modality, and the type of modality required when controlling the electronic devices 10, 20, 30, and 40 may vary depending on the context. Therefore, when the context is determined, the server 100 may determine for which modality to calculate a score.

[0047] For example, when controlling electronic devices 10, 20, 30, and 40, when the context is "watching TV," server 100 must consider both the visual modality and the auditory modality, but when the context is "communicating with an AI speaker," only the auditory modality may be considered. Therefore, according to an embodiment of the present disclosure, server 100 may select at least one modality according to the context, and to this end, information about the modality type corresponding to each context may be pre-stored in server 100.

[0048] 2) Get a score map for each selected modality When at least one modality is selected according to the context, the server 100 may obtain a score map for each selected modality. According to an embodiment of the present disclosure, the server 100 may generate a score map by displaying the locations of user 1 and electronic devices 10, 20, 30, and 40 on a map (e.g., a basic map of the space where the user and the electronic device are located) and for displaying the input / output kernel of each of user 1 and electronic devices 10, 20, 30, and 40. In this case, the input / output kernel of the electronic devices 10, 20, 30, and 40 may be determined based on the input / output characteristics of the electronic devices 10, 20, 30, and 40 associated with each modality. For example, when generating a score map for a visual modality, the server 100 may display the output kernels of the electronic devices that affect the field of view of user 1 on the score map.

[0049] According to an embodiment of the present disclosure, the score map may be a map including a two-dimensional grid form, and in this case, the size of each grid may be set to be proportional to the position measurement resolution of the server 100 .

[0050] According to an embodiment of the present disclosure, the score map may include a plurality of layers, and locations of electronic devices that do not move frequently (eg, a TV, a washing machine, etc.) may be displayed in advance in some layers.

[0051] According to an embodiment of the present disclosure, the server 100 may obtain a score map corresponding to the modality selected according to the context by extracting the score map from the entire score map. The entire score map may be a map showing all areas affected by inputs and outputs of the electronic devices 10, 20, 30, and 40 for multiple modalities. The server 100 may pre-generate and store the entire score map.

[0052] When generating the score map, the locations of the electronic devices 10, 20, 30, and 40 and the location of the user 1 may be measured using various methods. For example, as described above, the server 100 may measure the locations of the user 1 and the electronic devices 10, 20, 30, and 40 from images captured by the camera 300 and display the measured locations on the score map. Alternatively, for example, the server 100 may measure the locations using positioning technology that utilizes wireless communication (e.g., Bluetooth, ultra-wideband (UWB), etc.) between the electronic devices 10, 20, 30, and 40 and the mobile terminal 200.

[0053] The server 100 may set at least one of an input kernel and an output kernel for the user 1 and the electronic devices 10, 20, 30, and 40, and display the set kernel on the score map. Alternatively, the server 100 may not set either the input kernel or the output kernel for some of the user 1 and the electronic devices 10, 20, 30, and 40. For example, the server 100 may not set an output kernel for electronic devices that do not affect the modality corresponding to the score map (for example, when generating a score map for a visual modality, an output kernel may not be set for a washing machine operating at a fixed location).

[0054] According to an embodiment of the present disclosure, the output kernel can be divided into a kernel with positive characteristics and a kernel with negative characteristics, and the characteristics of the output kernel can be determined according to whether the output is a topic of interest. For example, when the context is "watching TV", the area where the operating noise of the robot cleaner 40 is heard may correspond to an output kernel with negative characteristics for the auditory modality. In addition, for example, when the context is "communicating with the AI speaker", the area where the voice input of the AI speaker 20 is heard may correspond to an output kernel with positive characteristics for the auditory modality.

[0055] Even the output of the same electronic device can be an output of interest or noise depending on the context. Therefore, the characteristics of the output kernel can be determined depending on the context. A kernel with positive characteristics can have a positive impact on the score calculation, while a kernel with negative characteristics can have a negative impact on the score calculation.

[0056] According to an embodiment of the present disclosure, input / output kernels may be divided into multiple levels according to the degree to which input is affected or the degree to which output is affected. For example, when setting an output kernel for an auditory modality for an electronic device that outputs sound, an area up to a certain distance from the electronic device may be set as a high-level output kernel, and an area further than the certain distance may be set as a low-level output kernel.

[0057] According to an embodiment of the present disclosure, the server 100 may set an input / output kernel based on the input / output characteristics of the electronic device (e.g., the magnitude of operating noise, the viewing angle of an image, etc.). Depending on the type of electronic device, default values for the input / output characteristics may be predetermined (e.g., a default value for the operating noise level of a washing machine, a default value for the operating noise level of a vacuum cleaner, etc.), and the server 100 may set the input / output kernel for the electronic device based on the predetermined default value.

[0058] According to an embodiment of the present disclosure, the server 100 may correct the input / output characteristics of the electronic device by using capability information provided by the manufacturer of the electronic device or by using other nearby electronic devices (for example, by analyzing images captured by a camera or by analyzing sounds received by using a microphone), and correct the input / output kernel accordingly.

[0059] In addition, according to an embodiment of the present disclosure, the server 100 may set or correct an input / output kernel based on information regarding an operating state (e.g., brightness of a light, volume of a speaker, etc.) received from the electronic devices 10, 20, 30, and 40. As described above, the server 100 may set an input / output kernel according to the type of electronic device and correct the input / output kernel when necessary.

[0060] For example, the server 100 may obtain capability information or information about operating states from the electronic devices 10, 20, 30, and 40, and change at least one of the size or shape of the input kernel or the output kernel based on the obtained capability information or information about operating states.

[0061] According to an embodiment of the present disclosure, the server 100 may set the range of the area recognizable by the user for the modality corresponding to the score map as the input kernel for user 1. For example, when generating a score map for the auditory modality, the area where user 1 can hear sound may be set as the input kernel for user 1.

[0062] According to an embodiment of the present disclosure, the server 100 may generate a score map by dividing the space where the electronic devices 10, 20, 30, and 40 are located into a plurality of areas, mapping each of the electronic devices 10, 20, 30, and 40 to at least one of the plurality of areas, and reflecting the input / output characteristics of the electronic devices 10, 20, 30, and 40 in the mapped areas and displaying the input / output kernel.

[0063] As described above, the input / output kernel set for each of user 1 and electronic devices 10, 20, 30, and 40 can be determined based on the context, modality, and input / output characteristics of the electronic device. For example, when the context is "watching TV" and a score map for the visual modality is to be generated, the area where the image of TV 10 can be viewed can be set as the output kernel for TV 10 (an output kernel with positive characteristics). Alternatively, when the context is "idle" and a score map for the auditory modality is to be generated, the area where the operating noise of robot cleaner 40 can be heard can be set as the output kernel for robot cleaner 40 (an output kernel with negative characteristics).

[0064] 3) Calculate the score for each modality based on the score map When the score map is generated for each modality, the server 100 may calculate a score for each modality by using the score map.

[0065] According to an embodiment of the present disclosure, server 100 may calculate scores based on the area of the input / output kernels displayed on a score map. For example, server 100 may calculate the score by calculating the area of the overlap between the input and output kernels included in the score map and then adding or subtracting the area corresponding to the output kernel (the area of the overlapped area) based on the output kernel's characteristic (positive or negative). In other words, the score may be obtained by adding the area of the overlap between the output kernel with positive characteristics and the input kernel and subtracting the area of the overlap between the output kernel with negative characteristics. In this case, the impact on the score may vary depending on the rank of the input / output kernel. For example, even if the overlapping area between the input and output kernels is the same, the higher the rank of the output kernel, the greater the impact on the score.

[0066] According to an embodiment of the present disclosure, the server 100 may calculate a score by performing a convolution operation on the input kernels and output kernels included in the score map. For example, the server 100 may assign a score to each input / output kernel according to its rank (for example, when the score map includes a two-dimensional grid, 2 points are assigned to each grid included in a high-rank output kernel, and 1 point is assigned to each grid included in a low-rank output kernel), and perform a convolution operation on all output kernels of each input kernel to calculate the score.

[0067] A detailed description of how the server 100 calculates the score through the convolution operation is provided below.

[0068] According to an embodiment of the present disclosure, a score map can be represented as a two-dimensional grid. In the score map, the space may include multiple cells, and each cell may be represented by coordinates (i, j). Server 100 may calculate the total score (the score per modality) corresponding to each of the cells by performing a convolution operation.

[0069] When the input kernel of user 1 is denoted as I u (i, j), the output kernel of any electronic device is represented as 0 d (i,j, a d ), and there are multiple electronic devices d1, d2, ..., d N When , the server 100 can calculate the score S for a specific modality by the following formula 1 m .

[0070] …Formula 1 In this case, a dIt is the action of any electronic device and can refer to the operating mode or operating state of any electronic device. d (i, j, a d ) indicates the kernel value corresponding to the cell at coordinate (i, j) in the output kernel of any electronic device, and this value may depend on a d The kernel value is based on the score assigned to the kernel by the unit. The higher the kernel's rank, the higher the kernel value that can be assigned to the unit.

[0071] In the case of certain types of electronic devices, the location of the output kernel (ie, the area where the output kernel is displayed) may depend on the action a d and change (for example, in the case of the robot cleaner 40, action a d The moving direction and speed of the robot cleaner 40 may be included, and thus, the size and position of the area displaying the output kernel of the robot cleaner 40 may be changed).

[0072] In summary, the score S for a particular mode m is to multiply all the units included in the input kernel of user 1 by the electronic devices d1, d2, ..., d N The following describes the score S calculated in this way. m To control electronic devices d1, d2, ..., d N Method of operation.

[0073] (3) Controlling the operation of electronic devices based on the calculated score When the score is calculated for each modality, the server 100 may control operations of the electronic devices 10 , 20 , 30 , and 40 based on the calculated score.

[0074] According to an embodiment of the present disclosure, the server 100 may control the electronic devices 10 , 20 , 30 , and 40 to maximize a score for each modality.

[0075] According to an embodiment of the present disclosure, the server 100 may add the scores for each modality and control the electronic devices 10, 20, 30, and 40 to maximize the result. When adding the scores for each modality, the server 100 may apply weights to the scores for each modality.

[0076] According to an embodiment of the present disclosure, the weight may be determined according to the context. For example, when the context is "idle", the score for the auditory modality may be given a higher weight than the score for the visual modality.

[0077] According to an embodiment of the present disclosure, the weight may be a determined setting of user 1. For example, when user 1 selects a mode that prioritizes video over sound when watching TV, when the context is "watching TV," a higher weight may be given to the score for the visual modality than to the score for the auditory modality.

[0078] A method in which the server 100 controls the electronic devices 10 , 20 , 30 , and 40 based on the scores is described below.

[0079] 1) Control the operation of the electronic device in a manner that increases the score from the current score According to an embodiment of the present disclosure, the server 100 may control the operation of the electronic devices 10, 20, 30, and 40 in such a manner as to increase the score for each modality from the current score. For example, when it is desired to increase the score for the auditory modality, the server 100 may allow some electronic devices to operate in a low noise mode or stop the operation of some electronic devices.

[0080] In addition, according to an embodiment of the present disclosure, the server 100 may control operations of the electronic devices 10 , 20 , 30 , and 40 in a manner of increasing the sum (simple sum or sum reflecting weight) of scores for each modality from the current score.

[0081] 2) Predict the score after a certain period of time and control the operation of the electronic device in a way that the expected score increases According to an embodiment of the present disclosure, the server 100 may predict the score after a certain period of time based on the current situation (user's location and activity, operating status of the electronic device, etc.), and control the operation of the electronic devices 10, 20, 30 and 40 in such a manner that the expected score increases.

[0082] For example, when the current moving direction of the robot cleaner 40 is a direction that will enter the field of view of the user 1 in a while, the score for the visual modality may be lowered due to the operation of the robot cleaner 40, and therefore, the server 100 may change the moving direction of the robot cleaner 40 to a direction away from the field of view of the user 1.

[0083] 3) Establish the optimal operation plan through iterative optimization and perform corresponding control According to an embodiment of the present disclosure, the server 100 can identify the operation of the electronic devices 10, 20, 30, and 40 to maximize the score by applying an iterative optimization technique to the score map for each modality, establish an optimal operation plan, and control the electronic devices 10, 20, 30, and 40 to operate according to the plan. For example, the server 100 can design a cost function whose value decreases as the score calculated from the score map increases, and control the electronic devices 10, 20, 30, and 40 to minimize the value of the cost function.

[0084] As mentioned above, when the score S for a particular modality is calculated by the convolution operation m When the server 100 is based on the score S m Control electronics d1, d2, ..., d N The method of operation is described below.

[0085] According to an embodiment of the present disclosure, the server 100 can arrange the electronic devices d1, d2, ..., d in the target time range T (eg, 60 seconds) by using the cost function represented by the following formula 2: N Action a d .

[0086] …Formula 2 As can be understood from Formula 2, the cost that the server 100 must maximize within the target time range T is equal to the fraction S of each time t. m In other words, the server 100 can arrange the electronic devices d1, d2, ..., d N Action a d To maximize the score S at each time t within the target time range T m to optimize the experience of user 1. In this case, the score S m The action of each electronic device can be d Therefore, in order to continue the optimization (maximization of the cost function), the server 100 may obtain the action a of each electronic device. d Score S m The gradient of , and apply gradient descent to increase the score S m Determine the action of each electronic device a d The server 100 may iterate the process to arrange the electronic devices d1, d2, ..., d N Action a d .

[0087] Hereinafter, how to apply the above-mentioned general rules to specific embodiments is described with reference to the accompanying drawings. Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 and Figure 6 is a diagram for describing a first embodiment (when the context is “watching TV”) of various embodiments according to the present disclosure. Figure 7 、 Figure 8 、 Figure 9 、 Figure 10 and Figure 11 2 is a diagram for describing a second embodiment (when the context is “communication with an AI speaker”) according to various embodiments of the present disclosure.

[0088] 2. Description of the first embodiment (when the context is "watching TV") (1) (First embodiment) Determining the user's context First, the server 100 may determine the context of the user 1. The server 100 may determine the context of the user 1 according to various methods, and Figure 2 One embodiment thereof is shown.

[0089] Reference Figure 2 , the camera 300 may photograph the user 1 and the TV 10 and transmit the captured image to the server 100. The server 100 may analyze the received captured image to recognize the positions of the user 1 and the TV 10 and the movement of the user 1.

[0090] exist Figure 2 In the illustrated embodiment, when the distance d1 between user 1 and TV 10 is a preset reference value (e.g., 3 meters) or less and the facing direction (gaze direction) of user 1 is toward TV 10 (e.g., when the angle formed by the facing direction of user 1 and the straight line connecting user 1 and TV 10 is a preset reference value or less), server 100 may determine that the context of user 1 is “watching TV.”

[0091] According to an embodiment of the present disclosure, the server 100 may receive information about an operation state (eg, turned on) from the TV 10 and determine the context of the user 1 by considering the information together with a captured image received from the camera 300 .

[0092] According to an embodiment of the present disclosure, the server 100 may apply the captured image received from the camera 300 as input to a pre-trained neural network, and determine the inference result of the neural network as the context of user 1.

[0093] (2) (First embodiment) Calculating scores based on context The server 100 may generate a score map according to the determined context (watching TV), and calculate a score by using the score map.

[0094] 1) (First embodiment) Select at least one modality based on context Because the determined context is "watching TV", the server 100 may select a visual modality and an auditory modality. Information indicating that the modalities corresponding to "watching TV" are visual and auditory modalities may be pre-stored in the server 100. The server 100 may calculate a score for the selected modality.

[0095] 2) (First embodiment) Generate a score map for each selected modality 2-1) (First embodiment) Generating a score map for a visual modality ( Figure 3 ) The server 100 may generate a score map for each selected modality. Figure 3 and Figure 4 Score maps 300 and 400 are shown for the visual modality, respectively, and Figure 5 and Figure 6 Score maps 500 and 600 are shown for the auditory modality, respectively.

[0096] Figure 3 The score map shown in 300 and Figure 5 The score map 500 shown in FIG. 5 is a score map for a situation before the server 100 controls the electronic devices 10, 30, 40, and 50, and Figure 4 The score map shown in 400 and Figure 6 The score map 600 shown in FIG. 6 is a score map for a case after the server 100 controls the electronic devices 10 , 30 , 40 , and 50 .

[0097] First, refer to Figure 3 The method by which the server 100 generates the score map 300 for the visual modality is described, and then, with reference to Figure 5 The method by which the server 100 generates the score map 300 for the auditory modality is described.

[0098] Server 100 may display the locations of user 1 and electronic devices 10, 30, 40, and 50 on score map 300, and may also display an input / output kernel for the visual modality. According to an embodiment of the present disclosure, server 100 may identify the locations of user 1 and electronic devices 10, 30, 40, and 50 from images captured by camera 300. Furthermore, according to an embodiment of the present disclosure, server 100 may identify the locations by using position measurement using wireless communication between electronic devices 10, 30, 40, and 50. Furthermore, according to an embodiment of the present disclosure, information regarding the locations of electronic devices that are not frequently moved, such as TV 10 or washing machine 50, may be pre-stored in server 100.

[0099] When generating the score map 300 for the visual modality, the score map 300 may display the output kernel for the electronic devices among the electronic devices 10, 30, 40, and 50 that affect the field of view of the user 1, and display the input kernel for the user 1 (the range of the area that the user can visually perceive).

[0100] A method in which the server 100 displays the output kernels 10pvok1 and 10pvok2 for the TV 10 is described below.

[0101] Since the image output from the TV 10 is an output in which the user 1 is interested, an output kernel having positive characteristics may be set and displayed for the TV 10 .

[0102] The output kernel of the TV 10 may be divided into a plurality of levels according to the degree to which the visual field of the user 1 is affected by the output kernel, and in the embodiment of the present disclosure, for the convenience of description, it is assumed that all input / output kernels are divided into two levels. Figure 3 , the first output kernel 10pvok1 and the second output kernel 10pvok2 are displayed for the TV on the score map 300. The first output kernel 10pvok1 is an area that has a greater influence on the field of view of the user 1 than the second output kernel 10pvok2, and this difference is reflected in the score calculation (for example, the higher the output kernel's rank, the greater its influence on the score). The specific method of reflecting the output kernel's rank in the score calculation is described below in the section describing score calculation.

[0103] For the reference numbers of the input / output kernels used in the present disclosure, "positive visual output kernel (pvok)" is used for the output kernel having positive characteristics for the visual modality, and "negative visual output kernel (nvok)" is used for the output kernel having negative characteristics for the visual modality. In addition, "visual input kernel (vik)" is used for the input kernel for the visual modality. For other types of input / output kernels, reference numbers are used in a similar manner (see "positive auditory output kernel (paok)" is used for the output kernel having positive characteristics for the auditory modality, "negative auditory output kernel (naok)" is used for the output kernel having negative characteristics for the auditory modality, and "auditory input kernel (aik)" is used for the input kernel for the auditory modality).

[0104] The output kernels 10pvok1 and 10pvok2 of the TV 10 may vary in shape and size (e.g., left-right width, length, etc.) depending on the characteristics or operating state of the TV 10. For example, when the display panel mounted on the TV 10 has a wide viewing angle, the left-right width of the output kernels 10pvok1 and 10pvok2 may increase, and as the brightness of the output image of the TV 10 increases, the size of the output kernels 10pvok1 and 10pvok2 may increase.

[0105] The server 100 may set the output kernels 10pvok1 and 10pvok2 according to the output characteristics set as the default of the TV 10, and, if necessary, correct the output kernels 10pvok1 and 10pvok2 using capability information provided by the manufacturer of the TV 10 or information about the operating state (e.g., brightness of the image) received from the TV 10. In addition, according to an embodiment of the present disclosure, the server 100 may recognize the operating state of the TV 10 by analyzing an image of the TV 10 captured by other electronic devices (e.g., a camera) within the space, and correct the output kernels 10pvok1 and 10pvok2.

[0106] The following describes a method in which the server 100 displays the output kernels 30nvok1 and 30nvok2 of the lamp 30.

[0107] The light radiated from the lamp 30 may interfere with viewing the TV 10, and therefore, an output kernel having negative characteristics may be set and displayed for the lamp 30. However, depending on the surrounding environment (such as the inclination of the user 1 or the position of the lamp 30), the output kernel of the lamp 30 may have positive characteristics, but in this embodiment of the present disclosure, it is assumed that the output kernel of the lamp 30 has negative characteristics.

[0108] The output kernel of the lamp 30 can be divided into multiple levels according to the degree to which the visual field of the user 1 is affected, and Figure 3 In the score map 300 in FIG, the first output kernel 30nvok1 and the second output kernel 30nvok2 are displayed for the lamp 30. Because the brightness of the lamp 30 increases as the distance from the lamp 30 decreases, Figure 3 , the first output kernel 30nvok1 is provided until a certain distance from the lamp 30, and the second output kernel 30nvok2 is provided for a certain area thereafter.

[0109] The output cores 30nvok1 and 30nvok2 of the lamp 30 may vary in shape and size depending on the characteristics or operating state of the lamp 30. For example, as the output of the lamp 30 becomes brighter, the sizes of the output cores 30nvok1 and 30nvok2 may increase, and when the output of the lamp 30 has directionality, the shapes of the output cores 30nvok1 and 30nvok2 may also be set to have directionality.

[0110] The server 100 may set the output kernels 30nvok1 and 30nvok2 according to the output characteristics set as the default for the lamp 30, and, if necessary, correct the output kernels 30nvok1 and 30nvok2 using capability information provided by the manufacturer of the lamp 30 or information about the operating status received from the lamp 30. Furthermore, the server 100 may analyze images of the lamp 30 captured by other electronic devices (e.g., cameras) within the space to correct the output kernels 30nvok1 and 30nvok2.

[0111] The following describes a method in which the server 100 displays the output kernels 40nvok1 and 40nvok2 of the robot cleaner 40.

[0112] Since the robot cleaner 40 performs cleaning while moving within the space, the robot cleaner 40 may interfere with the field of view of the user 1 while watching the TV 10. Therefore, the server 100 may set and display an output kernel having a negative characteristic for the surrounding environment of the robot cleaner 40.

[0113] The output kernel of the robot cleaner 40 may be divided into a plurality of levels according to the degree to which the field of view of the user 1 is affected, and Figure 3 In the score map 300 in FIG. 1 , a first output kernel 40nvok1 and a second output kernel 40nvok2 are displayed for the robot cleaner 40 .

[0114] The output kernels 40nvok1 and 40nvok2 of the robot cleaner 40 may vary in shape and size depending on the characteristics or operating state of the robot cleaner 40. For example, as the moving speed of the robot cleaner 40 increases, the size of the output kernels 40nvok1 and 40nvok2 may increase, and the shape of the output kernels may be determined according to the moving direction of the robot cleaner 40.

[0115] The server 100 may set the output kernels 40nvok1 and 40nvok2 according to the default output characteristics set for the robot cleaner 40, and, if necessary, correct the output kernels 40nvok1 and 40nvok2 using capability information provided by the manufacturer of the robot cleaner 40 or information about the operating state (e.g., movement speed and movement direction) received from the robot cleaner 40. Furthermore, the server 100 may analyze images of the robot cleaner 40 captured by other electronic devices (e.g., cameras) within the space to correct the output kernels 40nvok1 and 40nvok2.

[0116] Available from Figure 3 It is recognized that no output kernel is set for the washing machine 50 among the electronic devices 10, 30, 40, and 50 displayed on the score map 300. As described above, Figure 3The score map 300 in FIG. 3 is a score map for the visual modality, and because the washing machine 50 is operated at a fixed location, the washing machine 50 does not have an output that affects the field of view of the user 1. Therefore, when the score map 300 for the visual modality is generated, the output kernel for the washing machine 50 may not be displayed. When the score map for the visual modality is generated, the output kernel for other electronic devices operating at a fixed location may not be displayed.

[0117] The server 100 may set and display input kernels 1vik1 and 1vik2 for the range of the area that can be visually recognized by the user 1. The input kernel of the user 1 may also be divided into a plurality of levels. Figure 3 In the score map 300 in FIG, the first input kernel 1vik1 and the second input kernel 1vik2 are displayed for user 1. Figure 3 Not shown in , but a higher level input kernel may be set in a preset appropriate recommended distance region.

[0118] 2-2) (First embodiment) Generating a score map for the auditory modality ( Figure 5 ).

[0119] This time, refer to Figure 5 The method by which the server 100 generates the score map 300 for the auditory modality is described.

[0120] The server 100 may display the locations of the user 1 and the electronic devices 10, 30, 40, and 50 on the score map 300, and may also display the input / output kernel for the auditory modality. The method by which the server 100 identifies the locations of the user 1 and the electronic devices 10, 30, 40, and 50 and displays the locations on the score map 500 has been described above.

[0121] When generating the score map 500 for the auditory modality, the score map 500 may display the output kernel for the electronic devices affecting the hearing of user 1 among the electronic devices 10, 30, 40, and 50, and display the input kernel for user 1 (the range of the area that the user can auditorily perceive).

[0122] A method in which the server 100 displays the output kernels 10paok1 and 10paok2 for the TV 10 is described below.

[0123] Since the sound output from the TV 10 is an output in which the user 1 is interested, an output kernel having positive characteristics may be set and displayed for the TV 10 .

[0124] Reference Figure 5 , the first output kernel 10paok1 is displayed until a certain distance from the TV 10, and the second output kernel 10paok2 is set for a certain area thereafter.

[0125] The shape and size of the output kernels 10paok1 and 10paok2 of the TV 10 vary depending on the characteristics or operating state of the TV 10, and the server 100 may correct the output kernels 10paok1 and 10paok2 of the TV 10 as described above in the section regarding the server 100 for the visual modality.

[0126] The server 100 may also display output kernels for the robot cleaner 40 and the washing machine 50, and the operating noises occurring from the robot cleaner 40 and the washing machine 50 are factors that interfere with watching the TV 10, so the server 100 may display output kernels having negative characteristics for the robot cleaner 40 and the washing machine 50.

[0127] The shapes and sizes of the output kernels 40naok1, 40naok2, 50naok1 and 50naok2 of the robot cleaner 40 and the washing machine 50 may vary depending on the characteristics or operating status of the robot cleaner 40 or the washing machine 50, and the server 100 may correct the output kernels 40naok1, 40naok2, 50naok1 and 50naok2 of the robot cleaner 40 and the washing machine 50, as described above in the section about the server 100 for visual modality.

[0128] According to an embodiment of the present disclosure, the server 100 may set an input / output core by reflecting the structure of the space where the electronic devices 10, 30, 40, and 50 are located. Figure 5 As shown in the fractional map 500 in FIG, the space where the washing machine 50 is located and the spaces where the other electronic devices 10, 30, and 40 are located are separated by walls 510 and 520. Because the walls 510 and 520 can block noise to a certain extent, the second output kernel 50naok2 of the washing machine 50 can be set to extend through the space between the walls 510 and 520.

[0129] Available from Figure 5 It is recognized that no output kernel is set for the lamp 30 among the electronic devices 10 , 30 , 40 , and 50 displayed on the score map 500 in FIG. Figure 5 The score map 500 in is a score map for the auditory modality, and because the lamp 30 does not output sound and does not cause operating noise, the output kernel for the lamp 30 may not be displayed when the score map 500 for the auditory modality is generated.

[0130] The server 100 may set and display input kernels 1aik1 and 1aik2 for the range of the area that can be auditorily recognized by the user 1. The input kernel for the user 1 may also be divided into a plurality of levels and displayed in Figure 5The first input kernel 1aik1 and the second input kernel 1aik2 are displayed for user 1 on the score map 500 in FIG.

[0131] 3) (First embodiment) Calculating scores for each modality based on a score map The server 100 can be based on Figure 3 The score for the visual modality is calculated by using the area of the input / output kernel shown on the score map 300. Figure 3 A method of calculating a score using the score map 300 in the example. The score calculated using the score map 300 may increase as the area of a region where the output kernel and the input kernel having positive characteristics overlap with each other, included in the score map 300, increases, and conversely, may decrease as the area of a region where the output kernel and the input kernel having negative characteristics overlap with each other increases.

[0132] As described above, the higher the rank of the input / output kernel, the greater the impact on the score. In the embodiments of the present disclosure, it is assumed that the score is calculated according to the rules described below. The same rules apply to the embodiments described below. However, the weights applied according to the rank of the input / output kernel in the following examples are merely examples and can be set differently in various ways.

[0133] - Assume that in a two-level input (output) kernel, the higher-level kernel is the first input (output) kernel, and the lower-level kernel is the second input (output) kernel.

[0134] The area of the region where the first input kernel and the first output kernel overlap each other is multiplied by a.

[0135] The area of the region where the first input kernel and the second output kernel overlap each other multiplied by b.

[0136] The area of the region where the second input kernel and the first output kernel overlap each other is multiplied by b.

[0137] The area of the region where the first input kernel and the first output kernel overlap each other is multiplied by c.

[0138] - a>b>c - A value obtained by multiplying the weight by the area of the region where the output kernel having positive characteristics and the input kernel overlap with each other is added, and a value obtained by multiplying the weight by the area of the region where the output kernel having negative characteristics and the input kernel overlap with each other is subtracted.

[0139] The following describes the process of calculating the score of the score map 300 according to the above rules.

[0140] ① A value obtained by multiplying the area of a region where the first input kernel 1vik1 of the user 1 and the first output kernel 10pvok1 of the TV 10 overlap each other by a is referred to as P1.

[0141] ② A value obtained by multiplying the area of a region where the first input kernel 1vik1 of the user 1 and the second output kernel 10pvok2 of the TV 10 overlap each other by b is referred to as P2.

[0142] ③ A value obtained by multiplying the area of a region where the second input kernel 1vik2 of the user 1 and the first output kernel 10pvok1 of the TV 10 overlap each other by b is referred to as P3.

[0143] ④ A value obtained by multiplying the area of a region where the second input kernel 1vik2 of the user 1 and the second output kernel 10pvok2 of the TV 10 overlap each other by c is referred to as P4.

[0144] ⑤ A value obtained by multiplying the area of a region where the first input kernel 1vik1 of the user 1 and the first output kernel 30nvok1 of the lamp 30 overlap each other by a is referred to as N1.

[0145] ⑥ A value obtained by multiplying the area of a region where the first input kernel 1vik1 of the user 1 and the second output kernel 30nvok2 of the lamp 30 overlap each other by b is referred to as N2.

[0146] ⑦ A value obtained by multiplying the area of a region where the second input kernel 1vik2 of the user 1 and the first output kernel 30nvok1 of the lamp 30 overlap each other by b is referred to as N3.

[0147] ⑧ A value obtained by multiplying the area of a region where the second input kernel 1vik2 of the user 1 and the second output kernel 30nvok2 of the lamp 30 overlap each other by c is referred to as N4.

[0148] 9. A value obtained by multiplying the area of a region where the first input kernel 1vik1 of the user 1 and the first output kernel 40nvok1 of the robot cleaner 40 overlap each other by a is referred to as N5.

[0149] ⑩ A value obtained by multiplying the area of a region where the first input kernel 1vik1 of the user 1 and the second output kernel 40nvok2 of the robot cleaner 40 overlap each other by b is referred to as N6.

[0150] A value obtained by multiplying the area of a region where the second input kernel 1vik2 of the user 1 and the first output kernel 40nvok1 of the robot cleaner 40 overlap each other by b is referred to as N7.

[0151] A value obtained by multiplying the area of a region where the second input kernel 1vik2 of the user 1 and the second input kernel 40nvok2 of the robot cleaner 40 overlap each other by c is referred to as N8.

[0152] In the above case, the score calculated by the score map 300, that is, the score for the visual modality, is (P1 + P2 + P3 + P4 - N1 - N2 - N3 - N4 - N5 - N6 - N7 - N8).

[0153] This score calculation method is merely an example, and the score may be calculated in various ways as long as the rules described below are maintained.

[0154] (Rule) The larger the area of the region where the output kernel with positive characteristics and the input kernel overlap with each other, the higher the score. Conversely, the larger the area of the region where the output kernel with negative characteristics and the input kernel overlap with each other, the lower the score.

[0155] In addition, according to an embodiment of the present disclosure, as described above, the server 100 may calculate the score by performing a convolution operation on the input kernel and the output kernel included in the score map 300. A detailed method is described below.

[0156] The first input kernel 1vik1 and the second input kernel 1vik2 of user 1 can be expressed as I u (i, j). Although Figure 3 Not shown in , but a higher level input kernel may be set in a preset appropriate recommended distance region.

[0157] The first output kernel 10pvok1 and the second output kernel 10pvok2 of the TV 10 may be represented as 0 t (i, j, a t ), and the kernel value may depend on the action a of TV 10 t (e.g., brightness of the image, type of content being reproduced, etc.)

[0158] The first output kernel 30nvok1 and the second output kernel 30nvok2 of the lamp 30 can be represented as 0 l (i, j, a l ), and at least one of the positions or kernel values of the display output kernels 30nvok1 and 30nvok2 may depend on the action a of the lamp 30 l (e.g., illumination, color, etc.).

[0159] The first output kernel 40nvok1 and the second output kernel 40nvok2 of the robot cleaner 40 may be represented as 0 r(i, j, a r ), and at least one of the positions or kernel values of the display output kernels 40nvok1 and 40nvok2 may depend on the motion a of the robot cleaner 40. r (e.g., moving direction and speed, low noise mode / turbo mode, etc.).

[0160] The server 100 may calculate the score S for the visual modality according to the following formula 3: v .

[0161] …Formula 3 The server 100 can be referred to above Figure 3 The method described by using Figure 5 The scores for the auditory modality are calculated using the score map 500 in FIG.

[0162] (3) (First embodiment) Controlling the operation of an electronic device based on the calculated score When based on Figure 3 The score map 300 in Calculate the score for the visual modality and according to Figure 5 When the score map 500 in FIG. 5 calculates the scores for the auditory modality, the score map 300 may control operations of the electronic devices 10 , 30 , 40 , and 50 based on the scores.

[0163] According to an embodiment of the present disclosure, the server 100 can increase the Figure 3 The electronic devices 10, 30, 40, and 50 are controlled based on the scores for the visual modalities calculated by the score map 300 in the server 100. For example, the server 100 may reduce the brightness of the lamp 30 to reduce the area of the lamp 30's output kernels 30nvok1 and 30nvok2 that overlap with the input kernels 1vik1 and 1vik2 of the user 1. As a result, the interference caused by the light of the lamp 30 is reduced, and the user 1 can view the image of the TV 10 more clearly.

[0164] According to an embodiment of the present disclosure, the server 100 may Figure 3 The situation shown in the score map 300 in FIG. 1 predicts the score after a certain time, and controls the electronic devices 10, 30, 40, and 50 in a manner to increase the expected score. For example, in Figure 3In the score map 300 in FIG, the robot cleaner 40 is moving in the viewing direction of the user 1. After a certain period of time, the area of the region where the user 1's input kernels 1vik1 and 1vik2 overlap with the robot cleaner 40's output kernels 40nvok1 and 40nvok2 further increases, and thus the score (expected score after a certain period of time) may decrease. Therefore, the server 100 may change the direction of the robot cleaner 40's movement away from the user 1's field of view to increase the expected score after a certain period of time.

[0165] According to an embodiment of the present disclosure, the server 100 may control the operation of the electronic devices 10, 30, 40, and 50 through the cost function, as described above. When the score S for the visual modality is calculated by the above formula 3 v When , the server 100 can determine the actions of the electronic devices 10, 30, 40 and 50 within the target time range T by using the following formula 4.

[0166] …Formula 4 Figure 4 Shown in Figure 3 The score map 400 is a case where the server 100 controls the electronic devices 10, 30, 40, and 50 in the case of the score map 300. Figure 3 Compared to score map 300 in

[15] , score map 400 shows that the sizes of output kernels 30nvok1 and 30nvok2 have become smaller. Furthermore, the movement direction of robotic cleaner 40 has been changed to move away from user 1's input kernels 1vik1 and 1vik2. This control increases the score for the visual modality, and user 1 is less disturbed by the light from lamp 30 or the movement of robotic cleaner 40. Therefore, user 1's experience is optimized in the context of "watching TV."

[0167] According to an embodiment of the present disclosure, the server 100 can increase the Figure 5The server 100 controls the electronic devices 10, 30, 40, and 50 based on the scores for the auditory modality calculated by the score map 500 in FIG. For example, the server 100 may increase the volume of the TV 10 to increase the area of overlap between the TV 10's output kernels 10paok1 and 10paok2 and the user 1's input kernels 1aik1 and 1aik2. As a result, the user 1 can more clearly hear the sound output from the TV 10. Furthermore, for example, the server 100 may reduce the operating noise from the robot cleaner 40 and washing machine 50 (switching them to low-noise mode or pausing them) to reduce the area of overlap between the output kernels 40naok1, 40naok2, 50naok1, and 50naok2 of the two electronic devices 40 and 50 and the user 1's input kernels 1aik1 and 1aik2. As a result, the user 1 is less disturbed by the noise from the electronic devices 40 and 50 and can focus more on the sound from the TV 10.

[0168] According to an embodiment of the present disclosure, the server 100 may calculate the score S for the visual modality by v The cost function is used to arrange the actions of the electronic devices 10, 30, 40 and 50 in the same way to calculate the score S for the auditory modality. v , and controls the operations of the electronic devices 10 , 30 , 40 , and 50 based on the scores.

[0169] Figure 6 Shown in Figure 5 The score map 600 is a case where the server 100 controls the electronic devices 10, 30, 40, and 50 in the case of the score map 500. Figure 5 The score map in 500 is compared to that in Figure 6 In score map 600, it can be seen that the size of TV output kernels 10paok1 and 10paok2, which are output kernels with positive characteristics, has increased. On the other hand, it can be seen that the size of robot cleaner 40's output kernels 40naok1 and 40naok2, as well as washing machine 50's output kernels 50naok1 and 50naok2, which are output kernels with negative characteristics, has decreased. This control increases the score for the auditory modality, and user 1 is less disturbed by the operating noise of robot cleaner 40 or washing machine 50. Therefore, user 1's experience in the context of "watching TV" is optimized.

[0170] 3. Description of the second embodiment (when the context is "communication with an AI speaker") (1) (Second embodiment) Determining the user's context First, the server 100 may determine the context of the user 1. The server 100 may determine the context of the user 1 according to various methods, and Figure 7One embodiment thereof is shown.

[0171] Reference Figure 7 , the camera 300 may photograph the user 1 and the AI speaker 20 and transmit the captured image to the server 100. The server 100 may analyze the received captured image to identify the positions of the user 1 and the AI speaker 20 and the movement of the user 1.

[0172] exist Figure 7 In the illustrated embodiment, when the distance d2 between user 1 and the AI speaker 20 is a preset reference value (e.g., 1 meter) or less and the facing direction of user 1 is toward the AI speaker 20 (e.g., when the angle formed by the facing direction of user 1 and the straight line connecting user 1 and the AI speaker 20 is a preset reference value or less), the server 100 may determine that the context of user 1 is “communicating with the AI speaker”.

[0173] According to an embodiment of the present disclosure, the server 100 may receive information about the operating state (eg, received voice input) from the AI speaker 20 and determine the context of the user 1 by considering the information together with the captured image received from the camera 300 .

[0174] According to an embodiment of the present disclosure, the server 100 may apply the captured image received from the camera 300 as input to a pre-trained neural network, and determine the inference result of the neural network as the context of user 1.

[0175] (2) (Second embodiment) Calculating scores based on context The server 100 may generate a score map according to the determined context (communication with the AI speaker) and calculate a score by using the score map.

[0176] 1) (Second embodiment) Select at least one modality based on context Because the determined context is "communication with an AI speaker," the server 100 may select the auditory modality. Information indicating that the modality corresponding to "communication with an AI speaker" is the auditory modality may be pre-stored in the server 100. The server 100 may calculate a score for the selected modality.

[0177] 2) (Second embodiment) Generate a score map for each selected modality The server 100 may generate a score map for the selected auditory modality. In the case where the context is "communication with the AI speaker", when user 1 outputs a voice, the AI speaker 20 receives the output sound, and conversely, when the AI speaker 20 outputs a voice, the user 1 hears the output sound. Therefore, there may be input kernels corresponding to user 1 and AI speaker 20, respectively. For ease of description, a score map indicating the input kernel of user 1 and a score map indicating the input kernel of AI speaker 20 are shown separately. Figure 8 Score map 800 and Figure 9 In the score map 900 in FIG, the input kernels 1aik1 and 1aik2 of user 1 are shown, and Figure 10 The fraction map 1000 and Figure 11 The input kernels 20aik1 and 20aik2 are shown in the score map 1100.

[0178] Figure 8 The score map shown in 800 and Figure 10 The score map 1000 shown in FIG. 1 is a score map for a situation before the server 100 controls the electronic devices 10, 20, 30, and 40, and Figure 9 The score map shown in 900 and Figure 11 The score map 1100 shown in FIG. 1 is a score map for a case after the server 100 controls the electronic devices 10 , 20 , 30 , and 40 .

[0179] Reference Figure 8 and Figure 10 The method by which the server 100 generates score maps 800 and 1000 for the auditory modality is described.

[0180] The server 100 may display the locations of the user 1 and the electronic devices 10, 20, 30, and 40 on the score maps 800 and 1000, and may also display the input / output kernel for the auditory modality. The method by which the server 100 identifies the locations of the electronic devices 10, 20, 30, and 40 is as described above (see position measurement by analyzing captured images or wireless communication between electronic devices).

[0181] Reference Figure 8 , the server 100 can display the output kernels of the electronic devices that affect the hearing of the user 1 among the electronic devices 10, 20, 30 and 40 on the score map 800, and display the input kernel for the user 1 (the range of the area that can be perceived by the user's hearing).

[0182] Because the speech output from the AI speaker 20 is of interest to the user 1, the output kernels 20paok1 and 20paok2 having positive characteristics are displayed for the AI speaker 20. The output kernels 20paok1 and 20paok2 of the AI speaker 20 are divided into two levels, with the first output kernel 20paok1 being a region having a greater impact on the hearing of the user 1 than the second output kernel 20paok2, and this difference is reflected in the calculation of the score.

[0183] The output kernels 20paok1 and 20paok2 of the AI speaker 20 may increase in size as the volume increases and extend longer in the direction of the output voice. The server 100 may set the output kernels 20paok1 and 20paok2 according to the default output characteristics set for the AI speaker 20 and, when necessary, correct the output kernels 20paok1 and 20paok2 using capability information provided by the manufacturer of the AI speaker 20 or information about the operating status (e.g., volume) received from the AI speaker 20. In addition, the server 100 may correct the output kernels 20paok1 and 20paok2 based on information about the volume and voice output direction of the AI speaker 20 measured by other electronic devices in the space (e.g., smartphones).

[0184] When user 1 communicates with the AI speaker 20, the voice output from the TV 10 may be disruptive. Therefore, an output kernel having negative characteristics may be set and displayed for the TV 10. The shape and size of the output kernels 10naok1 and 10naok2 of the TV 10 may also be determined according to the output characteristics of the TV 10.

[0185] Because the operating noise of the robot cleaner 40 may also be disruptive, output kernels 40naok1 and 40naok2 having negative characteristics may be displayed for the robot cleaner 40 .

[0186] Figure 8 The score map 800 in is a score map for an auditory modality, and since the lamp 30 does not output sound and does not cause operating noise, the output kernel for the lamp 30 may not be displayed on the score map 800 .

[0187] The server 100 may display input kernels 1aik1 and 1aik2 for the range of an area that can be auditorily recognized by the user 1 .

[0188] Reference Figure 10 , the server 100 may display the output kernel for the electronic device affecting the voice input of the AI speaker 20, and may also display the output kernel for user 1.

[0189] Because the voice command of user 1 is an output of interest to the AI speaker 20 , output kernels 1paok1 and 1paok2 having positive characteristics may be displayed for user 1 .

[0190] However, because the sound of the TV 10 or the operating noise of the robot cleaner 40 interferes with the AI speaker 20 recognizing the voice command, output kernels 10naok1 , 10naok2 , 40naok1 , and 40naok2 having negative characteristics are displayed for the TV 10 and the robot cleaner 40 .

[0191] The sizes of the output kernels 1paok1 and 1paok2 of user 1 may be determined according to the size of the voice of user 1 recognized by the AI speaker 20 or other nearby electronic devices.

[0192] As in Figure 8 In the score map 800 , shapes and sizes of the output kernels 10naok1 and 10naok2 of the TV 10 and the output kernels 40naok1 and 40naok2 of the robot cleaner 40 may be determined according to output characteristics of the TV 10 and the robot cleaner 40 and may be corrected by the score map 800 when necessary.

[0193] Figure 10 The score map 1000 in is a score map for an auditory modality, and since the lamp 30 does not output sound and does not cause operating noise, the output kernel for the lamp 30 may be not displayed on the score map 1000 .

[0194] 3) (Second embodiment) Calculating scores for each modality based on a score map The server 100 can be based on Figure 8 Score map 800 and Figure 10 The score for the auditory modality is calculated by taking the area of the input / output kernel shown on the score map 1000. Figure 3 A specific method of calculating the score by using the score map is described, so a detailed description thereof is omitted below.

[0195] According to an embodiment of the present disclosure, Figure 8In the score map 800 in , the score is the result of adding a value obtained by multiplying the area of the region where the output kernels 20paok1 and 20paok2 of the AI speaker 20 and the input kernels 1aik1 and 1aik2 of the user 1 respectively overlap with each other by a certain weight, and conversely subtracting a value obtained by multiplying the area of the region where the output kernels 10naok1 and 10naok2 of the TV 10 and the input kernels 1aik1 and 1aik2 of the user 1 respectively overlap with each other by a certain weight, and subtracting a value obtained by multiplying the area of the region where the output kernels 40naok1 and 40naok2 of the robot cleaner 40 and the input kernels 1aik1 and 1aik2 of the user 1 respectively overlap with each other by a certain weight.

[0196] Similarly, in Figure 10 In the score map 1000 in , the score is the result of adding a value obtained by multiplying a certain weight by the area of the region where the output kernels 1paok1 and 1paok2 of the user 1 and the input kernels 20aik1 and 20aik2 of the AI speaker 20 respectively overlap with each other, and conversely subtracting a value obtained by multiplying a certain weight by the area of the region where the output kernels 10naok1 and 10naok2 of the TV 10 and the input kernels 20aik1 and 20aik2 of the AI speaker 20 respectively overlap with each other, and subtracting a value obtained by multiplying a certain weight by the area of the region where the output kernels 40naok1 and 40naok2 of the robot cleaner 40 and the input kernels 20aik1 and 20aik2 of the AI speaker 20 respectively overlap with each other.

[0197] The server 100 can Figure 8 The scores in the score map 800 are calculated by using Figure 10 The scores calculated in the score map 1000 are summed to obtain a score for the auditory modality.

[0198] (3) (Second embodiment) Controlling the operation of an electronic device based on the calculated score The server 100 may control the electronic devices 10 , 20 , 30 , and 40 in such a manner as to increase the scores for the auditory modality.

[0199] According to an embodiment of the present disclosure, the server 100 may control the electronic device so that the volume of the AI speaker 20 is increased and the voice or operating noise output from the robot cleaner 40 is reduced. In addition, when the robot cleaner 40 is moving toward the user 1 or the AI speaker 20, the server 100 may control the robot cleaner 40 to move in a direction away from the user 1 or the AI speaker 20. Because the light 30 does not affect the auditory modality, the server 100 may not control the operation of the light 30.

[0200] The following describes the influence of the above-mentioned control of the server 100 on the score.

[0201] and Figure 8 In comparison, Figure 9 , it can be understood that the size of the output kernels 20paok1 and 20paok2 of the AI speaker 20 has increased, and the size of the output kernels 10naok1 and 10naok2 of the TV 10 and the output kernels 40naok1 and 40naok2 of the robot cleaner 40 have decreased. In addition, due to the change in the moving direction of the robot cleaner 40, the area of the region where the output kernels 40naok1 and 40naok2 of the robot cleaner 40 overlap with the input kernels 1aik1 and 1aik2 of the user 1 may decrease. Therefore, according to Figure 9 The score calculated by the score map 900 in the example above can be higher than that calculated by Figure 8 The scores in the score map 800 are calculated.

[0202] Reference Figure 10 and Figure 11 , it can be understood that the sizes of the output kernels 10naok1 and 10naok2 of the TV 10 and the output kernels 40naok1 and 40naok2 of the robot cleaner 40 are reduced due to the control of the server 100. In addition, due to the change in the moving direction of the robot cleaner 40, the area of the region where the output kernels 40naok1 and 40naok2 of the robot cleaner 40 overlap with the input kernels 20aik1 and 20aik2 of the AI speaker 20 may be reduced. Figure 11 The score calculated by the score map 1100 in the example above may be higher than that calculated by the example Figure 10 The scores in the score map are 1000 calculated scores.

[0203] Due to this control, the score for the auditory modality increases and the user 1 is less disturbed by the noise due to the TV 10 or the robot cleaner 40, and thus the experience of the user 1 can be optimized in the context of “communicating with the AI speaker”.

[0204] Hereinafter, a configuration of a computing device (server 100 ) for executing an embodiment of the present disclosure is described with reference to the accompanying drawings, and then a method of controlling a plurality of electronic devices according to a user's context according to an embodiment of the present disclosure is described. Figure 12 The server 100 shown in FIG. 1 corresponds to a computing device for executing the above-described embodiments, and Figures 13 to 16 A flowchart summarizing the above-described embodiment in time series is shown. Therefore, even when omitted below, the above-described features can be similarly applied to the embodiment described below.

[0205] Figure 121 is a diagram showing a detailed configuration of a server according to an embodiment of the present disclosure. Figure 12 , the server 100 according to an embodiment of the present disclosure may include a communication interface 110, an input / output interface 120, a memory 130, and a processor 140. However, the elements of the server 100 are not limited to the above examples, and the server 100 may include more elements or fewer components than the above elements. In an embodiment of the present disclosure, some or all of the communication interface 110, the input / output interface 120, the memory 130, and the processor 140 may be implemented in the form of one chip, and the processor 140 may include one or more processors.

[0206] The communication interface 110 is a device for transmitting signals (control commands, data, etc.) to an external device and receiving signals (control commands, data, etc.) from an external electronic device via a cable or wirelessly, and may be configured to include a communication chipset that supports various communication protocols. The communication interface 110 may receive signals from the outside and output the signals to the processor 140, or may transmit signals output from the processor 140 to the outside.

[0207] The input / output interface 120 may include an input interface (e.g., a touch screen, hard buttons, a microphone, etc.) for receiving control commands or information from a user, and an output interface (e.g., a display panel, a speaker, etc.) for displaying the execution result of an operation or the status of the server 100 under the control of the user.

[0208] The memory 130 is a configuration for storing various programs or data and may include a storage medium such as a read-only memory (ROM), a random access memory (RAM), a hard disk, a compact disc ROM (CD-ROM), or a digital versatile disc (DVD), or a combination thereof. The memory 130 may not exist independently and may be included in the processor 140. The memory 130 may include a volatile memory, a nonvolatile memory, or a combination of volatile and nonvolatile memories. Programs for performing operations according to embodiments of the present disclosure may be stored in the memory 130. The memory 130 may provide the stored data to the processor 140 in response to a request from the processor 140.

[0209] The processor 140 is a configuration for controlling a series of processes for the server 100 to operate according to the embodiments of the present disclosure, and may include one or more processors. The one or more processors may include a general-purpose processor (such as a central processing unit (CPU), an application processor (AP), a digital signal processor (DSP), etc.), a dedicated graphics processor (such as a graphics processing unit (GP), a visual processing unit (VPU), etc.), or a dedicated AI processor (such as a neural processing unit (NPU)). For example, one or more processors are dedicated AI processors that can be designed as a hardware structure dedicated to processing a specific AI model. The processor 140 is communicatively coupled to the communication interface 110 and the memory 130.

[0210] The processor 140 may record data to the memory 130 or read data stored in the memory 130, and specifically, execute the program stored in the memory 130 to process the data according to predefined operating rules or AI models. Therefore, the processor 140 may perform the operations described in the embodiments of the present disclosure, and it can be seen that, unless otherwise specified, the operations described as being performed by the server 100 in the embodiments of the present disclosure are performed by the processor 140.

[0211] Figure 13 、 Figure 14 、 Figure 15 and Figure 16 is a flowchart illustrating a method of controlling multiple electronic devices according to a user's context according to various embodiments of the present disclosure. Figures 13 to 16 A method of controlling multiple electronic devices according to a user's context according to an embodiment of the present disclosure is described. Since the operations described below are performed by the server 100 described so far, the contents included in the above embodiments may be similarly applied even when omitted below.

[0212] Reference Figure 13 , in operation 1301 , the server 100 may determine a context of a user. Figure 14 The flowchart in includes detailed operations included in operation 1301. Figure 14 , the server 100 may obtain a captured image of a user and at least one electronic device among a plurality of electronic devices in operation 1401, and analyze the image and determine a context of the user in operation 1402. In addition, the server 100 may use information on an operating state of the electronic device when necessary and determine the context based on an input of the user.

[0213] In operation 1302 , the server 100 may select at least one modality to be considered when controlling the plurality of electronic devices based on the determined context. To this end, information about the corresponding modality for each context may be previously stored in the memory 130 of the server 100 .

[0214] In operation 1303, the server 100 may generate a score map indicating the areas affected by the input and output of multiple electronic devices for each selected modality. The server 100 may reflect the context, the selected modality, and the input / output characteristics of the electronic device and display the input / output kernel on the map to generate the score map. On the score map, an input kernel or an output kernel for at least one of the user and the multiple electronic devices may be displayed. Here, the input kernel may indicate the range of the area affected by the input of the user or the electronic device, and the output kernel may indicate the range of the area affected by the output of the user or the electronic device. In addition, according to an embodiment of the present disclosure, the score map may include multiple layers, and the location of at least one electronic device may be pre-displayed in some layers.

[0215] The flowcharts of the detailed operations included in operation 1303 are respectively in Figure 15 and Figure 16 Shown in.

[0216] Reference Figure 15 In operation 1501, the server 100 may display an input kernel or an output kernel based on basic input / output characteristics preset for each of a plurality of electronic devices. To this end, information about the corresponding basic input / output characteristics for each type of electronic device may be pre-stored in the memory 130 of the server 100.

[0217] In operation 1502, the server 100 may correct an input kernel or an output kernel by using capability information about a plurality of electronic devices or information about operating states of the plurality of electronic devices. According to an embodiment of the present disclosure, the server 100 may obtain capability information or information about operating states from the electronic devices and change at least one of a size or a shape of the input kernel or the output kernel based on the capability information or information about operating states.

[0218] Reference Figure 16 In operation 1601, the server 100 may divide a space where a plurality of electronic devices are located into a plurality of areas. In operation 1602, the server 100 may map each of the plurality of electronic devices to at least one area of the plurality of areas. In operation 1603, the server 100 may generate a score map by reflecting the input / output characteristics of each of the plurality of electronic devices in the at least one mapped area.

[0219] Refer again Figure 13In operation 1304, the server 100 may calculate a score for each modality using a score map. According to an embodiment of the present disclosure, the server 100 may calculate the score based on the characteristics of the output kernel and the area of the region where the input kernel and the output kernel overlap, as displayed on the score map. As described above, the server 100 may calculate the score by adding the area of the region where the output kernel and the input kernel with positive characteristics overlap and subtracting the area of the region where the output kernel and the input kernel with negative characteristics overlap, and in this case, the weight may be reflected depending on the level of the input / output kernel. In addition, according to an embodiment of the present disclosure, the server 100 may calculate the score by performing a convolution operation on the input kernel and the output kernel included in the score map.

[0220] In operation 1305, the server 100 may control the operation of the plurality of electronic devices based on the calculated scores. According to an embodiment of the present disclosure, the server 100 may control the operation of at least one electronic device among the plurality of electronic devices so that the score corresponding to the score map increases. Optionally, according to an embodiment of the present disclosure, the server 100 may predict the score after a certain period of time, and control the operation of the electronic device in a manner to increase the predicted score. Optionally, according to an embodiment of the present disclosure, the server 100 may identify the operation of the electronic device to maximize the score by applying an iterative optimization technique to the score map for each modality, establish an optimal operation plan, and control the electronic device to operate according to the plan (for example, designing a cost function whose value decreases as the score calculated according to the score map increases, and controlling the electronic device so that the value of the cost function is minimized).

[0221] According to the above-described embodiments, a plurality of electronic devices are controlled based on a context determined according to a user's intention, so that an effect of optimizing user experience in various situations can be expected.

[0222] According to an embodiment of the present disclosure, a method for controlling multiple electronic devices based on a user's context may include: determining the user's context by a computing device, selecting by the computing device at least one modality to be considered when controlling the multiple electronic devices based on the determined context, obtaining by the computing device a score map indicating areas affected by inputs and outputs of the multiple electronic devices for each selected modality, calculating by the computing device a score for each selected modality by using the obtained score map, and controlling the operations of the multiple electronic devices based on the calculated scores, wherein the scores may include a value indicating a degree of optimization of the user experience in the determined context.

[0223] According to an embodiment of the present disclosure, an input kernel or an output kernel for a user and at least one electronic device among a plurality of electronic devices may be displayed on a score map, and the input kernel may indicate the range of an area affected by the input of the user or the electronic device, and the output kernel may indicate the range of an area affected by the output of the user or the electronic device.

[0224] According to an embodiment of the present disclosure, obtaining a score map may include: displaying an input kernel or an output kernel based on basic input and output characteristics preset for each of a plurality of electronic devices, and correcting the input kernel or the output kernel by using capability information about the plurality of electronic devices or information about operating states of the plurality of electronic devices.

[0225] According to an embodiment of the present disclosure, correcting the input kernel or the output kernel may include: obtaining capability information or information about operating states from a plurality of electronic devices, and changing at least one of a shape or a size of the input kernel or the output kernel based on the capability information or the information about operating states.

[0226] According to an embodiment of the present disclosure, obtaining a score map may include: dividing a space where multiple electronic devices are located into multiple areas, mapping each of the multiple electronic devices to at least one of the multiple areas, and reflecting input and output characteristics of each of the multiple electronic devices in the at least one mapped area.

[0227] According to an embodiment of the present disclosure, calculating the score may include calculating the score based on characteristics of the output kernel and an area of a region where the input kernel and the output kernel overlap each other displayed on the score map.

[0228] According to an embodiment of the present disclosure, the controlling may include controlling an operation of at least one electronic device among the plurality of electronic devices so that a score corresponding to the score map increases.

[0229] According to an embodiment of the present disclosure, the score map may include a plurality of layers, and the location of at least one electronic device is pre-displayed in some layers of the plurality of layers.

[0230] According to an embodiment of the present disclosure, determining the context of the user may include obtaining a captured image of the user and at least one electronic device among a plurality of electronic devices, and determining the context of the user by analyzing the image.

[0231] According to an embodiment of the present disclosure, a computing device may include: a communication interface configured to perform communication with multiple electronic devices; a memory configured to store a program for controlling multiple electronic devices according to a user's context; and at least one processor, wherein the at least one processor is configured to execute the program to determine the user's context, select at least one modality to be considered when controlling multiple electronic devices based on the determined context, obtain a score map indicating areas affected by inputs and outputs of the multiple electronic devices for each selected modality, and calculate a score for each selected modality by using the obtained score map, and then control the operation of the multiple electronic devices based on the calculated score, and the score may be a value indicating the degree of optimization of the user experience in the determined context.

[0232] According to an embodiment of the present disclosure, an input kernel or an output kernel for a user and at least one electronic device among a plurality of electronic devices may be displayed on a score map, and the input kernel may indicate the range of an area affected by the input of the user or the electronic device, and the output kernel may indicate the range of an area affected by the output of the user or the electronic device.

[0233] According to an embodiment of the present disclosure, when generating a score map, at least one processor may be configured to: display an input kernel or an output kernel based on basic input and output characteristics preset for each of a plurality of electronic devices, and then correct the input kernel or the output kernel by using capability information about the plurality of electronic devices or information about the operating states of the plurality of electronic devices.

[0234] According to an embodiment of the present disclosure, when correcting an input kernel or an output kernel, at least one processor may be configured to: obtain capability information or information about operating states from a plurality of electronic devices, and then change at least one of a size or a shape of the input kernel or the output kernel based on the capability information or the information about the operating states.

[0235] According to an embodiment of the present disclosure, when generating a score map, at least one processor may also be configured to: divide the space where multiple electronic devices are located into multiple areas, and map each of the multiple electronic devices to at least one of the multiple areas, and then reflect the input and output characteristics of each of the multiple electronic devices in the at least one mapped area.

[0236] According to an embodiment of the present disclosure, when calculating the score, the at least one processor may be configured to calculate the score based on characteristics of the output kernel and an area of a region where the input kernel and the output kernel overlap each other displayed on the score map.

[0237] According to an embodiment of the present disclosure, when controlling operations of a plurality of electronic devices, at least one processor is configured to control the operation of at least one electronic device among the plurality of electronic devices so that a score corresponding to the score map increases.

[0238] According to an embodiment of the present disclosure, the score map may include a plurality of layers, and the location of at least one electronic device is pre-displayed in some layers of the plurality of layers.

[0239] According to an embodiment of the present disclosure, when determining the context of a user, at least one processor may be configured to obtain a captured image of the user and at least one electronic device among a plurality of electronic devices, and then determine the context of the user by analyzing the image.

[0240] Various embodiments of the present disclosure may be implemented or supported by one or more computer programs, and the computer programs may be formed according to computer-readable program code and stored in a computer-readable medium. In the present disclosure, "application" and "program" may refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, related data, or a portion thereof that are suitable for implementation in computer-readable program code. "Computer-readable program code" may include various types of computer code, including source code, object code, and executable code. "Computer-readable medium" may include various types of media that can be accessed by a computer, such as ROM, RAM, hard disk drive (HDD), compact disk (CD), digital video disk (DVD), or various types of memory.

[0241] In addition, the machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, a "non-transitory storage medium" is a tangible device and may exclude wired, wireless, optical or other communication links that transmit transient electrical signals or other signals. In addition, such a "non-transitory storage medium" does not distinguish between the case where data is semi-permanently stored in the storage medium and the case where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer that temporarily stores data. Computer-readable media can be any available media that can be accessed by a computer and may include all volatile and non-volatile media and detachable and non-detachable media. Computer-readable media may include media on which data can be permanently stored and media on which data can be stored and later rewritten, such as a rewritable optical disc or an erasable memory device.

[0242] It will be understood that various embodiments of the present disclosure according to the claims and description in the specification can be realized in the form of hardware, software, or a combination of hardware and software.

[0243] Any such software may be stored in a non-transitory computer-readable storage medium that stores one or more computer programs (software modules) that include instructions that, when executed by one or more processors in an electronic device, cause the electronic device to perform the methods of the present disclosure.

[0244] According to an embodiment of the present disclosure, the methods according to the various embodiments disclosed in the present disclosure can be provided by being included in a computer program product. A computer program product is a product that can be traded between a seller and a buyer. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., a CD-ROM), or distributed through an application store (e.g., downloaded or uploaded), or distributed directly or online between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable application) can be at least temporarily generated or temporarily stored in a machine-readable storage medium, such as a memory of a manufacturer's server, an application store's server, or a relay server.

[0245] The above description of the present disclosure is provided for illustration, and it will be understood by those skilled in the art that various changes in form and detail may be readily made therein without departing from the essential features and scope of the present disclosure as defined by the appended claims. For example, appropriate results may be achieved even when the described techniques are performed in a different order than the described methods, or when elements (such as systems, structures, devices, apparatuses, circuits, etc.) described using "and / or" are combined differently than the described methods or replaced with other elements or equivalents. Therefore, the embodiments described above are examples in all respects and are not limiting. For example, each element described as a single type may be implemented in a distributed manner, and similarly, elements described as distributed may be implemented in combination.

[0246] While the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. A method for controlling multiple electronic devices according to a user's context, the method comprising: determining, by a computing device, a context of a user; selecting, by the computing device, at least one modality to be considered when controlling the plurality of electronic devices based on the determined context; obtaining, by the computing device, for each selected modality, a score map indicating areas affected by inputs and outputs of the plurality of electronic devices; calculating, by the calculation device, a score for each selected modality by using the obtained score map; as well as controlling, by the computing device, operations of the plurality of electronic devices based on the calculated scores, The score is a value indicating the degree of optimization of the user experience in a certain context.

2. The method according to claim 1, in, An input kernel or an output kernel for a user and at least one of the plurality of electronic devices is displayed on the score map, The input kernel indicates the range of the area affected by the input of the user or electronic device, and The output kernel indicates the range of an area affected by the output of a user or an electronic device.

3. The method according to any one of claims 1 and 2, wherein The obtaining of the score map includes: displaying an input kernel or an output kernel based on basic input and output characteristics preset for each of the plurality of electronic devices; and An input kernel or an output kernel is corrected by using capability information about the plurality of electronic devices or information about operation states of the plurality of electronic devices.

4. The method according to any one of claims 1 to 3, wherein The correction input kernel or output kernel comprises: obtaining the capability information or the information on the operating status from the plurality of electronic devices; and At least one of a shape or a size of an input kernel or an output kernel is changed based on the capability information or the information about the operating state.

5. The method according to any one of claims 1 to 4, wherein The obtaining of the score map includes: Dividing the space where the multiple electronic devices are located into multiple areas; mapping each of the plurality of electronic devices to at least one of the plurality of regions; and Input and output characteristics of each of the plurality of electronic devices are reflected in at least one mapped region.

6. The method according to any one of claims 1 to 5, in, The obtaining of the score map comprises: extracting the score map corresponding to the selected at least one modality from the entire score map, and The entire score map is a map showing all areas affected by inputs and outputs of the multiple electronic devices for multiple modalities.

7. The method according to any one of claims 1 to 6, wherein Calculating the score includes calculating the score based on a characteristic of the output kernel and an area of a region where the input kernel and the output kernel overlap each other, as displayed on the score map.

8. The method according to any one of claims 1 to 7, wherein The controlling includes controlling an operation of at least one electronic device among the plurality of electronic devices so that a score corresponding to the score map increases.

9. The method according to any one of claims 1 to 8, in, The score map includes a plurality of layers, and The location of at least one electronic device is pre-displayed in some of the multiple layers.

10. The method according to any one of claims 1 to 9, wherein Determining the user's context includes: obtaining a captured image of the user and at least one electronic device among the plurality of electronic devices; and The user's context is determined by analyzing the image. 11 . A non-transitory computer-readable recording medium having recorded thereon a program for executing the method according to claim 1 on a computer.

12. A computing device comprising: a communication interface 110 configured to perform communications with a plurality of electronic devices; Memory 130; as well as One or more processors 140, communicatively coupled to the communication interface 110 and the memory 130, The memory 130 stores one or more computer programs including computer-executable instructions, which, when executed by the one or more processors 140, cause the computing device 100 to perform the following operations: executing said program to determine the context of the user, selecting, based on the determined context, at least one modality to be considered when controlling the plurality of electronic devices, obtaining, for each selected modality, a score map indicating areas affected by inputs and outputs of the plurality of electronic devices, calculating a score for each selected modality by using the obtained score map, and controlling operations of the plurality of electronic devices based on the calculated scores, and The score is a value indicating the degree of optimization of the user experience in a certain context.

13. The computing device of claim 12, wherein: An input kernel or an output kernel for a user and at least one electronic device among the plurality of electronic devices is displayed on the score map, and The input kernel indicates the range of an area affected by an input of a user or an electronic device, and the output kernel indicates the range of an area affected by an output of a user or an electronic device.

14. The computing device of any one of claims 12 and 13, wherein: When obtaining the score map, the one or more computer programs further include computer executable instructions that, when executed by the one or more processors 140 , cause the computing device 100 to perform the following operations: displaying an input kernel or an output kernel based on basic input and output characteristics preset for each of the plurality of electronic devices, and An input kernel or an output kernel is corrected by using capability information about the plurality of electronic devices or information about operation states of the plurality of electronic devices.

15. The computing device according to any one of claims 12 to 14, wherein: When correcting the input kernel or the output kernel, the one or more computer programs further include computer executable instructions that, when executed by the one or more processors 140, cause the computing device 100 to: obtaining the capability information or the information on the operating status from the plurality of electronic devices, and At least one of a size or a shape of an input kernel or an output kernel is changed based on the capability information or the information about the operating state.

Images