Electronic device for identifying external device, and operating method thereof

Abstract

This electronic device comprises a connection unit and a processor, wherein the processor is configured to: identify that a first external device is physically connected to the electronic device through the connection unit; identify connection information of the first external device, the identified connection information including device information related to the first external device, image information received from the first external device, and information related to a connection environment of the first external device; determine whether the first external device is a controllable device on the basis of the identified connection information of the first external device and at least one machine learning model for recognizing a plurality of external devices; identify at least one control method of the first external device in response to determining that the first external device is a controllable device; and determine one of the at least one control method as a priority control method.

Description

Electronic device and method of operation for identifying an external device

[0001] Embodiments disclosed in this document relate to an electronic device and an operating method for identifying an external device.

[0002] The display device can be connected to various input devices (e.g., a set-top box, a personal computer (PC), or a mobile device) and display a screen based on data received from the input device. In the case of an HDMI (high definition multimedia interface) device, the display device can be connected to the display device through at least one HDMI port provided on the display device. The display device can recognize the connected HDMI device and display a screen based on data received from the HDMI device.

[0003] As the types of external devices connected to display devices diversify, there's a need to improve the success rate of external device recognition. Furthermore, external devices can be controlled via one or more control methods, and the priority between control methods needs to be adaptively determined based on the device type and connection environment (e.g., network status, HDMI cable information, etc.).

[0004] Based on the discussion described above, the present disclosure provides a device and method for accurately recognizing an external device connected to an electronic device.

[0005] In addition, the present disclosure provides a device and method for determining a control method to be adaptively applied first according to a situation when controlling an external device that provides multiple control methods.

[0006] An electronic device according to one embodiment of the present disclosure includes a connection unit; and a processor, wherein the processor identifies that a first external device is physically connected to the electronic device through the connection unit, identifies connection information of the first external device, and the identified connection information includes device information about the first external device, image information from the first external device, and information about a connection environment of the first external device, and is configured to determine whether the first external device is a controllable device based on the identified connection information of the first external device and at least one machine learning model for recognizing a plurality of external devices, and in response to determining that the first external device is a controllable device, identify at least one control method of the first external device, and determine one control method of the at least one control method as a preferred control method.

[0007] In one embodiment of the present disclosure, the electronic device may further include a communication unit. In one embodiment, the processor may transmit network information to a server device via the communication unit. In one embodiment, the processor may receive the identified connection information of the first external device from the server device. In one embodiment, the network information may include information about another device connected to the same network as the electronic device.

[0008] In one embodiment of the present disclosure, the electronic device may further include a communication unit. In one embodiment, the processor may transmit the identified connection information of the first external device to a server device through the communication unit.

[0009] In one embodiment of the present disclosure, the processor may transmit the identified connection information of the first external device to the server device when the processor determines that the first external device is not a controllable device.

[0010] In one embodiment of the present disclosure, the processor can identify that the first external device is connected to the electronic device through the connection when the processor identifies that the first external device connected to the electronic device is powered on.

[0011] In one embodiment of the present disclosure, the at least one control method may include at least one of a first method using infrared (IR), a second method using internet protocol (IP), and a third method using consumer electronics control (CEC).

[0012] In one embodiment of the present disclosure, the processor, in response to identifying at least one control method of the first external device, may display a user interface for determining a preferred control method of the first external device. In one embodiment, the user interface may include at least one object each indicating at least one of the identified control methods.

[0013] In one embodiment of the present disclosure, the processor may, in response to identifying a user input selecting an object of at least one object indicating at least one control method identified above, determine a control method corresponding to the selected object as a preferred control method of the first external device.

[0014] In one embodiment of the present disclosure, the processor can identify authentication information received from the first external device. In one embodiment, if the authentication information received from the first external device is valid, the processor can determine that the first external device is a controllable device.

[0015] In one embodiment of the present disclosure, the processor may determine whether authentication information received from the first external device is valid using an optical character recognition (OCR) method.

[0016] In one embodiment of the present disclosure, at least one machine learning model for recognizing the plurality of external devices may include at least one of a first machine learning model learned using device information of the plurality of external devices, a second machine learning model learned using image information of the plurality of external devices, and a third machine learning model learned using data about a device among the plurality of external devices that failed to connect.

[0017] In one embodiment of the present disclosure, the at least one machine learning model may be received from a server device.

[0018] In one embodiment of the present disclosure, the connection may include a high definition multimedia interface (HDMI) connection.

[0019] A system according to one embodiment of the present disclosure comprises: a display device including a connecting portion; a first external device connected to the display device; a server device for managing a plurality of devices connected to the display device; And a second external device for remotely controlling the display device, wherein the display device: identifies that the first external device is physically connected to the display device through a connection portion, identifies connection information of the first external device, and the identified connection information includes device information about the first external device, image information from the first external device, and information about a connection environment of the first external device, and determines whether the first external device is a controllable device based on the identified connection information of the first external device and at least one machine learning model for recognizing a plurality of external devices, and in response to determining that the first external device is a controllable device, identifies at least one control method of the first external device, and determines one control method of the at least one control method as a preferred control method, and the server device: receives network information from the display device, identifies connection information of the first external device based on the network information, transmits the connection information of the first external device to the display device, and trains the at least one machine learning model using the connection information of the first external device.

[0020] In one embodiment of the present disclosure, the at least one machine learning model may include at least one of a first machine learning model learned using device information of the plurality of devices, a second machine learning model learned using image information of the plurality of devices, and a third machine learning model learned using data about a device among the plurality of devices that failed to connect.

[0021] In one embodiment of the present disclosure, the display device may, in response to determining that the first external device is not a controllable device, transmit data regarding the device with which the connection has failed to the server device. In one embodiment, the server device may learn the third machine learning model based on the transmitted data regarding the device with which the connection has failed.

[0022] In one embodiment of the present disclosure, the server device can transmit information about the at least one machine learning model to the display device.

[0023] In one embodiment of the present disclosure, the connection unit may include a high definition multimedia interface (HDMI) connection unit. In one embodiment, the at least one control method may include at least one of a first method using infrared (IR), a second method using internet protocol (IP), and a fourth method using consumer electronics control (CEC).

[0024] In one embodiment of the present disclosure, the second external device may be connected to the display device using a short-range wireless communication technology.

[0025] In one embodiment of the present disclosure, when the second external device is a user device, the priority control method may be determined as the second method.

[0026] Embodiments of the present disclosure provide an effect of minimizing cases of incorrect recognition of an external device connected to a display device and increasing the device recognition rate.

[0027] In addition, embodiments of the present disclosure provide the effect of controlling a display device and an external device through a control method optimized for the user's environment by dynamically changing the control method in consideration of the type of external device, connection environment, etc.

[0028] The effects that can be obtained from the present disclosure are not limited to the effects mentioned in the various embodiments, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0029] FIG. 1 is a block diagram illustrating an exemplary configuration of an electronic device according to one embodiment.

[0030] Figure 2 illustrates an example of an IoT network according to one embodiment.

[0031] FIG. 3 is a block diagram illustrating an exemplary configuration of devices constituting a system for controlling an electronic device according to one embodiment.

[0032] FIG. 4 illustrates an exemplary operational flow of an electronic device according to one embodiment.

[0033] FIG. 5 illustrates an exemplary signaling flow of an electronic device and a server device according to one embodiment.

[0034] FIG. 6 illustrates an example of control priority settings according to another electronic device connected to an electronic device according to one embodiment.

[0035] FIG. 7 illustrates an example of priority settings of an electronic device controlled by a user device according to one embodiment.

[0036] Figure 8 illustrates an example of a user interface according to one embodiment.

[0037] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0038] Hereinafter, various exemplary embodiments of the present disclosure will be described in detail with reference to the drawings. However, the present disclosure may be implemented in various different forms and is not limited to the various exemplary embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0039] FIG. 1 is a block diagram illustrating an exemplary configuration of an electronic device according to one embodiment. The electronic device (100) may be a wearable terminal, such as a smartwatch or glasses, capable of performing various computing functions, including video streaming and communication. The electronic device (100) may be a terminal of various forms without limitation to the above-described embodiments.

[0040] According to one embodiment, the memory (120) is a storage medium used by the electronic device (100) and can store data such as at least one command (121) or setting information corresponding to at least one program. The at least one program may include an operating system (OS) program and / or various application programs.

[0041] In one embodiment, the memory (120) may store pairing information regarding an external electronic device located adjacent to the electronic device (100) or within an area where a communication connection can be established with the electronic device (100). In one embodiment, the pairing information may include device information regarding the external electronic device, information regarding other external electronic devices or remote control devices paired with the external electronic device, information regarding a method by which the external electronic device and the other external electronic devices or remote control devices are paired (e.g., Bluetooth, Wi-Fi), and information regarding a pairing history between the external electronic device and the other external electronic devices or remote control devices.

[0042] In one embodiment, the memory (120) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., an SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0043] According to one embodiment, the video input unit (130) can receive video and video information through a tuner (not shown), an input / output unit (not shown), or a communication unit (150). The video input unit (130) can include at least one of the tuner and the input / output unit. The tuner can select and tune only the frequency of a broadcast channel to be received by the electronic device (100) among many radio wave components through amplification, mixing, resonance, etc. of a broadcast signal received wirelessly or wiredly. The broadcast signal can include video, audio, and additional data (e.g., an Electronic Program Guide (EPG)). The tuner can receive real-time broadcast channels (or real-time viewing images) from various broadcast sources such as terrestrial broadcasting, cable broadcasting, satellite broadcasting, and Internet broadcasting. The tuner can be implemented as an integrated unit with the electronic device (100) or as a separate tuner electrically connected to the electronic device (100). The input / output unit may include at least one of an HDMI input port, a component input jack, a PC input port, and a USB (universal serial bus) input jack, which can receive images and image information from an external device of the electronic device (100) under the control of the processor (110). It is obvious to those skilled in the art that the input / output unit may be added, deleted, and / or changed depending on the performance and structure of the electronic device (100).

[0044] According to one embodiment, the display (140) may perform functions for outputting information in the form of numbers, characters, images, and / or graphics. The display (140) may include at least one hardware module for outputting. The at least one hardware module may include, for example, at least one of an LCD, an LED (Light Emitting Diode), an LPD (Light Emitting Polymer Display), an OLED (Organic Light Emitting Diode), an AMOLED (Active Matrix Organic Light Emitting Diode), or an FLED (Flexible LED). The display (140) may display a screen corresponding to data received from the processor (110). The display (140) may be referred to as an “output unit,” a “display unit,” or other terms having equivalent technical meanings thereto. The “screen” may include an image displayed on the display of an electronic device. The image may be referred to by terms such as a frame. Various types of objects such as icons, text, photos, videos, widgets, etc. may be displayed on the screen.

[0045] According to one embodiment, the communication unit (150) may provide a wired / wireless communication interface that enables communication with an external device. The communication unit (150) may include at least one of a wired Ethernet, a wireless LAN communication unit, and a short-range communication unit. The wireless LAN communication unit may include, for example, Wi-Fi, and may support the wireless LAN standard (IEEE802.11x) of the Institute of Electrical and Electronics Engineers (IEEE). The wireless LAN communication unit may be wirelessly connected to an AP (Access Point) under the control of the processor (110). The AP may include a device that enables devices to be connected using a related standard using Wi-Fi in a computer network. The short-range communication unit may wirelessly perform short-range communication with an external device under the control of the processor (110). Short-range communication may include Bluetooth, Bluetooth Low Energy, Infrared Data Association (IrDA), Ultra Wide Band (UWB), and Near Field Communication (NFC). The above external devices may include server devices that provide video services, etc., and mobile terminals (e.g., phones, tablets, etc.).

[0046] According to one embodiment, the processor (110) may execute at least one instruction (121) stored in the memory (120) to perform calculations or data processing related to control and / or communication of at least one other component of the electronic device (100). The processor (110) may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), a micro controller unit (MCU), a sensor hub, a supplementary processor, a communication processor, an application processor, an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), and may have multiple cores.

[0047] In one embodiment, the processor (110) may control at least one other component (e.g., a hardware or software component) of the electronic device (100) connected to the processor (110), for example, by executing software, and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (110) may store commands or data received from other components in a volatile memory, process the commands or data stored in the volatile memory, and store result data in a non-volatile memory. According to one embodiment, the processor (110) may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, when the electronic device (100) includes a main processor and an auxiliary processor, the auxiliary processor may be configured to use lower power than the main processor or to be specialized for a given function. The auxiliary processor may be implemented separately from the main processor or as part of it.

[0048] In one embodiment, the processor (110) may obtain image frame data from at least one of the memory (120), the image input unit (130), or the communication unit (150). The processor (110) may receive the image frame data from at least one of the memory (120), the image input unit (130), or the communication unit (150). The image frame data may include data regarding frames constituting an image. For example, the image frame data may be identified from the memory (120) (e.g., a recorded and stored image). For example, the image frame data may include data obtained from the communication unit (150) or the image input unit (130) (e.g., a real-time streaming image).

[0049] Fig. 2 illustrates an example of an IoT network according to one embodiment. The electronic device (220) of Fig. 2 may be a device corresponding to the electronic device (100) of Fig. 1.

[0050] According to one embodiment, an IoT network may include an AP (210), an electronic device (220), and an external device (230).

[0051] In one embodiment, the AP (210) may include a device that enables devices to connect to the network using short-range wireless communication technology (e.g., Wi-Fi) that is adjacent to the AP (210) or within an area where a communication connection can be established with the electronic device (100).

[0052] In one embodiment, the electronic device (220) may include a device for receiving data from an external device (230) or an input device (not shown) (e.g., a set-top box, USB, PC, or HDMI device) and displaying the screen based thereon.

[0053] In one embodiment, the electronic device (220) can communicate data with a server device (not shown) through the AP (210). For example, the electronic device (220) can receive at least one machine learning model necessary for recognizing an external device (230) through the AP (210). For example, the electronic device (220) can transmit data related to information about an external device (230) connected to the electronic device (220) to the server device (not shown) through the AP (210), and the server device (not shown) can learn a machine learning model for recognizing the external device (230) based on information received from the electronic device (220) through the AP (210). For example, the electronic device (220) can transmit network information to the server device (not shown) through the AP (210). The network information can include a multicast domain name (mDNS), a simple service discovery protocol (SSDP), and a medium access control (MAC) address of a device connected to the same AP.

[0054] In one embodiment, the external device (230) may include a device for transmitting data such as video, images, audio, etc. to the electronic device (220).

[0055] In one embodiment, an external device (230) may receive data from a server device (not shown) via an AP (210) and provide the received data to an electronic device (220).

[0056] In one embodiment, the external device (230) may be physically connected (e.g., via a wired connection) to the electronic device (220). The external device (230) may be connected to the electronic device (220) via at least one connection provided to the electronic device (220).

[0057] In one embodiment, an external device (230) may be connected to the electronic device (220) via an HDMI port. The external device (230) may include an “HDMI device,” an “HDMI appliance,” an “OTT (over the top) device,” and / or an “external electronic device.”

[0058] In one embodiment, when the electronic device (220) is connected to an external device (230), it can identify information about the external device (230). When the electronic device (220) is connected to the external device (230), information about the external device (230) identified by the electronic device (220) can include Infoframe information, CEC, and HDMI port information. Information about the external device (230) can include an image received as an IR code information image of a remote control device (e.g., a remote control). Information about the external device (230) can include sound information and display peripheral sensor information.

[0059] In one embodiment, when the electronic device (220) is connected to an external device (230), it may perform a procedure for recognizing the external device (230). The procedure for recognizing the external device (230) may include a procedure for determining whether the electronic device (220) can support a function provided by the external device (230).

[0060] In one embodiment, the procedure for recognizing an external device (230) may include a procedure for determining whether the external device (230) can be controlled by the electronic device (220).

[0061] In one embodiment, the procedure for recognizing an external device (230) may include an authentication procedure of the external device (230).

[0062] In one embodiment, the authentication procedure may be performed using character recognition technology such as optical character recognition (OCR) based on a pincode.

[0063] In one embodiment, the electronic device (220) may be connected to multiple other external devices as well as the external device (230). The electronic device (220) may include multiple HDMI ports and may be physically or wirelessly connected to multiple devices via the multiple HDMI ports.

[0064] FIG. 3 is a block diagram illustrating an exemplary configuration of devices constituting a system for controlling an electronic device according to one embodiment.

[0065] According to one embodiment, a system (300) for controlling an electronic device may include an electronic device (310), a first external device (320), a second external device (340), and a server device (330). The electronic device (310) may include a device corresponding to the electronic device (100) of FIG. 1 and the electronic device (220) of FIG. 2. The first external device (320) may include a device corresponding to the external device (230) of FIG. 2.

[0066] In one embodiment, the electronic device (310) may be physically (or wirelessly) connected to a first external device (320). For example, the electronic device (310) may be connected to the first external device (320) via an HDMI port included in the electronic device (310).

[0067] In one embodiment, the second external device (340) may include a device for remotely controlling the electronic device (310) and the first external device (320). For example, the second external device (340) may include a remote control or a smartphone.

[0068] In one embodiment, the second external device (340) can control the electronic device (310) and the first external device (320) using a short-range communication technology (e.g., Bluetooth, Wi-Fi, IR).

[0069] In one embodiment, the first external device (320) can be controlled via at least one control method. The at least one control method may include an IR method, an IP method, and / or a CEC communication method. For example, the IR method may include a method of controlling a device using infrared rays. The first external device (320) may be controlled via an infrared signal received via a second external device (340). For example, the IP method may include a method of controlling an HDMI device via a network connection. To use the IP method, the first external device (320) may need to be connected to a network. The first external device (320) may be controlled via a mobile device (e.g., a smartphone). For example, the CEC method may include a method of controlling multiple HDMI devices via a single remote control device (e.g., the second external device (340)) using a cable connected to an HDMI port. Devices that support CEC communication can communicate with each other via the HDMI cable.

[0070] In one embodiment, the first external device (320) can be controlled through at least one method to transmit data (e.g., video, audio) for display on the screen to the electronic device (310).

[0071] In one embodiment, the server device (330) can manage information for connection of the electronic device (310) and the first external device (320).

[0072] In one embodiment, the server device (330) can communicate data with the electronic device (310) and the first external device (320). For example, the server device (330) can transmit image data to the first external device (320), and the first external device (320) can transmit image data received from the server device (330) to the electronic device (310). For example, the server device (330) can transmit data associated with at least one machine learning model that the electronic device (310) uses to recognize the external device to the electronic device (310). For example, the server device (330) can receive network information (e.g., mDNS, SDDP, MAC address, etc.) from the electronic device (310).

[0073] In one embodiment, the server device (330) may learn (e.g., train) and update at least one machine learning model for device recognition based on connection information of the electronic device (310) received from the electronic device (310). The connection information may include device information (e.g., model name, manufacturer, or identification information) of an external device connected to the electronic device (310), image information (e.g., specific screen elements (logos, or UI elements) of the external device), information about the connection status (e.g., Wi-Fi information, HDMI cable information, other connected devices (e.g., 2 nd (TV), information about a paired mobile device, or information about the location of a connected device).

[0074] In one embodiment, the electronic device (310) may perform a procedure for recognizing a first external device (320) connected to the electronic device (310). The procedure for recognizing the first external device (320) may be used in the same sense as “a procedure for determining whether the first external device (320) is a device that can be controlled by the electronic device (310).”

[0075] In one embodiment, the electronic device (310) can recognize the first external device (320) using CEC information.

[0076] In one embodiment, the electronic device (310) can recognize the first external device (320) based on video infoframe information included in an HDMI signal received from the first external device (320). For example, the infoframe information can include video and audio formats, resolution, color information, etc. The electronic device (310) can identify the device capability of the connected first external device (320) using the infoframe information. Based on the device capability of the identified first external device (320), the electronic device (310) can determine whether the electronic device (310) supports the device capability of the first external device (320), and if it determines that it supports the device capability, it can recognize the first external device (320).

[0077] In one embodiment, the electronic device (310) may identify the first external device (320) based on a specific screen element of the first external device (320) received from the first external device (320) (e.g., a logo of the first external device (320), an object included in the UI, an icon, etc.).

[0078] In one embodiment, the server device (330) may include at least one machine learning model.

[0079] In one embodiment, the electronic device (310) may input image data of the first external device (320) and analyze the image data through an image analysis engine (e.g., a convolutional neural network (CNN)). The electronic device (310) may extract features of the image through the image analysis engine. The electronic device (310) may then generate real device data through the image analysis engine. The electronic device (310) may again use the CNN with the real device data to form a final device decision model, and the electronic device (310) may use the final device decision model to identify the first external device (320).

[0080] In one embodiment, the electronic device (310) can collect various data generated from the first external device (320). In other words, the electronic device (310) can collect and filter various data through a big data collection engine and filtering. The electronic device (310) can cluster various data (e.g., Infoframe, CEC, brand-IRCode mapping) collected through the big data collection engine to identify data characteristics. The electronic device (310) can determine principal component variables for each manufacturer of the first external device (320). The electronic device (310) can then generate real device data. The actual device data may include information related to the length of the HDMI device's cable (e.g., the time of flight (TOF) of a signal transmitted and received from the HDMI device), the network environment (e.g., the stability of a Wi-Fi signal), the location of the external device (e.g., the back or front of the electronic device (310)), whether it is connected to a mobile device, or whether it is connected to another display device. The electronic device (310) may form a final device determination model based on the actual device data and the machine learning model. The electronic device (310) may identify the first external device (320) using the final device determination model.

[0081] In one embodiment, the electronic device (310) can perform reinforcement learning based on data about the misrecognized device (e.g., actual device data) when there is an error in the recognition of the device, i.e., when a false recognition occurs.

[0082] In one embodiment, the electronic device (310) can store all data generated through the image analysis engine, big data collection engine, CNN model, and machine learning model.

[0083] In one embodiment, the above-described learning (e.g., learning on image data of an external device or learning on environmental data of an external device) may be performed on the server device (330). The server device (330) may transmit an artificial intelligence model (e.g., a CNN model, a machine learning model) generated based on the learning to the electronic device (310). Although not shown in the drawing, not only the server device (330) but also the electronic device (310) may perform learning (or training) for the final device recognition model based on image information and device information of an external device connected to the electronic device (310). In other words, the electronic device (310) may use the machine learning model received from the server device (330) as is, or may independently (e.g., automatically or automatically) update the machine learning model based on information about external devices that are connected thereafter. The electronic device (310) can transmit information about devices connected to the electronic device (310) to the server device (330) to enable the server device (330) to perform generation and learning of the machine learning model described above.

[0084] FIG. 4 illustrates an exemplary operation flow of an electronic device according to one embodiment. The electronic device of FIG. 4 may include devices corresponding to the electronic device (100) of FIG. 1, the electronic device (220) of FIG. 2, and the electronic device (310) of FIG. 3. In the description of FIG. 4, the electronic device is described as an example of a TV, and the first external device is described as an example of an HDMI device. However, this is merely an example, and the following description may be equally applicable to devices that are connected wired or wirelessly. In other words, the same may be applied even when the electronic device is a user device (e.g., a smartphone) and the first external device is an IoT device (e.g., a smart light or a smart switch).

[0085] In one embodiment, at operation 410, the electronic device may identify that a first external device is physically connected to the electronic device via a connector.

[0086] In one embodiment, the connection unit may include an HDMI connection unit included in the electronic device. According to one embodiment, the electronic device may include a plurality of connection units. The first external device may be connected to the electronic device via the HDMI port.

[0087] In one embodiment, the electronic device may identify that the first external device is physically connected to the electronic device through the connector when the electronic device identifies that the first external device, which is already connected through the connector, changes from a powered-off state to a powered-on state.

[0088] According to one embodiment, in operation 420, the electronic device may identify connection information of the first external device.

[0089] In one embodiment, the electronic device may identify connection information of the first external device independently of the electronic device, or may identify connection information of the first external device from a server device.

[0090] In one embodiment, the electronic device may transmit network information (e.g., MDNS, SSDP, MAC address) to a server device. In response to receiving the network information from the electronic device, the server device may transmit connection information of a first external device connected to the electronic device to the electronic device.

[0091] In one embodiment, the connection information of the first external device includes device information (e.g., model name, manufacturer, and / or identification information) of the external device connected to the electronic device (310), image information (e.g., specific screen elements (logos, UI elements, etc.) of the external device), information about the connection status (e.g., environmental data) (e.g., Wi-Fi information, HDMI cable information, other connected devices (e.g., 2 nd (TV), information about a paired mobile device, or information about the location of a connected device).

[0092] In one embodiment, the electronic device may transmit connection information of a first external device to a server device. The server device may train a machine learning model for image analysis and a machine learning model for big data collection based on the connection information (image information, device information (environmental data)) received from the first external device, thereby generating or updating a final device decision model.

[0093] In one embodiment, the electronic device can independently (e.g., by itself or automatically) train at least one machine learning model stored in a memory of the electronic device based on connection information of the first external device. The at least one machine learning model can include a first machine learning model trained using device information of external devices (e.g., the first external device) connected to the electronic device, a second machine learning model trained using image information of the external devices, and a third machine learning model trained using data regarding a device among the plurality of external devices that failed to connect (e.g., a model for reinforce training in FIG. 3).

[0094] In one embodiment, data about a device among a plurality of external devices that failed to connect may include information about settings set by the electronic device for connecting the external device when recognition of the external device failed after connection.

[0095] According to one embodiment, at operation 430, the electronic device may determine whether the first external device is a controllable device. The operation of the electronic device determining whether the first external device is a controllable device may include the operation of the electronic device recognizing (or identifying) the first external device.

[0096] In one embodiment, the electronic device can determine whether the first external device is a controllable device based on connection information of the first external device and at least one machine learning model for device recognition. The electronic device can determine whether the electronic device can support a function provided by the first external device based on device information of the first external device. If the electronic device can support the function provided by the first external device, the electronic device can determine that the first external device is a controllable device. If the electronic device identifies that a network (e.g., Wi-Fi) to which the first external device is connected is the same as the network to which the electronic device is connected, the electronic device can determine that the first external device is a controllable device.

[0097] In one embodiment, the electronic device may determine that the first external device is a controllable device if the authentication procedure of the connected first external device is successful. For example, if the recognition of the first external device requires an authentication procedure, the electronic device may request a PIN code from the first external device, and the electronic device may use a character recognition method such as optical character recognition (OCR) to identify the PIN code received from the first external device, determine whether it is a valid PIN code, and perform authentication based on the identified PIN code.

[0098] In one embodiment, the electronic device can form major cluster and minor cluster patterns based on at least one machine learning model, even if the information about the first external device input through the first external device is previously input information, and can recognize the first external device only when it is a major cluster by mapping it with another machine learning model.

[0099] In one embodiment, at least one machine learning model may be received from the server. The at least one machine learning model may include a first machine learning model learned based on image information of the external device, a second machine learning model learned based on device information and environmental data of the external device, a third machine learning model learned based on information about external devices that failed to connect, and a fourth machine learning model representing a final device decision model generated based on the first, second, and third machine learning models.

[0100] According to one embodiment, if the electronic device determines that the first external device is a controllable device, the electronic device may identify at least one control method of the first external device in operation 440. The at least one control method may include a first method using IR, a second method using IP, and a third method using CEC. Table 1 below may represent information regarding the control methods of external devices connected or connected to the electronic device.

[0101] Device / Control MethodIRIPCECPriorityAOOIPBOOIPCOOIRDOIREOIPFOCECGOOOIP

[0102] Referring to Table 1 above, information regarding the control methods of external devices may include control methods and priority control methods that can be supported for devices A to G.

[0103] For example, a first method may include a method of controlling a device using IR signals, wherein the electronic device can transmit a control signal to the first external device in response to receiving a signal transmitted by manipulating a physical button on the first external device from a remote control device (e.g., a second external device). For example, a second method may include a method of controlling a device via a network (e.g., a Wi-Fi network). If the first external device is connected to the same network as the electronic device and has the same IP address, the electronic device can control the first external device. For example, a third method is a technology that enables communication and control between HDMI-connected devices, wherein the devices can be controlled by exchanging signals between the electronic device and the first external device via an HDMI cable. HDMI devices that support CEC can control each other's functions with the second external device.

[0104] According to one embodiment, in operation 450, the electronic device may determine one of at least one control method as a preferred control method. The electronic device may determine the preferred control method based on a machine learning model received from a server or a machine learning model generated through self-learning.

[0105] In one embodiment, if the first external device supports multiple control methods, priorities may exist among the multiple control methods. A control method that is preferentially selected among the multiple control methods may be referred to as a preferred control method.

[0106] In one embodiment, the electronic device may determine a priority control method based on device information of the first external device.

[0107] In one embodiment, the electronic device may determine a priority control method of the first external device based on a machine learning model for priority determination learned based on connection information of the first external device.

[0108] In one embodiment, the electronic device may assign scores to each item (also referred to as a “variable”) constituting device information of the first external device to determine a preferred control method among at least one control method of the first external device, and determine the control method with the highest (or higher) score as the preferred control method. Items constituting the device information of the first external device may include the length of the HDMI cable of the external device, the relative position of the external device with respect to the electronic device, the manufacturer of the external device, the network environment of the external device, the control history of the external device, whether it is connected to a mobile device, and whether it is connected to another electronic device.

[0109] In one embodiment, an external device having an HDMI cable length less than a predetermined value may be given a higher score than an external device having an HDMI cable length greater than the predetermined value. The electronic device may determine the cable length based on the ToF of signals transmitted and received between the electronic device and the external device connected to the electronic device.

[0110] In one embodiment, an electronic device may prioritize a specific control method if a machine learning model determines that a specific control method is more efficient. For example, an electronic device may prioritize IP as the control method for an external device manufactured by a specific manufacturer based on a machine learning model that has learned a predetermined number of data regarding failures in control via IR and CEC.

[0111] In one embodiment, the electronic device may assign a score based on information about the network environment to which the external device is connected. For example, if the network to which the external device is connected is the same as the electronic device, a higher score may be assigned to the IP control method. If the network is not the same, a higher score may be assigned to a control method other than IP. For example, if the network environment of the electronic device and the external device is poor (e.g., if the data transfer rate is below a threshold), a higher score may be assigned to a control method other than IP.

[0112] In one embodiment, the electronic device may determine a score for each item based on the control failure history of the external device. For example, if the electronic device determines that control failed or was not recognized while performing control via CEC, it may assign a higher score to a control method other than CEC.

[0113] Table 2 below shows an example of assigning scores to the control methods supported by each external device according to the above-described contents.

[0114] IRIPCECA76-10B---C--100D--29E2051050F-8530G--23

[0115] Referring to Table 2 above, the control method with the highest score for each device can be determined as the priority control method.

[0116] In one embodiment, the electronic device can determine the priority control method of the first external device based on the user's settings. While the priority control method may be determined based on scores, as shown in Table 2, the user may also determine the priority control method of the external device based on their own convenience (e.g., user selection).

[0117] In one embodiment, the electronic device may display a user interface for setting or changing the priority control method of the first external device. The user interface for setting or changing the priority control method of the first external device may be described in detail in the description of FIG. 8 below.

[0118] In one embodiment, the electronic device may transmit information regarding the priority control method of the external device and the score of at least one control method of the external device to the server device. The server device may generate a machine learning model for determining the connection and control method of the external device based on the received information regarding the priority control method and the score of at least one control method.

[0119] According to one embodiment, if the electronic device determines in operation 430 that the first external device is not a controllable device, the electronic device may transmit connection information of the first external device to the server device.

[0120] In one embodiment, the server device may use the connection information of the first external device received from the electronic device for reinforcement learning. In one embodiment, the electronic device may independently use the connection information of the first external device for reinforcement learning purposes.

[0121] FIG. 5 illustrates an exemplary signaling flow of an electronic device and a server device according to one embodiment. The electronic device of FIG. 5 may include devices corresponding to the electronic device (100) of FIG. 1, the electronic device (220) of FIG. 2, the electronic device (310) of FIG. 3, and the electronic device of FIG. 4. The server device of FIG. 5 may include a device corresponding to the server device (330) of FIG. 3. The operation described in FIG. 5 may correspond to the operation of the electronic device and the operation of the server device of FIG. 4.

[0122] According to one embodiment, in operation 510, the electronic device may identify a physical connection to the first external device. Operation 510 may include all of the operation contents of operation 410 of FIG. 4.

[0123] According to one embodiment, in operation 520, the electronic device may transmit network information to a server device.

[0124] According to one embodiment, in operation 530, the server device may, in response to receiving network information from the electronic device, transmit connection information of the first external device to the electronic device.

[0125] In one embodiment, the electronic device may transmit network information (e.g., MDNS, SSDP, MAC address) to a server device. In response to receiving the network information from the electronic device, the server device may transmit connection information of a first external device connected to the electronic device to the electronic device.

[0126] In one embodiment, the connection information of the first external device includes device information (e.g., model name, manufacturer, and / or identification information) of the external device connected to the electronic device (310), image information (e.g., specific screen elements (logos, UI elements, etc.) of the external device), information about the connection status (e.g., environmental data) (e.g., Wi-Fi information, HDMI cable information, other connected devices (e.g., 2 nd (TV), information about a paired mobile device, or information about the location of a connected device).

[0127] In one embodiment, the electronic device may transmit connection information of a first external device to a server device. The server device may train a machine learning model for image analysis or a machine learning model for big data collection based on the connection information (e.g., image information and / or device information (environmental data)) received from the first external device, thereby generating or updating a final device decision model.

[0128] In one embodiment, the electronic device can train at least one machine learning model stored in a memory of the electronic device independently of the electronic device (e.g., automatically or automatically) based on connection information of the first external device. The at least one machine learning model can include a first machine learning model trained using device information of external devices (e.g., the first external device) connected to the electronic device, a second machine learning model trained using image information of the external devices, and a third machine learning model trained using data regarding a device among the plurality of external devices that failed to connect (e.g., a model for reinforce training in FIG. 3).

[0129] According to one embodiment, in operation 540, the electronic device may identify connection information of the first external device. FIG. 5 may include all of the operation contents according to operation 420 of FIG. 4.

[0130] According to one embodiment, in operation 550, the electronic device may determine whether the first external device is a controllable device. Operation 550 may include all of the operation contents according to operation 430 of FIG. 4.

[0131] According to one embodiment, at operation 560, the electronic device may transmit information about the first external device to the server device.

[0132] In one embodiment, the information about the first external device may include both connection information of the first external device and, if the connection of the first external device fails, information related to the connection of the first external device (e.g., network information).

[0133] According to one embodiment, at operation 570, the server device can update at least one machine learning model based on the information about the first external device received at operation 560.

[0134] According to one embodiment, in response to determining that the first external device is a controllable device, the electronic device may determine a preferred control method of the first external device at operation 580. Operation 580 may include all of the operation contents according to operations 440 and 450 of FIG. 4.

[0135] In one embodiment, in response to determining that the first external device is not a controllable device, the electronic device may transmit to the server device all information obtained by the electronic device about the first external device (e.g., network information, connection information (e.g., device information, image information, or environmental data), data regarding connection failures, etc.). The server device may perform reinforcement learning based on this and update a machine learning model (e.g., a final device determination model) for device identification (e.g., device recognition).

[0136] FIG. 6 illustrates an example of control priority settings according to another electronic device connected to an electronic device according to one embodiment. The electronic device (610) and the electronic device (620) of FIG. 6 may correspond to the electronic device (100) of FIG. 1 . The AP (630) of FIG. 6 may correspond to the AP (210) of FIG. 2 . The second external device (640) of FIG. 6 may include a device corresponding to the second external device (340) of FIG. 3 .

[0137] Referring to FIG. 6, a first electronic device (610) can be controlled through a second external device (640). The first electronic device (610) can receive a control signal from the second external device (640) through short-range wireless communication technology. The first electronic device (610) can transmit the control signal to the second electronic device (620). The second electronic device (620) can receive data from the AP (630) through communication with the AP (630). The second electronic device (620) can transmit the data received from the AP (630) to the first electronic device (610), and accordingly, the first electronic device (610) can display a screen.

[0138] In one embodiment, the second electronic device (620) may be controlled using a second method using IP. If the second electronic device (620), the first electronic device (610), an external device (not shown) connected to the first electronic device (610), and an external device (not shown) connected to the second electronic device (620) exist (e.g., are provided or exist), these devices may also be controlled using the second method using IP. In other words, for other electronic devices connected to the electronic device, the method using IP may be determined as the preferred control method to improve control reliability.

[0139] FIG. 7 illustrates an example of priority settings of an electronic device controlled by a user device according to one embodiment.

[0140] Referring to FIG. 7, the electronic device (710) may include an electronic device corresponding to the electronic device (100) of FIG. 1. The user device (720) of FIG. 7 may include a mobile device (e.g., a smartphone). The AP (730) of FIG. 7 may include a device corresponding to the AP (210) of FIG. 2.

[0141] In one embodiment, when the electronic device (710) is controlled by the user device (720), the priority control method of the electronic device (710) and an external device (not shown) connected to the electronic device (710) may be determined as a second control method using IP. In order to improve control reliability by allowing control only between devices that use the same network (e.g., connected to the same network or connected to the same network), the second control method may be determined as the priority control method.

[0142] Figure 8 illustrates an example of a user interface according to one embodiment.

[0143] In one embodiment, the user interface (800) may be displayed in response to identifying that a first external device is physically connected to the electronic device via a connector.

[0144] In one embodiment, the user interface (800) may be displayed in response to identifying a user input for changing the control method of the first external device.

[0145] In one embodiment, the user interface (800) may be displayed in response to an event detecting a change in the control method of the electronic device. The event detecting a change in the control method may include detecting a change in the network environment or a change in a connected device (e.g., when another external device is additionally connected, when another previously connected external device is disconnected, etc.).

[0146] In one embodiment, the user interface (800) may include a second object (820) representing an external device connected to the electronic device. The second object (820) may be displayed based on image information of the external device connected to the electronic device. For example, the second object (820) may include an icon corresponding to a logo image of the external device.

[0147] In one embodiment, the user interface (800) may include a first object (810) representing an object for changing the control method of a selected external device.

[0148] In one embodiment, in response to identifying a user input selecting a second object (820), the electronic device may display a UI component that includes a first object (810).

[0149] In one embodiment, when user input is identified through the first object (810), the electronic device can change the control method of the selected external device. For example, referring to FIG. 8, when user input is identified through the first object (810), the control method of the selected "AAA TV" can be changed to a second control method using IP.

[0150] According to the above-described embodiments, the control method can be dynamically changed using various variables such as the location of the HDMI device connected to the electronic device (e.g., back, side, front of the TV), HDMI cable length, and WIFI environment, thereby providing a control method optimized for the user environment. In addition, even if there is a priority control method set for the electronic device or external device, it can be changed (or updated) if necessary, thereby improving the user experience. In addition, in the case of TV-mobile remote control or TV-second TV control, the reliability of device control can be improved by setting control priority through IP to ensure stable operation during remote control. In addition, at least one machine learning model transmitted by the server device can be continuously updated based on data exchange and learning between the electronic device and the server device, thereby preventing false recognition. Based on this machine learning model, it is possible to provide the effect of accurately recognizing the device even if the type of the connected external device or the connection environment changes.

[0151] An electronic device according to one embodiment of the present disclosure may include a connection unit, a memory storing at least one machine learning model for recognizing a plurality of external devices, and a processor electrically connected to the connection unit and the memory. In one embodiment, the processor may identify that a first external device is physically connected to the electronic device through the connection unit. In one embodiment, the processor may identify connection information of the first external device. In one embodiment, the processor may determine whether the first external device is a controllable device based on the connection information of the first external device and the at least one machine learning model. In one embodiment, the processor may identify at least one control method of the first external device in response to determining that the first external device is a controllable device. In one embodiment, the processor may determine any one of the at least one control method as a priority control method. In one embodiment, the connection information of the first external device may include device information, image information, and information regarding a connection environment of the first external device.

[0152] In one embodiment, the electronic device may further include a communication unit connected to the processor. In one embodiment, the processor may transmit network information to a server device via the communication unit. In one embodiment, the processor may receive connection information of the first external device from the server device. In one embodiment, the network information may include information about another device connected to the same network as the electronic device.

[0153] In one embodiment, the electronic device may further include a communication unit electrically connected to the processor. In one embodiment, the processor may transmit connection information of the first external device to a server device through the communication unit.

[0154] In one embodiment, the processor may transmit connection information of the first external device to the server device when the processor determines that the first external device is not a controllable device.

[0155] In one embodiment, the processor can identify that the first external device is connected to the electronic device via the connector when the processor identifies that the first external device connected to the electronic device is powered on.

[0156] In one embodiment, the at least one control method may include at least one of a first method using IR, a second method using IP, and a third method using CEC.

[0157] In one embodiment, the processor, in response to identifying at least one control method of the first external device, may display a user interface for determining a preferred control method of the first external device. In one embodiment, the user interface may include at least one object, each object indicating the at least one control method.

[0158] In one embodiment, the processor may, in response to identifying a user input selecting any one of the at least one object indicating the at least one control scheme, determine the control scheme corresponding to the selected object as the preferred control scheme of the first external device.

[0159] In one embodiment, the processor may identify authentication information from the first external device. In one embodiment, if the authentication information received from the first external device is valid, the processor may determine that the first external device is a controllable device.

[0160] In one embodiment, the processor may determine whether authentication information received from the first external device is valid using an optical character recognition (OCR) method.

[0161] In one embodiment, the at least one machine learning model for recognizing the plurality of external devices may include at least one of a first machine learning model learned using device information of the plurality of external devices, a second machine learning model learned using image information of the plurality of external devices, and a third machine learning model learned using data about a device among the plurality of external devices that failed to connect.

[0162] In one embodiment, the at least one machine learning model may be received from a server device.

[0163] In one embodiment, the connection may include an HDMI connection.

[0164] A system according to one embodiment of the present disclosure may include a display device, a first external device connected to the display device, a server device for managing a plurality of devices connected to the display device, and a second external device for remotely controlling the display device. In one embodiment, the display device may identify that the first external device is physically connected to the display device through a connection. In one embodiment, the display device may identify connection information of the first external device. In one embodiment, the display device may determine whether the first external device is a controllable device based on the connection information of the first external device and at least one machine learning model. In one embodiment, the display device may identify at least one control method of the first external device in response to determining that the first external device is a controllable device. In one embodiment, the display device may determine one of the at least one control method as a preferred control method. In one embodiment, the server device may receive network information from the display device. In one embodiment, the server device may identify connection information of the first external device based on the network information. In one embodiment, the server device can transmit connection information of the first external device to the display device. In one embodiment, the server device can train the at least one machine learning model using the connection information of the first external device.

[0165] In one embodiment, the at least one machine learning model may include at least one of a first machine learning model trained using device information of the plurality of devices, a second machine learning model trained using image information of the plurality of devices, and a third machine learning model trained using data about a device among the plurality of devices that failed to connect.

[0166] In one embodiment, the display device may, in response to determining that the first external device is not a controllable device, transmit data about the device with which the connection has failed to the server device. In one embodiment, the server device may train the third machine learning model based on the data about the device with which the connection has failed.

[0167] In one embodiment, the server device can transmit information about the at least one machine learning model to the display device.

[0168] In one embodiment, the connection may include an HDMI connection. In one embodiment, the at least one control method may include at least one of a first method using IR, a second method using IP, and a third method using CEC.

[0169] In one embodiment, the second external device may be connected to the display device using short-range wireless communication technology.

[0170] In one embodiment, when the second external device is a user device, the priority control method may be determined as the second method.

[0171] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, display devices, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments disclosed in this document are not limited to the aforementioned devices.

[0172] The various embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. For example, a component expressed in the singular should be understood to include a concept including plural components unless the context clearly indicates only the singular. It should be understood that the term "and / or" used in this document encompasses any and all possible combinations of one or more of the listed items. The terms "comprise," "have," "consist of," and the like used in this disclosure are intended to specify only the presence of a feature, component, part, or combination thereof described in this disclosure, and the use of such terms does not exclude the presence or addition of one or more other features, components, parts, or combinations thereof. In this document, phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can each include any one of the items listed together in that phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used merely to distinguish the corresponding element from other corresponding elements and do not limit the corresponding elements in any other respect (e.g., importance or order).

[0173] The terms "part" or "module" used in various embodiments of this document may include units implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. The "part" or "module" may be an integrally formed component or a minimum unit or part of the component that performs one or more functions. For example, according to one embodiment, the "part" or "module" may be implemented in the form of an application-specific integrated circuit (ASIC).

[0174] The term "if" as used in various embodiments of this document may be interpreted to mean "when", "when", "in response to determining", or "in response to detecting", depending on the context. Similarly, "if it is determined that" or "if ~ is detected" may be interpreted to mean "upon determining", "in response to determining", or "upon detecting", or "in response to detecting", depending on the context.

[0175] The program executed by the server device described in this document may be implemented as hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0176] Software may include a computer program, code, instructions, or a combination of one or more of these, which can configure a processing device to perform a desired operation or command the processing device, either independently or collectively. Software may be implemented as a computer program including instructions stored on a computer-readable storage medium. Examples of the computer-readable storage medium include magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.) and optical reading media (e.g., CD-ROMs, digital versatile discs (DVDs)). The computer-readable storage medium may be distributed across network-connected computer systems so that the computer-readable code can be stored and executed in a distributed manner. Computer programs can be distributed online (e.g., by download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0177] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, connection; and Contains a processor, The above processor: Identifying that a first external device is physically connected to the electronic device through the connection portion; Identifying connection information of the first external device, wherein the identified connection information includes device information about the first external device, image information from the first external device, and information about the connection environment of the first external device. Based on the identified connection information of the first external device and at least one machine learning model for recognizing a plurality of external devices, determining whether the first external device is a controllable device; In response to determining that the first external device is a controllable device, identifying at least one control method of the first external device; configured to determine one control method of at least one of the above control methods as the preferred control method; , device.

2. In claim 1, Including the Department of Communications, The above processor: Transmit network information to the server device through the above communication unit, configured to receive the identified connection information of the first external device from the server device, A device wherein the above network information includes information about other devices connected to the same network as the electronic device.

3. In claim 1, Including the Department of Communications, The above processor: A device configured to transmit the identified connection information of the first external device to a server device through the communication unit.

4. In claim 1, The above processor: A device configured to transmit connection information of the first external device to a server device when the device determines that the first external device is not a controllable device.

5. In claim 1, The above processor: A device wherein the processor identifies that the first external device is connected to the electronic device through the connection when the processor identifies that the first external device connected to the electronic device is powered on.

6. In claim 1, At least one of the above control methods, A device comprising at least one of a first method using IR (infrared), a second method using IP (internet protocol), and a third method using CEC (consumer electronics control).

7. In claim 1, The above processor: In response to identifying at least one control method of the first external device, configured to display a user interface for determining a preferred control method of the first external device; A device wherein said user interface comprises at least one object each indicating at least one of said identified control methods.

8. In claim 7, The above processor: A device configured to determine a control method corresponding to the selected object as the preferred control method of the first external device in response to identifying a user input selecting an object of at least one object indicating at least one control method identified above.

9. In claim 1, The above processor, Identifying the authentication information received from the first external device, A device configured to determine that the first external device is a controllable device if the authentication information received from the first external device is valid.

10. In claim 9, The above processor, A device configured to determine whether authentication information received from the first external device is valid using an optical character recognition (OCR) method.

11. In claim 1, At least one machine learning model for recognition of the above plurality of external devices, A device comprising at least one of a first machine learning model learned using device information of the plurality of external devices, a second machine learning model learned using image information of the plurality of external devices, and a third machine learning model learned using data about a device from which connection has failed among the plurality of external devices.

12. In claim 11, A device wherein at least one machine learning model is received from a server device.

13. In claim 1, The above connection part is a device including an HDMI (high definition multimedia interface) connection part.

14. A display device including a connecting portion; A first external device connected to the above display device; A server device for managing a plurality of devices connected to the display device; and Including a second external device for remotely controlling the display device; The above display device: Identifying that the first external device is physically connected to the display device through a connection portion, Identifying connection information of the first external device, wherein the identified connection information includes device information about the first external device, image information from the first external device, and information about the connection environment of the first external device. Based on the identified connection information of the first external device and at least one machine learning model for recognizing a plurality of external devices, determining whether the first external device is a controllable device; In response to determining that the first external device is a controllable device, identifying at least one control method of the first external device; Determine one control method of at least one of the above control methods as the preferred control method, The above server device: Receive network information from the above display device, Identifying the connection information of the first external device based on the above network information, Transmitting connection information of the first external device to the display device, A system that trains at least one machine learning model using connection information of the first external device.

15. In claim 14, At least one machine learning model, A system comprising at least one of a first machine learning model learned using device information of the plurality of devices, a second machine learning model learned using image information of the plurality of devices, or a third machine learning model learned using data about a device among the plurality of devices from which connection has failed.

Images