Server, media play device and method for providing voice control service

A system that analyzes screen images to generate a relationship tree for voice control enables universal voice control of IPTV applications, overcoming the need for specific voice AI integration.

KR103044944B1Active Publication Date: 2026-11-06KT CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
KR1020210118446
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-09-06
Publication Date
2026-11-06
Estimated Expiration
2041-09-06

AI Technical Summary

Technical Problem

Existing IPTV systems require voice AI integration for each application to enable voice control, limiting universal control across different applications and devices.

Method used

A server and media playback device system that analyzes screen images to extract objects and text, divides the image into regions, generates a relationship tree, and provides guide information for voice control, enabling control of various applications without specific voice AI integration.

Benefits of technology

Enables universal voice control of all applications on media playback devices, regardless of manufacturer or operating system, allowing control of web pages, set-top boxes, and mobile apps through voice commands.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112021102962816-PAT00003_ABST
    Figure 112021102962816-PAT00003_ABST
Patent Text Reader

Abstract

A server providing a voice control service includes a receiving unit that receives a capture image of a screen area from a media playback device; an analysis unit that analyzes the received capture image to extract a first object related to an object within the screen area and a second object related to text within the screen area; a region division unit that divides the capture image into a plurality of regions based on the extracted first object and second object; a relationship tree generation unit that generates a relationship tree based on a relationship diagram between the first object and the second object included in the plurality of divided regions; a guide information generation unit that generates guide information based on coordinate information of a region corresponding to at least one node included in the relationship tree; and a transmission unit that transmits the generated guide information to the media playback device.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to a server, a media playback device, and a method for providing voice control services. Background Technology

[0002] IPTV (Internet Protocol Television) is an interactive broadcasting and communication service that utilizes a high-speed internet network to provide various multimedia content, such as video content like movies and broadcast programs as well as internet browsing, to television receivers. Using a remote control, viewers can easily access a wide range of internet-provided content and value-added services via IPTV, including internet browsing, watching movies, home shopping, home banking, online games, and MP3s.

[0003] As recent IPTVs have enabled the installation of applications, they have become able to provide various additional services through applications. In this regard, prior art Korean Registered Patent No. 10-1044674 discloses a method for operating a smart device application using an application channel.

[0004] Conventionally, when it was desired to control a specific application installed on an IPTV without using a remote control, it was possible to control the specific application by installing drivers related to a mouse and keyboard on the IPTV.

[0005] In order to control the detailed menu of a specific application installed on IPTV via voice commands, control was only possible if voice AI (Artificial Intelligence) dependent on that specific application was integrated. The problem to be solved

[0006] The present invention provides a server, a media playback device, and a method for receiving a capture image of a screen area from a media playback device, analyzing the received capture image to extract a first object related to an object within the screen area and a second object related to text within the screen area.

[0007] The present invention aims to provide a server, a media playback device, and a method for dividing a captured image into multiple regions based on a first object and a second object.

[0008] The present invention aims to provide a server, a media playback device, and a method for generating a relationship tree based on a relationship diagram between a first object and a second object included in a plurality of regions, generating guide information based on coordinate information of a region corresponding to at least one node included in the relationship tree, and transmitting the generated guide information to a media playback device.

[0009] However, the technical problems that this embodiment aims to solve are not limited to the technical problems described above, and other technical problems may exist. means of solving the problem

[0010] As a means for achieving the technical problem described above, one embodiment of the present invention may provide a voice control service providing server comprising: a receiving unit that receives a capture image of a screen area from a media playback device; an analysis unit that analyzes the received capture image to extract a first object related to an object within the screen area and a second object related to text within the screen area; a region division unit that divides the capture image into a plurality of regions based on the extracted first object and second object; a relationship tree generation unit that generates a relationship tree based on a relationship diagram between the first object and the second object included in the plurality of divided regions; a guide information generation unit that generates guide information based on coordinate information of a region corresponding to at least one node included in the relationship tree; and a transmission unit that transmits the generated guide information to the media playback device.

[0011] Another embodiment of the present invention may provide a media playback device comprising: a capture unit for capturing a screen area displayed through a display linked to the media playback device; a transmission unit for transmitting a capture image of the captured screen area to a voice control service provider server; a guide information receiving unit for receiving guide information generated based on the capture image from the voice control service provider server; a display unit for displaying the received guide information on the screen area; an input unit for receiving a voice command spoken by a user based on the guide information displayed on the screen area; and a control unit for controlling the screen area based on the voice command and the guide information, wherein the capture image is divided into a plurality of areas by the voice control service provider server, a relationship tree is generated based on a relationship diagram between objects included in the plurality of divided areas, and the guide information is generated based on coordinate information of an area corresponding to at least one node included in the relationship tree.

[0012] Another embodiment of the present invention may provide a method for providing a voice control service, comprising the steps of: receiving a capture image of a screen area from a media playback device; analyzing the received capture image to extract a first object related to an object within the screen area and a second object related to text within the screen area; dividing the capture image into a plurality of areas based on the extracted first object and second object; generating a relationship tree based on a relationship diagram between the first object and the second object included in the plurality of divided areas; generating guide information based on coordinate information of an area corresponding to at least one node included in the relationship tree; and transmitting the generated guide information to the media playback device.

[0013] The above-described means for solving the problem are merely exemplary and should not be interpreted as intended to limit the invention. In addition to the exemplary embodiments described above, additional embodiments described in the drawings and the detailed description of the invention may exist. Effects of the invention

[0014] According to any one of the means for solving the problem of the present invention described above, a server, a media playback device, and a method can be provided that enable universal control of all applications installed on a media playback device through voice commands, regardless of the manufacturer or operating system of the media playback device.

[0015] A server, a media player, and a method can be provided that enable control of web pages provided on a set-top box, control of content provided on a large kiosk, and control of apps provided on a mobile device through voice commands.

[0016] Conventionally, service developers had to directly develop control operations for the results of a user's speech within an application, but the present invention can provide a server, a media playback device, and a method that allow control operations for the results of a user's speech to be automatically controlled through a general-purpose daemon. Brief explanation of the drawing

[0017] FIG. 1 is a configuration diagram of a voice control service providing system according to one embodiment of the present invention. FIG. 2 is a configuration diagram of a voice control service providing server according to one embodiment of the present invention. FIGS. 3a to 3c are exemplary drawings for explaining the process of extracting a first object and a second object by analyzing a captured image according to an embodiment of the present invention. FIGS. 4a to 4c are exemplary drawings for explaining the process of dividing a captured image into a plurality of regions according to an embodiment of the present invention. FIGS. 5A and 5B are exemplary drawings for explaining the process of generating guide information based on a relationship tree according to an embodiment of the present invention. FIGS. 6a to 6c are exemplary drawings for explaining the process of adding a parent node in a relationship tree according to an embodiment of the present invention. FIG. 7 is a flowchart of a method for providing voice control services in a voice control service providing server according to an embodiment of the present invention. FIG. 8 is a configuration diagram of a media playback device according to one embodiment of the present invention. FIGS. 9a and 9b are exemplary drawings for explaining a process of controlling a screen area based on voice commands and guide information according to an embodiment of the present invention. FIG. 10 is an exemplary drawing for explaining the process of using a voice control service in various environments according to an embodiment of the present invention. FIG. 11 is a flowchart of a method for providing voice control services in a media playback device according to an embodiment of the present invention. Specific details for implementing the invention

[0018] Embodiments of the present invention are described below with reference to the attached drawings so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0019] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected" but also cases where they are "electrically connected" with other elements interposed between them. Furthermore, when a part is described as "including" a component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components, and it should be understood that this does not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0020] In this specification, the term "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware.

[0021] Some of the operations or functions described in this specification as being performed by a terminal or device may instead be performed by a server connected to said terminal or device. Likewise, some of the operations or functions described as being performed by a server may also be performed by a terminal or device connected to said server.

[0022] An embodiment of the present invention will be described in detail below with reference to the attached drawings.

[0023] FIG. 1 is a configuration diagram of a voice control service providing system according to an embodiment of the present invention. Referring to FIG. 1, the voice control service providing system (1) may include a voice control service providing server (110), a media playback device (120), and a display (125). The voice control service providing server (110), the media playback device (120), and the display (125) are exemplary components that can be controlled by the voice control service providing system (1).

[0024] Each component of the voice control service providing system (1) of FIG. 1 is generally connected via a network. For example, as shown in FIG. 1, the voice control service providing server (110) can be connected to the media playback device (120) at the same time or at a time interval.

[0025] A network refers to a connection structure capable of exchanging information among respective nodes, such as terminals and servers, and includes local area networks (LAN), wide area networks (WAN), the internet (WWW: World Wide Web), wired and wireless data communication networks, telephone networks, wired and wireless television communication networks, etc. Examples of wireless data communication networks include, but are not limited to, 3G, 4G, 5G, WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth communication, infrared communication, ultrasonic communication, visible light communication (VLC), and LiFi.

[0026] The voice control service provider server (110) can receive a capture image of a screen area from a media playback device (120).

[0027] The voice control service providing server (110) can analyze the received capture image to extract a first object related to an object within a screen area and a second object related to text within a screen area. For example, the voice control service providing server (110) can extract the first object or the second object from the capture image using at least one of a first learning model learned about the first object or the second object through web crawling or a second learning model learned about the first object or the second object through analysis of the source code of a web page.

[0028] A voice control service providing server (110) can extract at least one text from a captured image and perform clustering on the extracted at least one text to create a second object. For example, the voice control service providing server (110) can derive center point coordinates for each of the first text and the second text among at least one text, calculate the distance between texts based on the derived center point coordinates for each of the first text and the second text, and if the calculated distance between texts is less than a threshold, create a second object including the first text and the second text.

[0029] The voice control service providing server (110) can extract at least one metadata from a first object or a second object and measure similarity between metadata based on the extracted at least one metadata.

[0030] The voice control service providing server (110) can divide the captured image into multiple regions based on the extracted first object and second object. For example, the voice control service providing server (110) can perform clustering based on the center coordinates of each of the first object and second object, and divide the captured image into multiple regions based on the result of the clustering performed.

[0031] The voice control service providing server (110) can generate a relationship tree based on a relationship diagram between a first object and a second object included in each of the divided multiple regions. For example, the voice control service providing server (110) can sequentially assign the first object and the second object included in each of the multiple regions to at least one node based on the level of the relationship tree, and generate guide information by performing numbering on the regions corresponding to the assigned nodes. The voice control service providing server (110) can assign either the first object or the second object included in the first region among the multiple regions to the top node included in the relationship tree, and assign either the first object or the second object included in the multiple second regions within the first region to a lower node located below the top node.

[0032] The voice control service provider server (110) can create an additional parent node for an object in the relationship tree if there is no intermediate node in the relationship tree or if at least two objects are connected to the same link.

[0033] The voice control service provider server (110) can create additional parent nodes for the first object or the second object of the relationship tree based on the measured similarity.

[0034] The voice control service providing server (110) can generate guide information based on coordinate information of an area corresponding to at least one node included in the relationship tree.

[0035] The voice control service providing server (110) can transmit the generated guide information to the media playback device (120).

[0036] The media playback device (120) can capture a screen area displayed through a display (125) linked to the media playback device (120) and transmit a captured image of the captured screen area to a voice control service provider server (110).

[0037] A media playback device (120) receives guide information generated based on a captured image from a voice control service provider server (110) and can display the received guide information on the screen area. Here, the guide information is generated based on coordinate information of an area corresponding to at least one node included in a relationship tree, and a first object and a second object included in each of a plurality of areas are sequentially assigned to at least one node based on the level of the relationship tree, and numbering is performed on the area corresponding to the assigned node, and may include numbering information and area boundary information, etc.

[0038] The media playback device (120) can receive voice commands spoken by the user based on guide information displayed on a screen area. For example, if either a first object or a second object included in a first area among a plurality of areas is assigned to the top node included in a relationship tree, the media playback device (120) can display guide information by overlaying it on a screen area corresponding to the first area and receive voice commands related to the first area from the user.

[0039] As another example, if either the first object or the second object included in a plurality of second areas within the first area is assigned to a sub-node located below the top-level node, the media playback device (120) can display guide information by overlaying it on a screen area corresponding to the second area and receive voice commands related to the second area from the user.

[0040] The media playback device (120) can control a screen area based on voice commands and guide information. For example, if the second area corresponds to the lowest node of the relationship tree, the media playback device (120) can control it to move to a link page associated with the second area based on voice commands associated with the second area and the guide information.

[0041] That is, the guide information can perform a function similar to a voice button to allow an area constituting a web page displayed through an application of the media playback device (120) to be selected via voice command.

[0042] As described above, this media playback device (120) may click or scroll a portion of the screen area like a mouse according to voice commands spoken based on guide information displayed on the screen area, but for specific voice commands, the specific voice command may be executed immediately through mapping with the ASCII code value of the keyboard. Here, specific voice commands may include, for example, "Up," "Down," "Right," "7," "Left," "Select," "Cancel," "Scroll Down," "Scroll Right," "To Home," "Exit," etc.

[0043] For example, let's assume that a media player (120) receives a voice command "Up" from a user. The media player (120) converts the voice command into text, and if the converted text is mapped to the code value '103' of the keyboard, it can generate a keyboard input 'sendKeySync(103)' through the 'sendKeySync()' method.

[0044] Developers of conventional media playback devices (120) were required to directly perform development to control operations according to the user's voice commands within the application, but the present invention allows operations according to the user's voice commands to be automatically controlled through a general-purpose daemon without a separate development process through a voice keyboard method.

[0045] FIG. 2 is a configuration diagram of a voice control service providing server according to an embodiment of the present invention. Referring to FIG. 2, the voice control service providing server (110) may include a receiving unit (210), an analysis unit (220), a region division unit (230), a relationship tree generating unit (240), a guide information generating unit (250), and a transmission unit (260).

[0046] The receiver (210) can receive a capture image of a screen area from the media playback device (120). For example, the receiver (210) can receive a capture image of a screen area captured at a predetermined interval from the media playback device (120).

[0047] The analysis unit (220) can analyze the received capture image to extract a first object related to an object within the screen area and a second object related to text within the screen area. The process of analyzing the capture image to extract the first object and the second object will be explained in detail through FIGS. 3a to 3c.

[0048] FIGS. 3a to 3c are exemplary drawings for explaining the process of extracting a first object and a second object by analyzing a captured image according to an embodiment of the present invention.

[0049] Referring to FIG. 3a, when the receiving unit (210) receives a capture image (300) of a screen area from a media playback device (120), the analysis unit (220) can analyze the capture image (300) and extract a first object related to an object and a second object related to text as meaningful objects within the screen area. Here, a meaningful object may include an object of a general class of objects provided by an existing deep learning framework and an object in which a class of objects is characteristically learned.

[0050] Referring to FIG. 3b, the analysis unit (220) can extract the first object or the second object using either the first learning model or the second learning model learned for the first object or the second object.

[0051] For example, referring to (a) (310) of FIG. 3b, the analysis unit (220) can extract the first object or the second object using a first learning model learned on the first object or the second object through web crawling. Here, the first learning model is a model in which transfer learning has been performed on an existing learning model provided by a deep learning framework, and the first learning model can not only extract a general-purpose object that has been learned, but also extract an object that is frequently used in general web and applications.

[0052] Below, we will briefly explain the learning process of the first learning model. The first learning model can be trained after web crawling is performed on commonly used objects. For example, through web crawling, button-type images related to objects provided based on JavaScript (311), objects provided based on MFC (Microsoft Foundation Class) which is mainly used for UI (User Interface) development (312), and objects provided based on PyQt5 (313) can be searched. Here, since the searched button-type images are objects extracted based on captured images applied to the application, there may be differences in color, size, etc. compared to the searched images, but there may not be a significant change in shape.

[0053] Subsequently, transfer learning can be performed on an existing training model provided by a deep learning framework. For example, when performing transfer learning, a first training model with improved accuracy in object extraction can be created by applying 'epoch: less than 10,' transformations such as image size and color, and data augmentation.

[0054] For another example, referring to (b) (320) of FIG. 3b, the analysis unit (220) can extract the first object or the second object from the captured image using a second learning model learned about the first object or the second object through the analysis of the source code of the web page. Here, the second learning model may be a model that analyzes the HTML (Hypertext Markup Language) source code of the web page to search for hyperlinks and learns the object corresponding to the searched hyperlink. In the case of the first learning model, the object can be learned using a button-type image, but since it is difficult to learn objects for non-button-type images, areas composed only of color, backgrounds, etc., the second learning model may be used in this case.

[0055] Below, we will briefly explain the learning process of the second learning model. For example, the second learning model can be generated by analyzing the HTML source code of a web page to search for the 'href' (321) tag of a hyperlink, and by searching for the 'Div' (322) tag containing the 'href' (321) tag, and by learning the corresponding area, image, background, etc., related object (323). Similarly, in the case of an Android OS, it can be generated by searching for the View-type object of an XML file that displays UI objects, and by learning the corresponding area, image, background, etc. As such, the learning process of the second learning model described here does not place restrictions on a separate OS or development language.

[0056] Through this process, it is possible to provide the advantage of navigating to the corresponding object from the hyperlinks on a web page and converting it into a button.

[0057] Referring to FIG. 3c, the analysis unit (220) can extract at least one text from a captured image, such as ‘mail’, ‘work’, ‘cafe’, ‘page’, ‘blog’, ‘logo’, ‘ge’, ‘ji’, and ‘sik’. Here, Optical Character Recognition (OCR) may be used to extract at least one text.

[0058] The analysis unit (220) can generate a second object by performing clustering on at least one extracted text. For example, the analysis unit (220) can generate a second object by performing clustering on at least one text based on the color, size, boldness, distance between texts, etc. of each text. Here, the analysis unit (220) can perform primary clustering on at least one text based on the color, size, and boldness of each text corresponding to objective indicators, and perform secondary clustering based on the distance between texts. Detailed classification of the distance between texts can be performed for cases where a user wants to move to a lower node in the relationship tree described later.

[0059] Below, we will briefly explain the process of performing clustering based on the distance between texts. The analysis unit (220) derives the center point coordinates for each of the first text and the second text among at least one text, and can calculate the distance between texts in pixels based on the derived center point coordinates for each of the first text and the second text. For example, the distance between texts between 'me' and 'il' can be calculated as 'a' (330), and the distance between texts between 'il' and 'ka' can be calculated as 'b' (331) by considering the distance between the next texts.

[0060] The analysis unit (220) can create a second object containing the first text and the second text when the distance between the calculated texts is less than a threshold. For example, if 'a=2' and 'b=4', this can be applied to 'ba>a / 5', and if the distance between each text differs by more than 20%, they are assumed to be objects of different clusters, and if the difference is less than 20%, they are assumed to be objects of the same cluster, thereby creating a second object.

[0061] Through this process, the analysis unit (220) can create second objects such as 'Mail' (340), 'Cafe' (341), 'Blog' (342), and 'Knowledge' (343).

[0062] Returning to FIG. 2, the region division unit (230) can divide the captured image into multiple regions based on the extracted first object and second object. The process of dividing the captured image into multiple regions will be explained in detail through FIG. 4a to 4c.

[0063] FIGS. 4a to 4c are exemplary drawings for explaining the process of dividing a captured image into a plurality of regions according to an embodiment of the present invention.

[0064] FIG. 4a is an exemplary drawing illustrating a web page divided into a plurality of regions according to an embodiment of the present invention. Referring to FIG. 4a, a web page can generally be divided into a plurality of regions (400) through 'div' tags in the source code.

[0065] In this way, to divide the captured image into multiple regions (400), the region division unit (230) can divide the captured image into multiple regions (400) based on the first object and the second object. Here, the reason for dividing the captured image into multiple regions (400) is that if all guide information related to the first object and the second object is exposed on the screen area, a problem may arise in which information loss occurs in some areas as the screen area is obscured by the guide information related to each object.

[0066] To improve this, the captured image can be divided into multiple regions considering the first object and the second object, and then a relationship tree can be used to assign each of the divided multiple regions to the top-level node corresponding to the root node so that only minimal guide information is displayed.

[0067] Guide information can be exposed sequentially starting from the top node of the relationship tree through these multiple regions (400). If a user selects a region they wish to select starting from the top node corresponding to each of the multiple regions (400), they can be guided to select a final node sequentially based on the relationship tree.

[0068] FIG. 4b is an exemplary drawing for explaining the process of performing clustering based on a first object and a second object according to an embodiment of the present invention. Referring to FIG. 4b, the region division unit (230) can perform clustering (420) based on the center coordinates of each of the first object and the second object within the screen area (410). For example, the region division unit (230) can repeatedly perform clustering (420) using a K-means algorithm with a K value between 2 and 8. Here, the K value is set to 2 or more because when 'K value=1', there is only one top node in the relationship tree described later, making region division into multiple regions meaningless, but the maximum value can be set to 8 so that it is not divided into too many regions.

[0069] FIG. 4c is an exemplary diagram illustrating the process of dividing a captured image into multiple regions according to an optimal K value according to an embodiment of the present invention. Referring to FIG. 4c, the region division unit (230) can divide the captured image into multiple regions based on the result (430) of the clustering performed. For example, the region division unit (230) can determine an optimal K value based on the result (430) of the clustering and divide the captured image into multiple regions based on the determined optimal K value. Here, the optimal K value can be determined as the K value before the value of 'inertia' decreases to 50 or less based on an elbow method (440) that determines the optimal number of clusters. The elbow method (440) refers to a method in which a section occurs where the 'inertia', which represents the sum of distances between clusters, decreases rapidly, and the K value of that section is used as the number of clusters.

[0070] Returning to FIG. 2, the relationship tree generation unit (240) can generate a relationship tree based on the relationship diagram between the first object and the second object included in each of the divided multiple regions.

[0071] The relationship tree generation unit (240) can sequentially assign a first object and a second object included in each of a plurality of regions to at least one node based on the level of the relationship tree. For example, the relationship tree generation unit (240) can assign either the first object or the second object included in the first region among the plurality of regions to the top node included in the relationship tree. Subsequently, the relationship tree generation unit (240) can assign either the first object or the second object included in a plurality of second regions within the first region to a child node located below the top node. The process of generating guide information based on the relationship tree will be explained in detail through FIGS. 5a and 5b.

[0072] FIGS. 5A and 5B are exemplary drawings for explaining the process of generating guide information based on a relationship tree according to an embodiment of the present invention.

[0073] Referring to FIG. 5a, the relationship tree generation unit (240) can assign a plurality of regions (500 to 502) to the top node of the region-specific relationship tree. For example, it can be assigned to Node A (511), which is the top node of the relationship tree (510) for the first region (500) among the plurality of regions (500 to 502).

[0074] Referring to FIG. 5b, a relationship tree (530) for the second area (501) can be generated based on the first object and the second object included in the second area (501) among the plurality of areas (500 to 502).

[0075] For example, let’s assume that ‘a object’ (520), ‘b object’ (521), and ‘c object’ (522) are included within the second area (501), that ‘a object’ (520, e.g., an image) is included within the rectangle of the second area (501), that ‘b object’ (521, e.g., a human face) is included within the rectangle containing ‘a object’ (520), and that ‘c object’ (522, e.g., text) is included within the rectangle containing ‘b object’ (521).

[0076] The relationship tree generation unit (240) can generate a relationship tree by defining another object as a child node when another object is included in a sub-region within the second area (501). For example, the relationship tree generation unit (240) can assign the second area (501) to the top node, 'A node' (531), assign 'a object' (520) to 'B node' (532), which is a sub-node of 'A node' (531), assign 'b object' (521) to 'E node' (533), which is a sub-node of 'B node' (532), and assign 'c object' (521) to 'K node' (534), which is a sub-node of 'E node' (533).

[0077] Returning to FIG. 2, the relationship tree generation unit (240) can create additional parent nodes for objects in the relationship tree. The process of adding parent nodes will be explained in detail through FIG. 6a and FIG. 6b.

[0078] FIGS. 6a to 6c are exemplary drawings for explaining the process of adding a parent node in a relationship tree according to an embodiment of the present invention.

[0079] Referring to FIG. 6a, the relationship tree generation unit (240) can additionally generate a parent node for an object in the relationship tree when there is no intermediate node in the relationship tree or when at least two objects are connected to the same link. Here, the parent node can be additionally generated only for sibling nodes corresponding to nodes of the same level.

[0080] For example, if an object corresponding to ‘card image’ (601), ‘A card’ (602), ‘unlimited point accumulation’ (603), and ‘70,000 won’ (604) is included within a partial area (600), the partial area (600) containing all four objects can be operated as a single button.

[0081] Meanwhile, in the same context, if the relationship tree generation unit (240) can view the object corresponding to each of the 'H node' (611) and 'I node' (612) as a single object, it can expose the object corresponding to the 'O node' (613) before the object corresponding to each of the 'H node' (611) and 'I node' (612) by additionally creating an 'O node' (613) so that a parent node containing both objects is exposed.

[0082] As described above, in order to add a parent node (or parent node), it must be confirmed that there is an association between the four objects, 'card image' (601), 'A card' (602), 'unlimited point accumulation' (603), and '70,000 won' (604), so that they can be considered as a single object. The determination of association will be explained through FIG. 6b.

[0083] Referring to FIG. 6b, the relationship tree generation unit (240) can additionally generate a parent node for the first object or the second object of the relationship tree based on the measured similarity.

[0084] The analysis unit (220) can extract at least one metadata from the first object or the second object and measure similarity between metadata based on the extracted at least one metadata. For example, the analysis unit (220) can extract metadata such as 'card', 'credit card', 'debit card', etc. from the card image (601) through deep learning. As another example, the analysis unit (220) can extract metadata such as 'card', 'limit', etc. related to 'A card' (602).

[0085] The analysis unit (220) can measure similarity by analyzing metadata extracted for each object and synonyms, similar words, and related words related to the metadata. To this end, the analysis unit (220) can extract synonyms, similar words, and related words related to the metadata. Here, synonyms include abbreviations, acronyms, etc. included in standard dictionaries, and related words may include superordinate words, subordinate words, etc.

[0086] For example, the analysis unit (220) can extract 'card' as a synonym because the 'card' of 'credit card' corresponding to 'object 1', which is metadata extracted through deep learning from the card image (601), and the 'card' of 'A card' (602) corresponding to 'object 2' match.

[0087] Additionally, the analysis unit (220) can extract 'card' as a synonym because it matches the 'card' of 'credit card' corresponding to 'object 1' and the 'card' of 'credit card' which is a synonym of 'credit card'.

[0088] Additionally, the analysis unit (220) can extract related words using a Korean related word dictionary that utilizes word2vec technology. For example, when the analysis unit (220) searches for 'credit card' (620) corresponding to 'object 1' using a Korean related word dictionary, it can extract 'point' (621) as a related word. As another example, when the analysis unit (220) searches for 'ten thousand won' (630) using a Korean related word dictionary, it can extract 'limit' (631) as a related word.

[0089] Through this process, the analysis unit (220) can measure similarity using the extracted synonyms, related words, and related words. Here, the following mathematical formula 1 may be used for the similarity measurement method.

[0090]

[0091] Referring to mathematical formula 1, the analysis unit (220) measures similarity using Jaccard similarity, but in the case of synonyms and related words, it can measure by multiplying by a weighting value of α (alpha).

[0092] For example, when comparing the metadata of the ‘card image’ (601) corresponding to ‘object 1’ with the metadata of the ‘A card’ (602) corresponding to ‘object 2’, the similarity can be measured as ‘0.334’ through the common metadata of ‘card’. Here, the α (alpha) value is set to 0.7 for synonyms and 0.3 for related words, and the similarity between the final objects can be measured through the sum of all calculation results.

[0093] If the similarity measured through this process exceeds a certain threshold, it is assumed that there is a close association between the two objects, and they can be grouped into objects of the same level to create a new parent node. Here, the α (alpha) value and the threshold may not be specified.

[0094] Referring to FIG. 6c, the analysis unit (220) can determine whether the combined images are a single image based on the similarity between objects through deep learning. At this time, the analysis unit (220) can predict the relationship between a first object related to an object and a second object related to text, and if it is predicted to be a single related object, it can determine that it is an object corresponding to a large cluster and assign it as a top node of the relationship tree.

[0095] For example, in the case of a web page (640) of a portal site, an image (641) and a description of the image (641) are often provided as a single hyperlink, so the analysis unit (220) can determine whether they are one object by analyzing the relationship between a first object related to the image (641) and a second object related to the text. For example, if a bear is included within a specific image (642), the analysis unit (220) can determine that the specific image (642) and the text and description of 'bear' are one object.

[0096] As another example, in the case of the text ‘Pre-character & culture generation’ (644), if it is predicted to be text located in a single banner image (643), the analysis unit (220) may not extract it as separate text and may assign the banner image (643) as a parent node. At this time, if text selection is required, a child node corresponding to a second object related to the text may be selected based on the relationship tree.

[0097] As another example, in the case of text, clustering can be performed based on font, font size, underline presence, italics, bold, color, line breaks, periods, etc., to determine the starting text and the ending text, thereby clustering parts expected to be second objects related to one text and assigning them to upper nodes. For example, the analysis unit (220) can extract a second object related to one text up to 'Police Chief selection is gaining momentum... as a variable' (655). At this time, the analysis unit (220) can extract 'c-news' as an object related to another text. Clustering can be performed using font, bold, size, etc. to divide and predict the area of ​​parts related to such text.

[0098] Returning to FIG. 2, the guide information generation unit (250) can generate guide information based on coordinate information of an area corresponding to at least one node included in the relationship tree. For example, the guide information generation unit (250) can generate guide information by performing numbering on an area corresponding to an assigned node. Here, the guide information may include numbering information and area boundary information, and the guide information may perform a function similar to a voice button to allow an area constituting a web page displayed through an application of the media playback device (120) to be selected via a voice command.

[0099] The transmission unit (260) can transmit the generated guide information to the media playback device (120).

[0100] FIG. 7 is a flowchart of a method for providing voice control services in a voice control service providing server according to an embodiment of the present invention. The method for providing voice control services in the voice control service providing server (110) shown in FIG. 7 includes steps that are processed chronologically by the voice control service providing system (1) according to the embodiment shown in FIG. 1 to 6b. Therefore, even if the following content is omitted, it is also applicable to the method for providing voice control services in the voice control service providing server (110) according to the embodiment shown in FIG. 1 to 6b.

[0101] In step S710, the voice control service providing server (110) can receive a capture image of a screen area from the media playback device (120).

[0102] In step S720, the voice control service providing server (110) can analyze the received capture image to extract a first object related to an object within the screen area and a second object related to text within the screen area.

[0103] In step S730, the voice control service providing server (110) can divide the captured image into multiple regions based on the extracted first object and second object.

[0104] In step S740, the voice control service providing server (110) can generate a relationship tree based on the relationship diagram between the first object and the second object included in each of the divided multiple regions.

[0105] In step S750, the voice control service providing server (110) can generate guide information based on coordinate information of an area corresponding to at least one node included in the relationship tree.

[0106] In step S760, the voice control service providing server (110) can transmit guide information to the media playback device (120).

[0107] In the description above, steps S710 to S760 may be further divided into additional steps or combined into fewer steps, depending on an embodiment of the present invention. Additionally, some steps may be omitted as necessary, and the order between steps may be changed.

[0108] FIG. 8 is a configuration diagram of a media playback device according to an embodiment of the present invention. Referring to FIG. 8, the media playback device (120) may include a capture unit (810), a transmission unit (820), a guide information receiving unit (830), a display unit (840), an input unit (850), and a control unit (860).

[0109] The capture unit (810) can capture a screen area displayed through a display (125) linked to a media playback device (120). For example, the capture unit (810) can generate a captured image by capturing a screen area based on a predetermined period.

[0110] The transmission unit (820) can transmit a captured image of the captured screen area to the voice control service provider server (110).

[0111] The guide information receiving unit (830) can receive guide information generated based on a captured image from the voice control service providing server (110). Here, it is generated based on coordinate information of an area corresponding to at least one node included in a relationship tree, and may further include numbering information and area boundary information. For example, the guide information may be generated by the voice control service providing server (110) sequentially assigning a first object and a second object included in each of a plurality of areas to at least one node based on the level of the relationship tree, and performing numbering on the area corresponding to the assigned node.

[0112] The display unit (840) can display the received guide information on the screen area.

[0113] The input unit (850) can receive voice commands spoken by the user based on guide information displayed on the screen area.

[0114] For example, if either the first object or the second object included in the first area among the multiple areas is assigned to the top node included in the relationship tree, the display unit (840) displays guide information by overlaying it on the screen area corresponding to the first area, and the input unit (850) can receive voice commands related to the first area from the user.

[0115] As another example, if either the first object and the second object included in a plurality of second areas within the first area are assigned to a sub-node located below the top-level node, the display unit (840) displays guide information by overlaying it on a screen area corresponding to the second area, and the input unit (850) can receive voice commands related to the second area from the user.

[0116] The control unit (860) can control the screen area based on voice commands and guide information. For example, if the second area corresponds to the lowest node of the relationship tree, the control unit (860) can control the movement to a link page associated with the second area based on voice commands associated with the second area and the guide information.

[0117] FIGS. 9a and 9b are exemplary drawings for explaining a process of controlling a screen area based on voice commands and guide information according to an embodiment of the present invention.

[0118] Referring to FIG. 9a, guide information can be displayed through the screen area of ​​the display (125) of the media playback device (120). At this time, when a user voices the numbering information included in the guide information, a specific area can be selected based on the guide information. When a user wishes to select a detailed area included in the guide information, the detailed area (corresponding to a lower node of the relationship tree) can be selected by voice-voicing an identification sound associated with the detailed area (e.g., 'again'). Here, the identification sound 'again' is merely an example and is not limited thereto. This process can be repeated from the top node to the bottom node of the relationship tree.

[0119] For example, if there are two areas (900, 910) corresponding to the top nodes of a relationship tree on the screen area, the user can select the area corresponding to the top node by uttering '1' (900) or '2' (910). For example, the user can select the area related to '1' (900) by uttering '1' (920).

[0120] Afterwards, if the user wishes to select a detailed area related to ‘A card’ or ‘70,000 won’ located within the ‘1’ (900) area, the user can select the area (901) corresponding to ‘1’ again within the ‘1’ (900) area by uttering ‘1 again’ (921).

[0121] Afterwards, if the user wishes to select a detailed area related to the ‘A card’ located inside the area (901) corresponding to ‘1 again’, the user can finally select the area (902) containing the text of the ‘A card’ corresponding to ‘2’ by uttering ‘2’ (922).

[0122] Referring to FIG. 9b, the input unit (850) can receive a voice command spoken by a user to control a scroll (931, 932) displayed on a screen area (930). Here, the scroll (931, 932) may be located on the right or bottom of the screen area (930), and the strength of the scroll may be adjusted according to the length of the voice command spoken by the user.

[0123] For example, the input unit (850) may receive a voice command, such as "scroll down," from a user to scroll a web page displayed through the screen area (930). Here, when a voice command such as "scroll down" is received, the control unit (860) may control the screen area (930) so that the web page displayed on the screen area (930) is slowly scrolled downwards.

[0124] Afterward, the input unit (850) may receive a voice command such as "straight" from the user to increase the strength of the scroll. Here, acceleration may be generated according to the length of the voice command spoken by the user, such as "straight," "straight," "straight," etc. The acceleration may be displayed on the screen area in the form of an icon (933), a round gauge (934), an avatar & color (935), a bar, etc.

[0125] FIG. 10 is an exemplary drawing for explaining the process of using a voice control service in various environments according to an embodiment of the present invention. Referring to FIG. 10, a media playback device (120) may include a set-top box, a large kiosk, a mobile phone, a tablet PC, etc.

[0126] For example, if the media playback device (120) is a set-top box (1000), the web page (1001) can be freely controlled universally through voice commands regardless of the manufacturer of the set-top box (1000) and the operating system (OS) installed on the set-top box (1000).

[0127] In another example, in the case of a large kiosk (1010) installed in a store, public place, etc., the media playback device (120) is not touchable, so it was difficult to control the large kiosk (1010) by the user. However, the present invention can provide the advantage of being able to universally control content (e.g., advertisements, games) displayed through the large kiosk (1010) using the user's voice command.

[0128] As another example, if the media player (120) is a mobile or tablet PC, the user can control an application installed on the mobile or tablet PC via voice commands. For example, if the user wants to order delivery food while driving, the user can be provided with the advantage of being able to complete the order by controlling it via voice commands without touch input to the delivery application.

[0129] FIG. 11 is a flowchart of a method for providing a voice control service in a media playback device according to an embodiment of the present invention. The method for providing a voice control service in the media playback device (120) illustrated in FIG. 11 includes steps that are processed sequentially by a voice control service providing system (1) according to the embodiments illustrated in FIG. 1 to FIG. 10. Therefore, even if the following is omitted, it applies to the method for providing a voice control service in the media playback device (120) according to the embodiments illustrated in FIG. 1 to FIG. 10.

[0130] In step S1110, the media playback device (120) can capture a screen area displayed through a display (125) connected to the media playback device (120).

[0131] In step S1120, the media playback device (120) can transmit a captured image of the captured screen area to the voice control service provider server (110).

[0132] In step S1130, the media playback device (120) can receive guide information generated based on a captured image from the voice control service provider server (110).

[0133] In step S1140, the media playback device (120) can display the received guide information on the screen area.

[0134] In step S1150, the media playback device (120) can receive voice commands spoken by the user based on guide information displayed on the screen area.

[0135] In step S1160, the media playback device (120) can control the screen area based on voice commands and guide information.

[0136] In the description above, steps S1110 to S1160 may be further divided into additional steps or combined into fewer steps, depending on an embodiment of the present invention. Additionally, some steps may be omitted as necessary, and the order between steps may be changed.

[0137] The method of providing voice control services in a voice control service providing server and a media playback device described through FIGS. 1 to 11 may also be implemented in the form of a recording medium containing a computer program stored on a medium executed by a computer or instructions executable by a computer. Additionally, the method of providing voice control services in a voice control service providing server and a media playback device described through FIGS. 1 to 11 may also be implemented in the form of a computer program stored on a medium executed by a computer.

[0138] A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and inremovable media. Additionally, a computer-readable medium may include a computer storage medium. A computer storage medium includes both volatile and non-volatile, removable and inremovable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data.

[0139] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0140] The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0141] 110: Voice control service provider server 120: Media playback device 125: Display 210: Receiver 220: Analysis Department 230: Zero Division 240 Relationship Tree Generation Section 250: Guide Information Generation Section 260: Transmission section 810: Capture section 820: Transmitter 830: Guide Information Receiver 840: Display unit 850: Input section 860: Control unit

Claims

Claim 1 A voice control service providing server comprising: a receiving unit that receives a capture image of a screen area from a media playback device; an analysis unit that analyzes the received capture image to extract a first object related to an object within the screen area and a second object related to text within the screen area; a region division unit that divides the capture image into a plurality of regions based on the extracted first object and second object; a relationship tree generation unit that generates a relationship tree based on a relationship diagram between the first object and the second object included in the plurality of divided regions; a guide information generation unit that generates guide information based on coordinate information of a region corresponding to at least one node included in the relationship tree; and a transmission unit that transmits the generated guide information to the media playback device, wherein the relationship tree generation unit sequentially assigns the first object and the second object included in each of the plurality of regions to the at least one node based on the level of the relationship tree, and the guide information includes numbering information and region boundary information for a region corresponding to the at least one node. Claim 2 A voice control service providing server according to claim 1, wherein the analysis unit extracts at least one text from the captured image and performs clustering on the extracted at least one text to generate the second object. Claim 3 A voice control service providing server according to claim 2, wherein the analysis unit derives center point coordinates for each of the first text and the second text among the at least one text, calculates the distance between texts based on the derived center point coordinates for each of the first text and the second text, and if the calculated distance between texts is less than a threshold value, generates the second object including the first text and the second text. Claim 4 A voice control service providing server according to claim 1, wherein the region division unit performs clustering based on the center coordinates of each of the first object and the second object, and divides the captured image into the plurality of regions based on the result of the performed clustering. Claim 5 delete Claim 6 A voice control service providing server according to claim 1, wherein the relationship tree generating unit assigns either the first object or the second object included in the first area among the plurality of areas to the top-level node included in the relationship tree. Claim 7 A voice control service providing server, wherein the relationship tree generating unit assigns either the first object or the second object included in the second area within the first area to a child node located below the top-level node. Claim 8 A voice control service providing server according to claim 1, wherein the relationship tree generating unit additionally generates a parent node for the object in the relationship tree when there is no intermediate node in the relationship tree or when at least two objects are connected to the same link. Claim 9 A voice control service providing server according to claim 1, wherein the analysis unit extracts at least one metadata from the first object or the second object and measures similarity between metadata based on the extracted at least one metadata, and the relationship tree generation unit additionally generates a parent node for the first object or the second object of the relationship tree based on the measured similarity. Claim 10 A voice control service providing server according to claim 1, wherein the analysis unit extracts the first object or the second object from the captured image using at least one of a first learning model learned about the first object or the second object through web crawling or a second learning model learned about the first object or the second object through source code analysis of a web page. Claim 11 A media playback device providing a voice control service comprises: a capture unit for capturing a screen area displayed through a display linked to the media playback device; a transmission unit for transmitting a capture image of the captured screen area to a voice control service providing server; a guide information receiving unit for receiving guide information generated based on the capture image from the voice control service providing server; a display unit for displaying the received guide information on the screen area; an input unit for receiving a voice command spoken by a user based on the guide information displayed on the screen area; and a control unit for controlling the screen area based on the voice command and the guide information, wherein the capture image is divided into a plurality of areas by the voice control service providing server, a relationship tree is generated based on a relationship diagram between objects included in the plurality of divided areas, and the guide information is generated based on coordinate information of an area corresponding to at least one node included in the relationship tree; a first object and a second object included in each of the plurality of areas are sequentially assigned to the at least one node based on the level of the relationship tree, and the guide information includes numbering information and area boundary information for the area corresponding to the at least one node. Claim 12 delete Claim 13 A media playback device according to claim 11, wherein either the first object or the second object included in the first area among the plurality of areas is assigned to the top-level node included in the relationship tree, the display unit displays the guide information by overlaying it on a screen area corresponding to the first area, and the input unit receives a voice command related to the first area from the user. Claim 14 A media playback device according to claim 13, wherein either the first object or the second object included in the second area within the first area is assigned to a sub-node located below the top-level node, the display unit displays the guide information by overlaying it on a screen area corresponding to the second area, and the input unit receives a voice command related to the second area from the user. Claim 15 A media playback device according to claim 14, wherein the control unit controls to move to a link page associated with the second area based on a voice command associated with the second area and guide information when the second area corresponds to the lowest node of the relationship tree. Claim 16 A method for providing a voice control service on a server comprises: receiving a capture image of a screen area from a media playback device; analyzing the received capture image to extract a first object related to an object within the screen area and a second object related to text within the screen area; dividing the capture image into a plurality of areas based on the extracted first object and second object; generating a relationship tree based on a relationship diagram between the first object and the second object included in the plurality of divided areas; generating guide information based on coordinate information of an area corresponding to at least one node included in the relationship tree; and transmitting the generated guide information to the media playback device, and further comprises the step of sequentially assigning the first object and the second object included in each of the plurality of areas to the at least one node based on the level of the relationship tree, wherein the guide information includes numbering information and area boundary information for an area corresponding to the at least one node.

Citation Information

Patent Citations

  • Video processing method, apparatus and computer program

    KR101843135B1

  • Method for Region Analysis of Documents

    KR1020020055454A

  • Media play device and method for controlling screen and server for analyzing screen

    KR1020200008341A

  • Autonomous navigation ship system for removing sea waste based on deep learning-vision recognition

    KR1020210007767A

  • Video reception device, and information-display control method for video reception device

    WO2014171046A1