method and system for Multi-view linkage

Abstract

A multi-view linkage method according to an embodiment of the present invention comprises: a step of selecting an object to be extracted from content output from a smart device; a step in which the smart device extracts the selected object and stores cursor position information and object information; a step in which the smart device executes an object movement app to display the object on the smart device screen; a step of moving the displayed object and the cursor in response to user input; and a step of retrieving and outputting object information stored in advance at the final position where the cursor has moved.

Description

Technology Field

[0001] The present invention relates to a multi-view linkage method as a smart device technology. Background Technology

[0002] Recently, digital TV services using wired or wireless communication networks have become commonplace. Digital TV services can provide a variety of services that were not available through existing analog broadcasting services.

[0003] In particular, as IPTV (Internet Protocol Television) and Smart TV have recently become capable of sharing and displaying content such as video with user terminals like smartphones, users are being provided with the convenience of viewing high-definition content filmed on mobile terminals on a larger screen.

[0004] Meanwhile, as the number of channels provided through TV increases, multi-screen technology is being developed to simultaneously display objects supplied through content on a single screen. The problem to be solved

[0005] Accordingly, one embodiment of the present invention aims to provide a multi-view linkage method and system that recognizes and separates a specific object within content currently in use, moves it to another app, and displays it.

[0006] The problems of the present invention are not limited to those mentioned above, and other unmentioned problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0007] To achieve the above objective, a multi-view linkage method according to an embodiment of the present invention comprises: a step of selecting an object to be extracted from content output from a smart device; a step in which the smart device extracts the selected object and stores cursor position information and object information; a step in which the smart device executes the object movement app to display the object on the smart device screen; a step of moving the displayed object and the cursor in response to user input; and a step of retrieving and outputting previously stored object information at the final position where the cursor has moved.

[0008] According to one embodiment, the method further includes a step of determining whether the multi-view mode is enabled or disabled.

[0009] According to one embodiment, when the smart device is in a multi-view mode activated, the method includes the step of configuring an area of ​​the screen of the smart device as a first window where the content is displayed; and dividing an area other than the first window into a second window.

[0010] According to one embodiment, when the smart device is in a state where multi-view mode is disabled, the step of executing the second window when the cursor moves to an edge area of ​​the smart device screen in response to user input is included.

[0011] According to one embodiment, the step of moving the exposed object to a multi-view area in response to user input includes: moving the object from a first viewer area where content is displayed within the screen of the smart device to the second window executed according to the multi-view mode; and exposing the object to the screen of the smart device in the second window.

[0012] According to one embodiment, the method further includes the step of operating a multi-view mode between the smart device and an external device connected thereto.

[0013] According to one embodiment, when a multi-view mode between an external device connected to the smart device is activated, the method includes the step of configuring the entire screen of the smart device into a first window where the content is displayed; and the step of dividing the entire screen of the external device into a second window.

[0014] According to one embodiment, when the multi-view mode between the smart device and the external device connected thereto is disabled, the method includes the step of displaying a popup menu corresponding to the execution of the multi-view on the screen of the smart device in response to user input; and when the multi-view is executed, the method includes the step of executing a second window on the entire screen of the external device.

[0015] Specific details of other embodiments are included in the detailed description and drawings. Effects of the invention

[0016] According to an embodiment of the present invention, there is one or more of the following effects.

[0017] The multi-view linkage method of the present invention allows users to easily separate necessary objects from content being viewed and utilize them in various apps, thereby promoting a richer user experience.

[0018] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description in the claims. Brief explanation of the drawing

[0019] FIG. 1 is a block diagram showing the configuration of a smart device according to one embodiment of the present invention. FIG. 2 is a block diagram illustrating a multi-view linkage system according to one embodiment of the invention. FIG. 3 is a diagram illustrating a multi-view linkage method in a multi-view mode activation state of a smart device according to an embodiment of the present invention. FIG. 4 is a diagram illustrating a multi-view linkage method in a state where the multi-view mode of a smart device is disabled according to an embodiment of the present invention. FIG. 5 is a diagram illustrating a multi-view linkage method in a multi-view mode activated state between a smart device and an external device according to an embodiment of the present invention. FIG. 6 is a diagram illustrating a multi-view linkage method in a state where the multi-view mode between a smart device and an external device is disabled, according to an embodiment of the present invention. FIG. 7 is a diagram illustrating an app operation method according to a multi-view linkage method according to an embodiment of the present invention. FIG. 8 is a diagram illustrating a multi-view linkage method between a smart device and an external device according to an embodiment of the present invention. FIG. 9 is a flowchart illustrating a multi-view linkage method between a smart device and an external device according to an embodiment of the present invention. Specific details for implementing the invention

[0020] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols will be assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0021] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0022] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0023] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0024] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0025] Hereinafter, a TV and a method of operation according to an embodiment of the present invention will be described in detail with reference to the attached drawings.

[0026] A TV according to an embodiment of the present invention is, for example, an intelligent TV that adds computer support functions to a broadcast reception function. While faithful to the broadcast reception function, it also adds internet functions and can be equipped with interfaces that are more convenient to use, such as a handwriting input device, a touch screen, or a spatial remote control. Furthermore, by supporting wired or wireless internet functions, it can connect to the internet and computers, and perform functions such as email, web browsing, banking, or games. A standardized general-purpose OS may be used for these various functions.

[0027] Accordingly, the TV described in the present invention allows various applications to be freely added or removed, for example, on a general-purpose OS kernel, thereby enabling various user-friendly functions to be performed. Meanwhile, the TV described in the present invention may be, more specifically, a network TV, HBB TV, smart TV, etc., and may also be applicable to smartphones depending on the circumstances.

[0028] Furthermore, embodiments of the present invention will be described in detail below with reference to the attached drawings and the contents described therein, but the present invention is not limited or restricted by the embodiments.

[0029] The terms used in this specification have been selected to be as widely used as possible, taking into account their functions in the present invention; however, these may vary depending on the intent or convention of those skilled in the art or the emergence of new technologies. In addition, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in the relevant description of the invention. Therefore, it should be noted that the terms used in this specification should be interpreted based on their actual meaning and the overall content of this specification, rather than merely their names.

[0030] FIG. 1 is a block diagram showing the configuration of a TV according to one embodiment of the present invention.

[0031] Referring to FIG. 1, the smart device (100) may include a broadcast receiver (130), an external device interface unit (135), a storage unit (140), a user interface unit (150), a control unit (170), a display unit (180), an audio output unit (185), and a power supply unit (190). Meanwhile, the broadcast receiver (130) may include a tuner (131), a demodulator (132), and a network interface unit (133).

[0032] The tuner (131) selects an RF (Radio Frequency) broadcast signal corresponding to a channel selected by the user or a pre-stored channel among the RF broadcast signals received through the antenna, and can convert the selected RF broadcast signal into an intermediate frequency signal or a baseband video or audio signal.

[0033] For example, the tuner (131) can convert the selected RF broadcast signal into a digital IF signal (DIF) if it is a digital broadcast signal, and into an analog baseband video or audio signal (CVBS / SIF) if it is an analog broadcast signal.

[0034] That is, the tuner (131) can process both digital broadcast signals and analog broadcast signals, and the analog baseband video or audio signal (CVBS / SIF) output from the tuner (131) can be directly input to the control unit (170).

[0035] Additionally, the tuner (131) can receive a single-carrier RF broadcast signal according to the ATSC (Advanced Television System Committee) method or a multiple-carrier RF broadcast signal according to the DVB (Digital Video Broadcasting) method.

[0036] Meanwhile, the tuner (131) can sequentially select RF broadcast signals of all broadcast channels stored through the channel memory function among the RF broadcast signals received through the antenna and convert them into intermediate frequency signals or baseband video or audio signals.

[0037] The demodulator (132) can receive the digital IF signal (DIF) converted by the tuner (131) and perform a demodulation operation. For example, if the digital IF signal output from the tuner (131) is in the ATSC format, the demodulator (132) can perform 8-VSB (8-Vestigal Side Band) demodulation.

[0038] Additionally, the demodulation unit (132) may perform channel decoding, and for this purpose, it may perform trellis decoding, deinterleaving, and Reed Solomon decoding by being equipped with a trellis decoder, a de-interleaver, and a Reed Solomon decoder.

[0039] For example, if the digital IF signal output from the tuner (131) is in the DVB format, the demodulator (132) can perform COFDMA (Coded Orthogonal Frequency Division Modulation) demodulation.

[0040] Additionally, the demodulation unit (132) may perform channel decoding, and for this purpose, it may perform convolution decoding, deinterleaving, and Reed-Soloman decoding by being equipped with a convolution decoder, a deinterleaver, and a Reed-Soloman decoder.

[0041] The demodulation unit (132) can output a stream signal (TS) after performing demodulation and channel decoding, and the stream signal may be a signal in which a video signal, an audio signal, or a data signal is multiplexed.

[0042] For example, the stream signal may be an MPEG-2 TS (Transport Stream) multiplexed with video signals of the MPEG-2 standard, audio signals of the Dolby AC-3 standard, etc. Specifically, the MPEG-2 TS may include a 4-byte header and a 184-byte payload.

[0043] The header contains information indicating the start of the signal and the Packet Identifier (PID) for the signal carried in the payload, and the payload contains video, audio, and channel-specific EPG information. The EPG information may include the System Time Table (STT), Master Guide Table (MGT), Virtual Channel Table (VCT), Rating Region Table (RRT), and Event Information Table (EIT) by the Program and System Information Protocol (PSIP).

[0044] Meanwhile, the demodulation unit (132) may separately include an ATSC demodulation unit and a DVB demodulation unit according to the ATSC method and the DVB method.

[0045] The stream signal output from the demodulation unit (132) can be input to the control unit (170), and the control unit (170) can output video to the display unit (180) and output voice to the audio output unit (185) after performing demultiplexing, video / audio signal processing, etc.

[0046] The external device interface section (135) can connect an external device and a smart device (100), and for this purpose, it may include an A / V input / output section (not shown) or a wireless communication section (not shown), etc.

[0047] The external device interface section (135) can be connected wirelessly or via wired connection to external devices such as DVD (Digital Versatile Disk), Blu-ray, game console, camera, camcorder, computer (laptop), etc.

[0048] Additionally, the external device interface unit (135) transmits video, audio, or data signals input from the outside through a connected external device to the control unit (170) of the smart device (100), and can output the video, audio, or data signals processed by the control unit (170) to the connected external device.

[0049] The A / V input / output unit (not shown) may include a USB terminal, a CVBS (Composite Video Banking Sync) terminal, a component terminal, an S-video terminal (analog), a DVI (Digital Visual Interface) terminal, an HDMI (High Definition Multimedia Interface) terminal, an RGB terminal, a D-SUB terminal, etc., so as to input video and audio signals from an external device into the smart device (100).

[0050] Meanwhile, a wireless communication unit (not shown) can perform short-range wireless communication with other electronic devices. For example, a smart device (100) can be network-connected with other electronic devices according to communication standards such as Bluetooth, RFID (Radio Frequency Identification), infrared communication (IrDA, infrared Data Association), UWB (Ultra Wideband), ZigBee, and DLNA (Digital Living Network Alliance).

[0051] Additionally, the external device interface section (135) may be connected to various set-top boxes through at least one of the various terminals described above, and may perform input / output operations with the set-top boxes.

[0052] Meanwhile, the external device interface unit (135) can receive an application or a list of applications within an adjacent external device and transmit it to the control unit (170) or storage unit (140).

[0053] The network interface section (133) may provide an interface for connecting the smart device (100) to a wired / wireless network including the internet. For example, the network interface section (133) may be equipped with an Ethernet terminal for connection to a wired network, and may connect to a wireless network using communication standards such as WLAN (Wireless LAN) (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), and HSDPA (High Speed ​​Downlink Packet Access).

[0054] Meanwhile, the network interface unit (133) can transmit or receive data with other users or other electronic devices through the connected network or another network linked to the connected network.

[0055] In addition, content data stored in the smart device (100) can be transmitted to another user or other electronic device selected among other users or other electronic devices that are previously registered in the smart device (100).

[0056] The network interface unit (133) can access a specific web page through a connected network or another network linked to the connected network. That is, it can access a specific web page through a network and transmit or receive data with the corresponding server.

[0057] In addition, the network interface unit (133) can receive content or data provided by a content provider or network operator. That is, the network interface unit (133) can receive content such as movies, advertisements, games, VOD, broadcast signals, and related information provided by a content provider or network provider through a network.

[0058] Additionally, the network interface unit (133) can receive firmware update information and update files provided by the network operator, and can transmit data to the internet, content provider, or network operator.

[0059] The network interface unit (133) can select and receive a desired application among the applications that are open to the public through the network.

[0060] The storage unit (140) can store programs for each signal processing and control within the control unit (170), and can store signal-processed video, audio, or data signals.

[0061] Additionally, the storage unit (140) may perform the function of temporarily storing video, audio, or data signals input from the external device interface unit (135) or the network interface unit (133), and may also store information regarding a predetermined broadcast channel through a channel memory function.

[0062] The storage unit (140) can store an application or a list of applications input from an external device interface unit (135) or a network interface unit (133).

[0063] The storage unit (140) may include at least one type of storage medium among, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory, etc.), RAM, and ROM (EEPROM, etc.).

[0064] The smart device (100) can play content files (video files, still image files, music files, document files, application files, etc.) stored in the storage unit (140) and provide them to the user.

[0065] The user interface unit (150) can transmit a signal input by the user to the control unit (170) or transmit a signal from the control unit (170) to the user. For example, the user interface unit (150) can receive and process control signals such as power on / off, channel selection, and screen setting from the remote control device (200) according to various communication methods, such as RF (Radio Frequency) communication or infrared (IR) communication, or process control signals from the control unit (170) to be transmitted to the remote control device (200).

[0066] Additionally, the user interface unit (150) can transmit control signals input from local keys (not shown), such as a power key, channel key, volume key, and setting key, to the control unit (170).

[0067] Meanwhile, the user interface unit (150) can transmit a control signal input from a sensing unit (not shown) that senses a user's gesture to a control unit (170), or transmit a signal from the control unit (170) to the sensing unit (not shown). Meanwhile, the sensing unit (not shown) may include a touch sensor, a voice sensor, a position sensor, a motion sensor, etc.

[0068] The control unit (170) can demultiplex a stream input from the tuner (131), the demodulator (132), or the external device interface unit (135), or process the demultiplexed signals to generate and output a signal for video or audio output.

[0069] The image signal processed by the control unit (170) can be input to the display unit (180) and displayed as an image corresponding to the image signal. Additionally, the image signal processed by the control unit (170) can be input to an external output device through the external device interface unit (135).

[0070] The voice signal processed by the control unit (170) can be output as audio to the audio output unit (185). Additionally, the voice signal processed by the control unit (170) can be input to an external output device through the external device interface unit (135).

[0071] In addition, the control unit (170) can control the overall operation within the smart device (100). For example, the control unit (170) can control the tuner (131) to select (Tuning) an RF broadcast corresponding to a channel selected by the user or a pre-stored channel.

[0072] Additionally, the control unit (170) can control the smart device (100) by means of a user command or internal program input through the user interface unit (150), and can connect to a network to allow the user to download an application or a list of applications desired by the user into the smart device (100).

[0073] For example, the control unit (170) controls the tuner (131) so that a signal of the selected channel is input according to a predetermined channel selection command received through the user interface unit (150), and can process the video, audio, or data signal of the selected channel.

[0074] The control unit (170) enables the processed video or audio signal, such as channel information selected by the user, to be output through the display unit (180) or audio output unit (185).

[0075] Additionally, the control unit (170) enables a video signal or audio signal from an external device, such as a camera or camcorder, input through the external device interface unit (135) according to an external device video playback command received through the user interface unit (150), to be output through the display unit (180) or audio output unit (185).

[0076] Meanwhile, the control unit (170) can control the display unit (180) to display an image, for example, a broadcast image input through the tuner (131), an external input image input through the external device interface unit (135), an image input through the network interface unit, or an image stored in the storage unit (140) can be controlled to be displayed on the display unit (180). In this case, the image displayed on the display unit (180) may be a still image or a video, and may be a 2D image or a 3D image.

[0077] Additionally, the control unit (170) can control the playback of content stored in the smart device (100), received broadcast content, or external input content input from the outside, and the content may be in various forms such as broadcast video, external input video, audio file, still image, connected web screen, and document file.

[0078] Meanwhile, although not illustrated in FIG. 1, the smart device (100) may further include a channel browsing processing unit that generates a thumbnail image corresponding to a channel signal or an external input signal.

[0079] The channel browsing processing unit receives a stream signal (TS) output from the demodulation unit (132) or a stream signal output from the external device interface unit (135), and can generate a thumbnail image by extracting an image from the input stream signal.

[0080] The generated thumbnail image can be input to the control unit (170) as is or encoded, or it can be input to the control unit (170) encoded in the form of a stream.

[0081] The control unit (170) can display a thumbnail list having multiple thumbnail images on the display unit (180) using the input thumbnail image, and the thumbnail images in the thumbnail list can be updated sequentially or simultaneously. Accordingly, the user can easily identify the content of multiple broadcast channels.

[0082] The display unit (180) can generate a driving signal by converting the video signal, data signal, OSD signal processed by the control unit (170) or the video signal, data signal, etc. received from the external device interface unit (135) into R, G, and B signals, respectively.

[0083] For this purpose, the display unit (180) may use a PDP, LCD, OLED, flexible display, 3D display, etc., or may be configured as a touch screen and used as an input device in addition to an output device.

[0084] The audio output unit (185) can receive a voice-processed signal from the control unit (170), for example, a stereo signal, a 3.1 channel signal, or a 5.1 channel signal, and output it as voice, and can be implemented using various types of speakers for this purpose.

[0085] Meanwhile, the smart device (100) may further be equipped with a shooting unit (not shown) for acquiring a user's image, and the image information acquired from the shooting unit (not shown) may be input to a control unit (170).

[0086] In this case, the control unit (170) may detect the user's gesture by each or in combination the image captured from the shooting unit (not shown) or the signal detected from the sensing unit (not shown).

[0087] The power supply unit (190) can supply power throughout the smart device (100) and can supply power to the control unit (170), display unit (180), and audio output unit (185), which may be implemented in the form of a System On Chip (SOC), for example.

[0088] For this purpose, the power supply unit (190) may be equipped with a converter (not shown) that converts AC power into DC power, and when the display unit (180) is implemented as a liquid crystal panel having a plurality of backlight lamps, it may be further equipped with an inverter (not shown) capable of PWM operation for brightness variation or dimming drive.

[0089] The remote control device (200) transmits user input to the user interface unit (150). To do this, the remote control device (200) may use Bluetooth, RF (Radio Frequency) communication, infrared (IR) communication, UWB (Ultra Wideband), ZigBee, etc.

[0090] Additionally, the remote control device (200) can receive video, voice, or data signals, etc., output from the user interface unit (150) and display them on the remote control device (200) or output voice or vibration.

[0091] The smart device (100) described above may be a digital broadcast receiver capable of receiving at least one of the following: a fixed type of ATSC digital broadcast (8-VSB method), a DVB-T digital broadcast (COFDM method), an ISDB-T digital broadcast (BST-OFDM method).

[0092] Meanwhile, the TV (100) illustrated in FIG. 1 may be a smart device (100) according to an embodiment of the present invention, and some of the illustrated components may be integrated, added, or omitted depending on the specifications of the smart device (100) actually implemented.

[0093] That is, as needed, two or more components may be combined into a single component, or a single component may be subdivided into two or more components. In addition, the functions performed in each block are intended to explain embodiments of the present invention, and the specific operations or devices do not limit the scope of the present invention.

[0094] According to another embodiment of the present invention, the smart device (100) may receive and play video through a network interface unit (133) or an external device interface unit (135) without having a tuner (131) and a demodulator (132), unlike as shown in FIG. 1.

[0095] For example, a smart device (100) may be implemented by separating it into a video processing device, such as a set-top box, for receiving broadcast signals or content according to various network services, and a content playback device for playing content input from the video processing device.

[0096] In this case, the method of providing content or settings suitable for multiple users according to the embodiment of the present invention described below may be performed not only by the smart device (100) described with reference to FIG. 1, but also by any one of the image processing device such as the separated set-top box described above, or a content playback device having a display unit (180) and an audio output unit (185).

[0097] FIG. 2 is a block diagram illustrating a multi-view linkage system according to one embodiment of the invention.

[0098] The smart device (100) transmits object information and cursor coordinate information to an external device (300) under the control of a remote control device (200), and the external device (300) can output an object based on the received object information and cursor information.

[0099] Here, object information may include recognizable object information among the content displayed on the screen of the smart device (100). Cursor information may include the coordinates where the cursor is located on the screen of the smart device (100).

[0100] FIG. 3 is a diagram illustrating a multi-view linkage method in a multi-view mode activation state of a smart device according to an embodiment of the present invention.

[0101] Referring to FIG. 3, when the smart device (100) operates in multi-view mode, a multi-screen including a first window (410) and a second window (420) can be displayed on the display unit (180). The number of configuration areas is not limited to two, but may be at least two or more.

[0102] The size of each of the multiple configuration areas constituting the multi-screen may be different or the same, as shown in FIG. 3.

[0103] The first window (410) may represent a main screen and may display content provided to the main screen of the multi-screen according to user input. The second window (420) may represent a sub-screen and may display objects separated from the content displayed in the first window (410).

[0104] As illustrated in FIG. 3, when content currently being viewed is displayed on the first window (410) of the smart device (100), an object (520) within the content (510) can be selected based on user input. Herein, user input includes selection via touch of the remote control device (200) or the display (180), and can be operated by recognizing the user's voice. In the following description of the invention, input via the remote control device (200) is described as user input.

[0105] A smart device (100) can display a cursor (530) at a first position corresponding to user input through a remote control device (200) in a first window (410), and can select a desired object (520) from the content (510) through the cursor (530). For example, the smart device (100) can determine that an object (520) within the content (510) is selected through a long-press input in which the object (520) is selected for a preset time or longer via the remote control device (200).

[0106] The smart device (100) can separate an object (520) when an object is selected within the output content (510). At this time, when an object (520) within the content (510) being output by the smart device (100) is separated, the current cursor position and object information displayed on the first window (410) can be saved. Specifically, the smart device (100) can save the separated object (520) in a separate temporary folder. In addition, the smart device (100) can save cursor information corresponding to the current cursor position along with the object information in a temporary folder (tmp).

[0107] When the object (520) is separated, the smart device (100) can run an overlay app to visually display the object's movement on the screen currently being viewed. When the app is run, the smart device (100) can load the object and cursor information stored in a temporary folder and display it in the first window (410). In addition, when the app is run, the smart device (100) can display a marker (540) around the object (520).

[0108] After the smart device (100) executes the overlay app, when an object (520) is moved to a desired second window (420) on the screen and a mouse-on command is input, the information of the second window (420), the position information of the cursor (530) within the second window (420), and the object information can be saved in a temporary folder.

[0109] The smart device (100) can operate by transmitting object information stored in a temporary folder in the LSM (Log Structured Merge) to the foreground app corresponding to the second window.

[0110] FIG. 4 is a diagram illustrating a multi-view linkage method in a state where the multi-view mode of a smart device is disabled according to an embodiment of the present invention.

[0111] Referring to FIG. 4, when a smart device (100) and an external device (300) are connected and operate in a multi-view mode, the smart device (100) and the external device (300) can operate as a multi-screen.

[0112] During multi-screen operation, the smart device (100) may represent the main screen and, depending on user input, may output content provided to the main screen of the multi-screen. The external device (300) may represent the sub-screen and may output an object separated from the content output from the external device (300).

[0113] As illustrated in FIG. 4, when content currently being viewed is displayed on the screen of a smart device (100), an object (520) within the content (510) can be selected based on user input.

[0114] A smart device (100) can have a cursor (530) at a first position corresponding to user input through a remote control device (200) displayed on the screen of the smart device (100), and can select a desired object (520) from the content (510) through the cursor (530). For example, the smart device (100) can determine that an object (520) within the content (510) has been selected through a long-press input in which the object (520) is selected for a preset time or longer via the remote control device (200).

[0115] When an object is selected within the output content (510), the smart device (100) can separate the object (520). At this time, when the object (520) within the content (510) being output by the smart device (100) is separated, the current cursor position and object information displayed on the screen of the smart device (100) can be saved. Specifically, the smart device (100) can save the separated object (520) in a separate temporary folder. In addition, the smart device (100) can save cursor information corresponding to the current cursor position along with the object information in a temporary folder (tmp).

[0116] When an object (520) is separated, the smart device (100) can run an overlay app to visually display the object's movement on the screen currently being viewed. When the app is run, the smart device (100) can load the object and cursor information stored in a temporary folder and display it on the screen of the smart device (100). In addition, when the app is run, the smart device (100) can display a marker (540) around the object (520).

[0117] After the smart device (100) executes the overlay app, when an object (520) is moved to the screen of a desired external device (300) on the screen and a mouse-on command is input, the screen information of the external device (300), the position information of the cursor (530) within the screen of the external device (300), and the object information can be saved in a temporary folder.

[0118] The smart device (100) can operate by transmitting object information stored in a temporary folder in LSM (Log Structured Merge) to the corresponding window (foreground App).

[0119] FIG. 5 is a diagram illustrating a multi-view linkage method in a multi-view mode activated state between a smart device and an external device according to an embodiment of the present invention.

[0120] Referring to FIG. 5, when the content currently being viewed on the smart device (100) is displayed in full screen, an object (520) within the content (510) can be selected based on user input.

[0121] A smart device (100) can display a cursor (530) at a first position corresponding to user input through a remote control device (200) on the smart device (100), and can select a desired object (520) from the content (510) through the cursor (530). For example, the smart device (100) can determine that an object (520) within the content (510) has been selected through a long-press input in which the object (520) is selected for a preset time or longer via the remote control device (200).

[0122] When an object is selected within the output content (510), the smart device (100) can separate the object (520). At this time, when the object (520) within the content (510) being output by the smart device (100) is separated, the current cursor position and object information displayed on the screen of the smart device (100) can be saved. Specifically, the smart device (100) can save the separated object (520) in a separate temporary folder. In addition, the smart device (100) can save cursor information corresponding to the current cursor position along with the object information in a temporary folder (tmp).

[0123] When an object (520) is separated, the smart device (100) can run an overlay app to visually display the object's movement on the screen currently being viewed. When the app is run, the smart device (100) can retrieve the object and cursor information stored in a temporary folder and display it on the screen of the smart device (100). In addition, when the overlay app is run, the smart device (100) can display a marker (540) around the object (520).

[0124] When the cursor (530) selecting an object (520) moves to the edge (550) of the screen, the smart device (100) can activate Multi-View mode.

[0125] Here, when in multi-view mode, the screen of the smart device (100) can display a multi-screen including a first window (410) and a second window (420). The number of configuration areas is not limited to two, but can be at least two or more.

[0126] When the smart device (100) operates in multi-view mode, the screen size of the content that was previously operating in full screen is reduced and can be displayed in the first window (410).

[0127] And, after the smart device (100) executes the overlay app, when the smart device (100) moves the object (520) to the desired second window (420) on the screen and a mouse-on command is input, the smart device (100) can save the second window (420) information, the position information of the cursor (530) within the second window (420), and the object information in a temporary folder.

[0128] The smart device (100) can operate by transmitting object information stored in a temporary folder in LSM (Log Structured Merge) to the corresponding window (foreground App).

[0129] FIG. 6 is a diagram illustrating a multi-view linkage method in a state where the multi-view mode between a smart device and an external device is disabled, according to an embodiment of the present invention.

[0130] Referring to FIG. 6, when a smart device (100) and an external device (300) are connected and operated, the smart device (100) and the external device (300) can operate as a multi-screen.

[0131] When the content currently being viewed is displayed on the screen of the smart device (100), an object (520) within the content (510) can be selected based on user input.

[0132] A smart device (100) can have a cursor (530) at a first position corresponding to user input through a remote control device (200) displayed on the screen of the smart device (100), and can select a desired object (520) from the content (510) through the cursor (530). For example, the smart device (100) can determine that an object (520) within the content (510) has been selected through a long-press input in which the object (520) is selected for a preset time or longer via the remote control device (200).

[0133] When an object is selected within the output content (510), the smart device (100) can separate the object (520). At this time, when the object (520) within the content (510) being output by the smart device (100) is separated, the current cursor position and object information displayed on the screen of the smart device (100) can be saved. Specifically, the smart device (100) can save the separated object (520) in a separate temporary folder. In addition, the smart device (100) can save cursor information corresponding to the current cursor position along with the object information in a temporary folder (tmp).

[0134] When an object (520) is separated, the smart device (100) can run an overlay app to visually display the object's movement on the screen currently being viewed. When the app is run, the smart device (100) can load the object and cursor information stored in a temporary folder and display it on the screen of the smart device (100). In addition, when the app is run, the smart device (100) can display a marker (540) around the object (520).

[0135] After the smart device (100) executes the overlay app, it can move the object (520) to the screen of the desired external device (300) on the screen and, when a mouse-on command is input, perform an input that can be operated on the screen of the external device (300).

[0136] The smart device (100) can output operable inputs to a pop-up menu (610). For example, operable inputs may include saving as a sticker, pasting an image, saving to storage, running an AI image generator, etc.

[0137] When the smart device (100) receives an input that can be operated on the screen of the external device (300), the external device (300) connected to the smart device (100) can execute a Multi-View mode.

[0138] Additionally, the smart device (100) can store screen information of the external device (300), location information of the cursor (530) within the screen of the external device (300), and object information in a temporary folder.

[0139] The smart device (100) can operate by transmitting object information stored in a temporary folder in LSM (Log Structured Merge) to the corresponding window (foreground App).

[0140] The external device (300) can display a window corresponding to the multi-view mode on the screen and then execute the selected command.

[0141] FIG. 7 is a diagram illustrating an app operation method according to a multi-view linkage method according to an embodiment of the present invention.

[0142] Referring to FIG. 7, when content currently being viewed is displayed on the screen of a smart device (100), a multi-viewer can be executed over the content being displayed on the screen of the smart device (100) based on user input.

[0143] Specifically, the smart device (100) can display a cursor (530) on the screen of the smart device (100) in response to user input through the remote control device (200), and can select a desired object (520) from the content (510) through the cursor (530). For example, the smart device (100) can determine that an object (520) within the content (510) has been selected through a long-press input in which the object (520) is selected for a preset time or longer via the remote control device (200).

[0144] When an object is selected within the output content (510) of the smart device (100), the smart device (100) may output an input that can be operated in the multi-viewer as a pop-up menu (710). For example, the input that can be operated in the multi-viewer may include tracing execution, capturing the current screen, etc.

[0145] For example, a smart device (100) can launch a tracing app by inputting specific keys (e.g., voice utterance, Short-Press, etc.) on the content currently being viewed. The smart device (100) can launch a multi-viewer as an overlay with the content currently being viewed as the background, and a sketch app can be launched in a window corresponding to the multi-viewer. When the sketch app is launched, the smart device (100) can perform actions such as tracing the captured content by capturing the current screen or adjusting transparency in response to user input, or drawing on the content currently being viewed.

[0146] For example, a smart device (100) receives input from a multi-viewer at the top layer, and the content being watched can be played on the layer below it.

[0147] For example, when a smart device (100) is used to receive existing content along with a multi-viewer, it can control the area that the user wants to control by separating the layer that receives the input according to information about the area that receives user input.

[0148] FIG. 8 is a diagram illustrating a multi-view linkage method between a smart device and an external device according to an embodiment of the present invention.

[0149] Referring to FIG. 8, the multi-view linkage system allows the smart device (100) and the external device (300) to perform multi-view linkage through a View layer, a system layer, and an object movement app layer.

[0150] The smart device (100) can separate the object of the content output from the first viewer on the viewer layer (View) and temporarily store the object information and cursor coordinates in a temporary folder (S10).

[0151] The smart device (100) can recognize a separated object and move the object from the View layer to the object move app layer (S20). At this time, the smart device (100) can execute the object move app along with object recognition and load object information at the previously stored cursor coordinates and display it on the screen.

[0152] The smart device (100) can move the cursor in the top layer, the object movement app layer (S30).

[0153] The smart device (100) can move a separated object within an object movement app (40) when the cursor moves. Specifically, the smart device (100) can transmit information to the current cursor coordinates and the foreground app running at that location when user input ends. If the smart device (100) is connected to an external device, the foreground app can be run on the external device.

[0154] The smart device (100) can load object information stored in a temporary folder in LSM (Log Structured Merge) and transmit it to a foreground app corresponding to a second viewer to expose the object (S50).

[0155] FIG. 9 is a flowchart illustrating a multi-view linkage method between a smart device and an external device according to an embodiment of the present invention.

[0156] The multi-view linkage system can select and separate an object to be extracted from content being viewed on a smart device (100) (S110).

[0157] The multi-view linkage system can save the separated object in a separate temporary folder and then run the object movement app to display the temporarily saved object at the cursor position (S120).

[0158] The multi-view linkage system can move the corresponding object together with the movement of the cursor (S130).

[0159] The multi-view linkage system can activate the app at the position where the cursor movement ends and transfer data of the object being stored in the temporary folder (S140).

[0160] The multi-view integration system can expose an app that can load the object by running a multi-viewer (S150).

[0161] The various embodiments described herein may be implemented, for example, in a recording medium readable by a computer or similar device using software, hardware, or a combination thereof.

[0162] According to hardware implementation, the embodiments described herein may be implemented using at least one of ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers, microcontrollers, microprocessors, and other electrical units for performing functions. In some cases, the embodiments described herein may be implemented as the control unit (180) itself.

[0163] According to the software implementation, embodiments such as the procedures and functions described herein may be implemented in separate software modules. Each of the software modules may perform one or more functions and operations described herein. Software code may be implemented as a software app written in a suitable programming language. The software code may be stored in memory (160) and executed by a control unit (180).

[0164] The detailed description of the preferred embodiments of the present invention disclosed above is provided to enable those skilled in the art to implement and practice the present invention. Although the present invention has been described with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the present invention without departing from the scope of the invention. For example, those skilled in the art may utilize each configuration described in the above embodiments in a manner that combines with one another.

[0165] Accordingly, the present invention is not intended to be limited to the embodiments shown herein, but rather to provide an optimal scope consistent with the principles and novel features disclosed herein.

Claims

Claim 1 A multi-view linkage method comprising: a step of selecting an object to be extracted from content output from a smart device; a step in which the smart device extracts the selected object and stores cursor position information and object information; a step in which the smart device executes the object movement app and displays the object on the smart device screen; a step of moving the displayed object and cursor in response to user input; and a step of retrieving and outputting previously stored object information at the final position where the cursor has moved. Claim 2 A multi-view linkage method according to claim 1, further comprising a step of determining whether the multi-view mode is in an active or inactive state. Claim 3 A multi-view linkage method according to claim 2, comprising the steps of: configuring an area of ​​the screen of the smart device as a first window where the content is displayed when the multi-view mode is activated of the smart device; and dividing an area other than the first window into a second window. Claim 4 A multi-view linkage method according to claim 3, comprising the step of executing the second window when the cursor moves to an edge area of ​​the smart device screen in response to user input when the multi-view mode of the smart device is disabled. Claim 5 In claim 4, the step of moving the exposed object to a multi-view area in response to user input comprises: the step of moving the object to the second window executed according to the multi-view mode in a first viewer area where content is displayed within the screen of the smart device; and the step of exposing the object to the smart device screen in the second window. Claim 6 A multi-view linkage method according to claim 2, further comprising the step of operating a multi-view mode between an external device connected to the smart device. Claim 7 A multi-view linkage method according to claim 6, comprising the step of configuring the entire screen of the smart device into a first window where the content is displayed when the multi-view mode between the smart device and the external device connected to the smart device is activated; and the further step of dividing the entire screen of the external device into a second window. Claim 8 A multi-view linkage method according to claim 6, further comprising: a step of displaying a popup menu corresponding to the execution of a multi-view on the screen of the smart device in response to user input when the multi-view mode between the smart device and the external device connected to the smart device is disabled; and a step of executing a second window on the entire screen of the external device when the multi-view is executed. Claim 9 A multi-view linkage system in which a smart device and an external device are connected to share content, wherein the smart device selects an object to be extracted from content displayed on a screen, the smart device extracts the selected object and stores cursor position information and object information, the smart device executes the object movement app to display the object on the smart device screen, moves the displayed object and cursor in response to user input, and retrieves and outputs the previously stored object information at the final position where the cursor has moved.

Images