Artificial intelligence system and action method thereof

Abstract

An artificial intelligence system provided by an embodiment of the present disclosure may include: a display device that receives a user input during a content playback process, and transmits the received user input and content information about the content to a first artificial intelligence (AI) server; the first AI server obtains an analysis result input by the user, and transmits the obtained analysis result and the content information to a second AI server; and the second AI server generates a cue word according to the analysis result and the content information, transmits the generated cue word to a generative AI server, receives a plurality of plot contents from the generative AI server, and transmits the received plot contents to the first AI server.

Description

Technical Field

[0001] This disclosure relates to an artificial intelligence system that generates content plots based on user input. Background Technology

[0002] Digital TV services, utilizing wired or wireless communication networks, are becoming increasingly popular. Digital TV services can provide a variety of services that are not available in existing analog broadcasting services.

[0003] For example, as a type of digital TV service, IPTV (Internet Protocol Television) and smart TV services provide interactivity, allowing users to actively choose the types of programs they want to watch and the duration of their viewing. IPTV and smart TV services can also leverage this interactivity to offer various value-added services, such as internet search, TV shopping, and online games.

[0004] In recent years, technologies that utilize artificial intelligence to generate plots have emerged one after another.

[0005] Existing AI systems generate stories based on texts such as plots, or edit highlights from complete footage (e.g., exciting clips from sports events), and provide edited footage.

[0006] However, the plots provided by existing artificial intelligence systems have certain limitations; the images can only be generated from a single content type, namely text or video.

[0007] Furthermore, existing AI systems cannot predict the plot of the next episode when generating plots. Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] The purpose of this disclosure is to automatically generate content based on image data and text data.

[0010] The purpose of this disclosure is to provide a modified version of the plot regarding content generated solely through simple language.

[0011] Methods for solving problems

[0012] The artificial intelligence system provided in the embodiments of this disclosure may include: a display device that receives user input during content playback and transmits the received user input and content information about the content to a first artificial intelligence (AI) server; the first AI server that obtains the analysis results of the user input and transmits the obtained analysis results and the content information to a second AI server; and the second AI server that generates prompt words based on the analysis results and the content information, transmits the generated prompt words to a generative AI server, receives multiple plot contents from the generative AI server, transmits the received multiple plot contents to the first AI server, the first AI server transmits the multiple plot contents to the display device, and the display device outputs the multiple plot contents.

[0013] The operating method of the artificial intelligence system provided in the embodiments of this disclosure may include the following steps: a display device receives user input during content playback; the display device transmits the received user input and content information about the content to a first artificial intelligence (AI) server; the first AI server obtains the analysis result of the user input; the first AI server transmits the obtained analysis result and the content information to a second AI server; the second AI server generates prompt words based on the analysis result and the content information; the second AI server transmits the generated prompt words to a generative AI server; the second AI server receives multiple plot contents from the generative AI server; the second AI server transmits the received multiple plot contents to the first AI server; the first AI server transmits the multiple plot contents to the display device; and the display device outputs the multiple plot contents.

[0014] Invention Effects

[0015] According to embodiments of this disclosure, users can obtain the next episode's plot information for the content they are currently watching simply through verbal communication. This allows users to access richer information about the next episode's plot, significantly improving the user experience.

[0016] According to embodiments of this disclosure, users can obtain content with a desired plot direction through simple feedback, thereby significantly improving the user experience. Attached Figure Description

[0017] Figure 1 A block diagram illustrates the structure of a display device provided in one embodiment of the present invention.

[0018] Figure 2 This is a block diagram of a remote control device provided in one embodiment of the present invention.

[0019] Figure 3 An actual structural example of a remote control device provided by an embodiment of the present invention is shown.

[0020] Figure 4 An example of using a remote control device according to an embodiment of the present invention is shown.

[0021] Figure 5 An artificial intelligence (AI) server provided in one embodiment of this disclosure is shown.

[0022] Figure 6 This is a ladder diagram used to illustrate the operation method of an AI system provided in one embodiment of this disclosure.

[0023] Figure 7 This diagram illustrates the process of generating the next episode of content that a user is currently watching, according to an embodiment of this disclosure.

[0024] Figure 8 This is a diagram illustrating the structure of an artificial intelligence system provided in one embodiment of the present disclosure.

[0025] Figure 9 This is a diagram illustrating the detailed structure of an artificial intelligence system provided in another embodiment of this disclosure.

[0026] Figure 10 Yes Figure 6 A ladder diagram illustrating the process of providing revised plot content based on user feedback.

[0027] Figure 11 This is an example illustrating the process of providing modified plot content based on user feedback. Detailed Implementation

[0028] Hereinafter, relevant embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The suffixes "module" and "part" used in the following description are assigned or used interchangeably for the convenience of writing the specification, and do not in themselves have a distinguishing meaning or function.

[0029] The display device provided in the embodiments of the present invention is, for example, an intelligent display device that adds computer support functionality to the basic function of broadcast reception. It retains the broadcast reception function while adding internet functionality, thus allowing for the use of more convenient interfaces such as manual input devices, touchscreens, or spatial remote controls. Furthermore, because it connects to the internet and computers via wired or wireless internet support, it can also perform functions such as email, web browsing, online banking, or gaming. To achieve these various functions, a standardized general-purpose OS (operating system) can be used.

[0030] Therefore, the display device described in this invention can freely add or remove various application software on a general-purpose OS kernel, thus enabling the execution of various user-friendly functions. More specifically, the display device can be, for example, a network TV, HBBTV (Hybrid Broadcast Broadband TV), a smart TV, an LED (Light Emitting Diode) TV, an OLED (Organic Light Emitting Diode) TV, etc., and may also be suitable for smartphones, depending on the circumstances.

[0031] Figure 1 A block diagram illustrates the structure of a display device provided in one embodiment of the present invention.

[0032] Reference Figure 1 The display device 100 may include a broadcast receiver 130, an external device interface 135, a memory 140, a user input interface 150, a controller 170, a wireless communication interface 173, a display 180, a speaker 185, and a power supply circuit 190.

[0033] The broadcast receiver 130 may include a tuner 131, a demodulator 132, and a network interface 133.

[0034] Tuner 131 can tune a specific broadcast channel according to a channel tuning command. Tuner 131 can receive broadcast signals about the tuned specific broadcast channel.

[0035] The demodulator 132 can separate the received broadcast signal into video signal, audio signal and data signal related to the broadcast program, and can restore the separated video signal, audio signal and data signal into an output form.

[0036] The external device interface 135 can receive application software or a list of application software from adjacent external devices and pass it to the controller 170 or the memory 140.

[0037] External device interface 135 provides a connection path between display device 100 and external devices. External device interface 135 can receive one or more video or audio signals output from external devices connected to display device 100 wirelessly or via wired connection, and transmit them to controller 170. External device interface 135 may include multiple external input terminals. These multiple external input terminals may include RGB terminals, one or more HDMI (High Definition Multimedia Interface) terminals, and component terminals.

[0038] Image signals from external devices input via external device interface 135 can be output via display 180. Audio signals from external devices input via external device interface 135 can be output via speaker 185.

[0039] The external device that can be connected to the external device interface 135 can be any of the following: set-top box, Blu-ray player, DVD player, game console, soundbar, smartphone, PC (computer), USB (Universal Serial Bus) storage device, home theater system, but this is just an example.

[0040] Network interface 133 provides an interface for connecting display device 100 to wired / wireless networks, including the Internet. Network interface 133 can send or receive data with other users or other electronic devices through the connected network or other networks linked to the connected network.

[0041] Furthermore, a portion of the content data stored in the display device 100 can be sent to a user or electronic device selected from other users or other electronic devices pre-registered in the display device 100.

[0042] Network interface 133 can access designated web pages through the connected network or other networks linked to the connected network. That is, it can access designated web pages through the network and send or receive data with the corresponding server.

[0043] Furthermore, network interface 133 can receive content or data provided by content providers or network operators. That is, network interface 133 can receive content such as movies, advertisements, games, VOD (video on demand), and broadcast signals, as well as related information, provided by content providers or network providers through the network.

[0044] Furthermore, network interface 133 can receive firmware update information and update files provided by network operators, and can send data to the Internet, content providers, or network operators.

[0045] Network interface 133 can select and receive desired application software from publicly available (open) application software via the network.

[0046] The memory 140 can store programs for processing and controlling various signals in the controller 170, and store processed image, voice or data signals.

[0047] Furthermore, the memory 140 can also perform the function of temporarily storing video, voice, or data signals input from the external device interface 135 or the network interface 133, and can store information about a specified image through the channel storage function.

[0048] The memory 140 can store application software or a list of application software input from the external device interface 135 or the network interface 133.

[0049] The display device 100 can play content files (video files, still image files, music files, document files, application software files, etc.) stored in the memory 140 and provide them to the user.

[0050] User input interface 150 can transmit user input signals to controller 170, or transmit signals from controller 170 to user. For example, user input interface 150 can receive and process control signals such as power on / off, channel selection, and screen settings from remote control device 200 according to various communication methods such as Bluetooth, WB (UltraWideband), Internet of Things (ZigBee), RF (Radio Frequency) communication, or infrared (IR) communication, or send control signals from controller 170 to remote control device 200 for processing.

[0051] Furthermore, the user input interface 150 can transmit control signals input from local keys (not shown) such as the power button, channel button, volume button, and setting value to the controller 170.

[0052] The image signal processed in the controller 170 can be input to the display 180 and displayed as an image corresponding to the image signal. Furthermore, the image signal processed in the controller 170 can be input to an external output device via the external device interface 135.

[0053] The voice signal processed in controller 170 can be output to speaker 185 in an audio manner. Furthermore, the voice signal processed in controller 170 can be input to an external output device via external device interface 135.

[0054] In addition, the controller 170 can control the overall operation of the display device 100.

[0055] Furthermore, the controller 170 can control the display device 100 using user commands or internal programs input through the user input interface 150, and can connect to a network to ensure that the application software or application software list desired by the user is downloaded to the display device 100.

[0056] The controller 170 can output user-selected channel information, along with processed video or audio signals, through a display 180 or a speaker 185.

[0057] Furthermore, the controller 170 can output image or audio signals from external devices, such as cameras or video recorders, input through the external device interface 135 via the display 180 or speaker 185, based on the external device image playback command received through the user input interface 150.

[0058] On the other hand, the controller 170 can control the display 180 to display images, such as broadcast images input through the tuner 131, externally input images input through the external device interface 135, images input through the network interface, or images stored in the memory 140. In this case, the images displayed on the display 180 can be still images or videos, or 2D or 3D images.

[0059] Furthermore, the controller 170 can control the playback of content stored in the display device 100, or received broadcast content, or externally input content, which can be various forms such as broadcast images, externally input images, audio files, still images, connected web pages, and document files.

[0060] The wireless communication interface 173 can communicate with external devices via wired or wireless communication. The wireless communication interface 173 can perform short-range communication with external devices. For this purpose, the wireless communication interface 173 can utilize at least one of the following technologies to support short-range communication: Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee (Internet of Things), NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus). This wireless communication interface 173 can support wireless communication between the display device 100 and the wireless communication system, between the display device 100 and other display devices 100, or between the network where the display device 100 (or an external server) resides, through wireless area networks. Short-range wireless communication networks can be short-range wireless personal area networks (SPANs).

[0061] Here, other display devices 100 may be mobile terminals such as wearable devices (e.g., smartwatches, smart glasses, HMDs, smartphones) capable of exchanging (or interacting with) data with the display device 100 provided by the present invention. The wireless communication interface 173 can sense (or identify) wearable devices that can communicate with the display device 100.

[0062] Furthermore, when the sensed wearable device is a certified device capable of communicating with the display device 100 provided by the present invention, the controller 170 can send at least a portion of the data to be processed in the display device 100 to the wearable device via the wireless communication interface 173. Therefore, the user of the wearable device can utilize the data to be processed in the display device 100 through the wearable device.

[0063] The display 180 can convert image signals, data signals, OSD (On-Screen Display) signals processed in the controller 170, or image signals and data signals received from the external device interface 135 into R, G, and B signals respectively, thereby generating drive signals.

[0064] on the other hand, Figure 1 The display device 100 shown is merely one embodiment of the present invention, and therefore some of the constituent elements shown may be integrated, added, or omitted according to the specifications of the actual implemented display device 100.

[0065] That is, it can be configured as needed to combine two or more constituent elements into one constituent element, or one constituent element can be subdivided into two or more constituent elements. Furthermore, the functions performed in each block are used to illustrate embodiments of the present invention, and their specific actions or devices do not limit the scope of the present invention.

[0066] According to another embodiment of the present invention, with Figure 1 Unlike the previous one, the display device 100 can also receive and play images via the network interface 133 or the external device interface 135 without being equipped with the tuner 131 and demodulator 132.

[0067] For example, the display device 100 can be implemented separately as two devices: an image processing device, such as a set-top box, for receiving broadcast signals or content related to various network services; and a content playback device for playing content input from the image processing device.

[0068] In this case, the operation method of the display device provided by the embodiments of the present invention described below can be derived not only from, but also from, as referred to Figure 1 The display device 100 described herein can be used for execution, but it can also be performed by any of the following devices: an image processing device such as the separate set-top box, or a content playback device equipped with a display 180 and an audio output unit 185.

[0069] Next, refer to Figures 2 to 3 A remote control device provided in one embodiment of the present invention will be described.

[0070] Figure 2 This is a block diagram of a remote control device provided in one embodiment of the present invention. Figure 3 An actual structural example of a remote control device 200 provided in an embodiment of the present invention is shown.

[0071] First, refer to Figure 2The remote control device 200 may include a fingerprint reader 210, a wireless communication circuit 220, a user input interface 230, a sensor 240, an output interface 250, a power supply circuit 260, a memory 270, a controller 280, and a microphone 290.

[0072] Reference Figure 2 The wireless communication circuit 220 can transmit and receive signals with any of the display devices provided in the embodiments of the present invention described above.

[0073] The remote control device 200 may be equipped with an RF (radio frequency) circuit 221 capable of transmitting and receiving signals with the display device 100 according to an RF communication standard, and may also be equipped with an IR (infrared) circuit 223 capable of transmitting and receiving signals with the display device 100 according to an IR communication standard. Furthermore, the remote control device 200 may be equipped with a Bluetooth circuit 225 capable of transmitting and receiving signals with the display device 100 according to a Bluetooth communication standard. Additionally, the remote control device 200 may be equipped with an NFC (Near Field Communication) circuit 227 capable of transmitting and receiving signals with the display device 100 according to an NFC (Near Field Communication) communication standard, and may also be configured with a WLAN (Wireless Local Area Network) circuit 229 capable of transmitting and receiving signals with the display device 100 according to a WLAN (Wireless Local Area Network) communication standard.

[0074] Furthermore, the remote control device 200 transmits a signal carrying information about the movement of the remote control device 200 to the display device 100 via the wireless communication circuit 220.

[0075] On the other hand, the remote control device 200 can receive signals transmitted by the display device 100 through the RF circuit 221, and can transmit commands regarding power on / off, channel change, volume change, etc. to the display device 100 through the IR circuit 223 as needed.

[0076] The user input interface 230 may consist of a keyboard, buttons, a touchpad, or a touchscreen. Users can operate the user input interface 230 to input commands related to the display device 100 to the remote control device 200. When the user input interface 230 is equipped with physical buttons, users can input commands related to the display device 100 to the remote control device 200 by pressing the physical buttons. For details, please refer to... Figure 3 Please provide an explanation.

[0077] Reference Figure 3The remote control device 200 may include multiple buttons. These buttons may include a fingerprint recognition button 212, a power button 231, a home button 232, a live button 233, an external input button 234, a volume adjustment button 235, a voice recognition button 236, a channel change button 237, a confirmation button 238, and a back button 239.

[0078] The fingerprint recognition button 212 can be a button for recognizing a user's fingerprint. As one embodiment, the fingerprint recognition button 212 can realize a pressing action, and therefore can also receive pressing actions and fingerprint recognition actions.

[0079] The power button 231 can be a button used to turn the power to / from the display device 100 on or off.

[0080] The home button 232 can be a button used to navigate to the home screen of the display device 100.

[0081] The live button 233 can be a button used to display live broadcast programs.

[0082] External input button 234 can be a button for receiving external input connected to display device 100.

[0083] The volume control button 235 can be a button used to adjust the volume output by the display device 100.

[0084] The voice recognition button 236 can be a button used to receive the user's voice and recognize the received voice.

[0085] The channel change button 237 can be a button used to receive broadcast signals from a specific broadcast channel.

[0086] The confirmation button 238 can be a button for selecting a specific function, and the back button 239 can be a button for returning to the previous screen.

[0087] Re-examine Figure 2 Please provide an explanation.

[0088] When the user input interface 230 is equipped with a touchscreen, the user can touch the soft keys on the touchscreen to input commands related to the display device 100 to the remote control device 200. Furthermore, the user input interface 230 can be equipped with various user-operable input methods such as scroll keys or joysticks; this embodiment does not limit the scope of the invention.

[0089] Sensor 240 may be equipped with gyroscope sensor 241 or accelerometer sensor 243. Gyroscope sensor 241 can sense information about the movement of remote control device 200.

[0090] For example, the gyroscope sensor 241 can sense information about the operation of the remote control device 200 with reference to the x, y, and z axes, and the accelerometer sensor 243 can sense information such as the moving speed of the remote control device 200. On the other hand, the remote control device 200 can also be equipped with a distance measurement sensor, which can sense the distance between itself and the display 180 of the display device 100.

[0091] The output interface 250 can output image or voice signals that correspond to the operation of the user input interface 230 or to the signals transmitted from the display device 100.

[0092] Users can sense whether the user input interface 230 is being operated or whether the display device 100 is being controlled through the output interface 250.

[0093] For example, the output interface 250 may be equipped with: an LED (light-emitting diode) 251, which is lit when the user input interface 230 is operated or when signals are transmitted and received between the wireless communication unit 225 and the display device 100; a vibrator 253, which generates vibration; a speaker 255, which outputs sound; or a display 257, which outputs images.

[0094] Furthermore, the power supply circuit 260 supplies power to the remote control device 200. When the remote control device 200 does not move within a specified time, power waste can be reduced by cutting off the power supply.

[0095] When a designated key equipped in the remote control device 200 is operated, the power supply circuit 260 can restore power supply.

[0096] The memory 270 can store various types of programs, application software data, etc. required for the control or operation of the remote control device 200.

[0097] When the remote control device 200 wirelessly transmits and receives signals with the display device 100 via the RF circuit 221, the remote control device 200 and the display device 100 transmit and receive signals via a specified frequency band.

[0098] The controller 280 of the remote control device 200 can store and refer to information such as the frequency band that enables wireless transmission and reception of signals with the display device 100, which is paired with the remote control device 200.

[0099] The controller 280 controls various matters related to the control of the remote control device 200. The controller 280 can transmit signals corresponding to the key operations of the user input interface 230 or signals corresponding to the movement of the remote control device 200 sensed in the sensor 240 to the display device 100 via the wireless communication unit 225.

[0100] Furthermore, the microphone 290 of the remote control device 200 can acquire voice.

[0101] Multiple microphones can be equipped with the 290.

[0102] Next, for Figure 4 Please provide an explanation.

[0103] Figure 4 An example of using a remote control device according to an embodiment of the present invention is shown.

[0104] Figure 4 (a) illustrates the case where the pointer 205 corresponding to the remote control device 200 is displayed on the display 180.

[0105] The user can move the remote control device 200 up and down, left and right, or rotate it. The pointer 205 displayed on the display 180 of the display device 100 corresponds to the movement of the remote control device 200. As shown, this remote control device 200 moves and displays the pointer 205 according to movement in 3D space, and therefore can be called a spatial remote control.

[0106] Figure 4 Example (b) illustrates the case where, when the user moves the remote control device 200 to the left, the pointer 205 displayed on the display device 180 also moves to the left accordingly.

[0107] Information about the movement of the remote control device 200, sensed by the sensors of the remote control device 200, is transmitted to the display device 100. The display device 100 can calculate the coordinates of the pointer 205 based on the information about the movement of the remote control device 200. The display device 100 can display the pointer 205 in a manner corresponding to the calculated coordinates.

[0108] Figure 4 Example (c) illustrates a scenario where, while holding down a specific button on the remote control device 200, the user moves the remote control device 200 away from the display 180. As a result, the selection area on the display 180 corresponding to the pointer 205 can be magnified.

[0109] Conversely, when the user moves the remote control device 200 closer to the display 180, the selection area on the display 180 corresponding to the pointer 205 can be reduced in size.

[0110] Alternatively, when the remote control device 200 is away from the display 180, the selection area is shrunk, and when the remote control device 200 is close to the display 180, the selection area is enlarged.

[0111] Furthermore, when a specific button on the remote control device 200 is pressed, up / down and left / right movement can be excluded from recognition. That is, when the remote control device 200 moves away from or towards the display 180, up / down, left / right movement is not recognized; only forward / backward movement is recognized. When the specific button on the remote control device 200 is not pressed, only the pointer 205 moves as the remote control device 200 moves up / down, left / right.

[0112] On the other hand, the moving speed or direction of the pointer 205 can correspond to the moving speed or direction of the remote control device 200.

[0113] On the other hand, the pointer in this specification refers to an object displayed on the display 180 corresponding to the action of the remote control device 200. Therefore, in addition to the arrow shape shown in the attached drawings, the pointer 205 can be implemented by objects of various shapes. For example, it can be a concept including a point, cursor, prompt word, thick outline, etc. Moreover, the pointer 205 can not only correspond to any point on the horizontal and vertical axes of the display 180, but also correspond to multiple points such as lines and surfaces.

[0114] Figure 5 An artificial intelligence (AI) server provided in one embodiment of this disclosure is shown.

[0115] Reference Figure 5 AI Server 500 can refer to a device that uses machine learning algorithms to enable artificial neural networks to learn, or a device that uses a learned artificial neural network.

[0116] The AI ​​Server 500 can also be composed of multiple servers to perform distributed processing and can be defined as a 5G network.

[0117] AI server 500 can also be part of the structure of AI device 100, jointly performing at least a portion of AI processing.

[0118] AI server 500 may include communication unit 510, memory 530, learning processor 540 and processor 560.

[0119] The communication unit 510 can send and receive data with external devices such as the display device 100.

[0120] The memory 530 may include a model storage unit 531.

[0121] The model storage unit 531 can store the model 531a (or artificial neural network) that is being learned or has been learned by the learning processor 540.

[0122] The learning processor 540 can use the learning data to enable the artificial neural network 531a to learn. The learning model can also be used in the presence of an AI server 500 installed on the artificial neural network, or installed on an external device such as a display device 100.

[0123] The learning model can be implemented by hardware, software, or a combination of hardware and software. When part or all of the learning model is implemented by software, one or more instructions constituting the learning model can be stored in memory 530.

[0124] The processor 560 can use the learning model to infer result values ​​from new input data and generate response or control commands based on the inferred result values.

[0125] Figure 6 This is a ladder diagram used to illustrate the operation method of an AI system provided in one embodiment of this disclosure.

[0126] Reference Figure 6 The AI ​​system may include a display device 100, a first AI server 500-1, a second AI server 500-2, and a generative AI server 610.

[0127] The first AI server 500-1 can be a Natural Language Processing (NLP) server that processes natural language and obtains the analysis results of voice commands.

[0128] The second AI server 500-2 can be an AI proxy server capable of obtaining response results from generative AI servers and providing search services.

[0129] The first AI server 500-1 and the second AI server 500-2 can respectively include Figure 5 The constituent elements of.

[0130] The first AI server 500-1 and the second AI server 500-2 can be composed of a single server.

[0131] The controller 170 of the display device 100 can receive voice commands related to plot generation (S601).

[0132] In one embodiment, the controller 170 may receive voice commands from a user via a microphone (not shown).

[0133] The controller 170 can receive voice commands from the remote control device 200. The remote control device 200 can receive voice commands issued by the user and transmit them to the display device 100.

[0134] Voice commands issued by users can be commands that express predictive intentions related to the plot of the content.

[0135] The controller 170 can receive voice commands related to plot generation while content is playing on the display 180.

[0136] The content can be any of the following: TV drama footage, sports event footage, photos, or images.

[0137] In this disclosure, voice commands are used as an example, but it is not limited to this; text input can also be received from the remote control device 200 instead of voice commands. In this case, the text input can be transmitted to the first AI server 500-1.

[0138] The controller 170 of the display device 100 can transmit voice data corresponding to the voice command and content information about the content being viewed to the first AI server 500-1 (S603) via the network interface 133.

[0139] Content information may include content identification information that identifies the content. Content identification information may include a content ID.

[0140] In addition to content identification information, content information may include one or more of the following: content name, episode number, current episode plot, previous episode plot, character information, next episode preview image, and main scene image representing the main scene of the previous episode. For example, when the content is a TV series video, the content name can be the TV series title, and the episode number can be the number of episodes in the TV series.

[0141] In another embodiment, the controller 170 may transmit content information and voice data separately to the first AI server 500-1. For example, the controller 170 may periodically transmit content information about the content the user is viewing to the first AI server 500-1.

[0142] The processor 560 of the first AI server 500-1 can obtain the analysis results of voice commands based on the received voice data (S605).

[0143] In one embodiment, processor 560 may utilize a Speech-to-Text (STT) engine stored in memory 530 to convert speech data into text data.

[0144] In another embodiment, processor 560 can transmit voice data to an STT server and receive converted text data from the STT server.

[0145] The processor 560 can use a natural language processing (NLP) engine to obtain analysis results of the converted text data.

[0146] The analysis results can include analysis results regarding the intent of the voice commands.

[0147] For example, when the voice command is "<How will the next episode unfold?>", the analysis result could be a request for information about the plot (or storyline) of the next episode.

[0148] The processor 560 of the AI ​​server 500 can determine whether a generative AI response is needed based on the analysis results of the voice command (S607).

[0149] In one embodiment, the processor 560 can determine whether a generative AI response is needed based on the analysis results of the voice command.

[0150] When the analysis result of a voice command is an intent to control the function of the display device 100, the processor 560 can determine that a generative AI response is not required.

[0151] When the analysis result of the voice command is not the intention for controlling the function of the display device 100, the processor 560 can determine that a generative AI response is needed.

[0152] When the analysis result of the voice command is not intended to control the function of the display device 100, the processor 560 can determine that a response from the generative AI server 600 is required.

[0153] For example, when the intent of a voice command is to control the display device 100's functions, such as screen brightness or volume, the processor 560 can determine that a generative AI response is not required. This is because these are functions that the display device 100 can handle autonomously even without the assistance of generative AI.

[0154] When the intent of a voice command is a plot prediction about the next episode of the currently viewed content, the processor 560 can determine that a generative AI response is required. This is because this is a function that the display device 100 would find difficult to handle autonomously without the assistance of generative AI.

[0155] If the processor 560 of the AI ​​server 500 determines that a generative AI response is not required, it can transmit the analysis results to the display device 100 via the communication unit 510 (S609).

[0156] The processor 560 can transmit the search results corresponding to the analysis results and intent analysis results to the display device 100 via the communication unit 510.

[0157] The processor 560 can transmit only one of the analysis results or search results corresponding to the intent analysis results to the display device 100 via the communication unit 510.

[0158] The controller 170 of the display device 100 can output the received analysis results (S611).

[0159] The controller 170 can display the search results included in the analysis results on the display 180.

[0160] The controller 170 can control the corresponding functions according to the function control commands of the display device 100 included in the analysis results.

[0161] When the processor 560 of the first AI server 500-1 determines that a generative AI response is needed, it can transmit the analysis results and content information of the voice command to the second AI server 500-2 (S613).

[0162] The processor 560 of the first AI server 500-1 can transmit the analysis results and content information of the voice command to the second AI server 500-2 through the communication unit 510 to generate prompt words.

[0163] The processor 560 of the second AI server 500-2 can generate prompts for generating plot content based on the received analysis results and content information (S615).

[0164] The second AI server 500-2 can be a server used to generate prompt words that will be input into the generative AI model.

[0165] The processor 560 of the second AI server 500-2 can generate prompt words based on the analysis results of the voice command <the request for prediction of the next episode's plot> and the content information <the name of the content (A), the next episode preview image extracted from the next episode preview image, and the main scene image representing the main scene of the current episode or the previous episode>.

[0166] The processor 560 of the second AI server 500-2 can generate prompts for predicting the plot based on the analysis results of voice commands and the plot information of the content.

[0167] The processor 560 of the second AI server 500-2 can obtain plot information by using the content name, the next episode preview image extracted from the next episode preview video, and the main scene image representing the main scene of the previous episode.

[0168] The processor 560 of the second AI server 500-2 can analyze the condition of each next episode preview image and generate a first condition text representing the analyzed condition.

[0169] To this end, the processor 560 of the second AI server 500-2 can utilize an image analysis model learned through artificial intelligence. The image analysis model can be a model based on an artificial neural network that infers the condition of an image from its contents.

[0170] The processor 560 of the second AI server 500-2 can analyze the condition of each main scene image and convert it into a second condition text representing the analyzed condition.

[0171] The processor 560 of the second AI server 500-2 can generate analysis results including voice commands, first situation text, and prompts for the second situation text.

[0172] For example, the processor 560 of the second AI server 500-2 can generate a prompt word representing a first instruction, which is <Please use the first situation text and the second situation text to generate the plot of the next episode of TV series A, the first situation text is ~~, the second situation text is ~~>.

[0173] The processor 560 of the second AI server 500-2 can transmit the generated prompts to the generative AI server 600.

[0174] As another example, the processor 560 of the second AI server 500-2 can generate a prompt word representing a second instruction, which is <Please use the preview image and main scene image of the next episode of TV series A to generate the plot of the next episode of TV series A>.

[0175] The processor 560 of the second AI server 500-2 can generate a prompt word representing a second instruction, which is <Please use the preview image of the next episode of TV series A and the plot of the previous episode to generate the plot of the next episode of TV series A>.

[0176] The prompts can include both text and images.

[0177] As an example, prompts may include a second instruction, a preview image of the next episode of TV series A, and images of key scenes.

[0178] As another example, prompts may include third instructions, preview images for the next episode of TV series A, and images of key scenes.

[0179] As another example, prompts may include third instructions, preview images for the next episode of a TV series, images of key scenes, and textual descriptions of the current or previous episode's plot.

[0180] The processor 560 of the second AI server 500-2 can generate various forms of prompts based on the analysis results and content information of voice commands.

[0181] The processor 560 of the second AI server 500-2 can transmit the generated prompts to the first AI server 500-1, and the first AI server 500-1 can transmit the prompts to the display device 100.

[0182] The display device 100 can display the prompt on the display 180.

[0183] The processor 560 of the second AI server 500-2 can transmit the generated prompt words to the generative AI server 600 (S617) via the communication unit 510.

[0184] In addition to the prompts, the processor 560 of the second AI server 500-2 can also transmit one or more of the following to the generative AI server 600: the next episode preview image, the current episode plot, the previous episode plot, or the main scene image.

[0185] The processor 560 of the second AI server 500-2 can transmit prompts and content information to the generative AI server 600.

[0186] Generative AI server 600 can generate multiple plot contents based on the received prompts and transmit the generated multiple plot contents to the second AI server 500-2 (S619).

[0187] The Generative AI Server 600 can be equipped with more than one generative AI model.

[0188] Generative AI models are artificial intelligence-based models that learn through deep learning or machine learning. They can be models that take prompt words as input and output response results.

[0189] Generative AI models can be one or more models that generate text or videos based on prompt words as input.

[0190] Generative AI models can be models that learn from large amounts of text data using recurrent neural networks (RNNs) and deep learning architectures such as transition models that use attention mechanisms.

[0191] The data used to train generative AI models can be collected from web pages or various text sources.

[0192] Generative AI models can take prompts as input and generate one or more episode images or episode videos representing the next episode's plot. These models can utilize GANs (Generative Adversarial Networks) to generate images as responses to prompts.

[0193] Generative AI models can take received prompts as input and output one or more plot points representing the next episode's plot.

[0194] Plot content can be text, images, or videos that indicate the plot of the next episode.

[0195] The processor 560 of the second AI server 500-2 can transmit multiple received plot contents to the first AI server 500-1 via the communication unit 510 (S621).

[0196] In one embodiment, the second AI server 500-2 can acquire plot content that is not based on user voice commands. That is, the second AI server 500-2 can store generated plot content that is most frequently selected by multiple viewers.

[0197] In addition to receiving multiple plot contents from the generative AI server 600, the second AI server 500-2 can also transmit the generated plot contents most selected by multiple viewers to the first AI server 500-1.

[0198] The processor 560 of the first AI server 500-1 can transmit multiple plot contents received to the display device 100 via the communication unit 510 (S623).

[0199] The controller 170 of the display device 100 can output multiple received plot contents (S625).

[0200] The controller 170 can display multiple plot contents on the display 180.

[0201] The controller 170 can display the content the user is watching, as well as multiple episodes, on the display 180.

[0202] The controller 170 can store the selected episodes from multiple episode contents according to the user's request into the memory 140 or the database.

[0203] The controller 170 can store one or more plot contents from multiple plot contents into the memory 140 or the database based on user feedback.

[0204] Figure 7 This diagram illustrates the process of generating the next episode of content that a user is currently watching, according to an embodiment of this disclosure.

[0205] The following is for reference Figure 6 The explanation is for Figure 7 Please provide an explanation.

[0206] Reference Figure 7 The display device 100 can play content images 700.

[0207] Users can press and hold the microphone button on the remote control device 200 to issue a voice command: "How will the next episode unfold?"

[0208] The remote control device 200 can transmit received voice commands to the display device 100.

[0209] The display device 100 can transmit voice data and content information corresponding to the voice command to the first AI server 500-1 via the network interface 133.

[0210] The first AI server 500-1 can convert speech data into text data and use a natural language processing engine to obtain the analysis results of the converted text data.

[0211] The first AI server 500-1 can transmit the analysis results and content information of the text data to the second AI server 500-2. The second AI server 500-2 can generate prompt words based on the analysis results and content information, and transmit the generated prompt words to the generative AI server 600.

[0212] Generative AI server 600 can generate multiple plot contents based on prompt words and transmit the generated multiple plot contents to the second AI server 500-2.

[0213] The second AI server 500-2 can transmit multiple plot contents to the first AI server 500-1. The first AI server 500-1 can transmit the received multiple plot contents to the display device 100.

[0214] As another example, the second AI server 500-2 can directly transmit multiple plot contents to the display device 100.

[0215] The display device 100 can display multiple plot contents 710, 730, 750 representing the next episode of the currently viewed content on the display 180.

[0216] Each plot point can consist of multiple plot images or plot videos.

[0217] The first plot content 710 can be content representing a happy ending.

[0218] The second plot point 730 could represent a tragic ending.

[0219] The third plot content 750 can represent the "viewer's favorite plot" that received the most votes.

[0220] The first plot content 710 and the second plot content 730 can be generated based on voice commands issued by the user.

[0221] The third plot content 750 can be the content sent by the second AI server 500-2 that is selected or shared most by multiple viewers.

[0222] The display device 100 can store the selected plot content from multiple plot contents according to the user's selection into the memory 140 or an external database.

[0223] The display device 100 can store the most popular plot content into the memory 140 or an external database based on user feedback obtained when multiple plot contents 710, 730, and 750 are played respectively.

[0224] The display device 100 can acquire user speech while playing multiple plot contents 710, 730, and 750 respectively. When the analysis result of the acquired speech indicates a positive response, the corresponding plot content is stored in the memory 140 or an external database.

[0225] Thus, according to embodiments of this disclosure, users can obtain the next episode's plot information from the content they are watching simply through verbal communication. This allows users to access richer information about the next episode's plot, significantly improving the user experience.

[0226] On the other hand, the display device 100 can also display text 701 corresponding to the user's voice command and notification 703 indicating that the next episode's plot has been predicted on the display 180.

[0227] Figure 8 This is a diagram illustrating the structure of an artificial intelligence system provided in one embodiment of the present disclosure.

[0228] Reference Figure 8 An embodiment of the present disclosure provides an artificial intelligence system 800 that may include a display device 100, an NLP (Natural Language Processing) server 810, an AI agent server 830, and a generative AI server 600.

[0229] NLP server 810 can be Figure 6 The first AI server 500-1.

[0230] AI Proxy Server 830 can be Figure 6 The second AI server 500-2.

[0231] The display device 100 can transmit log information to the second AI server 500-2.

[0232] Log information may include representations Figure 7 The log information includes information about the plot selected by the user based on their request from among the multiple plot options shown. In other words, the log information can include identification information for the plot content.

[0233] The AI ​​agent server 830 can determine the most frequently selected plot content by viewers based on the received log information.

[0234] Figure 9 This is a diagram illustrating the detailed structure of an artificial intelligence system provided in another embodiment of this disclosure.

[0235] Reference Figure 9 The artificial intelligence system 900 may include a display device 100, an ASR (Automatic Speech Recognition) server 910, an NLP server 810, an AI agent server 830, multiple AI large-scale generation servers 600-A to 600-Z, multiple service servers, and multiple portal servers.

[0236] The display device 100 can receive visual data via an internally equipped camera.

[0237] The display device 100 can receive voice data or text input corresponding to the voice commands issued by the user from the remote control device 200 through the user input interface 150.

[0238] The display device 100 can transmit voice data and content information corresponding to voice commands to the ASR server 910 via the network interface 133.

[0239] Content information may include one or more of the following: the name of the content, the plot of the current episode, the plot of the previous episode, the next episode trailer image extracted from the next episode trailer video, and the main scene image representing the main scene of the current episode or the previous episode.

[0240] The ASR server 910 can convert voice data into text data. The ASR server 910 can also be an STT server.

[0241] NLP server 810 can receive converted text data from ASR server 910.

[0242] The NLP server 810 can obtain the analysis results of voice commands from text data. The NLP server 810 can determine whether to use the AI ​​agent server 830 based on the obtained analysis results.

[0243] When the obtained analysis results are determined to require a generative AI model response, the NLP server 810 can determine that the AI ​​agent server 830 needs to be used.

[0244] When the obtained analysis results determine that a generative AI model response is not required, the NLP server 810 can determine that the AI ​​agent server 830 is not needed.

[0245] When the analysis results indicate a search intent for the content, the NLP server 810 can transmit the content search request to the service server or portal server. The NLP server 810 can then respond to the content search request by receiving the content search results from the service server or portal server.

[0246] A service server can be a content provider server, while a portal server can be a web server that provides a website.

[0247] The NLP server 810 can transmit the analysis results and content information to the AI ​​agent server 830.

[0248] The AI ​​proxy server 830 can generate prompts based on analysis results and content information.

[0249] AI Proxy Server 830 can extract plot information from content information.

[0250] AI Agent Server 830 can extract a first condition text representing the condition of the preview image from the next episode preview image included in the content information.

[0251] AI agent server 830 can extract second situation text representing the state of the main scene image from the main scene image included in the content information.

[0252] AI agent server 830 can combine first situation text and second situation text to generate plot information of content.

[0253] The AI ​​agent server 830 can generate prompts for the next episode's plot based on the analysis results of voice commands and the plot information of the content.

[0254] For example, the prompt word can be in the form of "<Please generate the plot of the next episode of the TV series based on the plot information below, the plot information is ~~~>".

[0255] The AI ​​agent server 830 can store the generated prompts in a text DB (database).

[0256] AI proxy server 830 can transmit the generated prompts to one or more of the generative AI servers 600-A to 600-Z.

[0257] Any one of the multiple generative AI servers 600-A to 600-Z can generate plots in text form.

[0258] Any one of the multiple generative AI servers 600-A to 600-Z can generate plots in the form of images.

[0259] Any one of the multiple generative AI servers 600-A to 600-Z can generate video-format plots.

[0260] Generative AI servers can include generative AI models that take textual prompts as input to generate images or videos.

[0261] Each generative AI server can store the response results of the text form of the prompt word in the text database and the response results of the image form of the prompt word in the image database.

[0262] Generative AI servers can use received prompts as input to generate plot content representing the next episode's storyline.

[0263] Multiple plot contents generated by one or more generative AI servers can be transmitted to AI agent server 830.

[0264] The AI ​​agent server 830 can store multiple plot contents into the image database.

[0265] AI agent server 830 can transmit multiple plot contents to NLP server 810, and NLP server 810 can transmit the received multiple plot contents to display device 100.

[0266] The display device 100 can display multiple received plot contents on the display 180.

[0267] Figure 10 Yes Figure 6 A ladder diagram illustrating the process of providing revised plot content based on user feedback.

[0268] Figure 10 yes Figure 6 The embodiment executed after step S625.

[0269] Reference Figure 10 The controller 170 of the display device 100 can receive user feedback through the user input interface 155 (S1001) while outputting multiple plot contents.

[0270] User feedback can be feedback on one or more scenes used to edit multiple scenes.

[0271] User feedback can be either a voice command or text input.

[0272] User feedback can be requests for changes to the characters or plot.

[0273] The controller 170 of the display device 100 can transmit the received user feedback to the first AI server 500-1 via the network interface 133 (S1003).

[0274] The first AI server 500-1 can obtain the feedback analysis results for the received user feedback (S1005) and transmit the obtained feedback analysis results to the second AI server 500-2 (S1007).

[0275] The first AI server 500-1 can use a natural language processing engine to obtain feedback analysis results based on user feedback.

[0276] The processor 560 of the second AI server 500-2 can generate modified prompts based on the feedback analysis results and the generated prompts (S1009).

[0277] The processor 560 of the second AI server 500-2 can generate modified prompt words based on the feedback analysis results and the prompt words generated in step S615.

[0278] For example, when the feedback analysis result is a request to change character A in the plot to character B, the modification prompt could be: <Please change character A to B. Use the first situation text and the second situation text to generate the plot of the next episode of TV series A. The first situation text is ~~, and the second situation text is ~~>.

[0279] The processor 560 of the second AI server 500-2 transmits the generated modified prompt words to the generative AI server 600 via the communication unit 510 (S1011), and receives multiple modified plot contents from the generative AI server 600 (S1013).

[0280] Generative AI servers can generate multiple modified plot contents based on modified prompts.

[0281] The processor 560 of the second AI server 500-2 can transmit multiple modified plot contents to the first AI server 500-1 (S1015).

[0282] The first AI server 500-1 can transmit multiple modified plot contents received to the display device 100 (S1017).

[0283] The display device 100 can output multiple modified plot contents (S1019).

[0284] Thus, according to embodiments of this disclosure, users can obtain content with a desired plot direction through simple feedback, thereby significantly improving the user experience.

[0285] Figure 11 This is an example illustrating the process of providing modified plot content based on user feedback.

[0286] Reference Figure 11 The display device 100 can display multiple plot contents 710, 730, and 750.

[0287] Users can press and hold the microphone button on the remote control device 200 to send feedback voice messages (like editing a story).

[0288] The remote control device 200 can transmit the received feedback voice to the display device 100.

[0289] The display device 100 can transmit the feedback voice data corresponding to the feedback voice to the first AI server 500-1 via the network interface 133.

[0290] The first AI server 500-1 can convert feedback voice data into text data and use a natural language processing engine to obtain feedback analysis results of the converted text data.

[0291] The first AI server 500-1 can transmit the feedback analysis results to the second AI server 500-2. The second AI server 500-2 can generate modification prompts based on the feedback analysis results and the generated prompts, and can transmit the generated modification prompts to the generative AI server 600.

[0292] Generative AI server 600 can generate multiple modified plot contents based on the modified prompt words, and transmit the generated multiple modified plot contents to the second AI server 500-2.

[0293] The second AI server 500-2 can transmit multiple modified plot contents to the first AI server 500-1. The first AI server 500-1 can then transmit the received multiple modified plot contents to the display device 100.

[0294] As another example, the second AI server 500-2 can directly transmit multiple modified plot contents to the display device 100.

[0295] The display device 100 can display multiple modified plot contents 1110, 1130, and 1150 on the display 180.

[0296] Thus, according to embodiments of this disclosure, users can obtain content with a desired plot direction through simple feedback, thereby significantly improving the user experience.

[0297] According to one embodiment of the present invention, the aforementioned method can be implemented as processor-readable code on a medium on which a program is recorded. Examples of processor-readable media include ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Read-Only Optical Disc Memory), magnetic tape, floppy disk, optical data storage device, etc.

[0298] The display device described above is not limited to the structure and method of the embodiments described above, but can be selectively constructed by combining all or part of the contents of the various embodiments to achieve various modifications of the embodiments.

Claims

1. An artificial intelligence system, comprising: A display device that receives user input during content playback and transmits the received user input and content information about the content to a first artificial intelligence server, i.e., a first AI server, where AI stands for Artificial Intelligence. The first AI server acquires the analysis results input by the user and transmits the acquired analysis results and the content information to the second AI server; and The second AI server generates prompts based on the analysis results and the content information, transmits the generated prompts to the generative AI server, receives multiple plot contents from the generative AI server, and transmits the received multiple plot contents to the first AI server. The first AI server transmits the multiple plot contents to the display device. The display device outputs the multiple plot contents.

2. The artificial intelligence system according to claim 1, wherein, The content information includes one or more of the following: the plot of the current episode, the plot of the previous episode, a teaser image indicating the next episode, and a main scene image indicating the main scene of the current episode or the previous episode.

3. The artificial intelligence system according to claim 2, wherein, The second AI server obtains a first status text representing the status of the next episode preview from the preview image, and a second status text representing the status of the main scene image from the main scene image. Based on the analysis results, the first situation text, and the second situation text, the prompt word is generated to request the generation of the next episode's plot for the content.

4. The artificial intelligence system according to claim 1, wherein, The display device displays the multiple plot contents on the display. The plot content selected by the user from the multiple plot contents is stored in the memory.

5. The artificial intelligence system according to claim 1, wherein, The display device receives user feedback and transmits the received user feedback to the first AI server. The first AI server obtains the feedback analysis results based on the user feedback and transmits the obtained feedback analysis results to the second AI server. The second AI server generates modification prompts based on the feedback analysis results and the prompts, and transmits the generated modification prompts to the generative AI server.

6. The artificial intelligence system according to claim 5, wherein, The feedback analysis results include the intention to request changes to the plot or characters of the multiple plot points.

7. The artificial intelligence system according to claim 6, wherein, The second AI server receives the multiple modified plot contents from the generative AI server, and transmits the received multiple modified plot contents to the first AI server. The first AI server transmits the multiple modified plot contents to the display device. The display device displays the multiple modified plot contents.

8. A method for operating an artificial intelligence system, comprising the following steps: The display device receives user input during content playback; The display device transmits the received user input and content information about the content to the first artificial intelligence server, i.e., the first AI server. AI stands for Artificial Intelligence; The first AI server acquires the analysis results input by the user; The first AI server transmits the obtained analysis results and the content information to the second AI server; The second AI server generates prompt words based on the analysis results and the content information; The second AI server transmits the generated prompts to the generative AI server; The second AI server receives multiple plot contents from the generative AI server; The second AI server transmits the received plot content to the first AI server; The first AI server transmits the multiple plot contents to the display device; and The display device outputs the multiple plot contents.

9. The action method of the artificial intelligence system according to claim 8, wherein, The content information includes one or more of the following: the plot of the current episode, the plot of the previous episode, a teaser image indicating the next episode, and a main scene image indicating the main scene of the current episode or the previous episode.

10. The action method of the artificial intelligence system according to claim 9, wherein, The operation method of the artificial intelligence system further includes the following steps: the second AI server obtains a first status text representing the status of the next episode preview from the preview image, and obtains a second status text representing the status of the main scene image from the main scene image. The step of generating the prompt word includes the following steps: generating the prompt word for requesting the generation of the next episode plot of the content based on the analysis results, the first situation text and the second situation text.

11. The action method of the artificial intelligence system according to claim 8, wherein, The output process includes the following steps: the display device displays the plurality of plot contents on the display. The operation method of the artificial intelligence system further includes the following steps: storing the plot content selected by the user from the multiple plot contents into a memory.

12. The method of action for the artificial intelligence system according to claim 8, wherein, The action method of the artificial intelligence system also includes the following steps: The display device receives user feedback; The display device transmits the received user feedback to the first AI server; The first AI server obtains the feedback analysis results in response to the user feedback; The first AI server transmits the obtained feedback analysis results to the second AI server; The second AI server generates modification prompts based on the feedback analysis results and the prompts. as well as The second AI server transmits the generated modification prompts to the generative AI server.

13. The action method of the artificial intelligence system according to claim 12, wherein, The feedback analysis results include the intention to request changes to the plot or characters of the multiple plot points.

14. The method of action for the artificial intelligence system according to claim 13, wherein, The action method of the artificial intelligence system also includes the following steps: The second AI server receives the multiple modified plot contents from the generative AI server; The second AI server transmits the received modified plot content to the first AI server; The first AI server transmits the multiple modified plot contents to the display device; as well as The display device displays the multiple modified plot contents.

Images