Display device control method and display device using the same

Abstract

The present application provides a display device control method and a display device using the method. The display device includes a display, a memory, and a controller, the memory is configured to store at least one instruction, and the controller includes at least one processor configured to execute at least one instruction to control the display device. According to the display device, since the user can watch an image with a unique style, for example, a third image based on AI style transfer, and will not feel bored when the image is style transferred and generated, the user's needs can be met and the user's convenience can be increased.

Description

Technical Field

[0001] The present disclosure relates to a method of controlling a display device and a display device using the method.

[0002] For example, the present disclosure relates to a method of controlling a display device for displaying a style transfer image and a display device using the method. Background Art

[0003] As displays become more popular and technology develops, display devices having various shapes and functions have been developed.

[0004] Therefore, the display device can be used to implement functions that meet various needs or intentions of consumers.

[0005] For example, products that use a display device as a picture frame for displaying an image, such as a portrait or a masterpiece painting, have been released. In detail, when a user is not viewing specific content through the display device, the display device can output a specific screen, such as a screen including a portrait or a masterpiece painting. In this case, the user can use the display device as a masterpiece painting frame or a large photo frame.

[0006] In addition, the user may want to display a style-transferred image on the display device, taking into account his or her own artistic taste or atmosphere at the location where the display device is located. In this case, the display device may need to quickly generate an image that satisfies the user's artistic taste or needs.

[0007] As described above, there is a need to provide a display device that meets various user needs and intentions. Summary of the invention

[0008] Technical issues

[0009] It is desirable to provide a display device that meets various user needs and intentions.

[0010] Technical Solution

[0011] A method of controlling a display device is provided, by which an image satisfying a user's artistic taste and needs is provided.

[0012] Beneficial Effects

[0013] Embodiments of the present disclosure provide a method of controlling a display device and a display device using the method, by which an image satisfying a user's artistic taste and needs can be quickly generated.

[0014] An embodiment of the present disclosure provides a method of controlling a display device and a display device using the method, by which an image can be generated and displayed so that a user does not feel bored by performing operations via, for example, a neural network during image generation time. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0016] Figure 1 is a diagram showing an exemplary general display device according to an embodiment of the present disclosure;

[0017] Figure 2 is a diagram illustrating an exemplary image transition by style transfer according to an embodiment of the present disclosure;

[0018] Figure 3 is a block diagram illustrating an exemplary display device according to an embodiment of the present disclosure;

[0019] Figure 4 is a flowchart illustrating an exemplary method of controlling a display device according to an embodiment of the present disclosure;

[0020] Figure 5 is a diagram illustrating an exemplary style transfer operation according to an embodiment of the present disclosure;

[0021] Figure 6 is a flowchart illustrating an exemplary method of controlling a display device according to an embodiment of the present disclosure;

[0022] Figure 7 is a diagram illustrating an exemplary style transfer operation through a neural network according to an embodiment of the present disclosure;

[0023] Figure 8 is a diagram showing an exemplary neural network for style transfer according to an embodiment of the present disclosure;

[0024] Fig. 9 is another diagram illustrating an exemplary neural network for style transfer according to an embodiment of the present disclosure;

[0025] Fig.10 is a diagram illustrating an exemplary simulation-based style transfer operation according to an embodiment of the present disclosure;

[0026] Fig.11 is a diagram showing an exemplary image output from a display device according to an embodiment of the present disclosure;

[0027] Fig.12 is a flowchart illustrating an exemplary method of controlling a display device according to another embodiment of the present disclosure;

[0028] Fig.13 is a block diagram illustrating an exemplary display device according to another embodiment of the present disclosure;

[0029] Fig.14 is another block diagram illustrating an exemplary display device according to another embodiment of the present disclosure; and

[0030] Fig.15 is another block diagram illustrating an exemplary display device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2018-0116581 filed in the Korean Intellectual Property Office on September 28, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0032] Embodiments of the present disclosure provide a method of controlling a display device and a display device using the method, by which an image satisfying a user's artistic taste and needs can be quickly generated.

[0033] Embodiments of the present disclosure provide a method of controlling a display device and a display device using the method, by which an image can be generated and displayed so that a user does not feel bored during image generation time by performing operations through, for example, a neural network.

[0034] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments of the disclosure.

[0035] According to an exemplary embodiment of the present disclosure, a display device includes a display, a memory configured to store at least one instruction, and a controller including at least one processor configured to execute the at least one instruction. The controller is configured to control the display device to: perform image processing based on a result of analyzing a first image to allow the first image to have a first style corresponding to a specific texture, control the generation of at least one second image for style transfer; control the generation of a third image by performing an operation for style transfer on the first image using a neural network; and control the output of at least one second image on the display and subsequently output the third image.

[0036] The controller can also be configured to control the display device to: generate at least one second image that can gradually change while performing image processing to allow the first image to have a first style; and perform image transition to output a third image on the display after the at least one second image that can gradually change can be sequentially output.

[0037] The controller may also be configured to control the display device to: generate at least one second image by performing simulation-based style transfer based on edge information of the first image to allow the first image to have a specific texture.

[0038] The controller may also be configured to control the display device to: control sequential output of the at least one second image on the display based on a generation order of the at least one second image.

[0039] The controller may also be configured to control the display device to: control generation of a third image corresponding to the first style by performing an operation of style transfer on the first image through a neural network.

[0040] The controller may also be configured to control the display device to obtain a content image and a style of the first image by performing operations via a neural network, and generate a third image by migrating the style of the first image to correspond to the first style while maintaining the content image.

[0041] The controller may include a first photo filter corresponding to the first style, the first photo filter configured to perform simulation-based style transfer, and the controller may be further configured to control the display device to control generation of at least one second image using the first photo filter.

[0042] The controller may also be configured to control the display device to: based on receiving an input selecting a first style, control generation of at least one second image by performing image processing using a first photo filter to allow the first image to have a first style, wherein the first style may be one of a plurality of different styles.

[0043] The display device may further include a communicator including a communication circuit configured to perform communication with an external device. The controller may also be configured to control the display device to receive an input selecting a first style from the plurality of styles from the external device through the communicator.

[0044] The display device may further include a communicator including a communication circuit configured to perform communication with an external device configured to perform an operation for style transfer through a neural network. The controller may also be configured to control the display device to control receiving a third image that can be obtained when the external device performs the operation through the neural network.

[0045] The neural network may include, for example, but not limited to, a convolutional neural network (CNN) that repeatedly performs convolution operations to allow the first image to correspond to the first style.

[0046] The controller may also be configured to control the display device to control output of at least one second image and a third image on the display during execution of the environmental service.

[0047] According to another exemplary embodiment of the present disclosure, a method for controlling a display device includes: generating at least one second image for style transfer by performing image processing based on a result of analyzing a first image having a first style corresponding to a specific texture; generating a third image by performing an operation for style transfer on the first image using a neural network; and outputting at least one second image on a display and then outputting the third image.

[0048] Invention Mode

[0049] The present disclosure will now be described in more detail with reference to the accompanying drawings, in which various exemplary embodiments of the present disclosure are shown. In the following disclosure, when it is determined that the detailed description of the related known functions or structures will make the main points of the present disclosure unclear, the detailed description may be omitted herein. In all the drawings, the same reference numerals represent the same elements.

[0050] In the present disclosure, when an element is “connected” or “connected” to another element, the element is not only directly in contact with or connected to another element, but also electrically connected through at least one other element interposed therebetween. In addition, when a component may “include” a certain element, unless otherwise specified, it may not be interpreted as excluding another element, but may be interpreted as further including other elements.

[0051] Throughout the disclosure, the phrases "in some embodiments" or "in an embodiment" may not necessarily refer to the same embodiment of the disclosure.

[0052] The present disclosure can be described in terms of function block components and various processing steps. Such function blocks can be implemented by any number of hardware and / or software components configured to perform a specified function. For example, the function blocks of the present disclosure can be implemented by one or more processors or microprocessors or by various integrated circuit components for various functions. In addition, the function blocks of the present disclosure can be implemented in any programming or scripting language. The function blocks can be implemented in various algorithms executed on one or more processors. In addition, the present disclosure can adopt any number of conventional techniques for electronic configuration, signal processing and / or control, data processing, etc. The words "module" and "configuration" can be used extensively, without limitation to the mechanical or physical embodiments of the present disclosure.

[0053] In addition, the connecting lines or connectors between the elements shown in the various figures are intended to represent the functional relationship and / or physical or logical connection between the various elements. It should be noted that many alternative or additional functional connections, physical connections or logical connections may exist in actual devices.

[0054] Furthermore, throughout the disclosure, the expression "at least one of a, b, or c" means only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0055] The present disclosure will be described in more detail below with reference to the accompanying drawings.

[0056] The display device according to the exemplary embodiment of the present disclosure may include all electronic devices that visually output specific content. For example, the display device according to the exemplary embodiment of the present disclosure may include all electronic devices that can selectively display at least one piece of content, and may exist in various forms, such as but not limited to a television (TV), a digital broadcast terminal, a tablet PC, a mobile phone, a computer, a laptop computer, etc. In addition, the display device may be not only a fixed type, but also a mobile type or a portable type carried by a user.

[0057] In an exemplary embodiment of the present disclosure, the display device may transmit and receive data with an external device via a wireless network. The external device may be, for example, but not limited to, a server or an electronic device capable of communicating with the display device via a wireless communication network. For example, the external device may be a server or an electronic device that may transmit and receive certain data with the display device, may be paired with the display device, may remotely control the display device, or may be remotely controlled by the display device. For example, the external device may include, for example, but not limited to, computing devices such as smart phones, tablet PCs, PCs, personal digital assistants (PDAs), laptop computers, media players, servers, micro servers, and the like.

[0058] Figure 1 is a diagram illustrating an exemplary display device 110 as a general display device according to an exemplary embodiment of the present disclosure.

[0059] Reference Figure 1 , a display device 110 may be set in a certain space 100. The display device 110 may be an electronic device that outputs an image through a screen 112. For example, the display device 110 may exist in various forms such as a TV or a digital broadcast terminal as a device for visually outputting advertisements and guide information, a user interface screen, etc. to a user. In addition, the display device 110 may be not only a fixed type but also a mobile type or a portable type carried by a user.

[0060] As in Figure 1In the example shown, the display device 110 is provided as a wall-mounted display device that can be detached on the wall surface by being attached to a specific wall surface 120. The user may want to use the display device 110 as a picture frame. For example, when the user cannot watch certain content, such as a TV channel, through the display device 110, the user may want to have an image that matches the user's taste or an image that matches well with the displayed space 100. In addition, the user may want not to have a general portrait or painting, but to have an image of unique artistic quality output through the display device 110 according to his or her own artistic taste.

[0061] The operation of displaying a specific image when the display device 110 is not reproducing specific content may be referred to as an ambient service, for example. For example, when the TV is turned off, for example, when the TV is in an off state where certain content is not reproduced, an ambient service (which is a new type of application or service for a display device such as a TV) may output a meaningful image such as a portrait or a representative painting on the screen instead of a black screen.

[0062] When the above-described environment service is used, in order to output an image, the display device 110 may be operated to generate an image corresponding to the user's intention, taste, and / or setting.

[0063] In order to generate a unique image corresponding to the user's intention and / or taste, in an embodiment of the present disclosure, a style transfer technique may be used. For example, in an embodiment of the present disclosure, an artistic style transfer technique may be used to generate an image.

[0064] Artistic style migration can refer to, for example, converting the style of an image into a specific form while maintaining the content of the image. In this regard, the image subjected to conversion can be an image digitized by a common digital camera or scanner, or an image newly created by using a drawing program. The style can be a painting style such as, but not limited to, watercolor, oil painting, ink painting, point painting, 3D painting, etc., or can refer to the painting style of a specific painter, such as Van Gogh style, Monet style, Mane style, Picasso style, etc. In addition, the style can be classified into different styles according to, for example, but not limited to, color, atmosphere, specific brightness, specific saturation of the image, etc. For example, the Starry Night style or the Sunflower style of a specific work of Van Gogh (Van Gogh) can be referred to as a style. In addition, in addition to the above examples, the "style" used in the embodiments of the present disclosure can also include any and all things in the painting style that can represent the style.

[0065] Figure 2 is a diagram showing an exemplary image transition through style transfer according to an exemplary embodiment of the present disclosure. Figure 2, an example is shown in which style transfer is performed by changing the color, brightness, and saturation of the original image 210. Therefore, the output image 220 can be generated by converting the style of the original image 210.

[0066] The above-mentioned artistic style transfer has evolved from a method in which a person directly edits an image using an image editing tool such as Photoshop to a method in which an image is automatically analyzed and then image processing is performed using a predetermined style.

[0067] A method for performing style transfer of an image using artificial intelligence (AI) technology that performs operations through a neural network is also provided.

[0068] In the following description, style migration that analyzes an image and then performs image processing using a predetermined style may be referred to as “simulation-based style migration.” In addition, style migration that performs image processing using AI technology may be referred to as “AI-based style migration,” for example.

[0069] For example, Figure 2 The example shown in is an example of generating an output image 220 by performing simulation-based style transfer. In simulation-based style transfer, the style used to generate the output image is a predetermined style. For example, it is necessary to develop a technology for each style applied to style transfer, and in order to express an appropriate style, a texture image or image resource corresponding to the style is developed and manufactured separately. Therefore, since simulation-based style transfer requires investment of time, manpower and cost for each development of a new style, there are limitations in the expansion of styles and the diversification of style types.

[0070] When style transfer is performed through AI, more unique images can be generated compared to simulation-based style transfer. In addition, by improving creativity, the artistic quality of the image can be improved, and images that further match the needs of users who do not want general images can be generated. In addition, for AI-based style transfer, there is no need to separately develop texture images or image resources corresponding to each style as in simulation-based style transfer. In other words, in AI-based style transfer, when the original image and the image with the style to be copied are input to the input of the neural network, the output is migrated to an image of a specific style through the operation in the neural network.

[0071] However, the time used to perform an AI-based style transfer operation may be greater than the time used to perform a simulation-based style transfer. For example, simulation-based style transfer can output a style transfer image in real time. On the other hand, for style transfer using AI, it may take tens of seconds to several minutes to perform style transfer on an input image and output a style-transferred image. While generating the AI-based style transfer image as the final result, the general display device may output a preset standby screen or a screen indicating that processing is being performed, such as a UI screen indicating the message "Processing". Therefore, for style transfer using AI, the user may not have immediate feedback about the generated image because it may take tens of seconds to several minutes to perform AI style transfer. Therefore, the user may feel bored during the time until the AI-based style transfer image is output.

[0072] Users usually want to quickly output the desired image, and when the time for outputting the style transfer image takes a long time, the user may feel more bored. In addition, when the output of the style transfer image is delayed, the user may mistakenly believe that an error has occurred in the execution of the style transfer operation.

[0073] Therefore, according to an embodiment of the present disclosure, a display device and a control method of the display device are provided, which can prevent and / or alleviate the user from feeling bored during the time spent outputting an AI-based style transfer image that better matches the user's needs by using an image generated by simulation-based style transfer. In addition, in an embodiment of the present disclosure, when the generation of the AI-based style transfer image is completed, by performing image migration starting from an image that has been output, for example, from an image generated by simulation-based style migration to an AI-based style transfer image, the user can at least see the AI-based style transfer image. Therefore, the user can eventually watch the AI-based style transfer image that has been migrated to a unique style with high artistic quality. In addition, the user can watch the simulation-based style transfer image that is output quickly without feeling bored during the time until the generation of the AI-based style transfer image is completed.

[0074] In the following description, reference will be made to Figures 3 to 15 A display device and a method of controlling the display device according to an embodiment of the present disclosure are described in more detail.

[0075] Figure 3 is a block diagram illustrating an exemplary display device 300 according to an exemplary embodiment of the present disclosure.

[0076] Reference Figure 3 , the display device 300 may include a memory 310 , a controller (eg, including a processing circuit) 320 , and a display 330 .

[0077] The memory 310 may store at least one instruction.

[0078] At least one instruction stored in the memory 310 may be executed by a processor included in the controller 320 .

[0079] In addition, the memory 310 may include a storage medium having at least one type, such as but not limited to flash memory, a hard disk, a multimedia card micro, a card type (such as SD or XD memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic storage, a disk, an optical disk, etc.

[0080] The controller 320 may include various processing circuits including at least one processor that executes the at least one instruction.

[0081] In this regard, each of the at least one processor may perform a specific operation by executing at least one instruction stored in the memory 310 .

[0082] In addition, each of the at least one processor may perform a specific operation by executing at least one instruction stored in the controller 320 .

[0083] In addition, at least one processor included in the controller 320 may control operations performed in the display device 300, and may control other elements included in the display device 300 to perform specific operations. Therefore, when, for example, the controller 320 is described as controlling certain operations, it is apparent that at least one processor included in the controller 320 controls certain operations.

[0084] In addition, the controller 320 may include an internal memory and at least one processor that executes at least one stored program. For example, the internal memory of the controller 320 may store at least one instruction. The at least one processor included in the controller 320 may perform a specific operation by executing at least one instruction stored in the internal memory of the controller 320, such as controlling the operation of the display device 300.

[0085] For example, the controller 320 may include a RAM (not shown) that stores signals or data input from the outside of the display device 300 or is used as a storage area corresponding to various tasks performed in the display device 300, a ROM (not shown) that stores a control program and / or multiple instructions for controlling the display device 300, and at least one processor. The processor may include a graphics processing unit (GPU) (not shown) for processing graphics corresponding to a video. The processor may be implemented by a system on chip (SoC) integrating a core (not shown) and a GPU (not shown). The processor may include a single core, a dual core, a triple core, a quad core, and multiple cores thereof.

[0086] In the following description, for convenience and ease of explanation, a case is described in which, when the controller 320 performs or controls a certain operation, the processor of the controller 320 performs the certain operation by executing at least one instruction stored in the memory 310, but the present disclosure is not limited thereto.

[0087] The display 330 outputs an image. For example, the display 330 may output an image corresponding to video data through a display panel (not shown) included inside so that a user can visually recognize the video data.

[0088] In an embodiment of the present disclosure, the display 330 displays at least one second image under the control of the controller 320, and sequentially displays a third image according to image transition.

[0089] In the following description, reference will be made to Figure 4 Operations performed by respective elements included in the display device 300 are described in more detail.

[0090] Figure 4 is a flowchart illustrating an exemplary method 400 for controlling a display device according to an embodiment of the present disclosure. Figure 4 is a flowchart illustrating an exemplary operation performed in the display device 300 according to an embodiment of the present disclosure.

[0091] Reference Figure 4 , display device (e.g., Figure 3 The method 400 of the display device 300 may include operation S410, operation S420 and operation S430.

[0092] Reference Figure 3 and Figure 4In an embodiment of the present disclosure, the controller 320 controls: by performing image processing on the first image based on the result of analyzing the first image so that the first image has a first style corresponding to a specific texture, at least one second image for style transfer is generated (S410). The first image may represent an original image subjected to style transfer. For example, the first image may include, for example but not limited to, an image received by the display device 300, an image already stored by the display device 300, an image input by a user for style transfer, an image sent by an external device (not shown) for style transfer, etc.

[0093] Furthermore, the controller 320 controls to generate a third image by performing an operation of style transfer of the first image using a neural network ( S420 ).

[0094] Despite Figure 4 In the flowchart of , after operation S410 is performed, operation S420 is performed, but operations S410 and S420 may be performed simultaneously. Operations S410 and S420 may be performed simultaneously in parallel or at adjacent time points.

[0095] For example, when receiving a user input corresponding to a style transfer request, the controller 320 may perform operations S410 and S420 in parallel. In another example, when receiving a first image as an original image subject to style transfer, the controller 320 may perform operations S410 and S420 in parallel.

[0096] The controller 320 controls: outputting at least one second image and sequentially outputting a third image on the display 330 (S430). For example, the controller 320 may control displaying at least one second image and finally displaying the third image by image migration from the last output image of the at least one second image to the third image.

[0097] The first image may be an original image that is subjected to style transfer. In addition, the above-mentioned simulation-based style transfer generates a result, such as a style transfer image, by analyzing an input image and performing image processing on the analyzed image using a previously produced texture image or image resource. Therefore, "performing style transfer by analyzing an image and performing image processing based on the analysis result" may refer to, for example, the above-mentioned "simulation-based style transfer". In other words, when migrating the style of the first image as the original image, the controller 320 may control the generation of at least one second image by performing simulation-based style transfer, and control the generation of a third image by performing AI-based style transfer.

[0098] In addition, the controller 320 can simultaneously and in parallel perform simulation-based style migration and AI-based style migration operations, where the simulation-based style migration is an operation to generate at least one second image, and the AI-based style migration operation is an operation to generate a third image. In this example, performing AI-based style migration may take tens of seconds to several minutes. Therefore, before the generation of the third image is completed, at least one second image can be displayed. Because the simulation-based style migration operation can provide real-time feedback, at least one image generated in the process of performing style migration, such as at least one second image, can be output in real time.

[0099] In an embodiment of the present disclosure, the user can finally watch the third image having high artistic quality and being transferred to a unique style as an AI-based style transfer image, and can watch the simulation-based style transfer image output quickly without getting bored during the time until the generation of the AI-based style transfer image (e.g., the third image) is completed. Therefore, according to an embodiment of the present disclosure, since the user can watch an image having a unique style, that is, an AI-based style transfer image, for example, the third image, and can watch the image without getting bored during the time until the style-transferred third image is generated, the user's needs can be met and the user's convenience can be increased.

[0100] according to Figure 3 and Figure 4 The embodiments of the present disclosure shown can improve user satisfaction and user convenience because the user can watch an image with a unique style, for example, a third image based on AI style transfer, without feeling bored during the time before the third image is generated by style transfer.

[0101] Figure 5 is a diagram illustrating an exemplary style transfer operation according to an exemplary embodiment of the present disclosure. For example, the controller 320 may perform Figure 5 The style transfer operation shown.

[0102] The controller 320 may obtain the first image 511 and may set the first style 513 as the style to be migrated. The first style 513 may be set according to the user's selection. Fig.13 An operation of setting or determining a first style corresponding to a user selection is described in more detail.

[0103] Fig.13 is a block diagram illustrating an exemplary display device 1300 according to another exemplary embodiment of the present disclosure. Fig.13 The display device 1300 shown may correspond to Figure 3 The display device 300 shown. Fig.13 In the figure, the same reference numerals are used to show Figure 3 Therefore, in the description of the display device 1300, descriptions that are redundant with the description of the display device 300 may not be repeated here.

[0104] Reference Fig.13 Compared with the display device 300 , the display device 1300 may further include a communicator (eg, including a communication circuit) 1310 and / or a user interface (eg, including a user interface circuit) 1320 .

[0105] The communicator 1310 may include various communication circuits and may transmit and receive data about at least one external device via a communication network under the control of the controller 320. The external device may include a server 1350, a mobile device 1370, or an electronic device (not shown) for performing image processing, but the present disclosure is not limited thereto.

[0106] The server 1350 may perform an operation according to a neural network. For example, the server 1350 may perform an AI-based style migration on a first image subjected to style migration to generate a third image. For example, the server 1350 may receive a first image through the communicator 1310 of the display device 1300, perform an operation according to a neural network, and migrate the style of the first image, thereby generating a third image. In addition, the server 1350 may send the generated third image to the communicator 1310 of the display device 1300.

[0107] Mobile device 1370 may include a mobile computing device such as, but not limited to, a wearable device, a smart phone, a tablet PC, a PDA, a laptop computer, a media player, a microserver, a global positioning system (GPS) device, and the like.

[0108] For example, the mobile device 1370 and the display apparatus 1300 may be connected to each other through a Home IoT platform, etc. In addition, the mobile device 1370 may be an electronic device paired with the display apparatus 1300 via a wireless network.

[0109] For example, the mobile device 1370 may remotely control the display apparatus 1300. In addition, the mobile device 1370 may transmit and receive data with respect to the display apparatus 1300 so that the display apparatus 1300 may perform a specific service or a specific operation.

[0110] The display device 1300 may operate under the control of the mobile device 1370. For example, the display device 1300 may recognize the location of the mobile device 1370 and automatically perform a specific service (e.g., an environmental service) or a specific operation. In another example, the display device 1300 may perform a style transfer operation upon request or under the control of the mobile device 1370.

[0111] The user interface 1320 may receive user input for controlling the display device 1300. The user interface 1320 may include various user interface circuits including user input devices, such as but not limited to a touch panel for sensing a user's touch, a button for receiving a user's pressing operation, a wheel for receiving a user's rotating operation, a keyboard, and a dome switch, but the present disclosure is not limited thereto.

[0112] In addition, the user interface 1320 may include a voice recognition device (not shown) for voice recognition. For example, the voice recognition device may be a microphone, and the voice recognition device may receive a user's voice command or voice request. Therefore, the controller 320 may control an operation corresponding to the voice command or voice request to be executed.

[0113] In addition, the user interface 1320 may include a motion detection sensor (not shown). For example, the motion detection sensor may detect the motion of the display device 1300, and may receive the detected motion as an input of the user. In addition, the above-mentioned voice recognition device and motion detection sensor may not be included in the user interface 1320, but may be included in the display device 1300 as a module independent of the user interface 1320.

[0114] Reference Figure 5 , shows an example in which the first style is a style corresponding to a painting by a specific painter. Figure 5 In the example, the first style 513 may refer to Van Gogh's "The Starry Night" itself or Van Gogh's style.

[0115] In the following description, reference is made to Figure 5 , Figure 6 and Fig.13 The style transfer operation according to an exemplary embodiment of the present disclosure is described in more detail.

[0116] Figure 6 is a flowchart illustrating an exemplary method of controlling a display device according to an exemplary embodiment of the present disclosure. In addition, Figure 6 is a flowchart illustrating an exemplary operation performed in the display device 300 or 1300 according to an exemplary embodiment of the present disclosure. Figure 6 In the figure, the same reference numerals are used to show Figure 4 Therefore, in Figure 6 In the description of the exemplary embodiments of the present disclosure shown, it may not be directed to Figure 6 Repeat with Figure 4 The description in the description is redundant.

[0117] For example, in order to perform style transfer, a first image as an original image is input ( S601 ), and a first style to be applied to the first image is determined ( S605 ).

[0118] The first style may be set or selected as any one of one or more styles ( S605 ).

[0119] For example, the first style 513 may be determined by the default setting of the controller 320 or the display device 1300. In other words, the first style 513 may be a preset style. For example, in the initial setting of the display device 1300, the user may set the first style in consideration of the user's taste, painting style, and / or favorite painter or work. Before the user changes the first style, the controller 320 may set the first style to a specific style according to the user's initial setting. In addition, when there is no initial setting by the user, the controller 320 may set the first style to a specific style according to the default setting.

[0120] In addition, the first style 513 may be determined based on a user input through the user interface 1320 of the display device 1300. For example, the controller 320 of the display device 1300 may control the display device 1300 to output a user interface screen (not shown) for selecting any one of a plurality of available styles. The user may select any one of the styles as the first style through the user interface screen. The controller 320 may perform style migration on the first image 511 according to the first style based on the user input for selecting the first style.

[0121] In another example, the controller 320 may determine the first style based on the user's input to an external device connected to the display device 1300 through a communication network. For example, the controller 320 may determine the first style based on the user's input through the mobile device 1370, which operates by pairing with the display device 1300. In this example, the mobile device 1370 may output a user interface screen 1375 to select any one of a plurality of styles that can be used by the display device 1300. The user may select a first style, such as "Style 1", through the user interface screen 1375 output by the mobile device 1370. The mobile device 1370 may send information indicating the first style selected based on the user input to the communicator 1310 of the display device 1300. The communicator 1310 may send the received information indicating that the first style is selected to the controller 320. Therefore, the controller 320 may perform style migration on the first image 511 according to the first style.

[0122] Reference Figure 5 and Fig.13 , the controller 320 may perform style migration on the first image 511 based on Van Gogh's "The Starry Night" as the first style.

[0123] In order to perform style transfer to the first style on the input first image, the controller 320 may perform AI-based style transfer 540 and simulation-based style transfer 550 ( S607 ).

[0124] Reference below Figure 7 , Figure 8 and Fig. 9 The operation of AI-based style transfer 540 is described in more detail.

[0125] For example, the controller 320 may perform an operation for performing style transfer on the first image 511 through a neural network (eg, based on AI) ( S621 ), and control generation of a third image 541 corresponding to the first style 513 ( S623 ).

[0126] In addition, the controller 320 may perform simulation-based style migration 550 (S611) to control the generation of at least one second image (S613 and S615). Here, the at least one image (556, 557, 558, 559, and 570) generated by the simulation-based style migration is referred to as a "second image". For example, the controller 320 may control the generation and output of at least one second image, which gradually changes during the image processing so that the first image 511 has the first style 513 (S613). The at least one second image 551 may include Figure 5 Second images 556 , 557 , 558 , and 559 and image 570 are shown.

[0127] For example, when the controller 320 performs simulation-based style transfer 550, an intermediate result 555 may be generated and output in real time during the process in which the first image is style transferred. In this state, the intermediate result 555 may include second images 556, 557, 558, and 559, which show the degree of progress of the style transfer (although the style transfer is not completed).

[0128] For example, the controller 320 may perform simulation-based style transfer on the first image 511 based on edge information of the first image 511 so that the first image 511 may have a specific texture, and generate at least one of the second images 556 , 557 , 558 , 559 , and 570 .

[0129] For example, the simulation-based style transfer 550 may analyze the shape of an image by image processing the first image 511 as an original image.

[0130] For example, simulation-based style migration 550 may be performed by the following process. For example, the image analysis method may include a method of extracting edges of features in an image by obtaining a gradient of the image. A portion to be drawn with a relatively large texture and a portion to be repeatedly drawn with a relatively small texture may be defined based on the extracted edges because the edges are finely arranged in the image. By repeatedly drawing a texture prepared according to the defined portion or region, the first image may be migrated to the first style as the desired style.

[0131] In other words, the controller 320 can extract the edges of the features of the image by analyzing the first image 511, and repeatedly draw a specific texture based on the extracted edges, thereby performing style migration. As the simulation-based style migration operation proceeds, the texture of the image can be further finely represented. For example, the second image 556, the second image 557, the second image 558, the second image 559, and the image 570 can be sequentially generated in the process of performing the simulation-based style migration 550. The controller 320 can generate the image 570 as the final result of the simulation-based style migration 550 (S615).

[0132] Reference below Fig.10 The operation of simulation-based style transfer 550 is described in more detail.

[0133] In addition, the controller 320 controls at least one of the second images 556, 557, 558, 559, and 570 to be sequentially output on the display 330 according to the generation order of at least one of the second images 556, 557, 558, 559, and 570. Figure 7 The screen output on the display 330 is described in more detail.

[0134] As described above, the time for performing the operation of the AI-based style migration 540 is greater than the time for performing the operation of the simulation-based style migration 550. In addition, the operation of the AI-based style migration 540 may not output the intermediate result 555 like the operation of the simulation-based style migration 550, but may output only the third image 541 as the final result. Therefore, assuming that the AI-based style migration 540 and the simulation-based style migration 550 are performed simultaneously and in parallel, the time for completing the generation of the third image 541 may be greater than the time for completing the generation of at least one of the second images 556, 557, 558, 559, and 570. Therefore, the display device 300 or 1300 according to an embodiment of the present disclosure may output the third image 541 after outputting at least one of the second images 556, 557, 558, 559, and 570 on the display 330.

[0135] For example, the controller 320 may perform image transition so that the third image 541 may be output after at least one of the gradually changing second images 556 , 557 , 558 , 559 , and 570 is sequentially output on the display 330 ( S631 and S633 ).

[0136] Reference below Fig.11 A screen output on the display 330 according to the image transition operation (S631) is described in more detail.

[0137] According to the implementation of this disclosure Figure 5 and Figure 6 In the embodiment of the operation shown in , the simulation-based style transfer operation and the AI-based style transfer operation can be performed in parallel, thereby sequentially outputting at least one of the second images gradually changed by the simulation-based style transfer, and the third image of the AI-based style transfer can be output. Therefore, during the time taken to generate the AI-based style transfer image, the user can watch the simulation-based style transfer image without feeling bored.

[0138] Furthermore, by ultimately viewing AI-based style transfer images, users can view highly creative and artistic images.

[0139] Figure 7 is a diagram illustrating an exemplary style transfer operation by a neural network according to an exemplary embodiment of the present disclosure. In other words, Figure 7 is a diagram illustrating an exemplary AI-based style transfer operation.

[0140] Reference Figure 7 , the original image 710, the style 720 and the migration image 730 correspond to Figure 5 The first image 511, the first style 513 and the third image 541 described in FIG.

[0141] When the original image 710 and the style 720 are input to the neural network, the neural network performs internal operations to migrate the original image 710 according to the style 720 and generate a migrated image 730 .

[0142] In an embodiment of the present disclosure, a neural network is used to generate a third image. For example, Figure 5The AI-based style transfer 540 is a method of classifying an original image 710 into a content image and a style image and transferring the style of the style image to the style 720 while maintaining the content of the content image. For example, there is a method using a convolutional neural network (CNN) named "Neural Algorithm of Artistic Style". For example, in a layer of a neural network, a transfer image 730 is generated by reconstructing the style and content in a feature map using the extracted feature map style.

[0143] In an embodiment of the present disclosure, a neural network performs an image generation operation according to AI technology. For example, the neural network may be a deep neural network (DNN) that performs operations through multiple layers. A neural network may be classified as a DNN according to the number of inner layers that perform operations, and when the number of layers is multiple, that is, the depth of the neural network that performs the operation increases. In addition, the DNN operation may include a CNN operation.

[0144] The neural network used to generate the third image may be a neural network that has been trained to achieve the desired results.

[0145] For example, a neural network may implement a recognition model to classify or recognize a content image in the original image 710, and the implemented recognition model may be trained using training data. The neural network is trained by repeatedly performing the above training, and a plurality of weights applied to each of a plurality of nodes forming the neural network may be set. The weight may represent the connection strength between the various nodes forming the neural network. The weight value may be optimized by repeated training, and the correction may be repeated until the accuracy of the result meets a certain degree of reliability. The neural network used in the embodiments of the present disclosure may be a neural network formed by the weight values ​​finally set by the above training process.

[0146] Content included in the original image 710 may be classified or extracted by analyzing or classifying the original image 710 as input data using a trained recognition model forming a neural network. A migration image 730 may be generated by applying a style 720 to a content image corresponding to the extracted content. The migration image 730 may include the extracted content and may be an image having the style 720. Figure 7 , the original image 710 represents a building following a medieval European architectural style and a river adjacent thereto. The neural network can generate a migration image 730 by performing an operation that allows the image to be represented in the painting style of Van Gogh's "Starry Night" corresponding to the set style 720 while maintaining the "medieval European architectural style and the river adjacent thereto", which is the content included in the original image 710.

[0147] In an embodiment of the present disclosure, the operation of generating the third image through the neural network may be performed in an external device that transmits and receives data with the display device 300 through a communication network.

[0148] For example, the communicator 1310 may communicate with an external device (e.g., a server 1350) that performs an operation of style transfer through a neural network. The controller 320 may control an external device, such as the server 1350, to receive a third image obtained by performing an operation through a neural network. As described above, in order to perform style transfer through a neural network, a trained neural network trained by repeatedly performing a training process is required. Many operation processes are performed until a trained neural network is generated. Therefore, in order to establish a highly perfect neural network, a processor with a large data processing capability and a fast data processing speed is required. Therefore, in this exemplary embodiment of the present disclosure, the third image can be obtained by the server 1350 that quickly and accurately performs AI-based style transfer.

[0149] For example, the controller 320 may transmit the original image 710 and the style 720 to the server 1350 through the communicator 1310, and request the server 1350 to migrate the style of the original image 710. Therefore, the server 1350 may migrate the style of the original image 710 to generate a migrated image 730, and transmit the generated migrated image 730 to the communicator 1310 of the display device 1300.

[0150] In addition, the controller 320 may establish a neural network for performing style transfer in the controller 320 or the memory 310, and perform an AI-based style transfer operation to generate a third image through the established neural network. In addition, according to an exemplary embodiment of the present disclosure, the neural network for performing style transfer may be established as a separate processor.

[0151] Figure 8 is a diagram showing an exemplary neural network for style transfer according to an exemplary embodiment of the present disclosure. Figure 8 , a neural network used in an exemplary embodiment of the present disclosure is shown as an example.

[0152] In an embodiment of the present disclosure, the controller 320 may control the generation of the third image using, for example, but not limited to, CNN, a deep convolutional neural network (DCNN), a Capsnet neural network (not shown), etc. The Capsnet neural network may be developed to compensate for the weaknesses of CNN. In addition, all of CNN, DCNN, and Capsnet may be neural networks based on CNN.

[0153] When the correlation between information fragments included in an image is local, a CNN-based neural network can introduce the concept of filters corresponding to specific local areas included in the image, and generate new features or representations by convolving the information fragments in the filters.

[0154] Figure 8 A CNN-based neural network 800 is shown. For example, Figure 8 In the figure, a DCNN including multiple layers and thus having multiple depths is shown as a CNN-based neural network 800.

[0155] Reference Figure 8 , the controller 320 may control the generation of the migration image 730 (which is the third image) through the CNN-based neural network 800.

[0156] Reference Figure 8 In the CNN-based neural network 800, convolution layers and pooling layers are arranged alternately, and the depth of each layer filter increases from left to right. In addition, the last layer of the CNN-based neural network 800 can be formed by a fully connected layer. The convolution layer may, for example, refer to a layer of data generated according to a convolution operation, and the pooling layer may, for example, refer to a layer for reducing the amount or size of data through subsampling or pooling operations. When the input image 810 passes through the convolution layer and the pooling layer, a data fragment indicating the features of the input image, such as a feature map, is generated. The data fragments generated by passing through the convolution layer and the pooling layer are processed by the hidden layer to form a fully connected layer, and then a "third image" as data with new features or representations can be generated.

[0157] For example, the CNN-based neural network 800 may include an input layer 810, a first convolutional layer 820, a first pooling layer 830, a second convolutional layer 840, a second pooling layer 850, a hidden layer 860, and an output layer 870. The depth of the convolutional layer and the pooling layer may be variable, and the depth of the hidden layer may be variable. In addition, as the depth of the convolutional layer and the pooling layer increases, more different data fragments may be generated. For example, as the depth of the convolutional layer and the pooling layer increases, the information fragment indicating the features of the input image may represent more detailed features of the input image. In addition, the depth and shape of the CNN-based neural network 800 may be designed to consider the accuracy of the result, the reliability of the result, or the operation processing speed and capacity of the processor very differently.

[0158] Fig. 9 is another diagram showing an exemplary neural network for style transfer. Fig. 9 is a diagram illustrating an example of a DNN in which the hidden layer has a depth of three. Fig. 9 , blocks 910 and 930 may correspond to forming a fully connected layer Figure 8The hidden layer 860 of , and the box 950 may correspond to Figure 8 The output layer 870.

[0159] Fig. 9 The neural network shown may include an input layer 910, a hidden layer 930, and an output layer 950. The input layer 910 may, for example, be a layer that receives Figure 8 The hidden layer 930 may refer to a layer for processing input data segments, and the output layer 950 may refer to a layer for outputting data segments processed by the hidden layer 930, for example.

[0160] In addition, the formation Fig. 9 Each of the multiple layers of the neural network shown may include one or more nodes. For example, the input layer 910 may include one or more nodes, such as node 901, that receive data. In addition, the case where the input layer 910 includes multiple nodes is shown. The subsampled image may be input to each node forming the input layer 910 through a pooling operation.

[0161] As shown, two adjacent layers are connected to a plurality of edges 920, such as edge 921. Each node may have a weight value corresponding thereto, and thus, Fig. 9 The neural network shown may obtain output data based on values ​​obtained by operating input signals and weight values, such as via a convolution operation.

[0162] when Fig. 9 When the neural network shown is trained to generate style-transferred images by transferring the style of an image, Fig. 9 The neural network shown may output a migration image 730 having the set style 720 .

[0163] Execution Reference Figure 7 , Figure 8 and Fig. 9 The neural network of the operation of the described AI-based style transfer may be referred to as an AI-based image generation model, and the AI-based image generation model may be implemented by software and / or hardware. The neural network as the AI-based image generation model may be formed in the display device 1300 as described above, and may be formed in an external device (e.g., server 1350) that is connected to the display device 1300 via a communication network.

[0164] Fig.10 is a diagram illustrating an exemplary simulation-based style transfer operation according to an exemplary embodiment of the present disclosure.

[0165] The display device 1300 according to an embodiment of the present disclosure may perform a simulation-based style transfer operation. For example, a configuration that performs simulation-based style transfer may be referred to as, for example, a "photo filter". For example, the controller 320 may perform simulation-based style transfer, and may include a first photo filter corresponding to a first style. The controller 320 may control the execution of the simulation-based style transfer using the first photo filter stored in the memory 310. The display device 1300 may include the first photo filter as a separate configuration, and use the first photo filter to control the execution of the simulation-based style transfer under the control of the controller 320.

[0166] In addition, the display device 1300 may include a plurality of photo filters corresponding to each of the plurality of styles. A photo filter corresponding to a first style among the plurality of photo filters may be referred to as a first photo filter, for example. In addition, when a style to be applied to style migration is added or developed, the controller 320 may additionally install a photo filter corresponding to the added or developed style.

[0167] The controller 320 may control generation of at least one of the second images 556 , 557 , 558 , 559 , and 570 , for example, using a first photo filter.

[0168] In addition, when receiving an input of a user selecting a first style as any one of a plurality of different styles, the controller 320 may control generation of at least one second image by performing image processing using a first photo filter so that the first image has the first style. Fig.13 As shown, user input may be received via user interface 1320 or mobile device 1370 .

[0169] Fig.10 is a diagram illustrating an exemplary style transfer operation performed by a photo filter according to an exemplary embodiment of the present disclosure. Fig.10 , the controller 320 may adjust the size of the original image 1001 as the input image and change its color to black and white, thereby generating the size and color adjusted image 1003, and may calculate the gradient and / or edge density by analyzing the size and color adjusted image 1003, thereby obtaining the edge image 1005. The controller 320 may generate the image 1020 by changing the color of the original image 1001 to correspond to the set style. Through the image processing 1030, by repeatedly drawing the previously prepared texture on the color-changed image 1020, a style transfer image 1050 may be generated. For example, the style transfer image 1050 may be generated by performing image processing based on the edge image 1005 to change at least one of, for example, but not limited to, the color, size, direction, texture, brightness, saturation, etc. of the image 1020.

[0170] The photo filter can perform the above-mentioned style transfer operation in real time and generate a style transfer image within a few seconds, and can immediately output the intermediate result in the process of performing the style transfer operation. Therefore, the display device 1300 can display at least one second image generated by the simulation-based style transfer operation in real time, which includes the intermediate result and the final result. Therefore, the user can watch the image of interest by watching the images that are sequentially style-transferred in real time without feeling bored.

[0171] Fig.11 is a diagram showing an exemplary image output from a display device according to an exemplary embodiment of the present disclosure. Fig.11 , at least one second image as one of the images to be generated by simulation-based style transfer may include image 1110 , image 1120 , image 1130 , image 1140 , and image 1150 . Fig.11 The images 1110, 1120, 1130, 1140 and 1150 may correspond to Figure 5 second image 556 , second image 557 , second image 558 , second image 559 and image 570 . Fig.11 The image 1160 may correspond to Figure 5 The third image 541.

[0172] The controller 320 may perform image transition to sequentially output the gradually changing second image and then the third image on the display 330 .

[0173] In other words, after sequentially outputting the image 1110 , the image 1120 , the image 1130 , the image 1140 , and the image 1150 , the display 330 may output the image 1160 through image transition.

[0174] For the image transition from image 1150 to image 1160, the following is used for execution: for example, but not limited to, an image transition method for operations in 3D, an image transition method using a dissolving method, an image transition method as an enlargement or reduction method using a specific 2D texture / mask, etc.

[0175] like Fig.11 As shown, when the user watches at least one second image based on a simulation style that is migrated and output in real time through the image transition screen, until the third image that is an AI-based style migration image is displayed, the user can finally watch high-quality images without feeling bored. Therefore, user satisfaction can be improved.

[0176] In addition, in an embodiment of the present disclosure, the controller 320 may control the output of at least one second image and a third image on the display during the execution of the environmental service. For example, when setting the environmental service screen, the controller 320 may perform the above-mentioned AI-based style migration 540 and simulation-based style migration 550, and control the output of the generated at least one second image and a third image.

[0177] In addition, the controller 320 may control the at least one second image and the third image to be repeatedly displayed on the display 330 by an image transition method. In addition, the controller 320 may control the at least one image selected from the at least one second image and the third image to be repeatedly displayed on the display 330 by an image transition method.

[0178] like Fig.11 As shown, through the image transition screen, the user can watch at least one of the second images that are subjected to simulation-based style transfer and output in real time until the third image that is an AI-based style transfer image is displayed. Therefore, the user can finally watch high-quality images without feeling bored. Therefore, the display device according to an embodiment of the present disclosure can improve user satisfaction.

[0179] Fig.12 1 is a flowchart showing an exemplary method 1200 of controlling a display device according to another exemplary embodiment of the present disclosure. In addition, according to an exemplary embodiment of the present disclosure, the control may be performed in the display device 300 or 1300. Fig.12 The method 1200 of the display device is shown. Therefore, Fig.12 Operations performed in the display device 300 or 1300, for example, are shown.

[0180] In addition, Fig.12 In the figure, the same reference numerals are used to show Figure 6 Therefore, in the description of the method 1200 for controlling the display device, the same elements as those in the method 1200 may not be repeated. Figure 6 Those repeated descriptions.

[0181] In an embodiment of the present disclosure, the controller 320 may determine whether the quality of the third image satisfies a certain reference (S1250) before an image transition from the last second image (e.g., image 1150) to the third image (e.g., image 1160), and perform the image transition based on the determination result.

[0182] For example, the controller 320 may determine whether the quality of the third image (e.g., image 1160) is equal to or greater than or exceeds a certain reference value (S1250), and when the quality of the third image (e.g., image 1160) is equal to or greater than the certain reference value, the image transition may be performed (S631). When the quality of the third image (e.g., image 1160) is less than or equal to or less than the certain reference value, the controller 320 may not perform the image transition and continuously output the last second image (e.g., image 1150) (S1260).

[0183] The quality of the third image may correspond to the image quality of the third image, and the controller 320 may perform an evaluation of the image quality of the third image (S1250). For example, the image quality evaluation method may include, for example, but not limited to, a non-reference image quality assessment (NRIQA) method. The NRIQA method may evaluate image quality without information about a reference image (e.g., an original image).

[0184] For example, according to the NRIQA method, the quality of an image can be evaluated by detecting blur or wave phenomena. To this end, an NRIQA algorithm for detecting visual artifacts to find specific distortions is used to extract contour information from the image, and then contour diffusion or contour distribution is found. When a specific distribution is found in a predetermined database (e.g., a database indicating the presence of distorted contour distribution), the generated image (e.g., a third image based on AI style migration) can be determined as a case where the image cannot be used because the image contains severe distortion. Therefore, the controller 320 can prevent the third image from being displayed, and can continuously output the last second image, such as image 1150 (S1260).

[0185] According to the AI-based style transfer, the user can neither check the intermediate results nor anticipate the shape of the third image as the final result. Therefore, an image recognized by the user as having distortion may be output. For the image recognized by the user as having distortion, the user may determine that the output third image has no or low artistic quality, work quality, and / or creativity, and in this example, user satisfaction may be improved by preventing the third image from being displayed.

[0186] In an exemplary embodiment of the present disclosure, after outputting the third image ( S633 ), when the user inputs a request to prevent output of the third image through the user interface 1320 , the controller 320 may control the last second image, for example, image 1150 , to be output again.

[0187] Fig.14 is another block diagram illustrating an exemplary display device 1400 according to another exemplary embodiment of the present disclosure. Fig.14 The display device 1400 shown may correspond to Fig.13 The display device 1300 shown. In addition, Fig.14 In the figure, the same reference numerals are used to show Fig.13 Therefore, in the description of the display device 1400, descriptions that are redundant with the description of the display device 1300 may not be repeated.

[0188] Reference Fig.14 ,and Fig.13 Compared to the display device 1300 shown, the display device 1400 may further include a neural network processor 1410 .

[0189] The display device 1400 may perform operations through a neural network at the controller 320 or an external device (eg, the server 1350). In addition, the display device 1400 may further include a neural network processor 1410, which may be a separate processor that performs operations through a neural network.

[0190] The neural network processor 1410 may perform operations through a neural network. For example, in an embodiment of the present disclosure, the neural network processor 1410 may perform operations for style transfer through a neural network by executing one or more instructions. For example, the neural network processor 1410 may generate a third image by performing an AI-based style transfer operation.

[0191] according to Fig.13 and Fig.14 The embodiments of the present disclosure shown can further improve the speed, time and quality of AI-based style transfer when the operation of the neural network is performed by an external device such as a server.

[0192] Fig.15 1 is another block diagram showing an exemplary display device 1500 according to another exemplary embodiment of the present disclosure. The display device 1500 according to another exemplary embodiment of the present disclosure may correspond to the display device 1500 according to the reference Figures 1 to 14 The display device 300, 1300, or 1400 of the exemplary embodiment of the present disclosure described herein is described. Therefore, in the description of the display device 1500, the description of the display device 1500 may not be repeated. Figures 1 to 14 The description in the description is redundant.

[0193] Reference Fig.15, the display device 1500 may include a video processor (e.g., including a video processing circuit) 1510, a display 1515, an audio processor (e.g., including an audio processing circuit) 1520, an audio outputter (e.g., including an audio output circuit) 1525, a power supply 1530, a tuner 1540, a communicator (e.g., including a communication circuit) 1550, a detector (e.g., including a detection or sensing circuit) 1560, an input / outputter (e.g., including an input / output circuit) 1570, a controller (e.g., including a processing circuit) 1580, and a memory 1590.

[0194] The controller 1580 may correspond to Figure 3 , Fig.13 and Fig.14 The controller 320 shown in FIG. The communicator 1550, the display 1515, and the memory 1590 of the display device 1500 may correspond to Fig.13 and Fig.14 The communicator 1310, the display 330 and the memory 310 are shown. Fig.15 In the description of the display device 1500 according to the exemplary embodiment of the present disclosure, descriptions that are redundant with the description of the display device 300, 1300, or 1400 may not be repeated.

[0195] The video processor 1510 may include various video processing circuits and perform processing on the video data received by the display device 1500. In the video processor 1510, various image processing may be performed on the video data, such as (but not limited to) decoding, scaling, noise filtering, frame rate conversion, resolution conversion, etc.

[0196] Controller 1580 may include various processing circuits and may control receiving write requests for video data processed in video processor 1510 and encryption of video data to be written to a memory device (not shown) (e.g., RAM (not shown)) included in controller 1580 or memory 1590.

[0197] The display 1515 may display a video included in a broadcast signal received through the tuner 1540 on a screen under the control of the controller 1580. In addition, the display 1515 may display content input through the communicator 1550 or the input / output 1570, such as a moving picture.

[0198] In addition, the display 1515 can output an image stored in the memory 1590 under the control of the controller 1580. In addition, the display 1515 can display a voice user interface (UI) including, for example, a voice instruction guide to perform a voice recognition task corresponding to voice recognition, or a motion UI including, for example, a motion guide for motion recognition of the user to perform a motion recognition task corresponding to motion recognition.

[0199] The audio processor 1520 may include various audio processing circuits and perform audio data processing. In the audio processor 1520, various processing on audio data may be performed, such as but not limited to decoding, amplification, noise filtering, etc. The audio processor 1520 may include multiple audio processing modules to process audio corresponding to multiple contents.

[0200] The audio outputter 1525 may include various audio output circuits, and output the audio included in the broadcast signal received through the tuner 1540 under the control of the controller 1580. In addition, the audio outputter 1525 may output the audio stored in the memory 1590 under the control of the controller 1580. The audio outputter 1525 may output the audio, such as voice or sound, input through the communicator 1550 or the input / outputter 1570. The audio outputter 1525 may include, for example, but not limited to, at least one of a speaker 1526, an earphone output terminal 1527, an S / PDIF (Sony / Philips Digital Interface) output terminal 1528, and the like. The audio outputter 1525 may include a combination of the speaker 1526, the earphone output terminal 1527, and the S / PDIF output terminal 1528.

[0201] In an embodiment of the present disclosure, the speaker 1526 may output a sound wave signal. For example, the speaker 1526 may output an ultrasonic signal.

[0202] The power supply 1530 provides power input from an external power source to the constituent elements 1510 to 1590 in the display device 1500 under the control of the controller 1580. In addition, the power supply 1530 may provide power output from one or more batteries (not shown) located within the display device 1500 to the constituent elements 1510 to 1590 under the control of the controller 1580.

[0203] The tuner 1540 may select a broadcast signal received by wire or wirelessly using amplification, mixing, or resonance by tuning only the frequency of a channel to be received by the display device 1500 from among many radio wave components. The broadcast signal may include, for example, but not limited to, audio, video, additional information (e.g., electronic program guide (EPG)), and the like.

[0204] Tuner 1540 can receive a broadcast signal within a frequency range corresponding to a channel number, such as cable broadcast No. 506, based on user input, such as a control signal received by an external control device (not shown) (e.g., a remote controller on an EPG screen), such as a channel number input, a channel up-down input, and a channel input.

[0205] The tuner 1540 may receive broadcast signals from various sources such as terrestrial broadcasting, cable broadcasting, satellite broadcasting, and Internet broadcasting. The tuner 1540 may receive broadcast signals from sources such as analog broadcasting or digital broadcasting. The broadcast signals received by the tuner 1540 may be decoded, such as audio decoding, video decoding, or additional information decoding, so as to be classified into audio, video, and / or additional information. The classified audio, video, and / or additional information may be stored in the memory 1590 under the control of the controller 1580.

[0206] The tuner 1540 of the display device 1500 may be single or plural. According to one embodiment of the present disclosure, when the tuner 1540 includes a plurality of tuners, a plurality of broadcast signals may be output to a plurality of windows forming a multi-window screen provided to the display 1515 .

[0207] The tuner 1540 may be implemented as an integrated type with the display device 1500 , or as a separate device having a tuner electrically connected to the display device 1500 , for example, a set-top box (not shown) or a tuner (not shown) connected to the input / output 1570 .

[0208] The communicator 1550 may include various communication circuits, and connects the display device 1500 to an external device such as an audio device under the control of the controller 1580. The controller 1580 may transmit / receive content to / from an external device connected through the communicator 1550, or download an application from the external device or perform web browsing. In detail, the communicator 1550 may be connected to a network to receive content from an external device (not shown).

[0209] As described above, the communicator 1550 may include various modules, including various communication circuits, including, for example but not limited to, at least one of a short-range communication module (not shown), a wired communication module (not shown), a mobile communication module (not shown), etc.

[0210] exist Fig.15 , an example in which the communicator 1550 includes one of a wireless LAN 1551, a Bluetooth communicator 1552, and a wired Ethernet 1553 is shown.

[0211] In addition, the communicator 1550 may include a module combination including one or more of a wireless LAN 1551, a Bluetooth communicator 1552, and a wired Ethernet 1553. In addition, the communicator 1550 may receive a control signal of a control device (not shown) under the control of the controller 1580. The control signal may be implemented in a Bluetooth type, an RF signal type, or a WIFI type.

[0212] In addition to Bluetooth, the communicator 1550 may also include other short-range communications such as near field communication (NFC; not shown) or a separate Bluetooth Low Energy (BLE) module (not shown).

[0213] The detector 1560 may include various detection or sensing circuits and detect a user's voice, a user's image, or a user's interaction.

[0214] The microphone 1561 may receive a voice issued by a user. The microphone 1561 may convert the received voice into an electrical signal and output the electrical signal to the controller 1580. The user's voice may include, for example, a voice corresponding to a menu or function of the display device 1500. For example, the recommended recognition range of the microphone 1561 is within 4 m from the microphone 1561 to the user's position, and the recognition range of the microphone 1561 may vary corresponding to the volume of the user's voice and the environment (e.g., speaker sound or ambient noise).

[0215] The microphone 1561 may be implemented as an integral type and a separate type with respect to the display device 1500. The separate type microphone 1561 may be electrically connected to the display device 1500 through the communicator 1550 or the input / output 1570.

[0216] It is easily understood by those skilled in the art that the microphone 1561 may be removed according to the performance and structure of the display device 1500 .

[0217] The camera 1562 receives images, such as continuous frames, corresponding to user movements including gestures within the camera recognition range. For example, the recognition range of the camera 1562 may be within 0.1m to 5m from the camera 1562 to the user. The user's movement may include, for example, a part of the user's body or a part of the body or the movement of the part of the user's body, such as the user's face, facial expression, hand, fist or finger. The camera 1562 may convert the received image into an electrical signal under the control of the controller 1580 and output the electrical signal to the controller 1580.

[0218] The controller 1580 may select a menu displayed on the display device 1500 by using the received recognition result of the motion, or perform control corresponding to the motion recognition result. For example, the control according to the received recognition result of the motion may include channel change, volume control, or cursor movement.

[0219] The camera 1562 may include a lens (not shown) and an image sensor (not shown). The camera 1562 may support optical zoom or digital zoom by using multiple lenses and image processing. The recognition range of the camera 1562 may be set differently according to angles or camera and environmental conditions. When the camera 1562 includes multiple cameras, a 3D still image or 3D motion may be received by using multiple cameras.

[0220] The camera 1562 may be implemented as an integral type or a separate type with respect to the display apparatus 1500. A separate device (not shown) including the separate type camera 1562 may be electrically connected to the display apparatus 1500 through the communicator 1550 or the input / output 1570.

[0221] It is easily understood by those skilled in the art that the camera 1562 may be removed according to the performance and structure of the display device 1500 .

[0222] The optical receiver 1563 may receive an optical signal including a control signal received from an external control device (not shown) through an optical window (not shown) in the frame of the display 1515. The optical receiver 1563 may receive an optical signal corresponding to a user input such as touch, press, touch gesture, voice, or motion from the control device (not shown). Under the control of the controller 1580, the control signal may be extracted from the received optical signal.

[0223] For example, the optical receiver 1563 may receive a signal corresponding to the pointing position of the control device (not shown) and transmit the signal to the controller 1580. For example, a user interface screen for receiving data or a command input by a user is output through the display 1515, and when the user inputs data or a command to the display device 1500 through the control device, when the user moves the control device while contacting a touch pad (not shown) provided in the control device (not shown) with a finger, the optical receiver 1563 may receive a signal corresponding to the movement of the control device and may transmit the signal to the controller 1580. In addition, the optical receiver 1563 may receive a signal indicating that a specific button provided in the control device is pressed and may transmit the signal to the controller 1580. For example, when the user presses a button-type touch pad (not shown) provided in the control device with a finger, the optical receiver 1563 may receive a signal indicating that the button-type touch pad is pressed and may transmit the signal to the controller 1580. For example, the signal indicating that the button-type touch pad is pressed may be used as a signal for selecting an item.

[0224] The input / output device 1570 may include various input / output circuits, and receives video (e.g., moving images), audio (e.g., voice or music), and additional information (e.g., EPG) from the outside of the display device 1500 under the control of the controller 1580. The input / output device 1570 may include, for example, but is not limited to, one or more of a high-definition multimedia interface (HDMI) port 1571, a component socket 1572, a PC port 1573, and a USB port 1574. The input / output device 1570 may include a combination of the HDMI port 1571, the component socket 1572, the PC port 1573, and the USB port 1574.

[0225] It will be easily understood by those skilled in the art that the configuration and operation of the input / output device 1570 may be variously implemented according to exemplary embodiments of the present disclosure.

[0226] The controller 1580 may include various processing circuits and control the overall operation of the display device 1500 and the signal flow between the internal elements (not shown) of the display device 1500, and may perform a function of processing data. When there is a user input or a preset storage condition is met, the controller 1580 may execute an operating system (OS) and various applications stored in the memory 1590.

[0227] The controller 1580 may include a RAM (not shown), a ROM (not shown), and a processor (not shown), wherein the RAM is used to store signals or data input from the outside of the display device 1500, or is used as a storage area corresponding to various tasks performed in the display device 1500, and the ROM stores a control program for controlling the display device 1500.

[0228] The processor may include a graphics processing unit (GPU; not shown) for performing graphics processing corresponding to the video. The processor may be implemented by a system on a chip (SoC) that combines a core (not shown) and a GPU (not shown). The processor may include a single core, a dual core, a triple core, a quad core, and more cores.

[0229] In addition, the processor may include a plurality of processors. For example, the processor may be implemented by a main processor (not shown) and a sub-processor (not shown) operating in a sleep mode.

[0230] The graphics processor (not shown) can generate a screen including various objects such as icons, images, and texts by using an operator (not shown) and a renderer (not shown). The operator can calculate attribute values ​​such as coordinate values, shapes, sizes, or colors for displaying each object according to the layout of the screen by using user interactions detected by a detector (not shown). The renderer can generate screens including various layouts of objects based on the attribute values ​​calculated by the operator. The screen generated by the renderer is displayed in the display of the display 1515.

[0231] As described above, according to an embodiment of the present disclosure, a user can watch an image with a unique style, such as a third image based on AI style transfer, and will not feel bored during the time until the image is style transferred and generated. Therefore, according to an exemplary embodiment of the present disclosure, user satisfaction and user convenience can be improved.

[0232] The exemplary embodiments of the present disclosure may be embodied in the form of program commands that can be executed by various computing devices and may be recorded on a computer-readable medium. In addition, the exemplary embodiments of the present disclosure may be implemented by a computer-readable recording medium having a program recorded thereon, the program including instructions for executing the control method of the above-mentioned display device.

[0233] The computer-readable recording medium may include program commands, data files, data structures, etc., either individually or in combination. The program commands recorded on the medium may be specially designed and configured for the present disclosure, or may be available program commands known to those of ordinary skill in the art to which the present disclosure belongs, such as computer software. The computer-readable recording medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs or DVDs, magneto-optical media such as optical disks, and hardware devices such as ROMs, RAM flash memories, which are configured to store and execute program commands. Examples of program commands may include not only machine codes generated by a compiler, but also codes executed by an interpreter.

[0234] In addition, the control method of the above-mentioned display device according to various exemplary embodiments of the present disclosure can be implemented as a computer program product, which includes a recording medium, which stores a program, and the program performs the following operations: obtaining an operation of a sentence configured by multiple languages; and obtaining vector values ​​corresponding to each word contained in the sentence configured by multiple languages, and converting the obtained vector values ​​into vector values ​​corresponding to the target language, and obtaining the operation of the sentence configured in the target language based on the converted vector values.

[0235] Various exemplary embodiments of the present disclosure are described with reference to the accompanying drawings, but the present disclosure is not limited thereto, and it should be understood that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure.

Claims

1. A display device, comprising: monitor; a memory configured to store at least one instruction; as well as a controller comprising at least one processor configured to execute the at least one instruction, Wherein, the controller is configured to control the display device to: controlling generation of at least one second image of gradually changing style transfer by performing simulation-based style transfer on the first image to allow the first image to have a first style corresponding to a specific texture; In parallel with the generation of the at least one second image, controlling the generation of a third image by performing AI-based style transfer on the first image using a neural network, wherein a time for completing the generation of the third image is greater than a time for completing the generation of the at least one second image; and The image transition is performed such that the third image is output after the at least one second image which is gradually changed is sequentially output on the display.

2. The display device according to claim 1, wherein: The controller is configured to further control the display device to: The at least one second image is generated by performing the simulation-based style transfer based on edge information of the first image to allow the first image to have the specific texture.

3. The display device according to claim 1, wherein: The controller is configured to further control the display device to: Based on the generation order of the at least one second image, the at least one second image is controlled to be output sequentially on the display.

4. The display device according to claim 1, wherein: The controller is configured to further control the display device to: By performing the AI-based style transfer on the first image via the neural network, control is provided to generate the third image corresponding to the first style.

5. The display device according to claim 1, wherein: The controller is configured to further control the display device to: The content image and style of the first image are acquired by operating through the neural network, and the third image is generated by transferring the style of the first image to correspond to the first style while maintaining the content image.

6. The display device according to claim 1, wherein: The controller includes a first photo filter corresponding to the first style by performing the simulation-based style transfer, and The controller is further configured to control the display device to control generation of the at least one second image using the first photo filter.

7. The display device according to claim 6, wherein: The controller is configured to further control the display device to: Based on receiving the input selecting the first style, controlling the generation of the at least one second image by performing image processing using the first photo filter to allow the first image to have the first style, the first style being one of a plurality of different styles.

8. The display device according to claim 7, further comprising: a communicator including a communication circuit configured to perform communication with an external device, Wherein, the controller is configured to further control the display device to: An input selecting the first style from among the plurality of styles is received from the external device through the communicator.

9. The display device according to claim 1, further comprising: a communicator including a communication circuit configured to perform communication with an external device, wherein the external device is configured to perform the AI-based style transfer through the neural network, Wherein, the controller is configured to further control the display device to: Controlling receiving the third image obtained when the external device performs an operation through the neural network.

10. The display device according to claim 1, wherein: The neural network includes a convolutional neural network configured to repeatedly perform convolution operations to allow the first image to correspond to the first style.

11. The display device according to claim 1, wherein: The controller is configured to further control the display device to: The at least one second image and the third image are controlled to be output on the display during execution of the ambient service.

12. A method for controlling a display device, the method comprising: generating at least one second image of gradually changing style transfer by performing simulation-based style transfer on the first image to allow the first image to have a first style corresponding to a specific texture, In parallel with the generation of the at least one second image, generating a third image by performing AI-based style transfer on the first image using a neural network, wherein a time to complete the generation of the third image is greater than a time to complete the generation of the at least one second image, and The image transition is performed such that the third image is output after the at least one second image which is gradually changed is sequentially output on the display.

13. The method of claim 12, wherein: Generating the third image further includes: generating the third image by the neural network by an external device connected to the display device through a communication network; and The third image is transmitted from the external device to the display device.

Images