Display device and operation method thereof

Abstract

A display device and an operation method thereof are provided. The display device comprises: a memory for storing one or more instructions; and one or more processors for executing the one or more instructions stored on the memory. The processor according to one embodiment of the present disclosure executes the one or more instructions stored on the memory so as to: acquire an input image; input the input image into a contour prediction model so as to acquire a contour prediction result including information predicting whether the input image has contours; and perform, on the basis of the contour prediction result, a first blurring process and / or a second blurring process differing from the first blurring process, on the input image.

Description

Display device and method of operation thereof

[0001] The disclosed embodiments relate to a display device and a method of operating the same. Specifically, the disclosed embodiments relate to a display device and a method of operating the display device for reducing contours due to color banding.

[0002] A display device is a device that processes a video signal provided from a video signal source and displays it on a display, and includes a television receiver that receives and displays a video signal provided from a broadcasting station. Here, the video signal can correspond to an input image.

[0003] A display device can process an image to increase the user's satisfaction, aesthetics, and three-dimensionality when outputting an input image onto a display.

[0004] For example, a display device can perform a blurring process on an image. Blurring or blurring refers to reducing the sharpness of an image and applying a soft blurring effect, making the image appear blurry. The display device can output the blurred image (blurred image) on the display.

[0005] Meanwhile, when an image is displayed on a display, the image may exhibit color banding. Color banding occurs when the color of each pixel in an image composed of multiple pixels is rounded to the nearest color among the digital color levels. In an image with color banding, the gradient (or gradation) changes are not smooth, but rather the contour (or outline, border, contour line) appears as a band.

[0006] In particular, the greater the difference in color levels between adjacent regions, the more likely color banding is to occur. For example, color banding may be more likely to occur in images with prominent gradual color changes, such as photographs of a sunset sky, or in blurred images.

[0007] Color banding and contours due to color banding reduce user visibility, so technologies are being developed to minimize contours in images.

[0008] A display device according to one embodiment of the present disclosure includes a memory storing one or more instructions, and one or more processors executing the one or more instructions stored in the memory.

[0009] A processor according to one embodiment of the present disclosure obtains an input image by executing one or more instructions stored in a memory.

[0010] According to one embodiment of the present disclosure, a processor obtains a contour prediction result including information predicting whether a contour occurs in an image output by the display device based on the input image by executing one or more instructions stored in a memory, thereby inputting the input image into a contour prediction model.

[0011] According to one embodiment of the present disclosure, a processor performs at least one of a first blurring process and a second blurring process different from the first blurring process on the input image based on the contour prediction result by executing one or more instructions stored in a memory.

[0012] A method of operating a display device according to one embodiment of the present disclosure includes the steps of: obtaining an input image; inputting the input image into a contour prediction model, thereby obtaining a contour prediction result including information predicting whether a contour occurs for an image output by the display device based on the input image; and performing at least one of a first blurring process and a second blurring process different from the first blurring process on the input image based on the contour prediction result.

[0013] According to one embodiment of the present disclosure, a computer-readable recording medium having recorded thereon a program for performing at least one of the operating methods of a display device on a computer may be provided.

[0014] FIG. 1 is a drawing for explaining an operation of a display device according to one embodiment of the present disclosure to output a blur image.

[0015] FIG. 2 is a drawing for explaining color banding and contour according to color banding according to one embodiment of the present disclosure.

[0016] FIG. 3 is a flowchart illustrating an operation method of a display device for preventing contour occurrence according to one embodiment of the present disclosure.

[0017] FIG. 4 is a flowchart illustrating an operation method of a display device for preventing contour occurrence according to one embodiment of the present disclosure.

[0018] FIG. 5 is a block diagram showing the configuration of a display device according to one embodiment of the present disclosure.

[0019] FIG. 6 is a block diagram illustrating a first blur processing according to one embodiment of the present disclosure.

[0020] FIG. 7 is a block diagram illustrating an example of a second blur processing according to one embodiment of the present disclosure.

[0021] FIG. 8 is a block diagram illustrating an example of a second blur processing according to one embodiment of the present disclosure.

[0022] Figure 9 is a drawing for explaining the second blur processing of Figure 8.

[0023] FIG. 10 is a block diagram illustrating an example of a second blur processing according to one embodiment of the present disclosure.

[0024] FIG. 11 is a block diagram illustrating blur processing for a video image according to one embodiment of the present disclosure.

[0025] FIG. 12 is a flowchart illustrating a method for providing learning data to a contour prediction model according to one embodiment of the present disclosure.

[0026] FIG. 13 is a diagram illustrating a method for performing reinforcement learning on a contour prediction model using learning data according to one embodiment of the present disclosure.

[0027] FIG. 14 is an example of a situation in which an image is blurred in a display device according to one embodiment of the present disclosure.

[0028] FIG. 15 is a schematic block diagram of a display device according to one embodiment of the present disclosure.

[0029] FIG. 16 is a detailed block diagram of a display device according to one embodiment of the present disclosure.

[0030] In this disclosure, the expression “at least one of a, b or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “all of a, b and c”, or variations thereof.

[0031] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0032] The terms used in this disclosure are described as currently common terms, taking into account the functions mentioned herein. However, these terms may mean various other terms depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Therefore, the terms used in this disclosure should not be interpreted solely based on their names, but rather based on the meanings of the terms and the overall content of this disclosure.

[0033] Additionally, the terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure.

[0034] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where it is "directly connected" but also the cases where it is "electrically connected" with another element in between.

[0035] As used herein, and particularly in the claims, the terms "above" and "above" and similar referents may refer to both the singular and the plural. Furthermore, unless the order of steps in a method according to the present disclosure is explicitly specified, the steps described may be performed in any appropriate order. The present disclosure is not limited by the order in which the steps are described.

[0036] The appearances of phrases such as “in some embodiments” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment.

[0037] Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a given function. Furthermore, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented by algorithms that execute on one or more processors. Furthermore, the present disclosure may employ conventional techniques for electronic configuration, signal processing, and / or data processing. Terms such as “mechanism,” “element,” “means,” and “configuration” may be used broadly and are not limited to mechanical and physical configurations.

[0038] In the present disclosure, a processor may include various processing circuits and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits, including at least one processor. At least one processor, one or more processors, may be configured to perform various functions described herein, individually and / or collectively, in a distributed fashion. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform various functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, the at least one processor may include a combination of processors that perform various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0039] Additionally, the connecting lines or connecting members between components depicted in the drawings are merely exemplary representations of functional connections and / or physical or circuit connections. In an actual device, connections between components may be represented by various functional connections, physical connections, or circuit connections that may be replaced or added.

[0040] Additionally, terms such as “part”, “module”, etc. described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

[0041] Additionally, the term "user" in this specification refers to a person who uses the display device, and may include a consumer, evaluator, viewer, administrator, or installer. Furthermore, the term "manufacturer" or "provider" in this specification may refer to a manufacturer that manufactures the display device and / or components included in the display device.

[0042] In the present disclosure, an 'image' may include a still image, a graphic, a picture, a frame, a moving image composed of a plurality of consecutive still images, or a video.

[0043] In this disclosure, "neural network" refers to a representative example of an artificial neural network model that mimics brain neurons, and is not limited to an artificial neural network model using a specific algorithm. A neural network may also be referred to as a deep neural network.

[0044] FIG. 1 is a drawing for explaining an operation of a display device according to one embodiment of the present disclosure to output a blur image.

[0045] Referring to FIG. 1, the display device (100) may refer to a device that has a display and can display image content, video content, game content, graphic content, etc. For example, the display device (100) may include various types of electronic devices that can receive and output content, such as network TV, smart TV, Internet TV, web TV, IPTV, PC, and various smart devices such as smartphones. The display device (100) may be referred to as a display device in that it receives and displays content, and may also be referred to as a content receiving device, a sink device, an electronic device, a computing device, etc.

[0046] A display device (100) according to one embodiment of the present disclosure can perform image processing. For example, the display device (100) can perform image processing on an input image (10) to generate an output image (20). The display device (100) can output the output image (20) on a display. Here, the image can include a graphic image, a video image, a mixing image between a graphic and a video, etc.

[0047] A display device (100) according to one embodiment of the present disclosure can perform blurring (30). Blurring (30) refers to making an image blurry by reducing the sharpness of the image and applying a soft blur effect. The display device (100) can perform blurring (30) on an input image (10) to generate a blurred image (blurred image). The blurred image can have a gradient effect in which the outlines of objects included in the image are blurred and colors are continuously continued. The display device (100) can output the blurred image to the display. Here, the blurred image can correspond to the output image (20).

[0048] In one embodiment of the present disclosure, color banding may occur in a blurred image. The blurred image may have contours due to the color banding. This will be further described with reference to FIG. 2.

[0049] FIG. 2 is a drawing for explaining color banding and contour according to color banding according to one embodiment of the present disclosure.

[0050] Figure 2 illustrates images, a first image (210) and a second image (220), which have a large difference in color levels at the boundary between two adjacent areas. For example, the first image (210) and the second image (220) may be images in which gradual color changes stand out, such as sunset photographs, or may be blurred images. The images (210, 220) may be images susceptible to color banding.

[0051] The first image (210) exemplifies an image in which color banding and contours according to color banding exist, and the second image (220) exemplifies an image in which color banding and contours according to color banding are minimized.

[0052] For example, a graphic image may contain multiple pixels. Each of the pixels can be expressed in grayscale. For example, if a graphic image has 8-bit grayscale data, each pixel can be expressed in one of 256 color levels.

[0053] Alternatively, each of the plurality of pixels can be represented as RGB data. For example, if a graphic image has 8-bit RGB data, red (R, red) data, green (G, green) data, and blue (B, blue) data can each be represented as one of 256 color levels. The red, green, and blue colors represented by their respective levels can be mixed to represent a color. For example, if the RGB data is (255, 255, 255), it represents white, and if the RGB data is (0, 0, 0), it represents black.

[0054] For example, in the first image (210), the color of a pixel at the boundary between two adjacent color areas may be rounded to a nearby color level. For example, as the RGB data of each pixel is rounded to the nearest color level, the color level may increase in a stepwise (or gradual) manner rather than gradually. Accordingly, a boundary line may exist for each color area in the first image (210).

[0055] For example, the second image (220) may be an image in which there is no boundary between color areas and the color levels gradually increase.

[0056] Hereinafter, examples in which a contour is generated, such as the first image (210), when displaying a blurred image on a display device (100) will be described in detail.

[0057] In a display device (100), contours may occur at the grayscale boundaries of a blurred image even after a graphic processing (e.g., blur processing) operation, depending on factors such as the resolution or bit count of the display, and the performance of the processor. Contours occurring in an image after a graphic processing operation may be difficult to remove.

[0058] For example, in a display device (100), the number of bits of data on which graphic processing occurs may be different from the number of bits of the output display panel. For example, while graphics may be processed as 8 bits, the display panel may process it as 10 bits. For example, 8-bit RGB data can express approximately 16 million color levels, and 10-bit RGB data can express approximately 1 billion color levels.

[0059] When the display device (100) converts an 8-bit graphic image into a 10-bit one, color banding that did not occur in the graphic processing operation may occur because more color expressions must be expressed. For example, the display device (100) may involve a dithering process that provides dither, which is noise data that replaces the difference in grayscale (e.g., the difference between about 16 million color levels and about 1 billion color levels). The dithering process is one of the post-processing methods for removing color banding, but noise may increase in proportion to the number of dithers. In addition, the degree of dithering varies depending on the performance of the processor of the display device (100), and color banding may occur.

[0060] For example, when a display device (100) outputs an image, even images with the same color level may be expressed brighter or darker depending on the display panel. In particular, in the case of black and low-grayscale images, color banding may be more or less noticeable depending on the display panel.

[0061] For example, since the viewing angles of the display panels of the display device (100) differ, the luminance increase rate may vary depending on the viewing angle. In this case, when the display device (100) outputs a blurred image, color banding may be more or less noticeable to the user. Furthermore, color banding may be more or less noticeable depending on the panel specifications of the display device (100).

[0062] Additionally, as an example, the display device (100) can output a mixed image that combines or mixes graphic images on a video image. The display device (100) can apply a blur effect to the mixed image. Contours present in the mixed image to which the blur effect has been applied may not be sufficiently removed according to a general blur processing algorithm. Meanwhile, the blur effect of the mixed image will be described later in FIG. 14.

[0063] In this way, contours that occur after a blur processing operation and contours that occur at the hardware level may not be sufficiently removed by general blur processing algorithms.

[0064] A display device (100) according to one embodiment of the present disclosure may use a blur algorithm that minimizes contour occurrence in order to prevent contour occurrence in advance after a blur processing operation.

[0065] For example, the display device (100) can predict in advance whether color banding will occur when the original image is blurred, and perform different blur processing on the original image according to the prediction result.

[0066] For example, if the display device (100) predicts that color banding will occur when the original image is blurred, the display device (100) can apply a blur algorithm that minimizes the occurrence of contours. By performing different blur processing on the original image, the display device (100) can minimize color banding of the blurred image outputted on the display. For example, the display device (100) can output the second image (220) instead of outputting the first image (210). The user can recognize an image with minimized color banding and contours through the display device (100).

[0067] Hereinafter, a method for minimizing color banding and contour while processing a blur image by a display device (100) according to one embodiment of the present disclosure is described.

[0068] FIG. 3 is a flowchart illustrating an operation method of a display device for preventing contour occurrence according to one embodiment of the present disclosure.

[0069] Referring to FIG. 3, in operation 310, the display device (100) can obtain an input image.

[0070] In one embodiment, the display device (100) can obtain an input image having a predetermined format. For example, the input image having a predetermined format may include a video image, a graphic image, and the like.

[0071] A given format may mean a format having a given color space that the display device (100) can represent, in order to output an image from the display device (100) or store it in memory before outputting it. For example, the input image may have an image format such as YUV, RGB, RGBA, etc. For example, the input image may be an image in YUV format stored in a video plane, or an image in RGB or RGBA format stored in a graphic plane. Each pixel value may be written in the input image.

[0072] In one embodiment, the display device (100) can acquire an input image through a capture tool. The display device (100) can store an image previously stored in a buffer in a separate buffer through the capture tool. The capture tool may be software, hardware, or a combination thereof for storing an image with pixel values ​​entered in a separate buffer.

[0073] In operation 320, the display device (100) can obtain a contour prediction result for the input image by inputting the input image into the contour prediction model.

[0074] A contour prediction model according to one embodiment may be a model trained to predict contours for an image. For example, the contour prediction model may determine whether an image is likely to have contours when the original image is blurred. The contour prediction model may be applied to an unblurred original image.

[0075] A contour prediction model according to one embodiment may correspond to a reinforcement learning model. A reinforcement learning model may be a learning model that utilizes feedback on whether the results of situational judgment based on learning are correct. For example, a contour prediction model may be trained using pairs of learning images and feedback on the learning images as training data. Feedback on the learning images may mean a rating, score, reward, etc. regarding the presence of a contour in a blurred image. The operation of the contour prediction model undergoing reinforcement learning is described below with reference to FIGS. 12 and 13.

[0076] A contour prediction model according to one embodiment is a model that is pre-trained to output contour prediction results for an image, and may be referred to as a trained reinforcement learning model or a deployed reinforcement learning model.

[0077] A display device (100) according to one embodiment can obtain a contour prediction result for an input image using a contour prediction model. For example, the display device (100) can input an input image to the contour prediction model. The input of the contour prediction model may be an unblurred original image.

[0078] According to one embodiment, a display device (100) can convert an input image into a vector representation, generate an image embedding, and input this into a contour prediction model. Image embedding refers to data that compresses input image information. Instead of inputting the input image into the contour prediction model, the display device (100) can reduce the amount of data processing of the display device (100) by inputting an image embedding that converts the input image into a vector representation. Image embedding is described in more detail in FIG. 12.

[0079] A contour prediction model according to one embodiment can output a contour prediction result including information predicting whether a contour occurs for an input image.

[0080] For example, a contour prediction model can output prediction results as probability values. For example, a contour prediction model can output a high probability value corresponding to an image with a high probability of contour occurrence. A contour prediction model can output a low probability value corresponding to an image with no or low probability of contour occurrence. The probability values ​​can be compared to a threshold.

[0081] Alternatively, for example, a contour prediction model may output prediction results as binary values ​​such as 0 and 1. For example, a contour prediction model may output 1 in response to an image with a high probability of contour occurrence. A contour prediction model may output 0 in response to an image with no or low probability of contour occurrence. However, the form of the contour prediction results is not limited to the examples described above.

[0082] In operation 330, the display device (100) may perform at least one of a first blurring process and a second blurring process on the input image based on the contour prediction result.

[0083] According to one embodiment, the display device (100) may perform different blur processing on the original image depending on the contour prediction result. For example, the display device (100) may apply a first blur processing to an image with no or low possibility of contour occurrence, and apply a second blur processing to an image with or high possibility of contour occurrence.

[0084] For example, when the prediction result of the contour prediction model is output as a probability value, the display device (100) can determine an image having a probability value greater than or equal to a threshold as an image likely to have a contour, and determine an image having a probability value less than or equal to a threshold as an image likely to have a contour.

[0085] For example, if the prediction result of the contour prediction model is output as a binary value such as 0 and 1, the display device (100) can determine an image with a prediction result of 1 as an image with a possibility of contour occurrence, and can determine an image with a prediction result of 0 as an image with no possibility of contour occurrence.

[0086] The first blur processing and the second blur processing may be different from each other. The first blur processing and the second blur processing may each be implemented using software, hardware, or a combination thereof.

[0087] For example, the display device (100) may generate a blur image for an input image through separate paths or separate pipelines based on the contour prediction result. For example, the display device (100) may include a first pipeline that receives an image with no or low possibility of contour occurrence and generates a blur image. The display device (100) may include a second pipeline that receives an image with a high or no possibility of contour occurrence and generates a blur image. The first blur processing may correspond to the first pipeline (510) of FIG. 5, and the second blur processing may correspond to the second pipeline (520) of FIG. 5.

[0088] The first blur processing according to one embodiment may include general operations for blurring an image. For example, the first blur processing may include downscaling, blurring, and upscaling operations on the input image. Furthermore, the first blur processing according to one embodiment may further include a dithering operation on the blurred image.

[0089] The second blur processing according to one embodiment may include a blur processing operation that minimizes contour generation. The second blur processing may include an operation that preemptively reduces contours generated at the hardware level. For example, for the same original image, the second blurred image may have fewer contours than the first blurred image.

[0090] For example, the second blur processing may further include a dimming operation that lowers the color gradation of the image, a contour interpolation operation that softens the boundary area where a contour exists, and an operation that blurs by lowering the blur intensity. For example, the second blur processing may further include the operations described above in the general operation for blur processing (e.g., the first blur processing).

[0091] For example, the second blur processing may include a first subprocess that further performs a dimming operation after performing downscaling, blurring, and upscaling on the input image. The first subprocess may perform a dimming operation that lowers the color gradation of the blurred image, thereby minimizing the contours that are visible to the user.

[0092] For example, the second blur processing may include a second sub-process that further performs a contour interpolation operation after performing downscaling, blurring, and upscaling on the input image. The contour interpolation operation may include an operation of alleviating the contour of the boundary area by interpolating the color gradation of each pixel in the boundary area where the color difference between pixels belonging to the blurred image is large. The second sub-process may reduce the contour of the blurred image by performing the contour interpolation operation.

[0093] For example, the second blur processing may include a third sub-process that performs blur processing by adjusting the blur intensity. For example, the third sub-process may include downscaling, blurring, and upscaling operations on the input image, but may be set to a relatively low blur intensity. For example, the second blur intensity used in the third sub-process (the second blur processing) may be lower than the first blur intensity used in the first blur processing. The blur intensity may be adjusted by a scaling factor, a blur filter size, etc. The third sub-process may minimize the contour being visible to the user by lowering the blur intensity.

[0094] According to one embodiment, the first sub-process, the second sub-process, and the third sub-process may each be implemented as separate paths or separate pipelines. For example, the first sub-process may correspond to the first sub-pipeline (521), the second sub-process may correspond to the second sub-pipeline (522), and the third sub-process may correspond to the third sub-pipeline (523).

[0095] A display device (100) according to one embodiment can predict contours (or color banding) that occur when applying a blur effect to an input image, and apply an adaptive blur effect accordingly. The display device (100) can prevent contours that occur after graphic processing (e.g., blur processing). In addition, since the display device (100) performs different blur processing on the original image in advance before applying the blur effect, computational costs and computational time can be reduced.

[0096] FIG. 4 is a flowchart illustrating an operation method of a display device for preventing contour occurrence according to one embodiment of the present disclosure.

[0097] Referring to FIG. 4, operation 410 may correspond to operation 310 of FIG. 3, and operation 420 may correspond to operation 320 of FIG. 3.

[0098] In operation 430, the display device (100) may perform operation 440 or operation 450 based on the prediction result regarding whether a contour is likely to occur.

[0099] In operation 440, the display device (100) may perform a first blur processing on the input image as contour occurrence is not predicted.

[0100] In operation 450, the display device (100) may perform a second blur processing on the input image as contour occurrence is predicted.

[0101] For example, if the contour prediction model outputs the prediction result as a probability value, the display device (100) can compare the probability value with a threshold. If the probability value indicating the prediction result is less than the threshold, the display device (100) can perform a first blur process on the input image. If the probability value is greater than or equal to the threshold, the display device (100) can perform a second blur process on the input image.

[0102] Alternatively, for example, a contour prediction model may output a prediction result of 0 or 1. 0 corresponds to an image with a low probability of contour occurrence, and 1 corresponds to an image with a high probability of contour occurrence, but the opposite may also apply. The display device (100) may perform a first blur process on the input image when the prediction result is 0. The display device (100) may perform a second blur process on the input image when the prediction result is 1.

[0103] The first blur processing and the second blur processing are described in operation 330 of FIG. 3.

[0104] In operation 460, the display device (100) can generate a blurred image with the contour removed.

[0105] In operation 470, the display device (100) can display a blurred image. The display device (100) can minimize the occurrence of contours that occur during the step of outputting the blurred image to the display panel. For example, the display device (100) can apply blur processing to minimize contours for input images that are likely to have contours at the hardware level after graphic processing.

[0106] For example, color banding that occurs during the process of converting an 8-bit graphic image into 10-bit by a display device (100) can be minimized.

[0107] For example, color banding that occurs due to differences in brightness increase rates depending on different viewing angles for each display panel of the display device (100) can be minimized.

[0108] For example, color banding that occurs when the display device (100) outputs a mixing image can be minimized.

[0109] FIG. 5 is a block diagram showing the configuration of a display device according to one embodiment of the present disclosure.

[0110] Referring to FIG. 5, a display device (100) according to one embodiment may include a processor (110), a contour prediction model (113), an image processing module (117), and a display (130).

[0111] The processor (110) can obtain an input image having a predetermined format. For example, the input image may include a graphic image stored in a graphic plane and / or a video image stored in a video plane. The graphic plane may process an image in an RGB or RGBA color space. The video plane may process an image in a YUV color space. The image stored in the video plane or the graphic plane may be data in which pixel values ​​are written.

[0112] For example, the processor (110) may acquire an input image through a capture tool. The capture tool may be software, hardware, or a combination thereof for storing an image with pixel values ​​written in a separate buffer. For example, the capture tool may include, but is not limited to, a main capture tool, a subscaler capture tool, a post capture tool, a graphic capture tool, and the like.

[0113] In one embodiment, the processor (110) may input an input image to a contour prediction model (113). The contour prediction model (113) may be an artificial neural network model trained to infer whether a contour will occur when the input image is blurred through neural network operations. The contour prediction model (113) may receive an input image and output a contour prediction result of the input image. The contour prediction result may include information regarding whether a contour occurs in the input image.

[0114] The artificial neural network may include a deep neural network (DNN), for example, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks, but is not limited to the examples described above.

[0115] An artificial neural network can be a supervised learning neural network model, an unsupervised learning neural network model, or a reinforcement learning model.

[0116] Meanwhile, in one embodiment, the processor (110) may generate an image embedding that converts an input image into a vector representation. Instead of the input image, the processor (110) may input the image embedding containing input image information into the contour prediction model (113).

[0117] In one embodiment, the contour prediction model (113) may transmit the contour prediction result to the processor (110). The processor (110) may perform different blur processing on the input image based on the contour prediction result received from the contour prediction model (113).

[0118] The processor (110) may include a processor that controls the overall operation of the display device (100). The processor (110) may be implemented with at least one processor. The processor (110) may include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an APU (Accelerated Processing Unit), a MIC (Many Integrated Core), a DSP (Digital Signal Processor), and an NPU (Neural Processing Unit).

[0119] In one embodiment, the processor (110) and the contour prediction model (113) may be implemented together in a single chip. For example, the processor (110) may be implemented in the form of a System On Chip (SoC) in which the contour prediction model (113) is embedded. The processor (110) may perform operations to control the contour prediction model (113). For example, the processor (110) may execute the contour prediction model (113) stored in memory.

[0120] Additionally, in one embodiment, the processor (110) may include a graphic processor (Graphic Processing Unit, not shown) that processes data used to generate a video signal or image. Here, 'processing' may mean performing at least one of the following operations: receiving, converting (specifically, converting format, size, and / or characteristics, etc.), generating, and processing an image using a signal or data.

[0121] In one embodiment, the processor (110) may include an image processing module (117). The image processing module (117) may include various paths or pipelines that perform image generation and / or processing. For example, the image processing module (117) may perform image processing and generation operations through each independent pipeline (or path). Meanwhile, the image processing module (117) may be operated by a graphics processor (not shown), but is not limited thereto. For example, the image processing module (117) may be implemented by hardware, software, or a combination of hardware and software.

[0122] In one embodiment, the image processing module (117) may include at least two pipelines for performing blur processing. For example, the image processing module (117) may include a first pipeline (510) for performing first blur processing and a second pipeline (520) for performing second blur processing. A blurred image generated through the first pipeline (510) may be different from a blurred image generated through the second pipeline (520).

[0123] In one embodiment, the processor (110) may perform different blur processing for each input image based on the contour prediction result received from the contour prediction model (113).

[0124] For example, the processor (110) may perform the first blur processing when contour generation is not predicted for the input image. The processor (110) may generate a blurred image for the input image using the first pipeline (510).

[0125] For example, the processor (110) may perform a second blur process when contour generation is predicted for the input image. The processor (110) may generate a blurred image for the input image using the second pipeline (520).

[0126] In one embodiment, the operation of determining an appropriate pipeline for each input image may be performed directly in the image processing module (117) or may be performed through a separate management module (not shown) included in the processor (110).

[0127] In one embodiment, the processor (110) can transmit a blur image generated through the image processing module (117) to the display (130). The processor (110) can output the blur image by controlling the display (130).

[0128] The display (130) outputs an image on the screen. The display (130) can output an image corresponding to video data or an image signal. For example, the display (130) can output a blurred image. The blurred image output to the display (130) can have contour and color banding minimized.

[0129] Meanwhile, in one embodiment, data generated and / or processed in each pipeline may have a graphic format (e.g., RGB or RGBA). If the input image has a YUV video format, the processor (110) may perform format conversion on the input image before transmitting the input image to each pipeline. Here, YUV is an abbreviation used when referring to analog luminance signals and chrominance signals in a component video system, where Y represents intensity (brightness and darkness), U (Cb) represents the red component in intensity, and V (Cr) represents the blue component in intensity.

[0130] Hereinafter, the first pipeline and the second pipeline according to one embodiment will be described in detail.

[0131] FIG. 6 is a block diagram illustrating a first blur processing according to one embodiment of the present disclosure.

[0132] FIG. 6 illustrates a first pipeline (510) corresponding to the first blur processing. The first pipeline (510) may include downscaling (610), blurring (620), and upscaling (630) operations for an input image (601). At 630, the first pipeline (510) may further include a dithering operation. The first pipeline (510) may process the input image (601) to generate a blurred image (602).

[0133] The input image (601) may be an image in RGB, RGBA, and / or YUV format. The input image (601) may be an image with no or low possibility of contour occurrence when blurred.

[0134] Downscaling (610) may be a process of reducing the size of an input image (601). Downscaling (610) may reduce an image by 1 / scaling factor using a scaling factor determined for the image.

[0135] The blur intensity can be determined through the scaling factor of downscaling (610). For example, a larger scaling factor can lead to a larger blur intensity. For example, a smaller scaling factor can lead to a smaller blur intensity. The larger the blur intensity, the greater the blur effect on the blurred image.

[0136] Blurring (620) may be a process of applying a blur effect to an input image (601). In one embodiment, blurring (620) may be Gaussian blurring using a Gaussian filter, but is not limited thereto. Blurring (620) may use a separable filter. The separable filter may include a two-pass filter including a horizontal pass and a vertical pass.

[0137] The blur intensity can be determined by the filter size of the blurring (620). For example, if the size of the blur filter is large, the blur intensity can be large. For example, if the size of the blur filter is small, the blur intensity can be small.

[0138] Upscaling (630) may be a process of returning an image to its original size. Upscaling (630) may increase the size of a blurred image to match the output size of the display panel.

[0139] In one embodiment, dithering may be performed to reduce color banding effects that occur during upscaling. Dithering may be the process of applying a small amount of noise data, called dither, to minimize quantization errors. Dithering may remove contours present in the image.

[0140] Meanwhile, dithering is exemplified as being performed together with upscaling, but is not limited thereto and may be performed separately from upscaling.

[0141] When contour generation for the input image (601) is not predicted, the display device (100) can input the input image (601) into the first pipeline (510). The display device (100) can generate a blurred image (602) according to the first pipeline (510).

[0142] FIG. 7 is a block diagram illustrating an example of a second blur processing according to one embodiment of the present disclosure. In FIG. 7, descriptions overlapping with those in FIG. 6 are omitted.

[0143] FIG. 7 illustrates a second pipeline (520) corresponding to the second blur processing. The second pipeline (520) may include a first sub-pipeline (521). The first sub-pipeline (521) may include downscaling (710), blurring (720), and upscaling (730) operations for an input image (701). In 730, the first sub-pipeline (521) may further include a dimming operation. The first sub-pipeline (521) may process the input image (701) to generate a blurred image (702).

[0144] Here, downscaling (710), blurring (720), and upscaling (730) may correspond to downscaling (610), blurring (620), and upscaling (630) of FIG. 6.

[0145] The input image (701) may be an image in RGB, RGBA, and / or YUV format. The input image (701) may be an image that may have a possibility of generating a contour when blurred.

[0146] In one embodiment, dimming may be performed to reduce color banding effects in a blurred image. Dimming may be a process of lowering the color gradation of an image. For example, dimming may lower the RGB color levels per pixel of the image. Dimming may reduce the brightness, sharpness, etc. of the image. Images with reduced brightness or sharpness may have minimal occurrence of contours.

[0147] Meanwhile, dimming is exemplified as being performed in conjunction with upscaling, but is not limited thereto and may be performed separately from upscaling. Furthermore, the first sub-pipeline (521) may further include a dithering operation.

[0148] In one embodiment, when contour generation is predicted for the input image (701), the processor (110) may input the input image (701) into a first sub-pipeline (521), which is an example of the second pipeline (520). The processor (110) may generate a blurred image (702) according to the first sub-pipeline (521).

[0149] FIG. 8 is a block diagram illustrating an example of a second blur processing according to an embodiment of the present disclosure. FIG. 9 is a diagram illustrating the second blur processing of FIG. 8. In FIGS. 8 and 9, any description overlapping with that of FIG. 6 is omitted.

[0150] FIG. 8 illustrates a second pipeline (520) corresponding to the second blur processing. The second pipeline (520) may include a second sub-pipeline (522). The second sub-pipeline (522) may include downscaling (810), blurring (820), and upscaling (830) operations for an input image (801). The second sub-pipeline (522) may further include a contour interpolation (840) operation. The second sub-pipeline (522) may process the input image (801) to generate a blurred image (802).

[0151] Here, downscaling (810), blurring (820), and upscaling (830) may correspond to downscaling (610), blurring (620), and upscaling (630) of FIG. 6.

[0152] The input image (801) may be an image in RGB, RGBA, and / or YUV format. The input image (801) may be an image that may have a possibility of generating a contour when blurred.

[0153] In one embodiment, a contour interpolation (840) operation may be performed in the second sub-pipeline (522) to reduce the color banding effect for the blurred image (802). The contour interpolation (840) operation may include an operation of alleviating the contour of the boundary area by interpolating the color gradation of each pixel for the boundary area where the color difference between pixels belonging to the blurred image (802) is large.

[0154] For example, the processor (110) can detect a boundary area where the difference in grayscale between a first pixel and a second pixel included in an image is greater than a preset value. The processor (110) can interpolate an intermediate grayscale between the grayscale of the first pixel and the grayscale of the second pixel in the boundary area. The processor (110) can alleviate the contour of the boundary area by interpolating an intermediate grayscale for a boundary area where the difference in color levels is large.

[0155] Referring to FIG. 9, an original image (910) corresponding to the input image (801), a blurred intermediate image (920), and an interpolated image (930) corresponding to the blurred image (802) are illustrated.

[0156] The original image (910) may have a color combination in which the difference in grayscale is greater than or equal to a preset value. For example, in the boundary area (or boundary line) of the original image (910), the difference in color levels between pixels belonging to the image may exceed the preset color level. For example, if the boundary area has a B (0,0,255) pixel on the first side and an R (255,0,0) pixel on the second side, it may be the color combination described above.

[0157] If the color level of the original image (910) changes rapidly, the contour of the blurred image may be prominent. For example, the intermediate image (920) obtained through blurring (820) of the original image (910) may have a contour in the boundary area.

[0158] When the intermediate image (920) is contour interpolated, the contour of the interpolated image (930) can be minimized. For example, the processor (110) can calculate an intermediate grayscale between the color level of a first pixel located on a first side and the color level of a second pixel located on a second side based on a boundary area. The processor (110) can interpolate the grayscale of the first pixel and the grayscale of the second pixel using the intermediate grayscale. Here, the operation of interpolating the grayscale can mean increasing or decreasing the color level of the pixel. The processor (110) can alleviate the contour of the boundary area by reducing the grayscale difference between pixels existing in the boundary area.

[0159] Meanwhile, the processor (110) may detect a boundary area where the difference in grayscale between the first pixel and the second pixel included in the original image (910) is greater than or equal to a preset value. Alternatively, the processor (110) may detect a boundary area where the difference in grayscale between the first pixel and the second pixel included in the intermediate image (920) is greater than or equal to a preset value.

[0160] Meanwhile, in one embodiment, the contour interpolation (840) operation is exemplified as being performed on the blurred image (intermediate image (920)) that has been processed by blurring (820), but is not limited thereto. For example, the contour interpolation (840) operation may be performed before blurring (820). In addition, the second sub-pipeline (522) may further include a dithering operation.

[0161] In one embodiment, when contour generation is predicted for the input image (801), the processor (110) may input the input image (801) into a second sub-pipeline (522), which is an example of the second pipeline (520). The processor (110) may generate a blurred image (802) according to the second sub-pipeline (522).

[0162] Fig. 10 is a block diagram illustrating an example of a second blur processing according to one embodiment of the present disclosure. In Fig. 10, descriptions overlapping with those in Fig. 6 are omitted.

[0163] FIG. 10 illustrates a second pipeline (520) corresponding to the second blur processing. The second pipeline (520) may include a third sub-pipeline (523). The third sub-pipeline (523) may include downscaling (1010), blurring (1020), and upscaling (1030) operations for an input image (1001). The third sub-pipeline (523) may process the input image (1001) to generate a blurred image (1002).

[0164] The input image (1001) may be an image that has the possibility of generating a contour when blurred.

[0165] The second blur intensity of the blur image (1002) generated through the third sub-pipeline (523) may be lower than the first blur intensity of the blur image (602) generated through the first pipeline (510) of FIG. 6.

[0166] For example, the scaling factor (1011) of the down scaling (1010) used in the third sub-pipeline (523) may be smaller than the scaling factor of the down scaling (610) used in the first pipeline (510).

[0167] For example, the filter size (1012) of the blurring (1020) used in the third sub-pipeline (523) may be smaller than the filter size of the blurring (620) used in the first pipeline (510).

[0168] If the blur intensity of a blurred image is low, the visibility of the contour of the blurred image may be reduced. The processor (110) can minimize the visibility of the contour to the user by lowering the blur intensity of the blurred image (1002) through the third sub-pipeline (523).

[0169] Meanwhile, the third sub-pipeline (523) may further include a dithering operation.

[0170] FIG. 11 is a block diagram illustrating blur processing for a video image according to one embodiment of the present disclosure. In FIG. 11, descriptions overlapping with those in FIGS. 6 to 10 are omitted.

[0171] Referring to Fig. 11, when the input image is a video image, images of the same format can be input continuously.

[0172] For example, a first input image (1101) and a second input image (1102) have the same format and can be sequentially input to the image processing module (117). The first input image (1101) and the second input image (1102) can be image frames included in a video image. Here, if the input image has a YUV format, which is a video format, the input image can be converted to a graphic format before being processed in the pipeline.

[0173] Each of the first input image (1101) and the second input image (1102) may be an image for which a contour is predicted or an image for which a contour is not predicted.

[0174] The processor (110) can predict whether a contour occurs for each of the first input image (1101) and the second input image (1102) through a contour prediction model (113) and input the result into the first pipeline (510) or the second pipeline (520).

[0175] The processor (110) can generate a first blurred image (1111) and a second blurred image (1112) by blurring the first input image (1101) and the second input image (1102) in the first pipeline (510) or the second pipeline (520), respectively.

[0176] The processor (110) can perform a frame interpolation (1120) operation on the first blur image (1111) and the second blur image (1112). The processor (110) can generate interpolated images (1130) by interpolating the first blur image (1111) and the second blur image (1112). The frame interpolation (1120) operation can include, but is not limited to, linear interpolation.

[0177] The processor (110) can generate a video image with a high frame rate by performing blur processing for each video frame and interpolating according to the time difference between the video frames. For example, the processor (110) can generate a video image with a frame rate of 60 fps (frames per second) corresponding to an input image obtained at a rate of four per second.

[0178] Hereinafter, with reference to FIGS. 12 and 13, an operation of a display device (100) providing learning data to a contour prediction model (113) and obtaining an updated contour prediction model (113) will be described.

[0179] FIG. 12 is a flowchart illustrating a method for providing learning data to a contour prediction model according to one embodiment of the present disclosure.

[0180] Referring to FIG. 12, in operation 1210, the display device (100) can blur the learning image to generate a blurred image.

[0181] Training images may include broadcast streams, video streams, video images, graphic images, etc. Training images may be stored in a buffer.

[0182] A blurred image may be an image in which a learning image is blurred. The display device (100) can generate a blurred image by applying a first blur process to the learning image. However, the present invention is not limited to the first blur process.

[0183] A blurred image may or may not have a contour.

[0184] In one embodiment, the display device (100) can obtain a blur image through the image processing module (117) of FIG. 5.

[0185] In operation 1220, the display device (100) can obtain feedback on the result of contour generation of the blurred image.

[0186] In one embodiment, the display device (100) may receive user feedback via a user interface that includes an inquiry as to whether a contour of a blurred image has occurred.

[0187] For example, the display device (100) may display a blurred image and display a user interface including an inquiry regarding whether a contour is visible in the blurred image. If a contour is visible in the blurred image, the user may input a response thereto, such as Yes, a probability of visibility (%), a visibility level, etc., into the display device (100). Alternatively, if a contour is not visible in the blurred image, the user may input a response thereto, such as No, a probability of visibility (%), a visibility level, etc., into the display device (100). The display device (100) may obtain user feedback corresponding to the user input obtained through the user interface. The display device (100) may obtain user feedback for each learning image.

[0188] In one embodiment, the display device (100) can obtain feedback in response to a contour generation result for a blurred image obtained through a contour detection algorithm.

[0189] For example, the display device (100) can input a blurred image into a contour detection algorithm. The contour detection algorithm can output whether a contour exists in the blurred image. The blurred image can be captured through an internal camera or an external camera of the display device (100) and input into the contour detection algorithm, but is not limited thereto.

[0190] For example, the display device (100) can obtain whether a contour occurs (Yes or No) in a blurred image, the probability of contour occurrence (%), the level of contour occurrence, etc. through a contour detection algorithm. The display device (100) can obtain feedback corresponding to the output of the contour detection algorithm.

[0191] For example, a contour detection algorithm may specify a color buffer area of ​​a blurred image and then calculate a contour using the differences between adjacent pixels within the color buffer area. Alternatively, the contour detection algorithm may calculate a contour using the complexity of pixel values. Alternatively, the contour detection algorithm may include a neural network model (contour detection model) that infers a contour for a blurred image. Without being limited thereto, the contour detection algorithm may include suitable logic, circuitry, interfaces, and / or code for determining a contour for a blurred image.

[0192] Meanwhile, contour detection algorithms differ from contour prediction models in that they detect the presence of contours in blurred images. Contour prediction models can predict whether contours will appear in a blurred image before blurring.

[0193] In one embodiment, the display device (100) can label the acquired feedback. For example, the display device (100) can label the feedback by assigning a label of 1 in response to the occurrence of a contour and a label of 0 in response to the absence of a contour.

[0194] For example, the display device (100) may assign a label of 0 or 1 in comparison with a threshold value if the acquired feedback is a real number between 0 and 1 or a percentage value between 0% and 100%. Or, for example, the display device (100) may assign a label of 1 and 0, respectively, if the acquired feedback is Yes or No. Or, for example, the display device (100) may assign a label between 0 and 1 (e.g., 0.5) in response to the feedback.

[0195] In one embodiment, the display device (100) can obtain feedback through the feedback obtaining module (1310) of FIG. 13. The feedback obtaining module (1310) can assign a label to the feedback.

[0196] In operation 1230, the display device (100) can convert the learning image into a vector representation to generate an image embedding.

[0197] An embedding is data that compresses existing image information through a neural network. Embeddings are represented as vectors, which can be displayed as a list of numerical values ​​in a standardized format. Image embeddings are the conversion of an image into a vector representation.

[0198] For example, the display device (100) can perform tokenization, which divides data belonging to a learning image into small unit elements, and convert each token into a unique vector value.

[0199] In one embodiment, the display device (100) can generate an image embedding through the embedding processing module (1320) of FIG. 13.

[0200] In operation 1240, the display device (100) can provide a label corresponding to the image embedding and feedback as learning data for the contour prediction model.

[0201] In one embodiment, the display device (100) may provide the image embedding generated by converting the training image into a vector representation to the contour prediction model. For example, if the contour prediction model is trained on an external server for artificial intelligence learning, the display device (100) may provide the image embedding to the external server.

[0202] In one embodiment, the display device (100) can reduce the amount of data processing of the external server by transmitting an image embedding that converts the learning image into a vector representation to the external server instead of transmitting the learning image to the external server.

[0203] In one embodiment, the display device (100) may associate and store training images and labels including feedback for the training images. The display device (100) may associate the training images and labels and transmit them to an external server.

[0204] For example, the display device (100) can express vector values ​​and labels corresponding to a learning image as a single vector. For example, an image embedding may include not only information about the learning image, but also label information for the learning image. For example, the image embedding may be displayed in a form in which vector values ​​and labels corresponding to the learning image are listed. The display device (100) may transmit the image embedding including the label information to an external server.

[0205] Alternatively, in one embodiment, the display device (100) may separately store training images with a label of 0 and training images with a label of 1. The display device (100) may separately transmit the image embedding corresponding to the training image with a label of 0 and the image embedding corresponding to the training image with a label of 1 to an external server.

[0206] The contour prediction model can use image embeddings corresponding to training images and labels corresponding to feedback obtained for each training image as training data.

[0207] Contour prediction models can be reinforced through training data. They can be trained using pairs of image embeddings and labels as training data. Contour prediction models can be trained using feedback on the accuracy of classification results for training images. For example, a contour prediction model can predict the likelihood of contour occurrence (inference result) for training images and, using the inference result and the actual label, learn whether the inference result is correct.

[0208] However, without limitation, the contour prediction model may be a supervised learning model or an unsupervised learning model.

[0209] In one embodiment, the display device (100) can periodically provide training data to the contour prediction model, thereby obtaining an updated contour prediction model. For example, the contour prediction model can be trained on an external server and then distributed to the embedded system. The contour prediction model can be executed on the embedded system, and the display device (100) can execute the contour prediction model stored in memory via the processor (110).

[0210] FIG. 13 is a diagram illustrating a method for performing reinforcement learning on a contour prediction model using learning data according to one embodiment of the present disclosure.

[0211] Referring to FIG. 13, the processor (110) may include an image processing module (117), a feedback acquisition module (1310), and an embedding processing module (1320). For example, the processor (110) may perform operations corresponding to the image processing module (117), the feedback acquisition module (1310), and / or the embedding processing module (1320) by executing one or more instructions stored in a memory.

[0212] The image processing module (117) can generate a blurred image by blurring the learning image. For example, the learning image can be blurred through the first pipeline (e.g., 510).

[0213] The image processing module (117) can transmit the blurred image to the feedback acquisition module (1310).

[0214] The feedback acquisition module (1310) can obtain feedback on the contour generation result of the blur image through the blur image obtained from the image processing module (117).

[0215] For example, the feedback acquisition module (1310) can receive user feedback through a user interface that includes an inquiry about whether a contour has occurred in a blurred image.

[0216] Alternatively, for example, the feedback acquisition module (1310) may acquire a contour generation result for a blurred image through a contour detection algorithm. The feedback acquisition module (1310) may acquire feedback in response to the contour generation result.

[0217] Feedback on a blurred image may include whether a contour occurs in the blurred image (Yes or No), the probability of a contour occurring (%), the level of a contour occurring, etc.

[0218] The feedback acquisition module (1310) can label the acquired feedback. For example, the feedback acquisition module (1310) can label the feedback by assigning a label of 1 in response to the occurrence of a contour and a label of 0 in response to the absence of a contour.

[0219] The feedback acquisition module (1310) can transmit a label corresponding to the feedback to the embedding processing module (1320).

[0220] The embedding processing module (1320) can generate image embeddings by converting training images into vector representations. The embedding processing module (1320) can obtain labels corresponding to the evaluated feedback for each training image.

[0221] The embedding processing module (1320) can associate and store training images and labels. For example, the embedding processing module (1320) can express the training images and labels as a single vector. In this case, the image embedding can be displayed in a form where vector values ​​and labels corresponding to the training images are listed. Alternatively, for example, the embedding processing module (1320) can classify and store training images with the same label. The method of associating training images and labels is not limited to the examples described above. Table (1301) shows an example of labels associated with training images.

[0222] The processor (110) can transmit the image embedding generated by the embedding processing module (1320) and the label associated with the image embedding as learning data to the contour prediction model (1330). For example, if the contour prediction model (1330) is trained on an external server, the processor (110) can transmit the learning data to the external server. For example, the processor (110) can transmit the learning data to the external server through a communication module (not shown) or an input / output unit (not shown).

[0223] The processor (110) can obtain a learned contour prediction model (1330) using learning data. The contour prediction model (1330) can be periodically updated using learning data received periodically.

[0224] FIG. 14 is an example of a situation in which an image is blurred in a display device according to one embodiment of the present disclosure.

[0225] Referring to FIG. 14, the display device (100) can output a mixed image (1403) that is a mixture of a video image (1401) and a graphic image (1402) to the display (130). For example, the video image (1401) is an image corresponding to content, and the graphic image (1402) can include an OSD (On Screen Display), subtitles, or a sub-window. For example, the video image (1401) can be generated in a video plane that processes a YUV color space, and the graphic image (1402) can be generated in a graphic plane that processes an RGB or RGBA color space. The display device (100) can generate a mixed image (1403) by combining the video image (1401) generated in the video plane and the graphic image (1402) generated in the graphic plane. Meanwhile, the graphic plane can exist on the video plane.

[0226] The display device (100) can apply a blur effect to the mixing image (1403) and output the blurred mixing image (1403) to the display (130). For example, the display device (100) can output an image with a blur effect applied when creating a screen such as a home background or a settings menu.

[0227] The display device (100) may perform blur processing on each of the video image (1401) and the graphic image (1402) to output a blurred mixed image (1403), and mix the blurred video image and the blurred graphic image. For example, the display device (100) may output a mixed image (1403) including a blur non-display area (1411) and a blur display area (1412) to the display (130). The video image may be displayed in the blur non-display area (1411), and the image in which the blurred video image and the blurred graphic image are combined may be displayed in the blur display area (1412).

[0228] In one embodiment, a contour (e.g., area 1413) may be generated in the blur display area (1412). The contour generated at the hardware level after the blur processing operation may not be sufficiently removed by the first blur processing.

[0229] The display device (100) can predict the possibility of contour occurrence in the mixed image (1403) through the contour prediction model (113) and, if a contour is predicted, can apply a second blur processing. The display device (100) can predict the possibility of contour occurrence in the mixed image (1403) and, to prevent contour occurrence, can perform a second blur processing.

[0230] Accordingly, the display device (100) can output a mixed image (1403) with minimized contours.

[0231] FIG. 15 is a schematic block diagram of a display device according to one embodiment of the present disclosure.

[0232] Referring to FIG. 15, a display device (100) according to one embodiment may include a processor (110), a memory (120), and a display (130). However, the present invention is not limited thereto, and the display device (100) may include more or fewer components. In addition, the functions performed by each block are for the purpose of explaining an embodiment, and the specific operations or devices thereof do not limit the scope of the present invention.

[0233] According to one embodiment, a processor (110) controls the overall operation of the display device (100) and the signal flow between internal components of the display device (100), and performs a function of processing data.

[0234] The processor (110) may include single cores, dual cores, triple cores, quad cores, and multiples thereof. Additionally, the processor (110) may include multiple processors. For example, the processor (110) may be implemented as a main processor (not shown) and a sub processor (not shown).

[0235] Additionally, the processor (110) may include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a VPU (Video Processing Unit). Alternatively, according to an embodiment, the processor (110) may be implemented in the form of a SoC (System On Chip) that integrates at least one of a CPU, a GPU, and a VPU. Alternatively, the processor (110) may further include an NPU (Neural Processing Unit).

[0236] According to one embodiment, the memory (120) can store various data, programs or applications for driving and controlling the display device (100).

[0237] Additionally, the program stored in the memory (120) may include one or more instructions. The program (one or more instructions) or application stored in the memory (120) may be executed by the processor (110).

[0238] According to one embodiment, a processor (110) obtains an input image by executing one or more instructions stored in a memory (120).

[0239] According to one embodiment, a processor (110) obtains a contour prediction result including information predicting whether a contour occurs in the input image by inputting the input image into a contour prediction model (113) by executing one or more instructions stored in a memory (120).

[0240] According to one embodiment, the processor (110) performs at least one of a first blurring process and a second blurring process different from the first blurring process on the input image based on the contour prediction result by executing one or more instructions stored in the memory (120).

[0241] According to one embodiment, a processor (110) can output a contour occurrence probability for the input image through the contour prediction model (113) by executing one or more instructions stored in a memory (120).

[0242] According to one embodiment, the processor (110) can perform the first blur processing on the input image by executing one or more instructions stored in the memory (120), as the contour occurrence probability is less than a threshold value.

[0243] According to one embodiment, the processor (110) may perform the second blur processing on the input image when the contour occurrence probability is greater than or equal to the threshold value by executing one or more instructions stored in the memory (120).

[0244] According to one embodiment, a processor (110) can generate an image embedding by converting the input image into a vector representation by executing one or more instructions stored in a memory (120).

[0245] According to one embodiment, the processor (110) can input the image embedding into the contour prediction model (113) by executing one or more instructions stored in the memory (120).

[0246] According to one embodiment, the processor (110) can output a contour prediction result for the input image through the contour prediction model (113) by executing one or more instructions stored in the memory (120).

[0247] The first blur processing according to one embodiment may include down scaling, blurring, and upscaling operations on the input image.

[0248] The second blur processing according to one embodiment may include a dimming operation that lowers the color gradation of the blurred image according to the first blur processing.

[0249] A second blur processing according to one embodiment may include a contour interpolation operation on the blurred image.

[0250] A contour interpolation operation according to one embodiment may include an operation of detecting a boundary area in which a difference in grayscale between a first pixel and a second pixel included in the input image is greater than or equal to a predetermined value, and an operation of interpolating an intermediate grayscale between the grayscale of the first pixel and the grayscale of the second pixel in the boundary area.

[0251] The second blur processing according to one embodiment includes downscaling, blurring, and upscaling operations for the input image, wherein a second blur intensity used in the second blur processing may be lower than a first blur intensity used in the first blur processing.

[0252] A second scaling factor used in the second blur processing according to one embodiment may be smaller than a first scaling factor used in the first blur processing.

[0253] The size of the second blur filter used in the second blur processing according to one embodiment may be smaller than the size of the first blur filter used in the first blur processing.

[0254] In an input image according to one embodiment, the second blurred blurred image may have fewer contours than the first blurred blurred image.

[0255] According to one embodiment, a processor (110) can blur a learning image and generate a blurred image by executing one or more instructions stored in a memory (120).

[0256] According to one embodiment, a processor (110) can obtain feedback on the result of contour generation of the blurred image by executing one or more instructions stored in a memory (120).

[0257] According to one embodiment, a processor (110) can generate an image embedding by converting the learning image into a vector representation by executing one or more instructions stored in a memory (120).

[0258] According to one embodiment, the processor (110) can provide the image embedding and the label corresponding to the feedback as learning data of the contour prediction model (113) by executing one or more instructions stored in the memory (120).

[0259] According to one embodiment, the processor (110) may perform an operation of receiving user feedback through a user interface including an inquiry about whether a contour has been generated in the blurred image by executing one or more instructions stored in the memory (120). According to one embodiment, the processor (110) may perform an operation of obtaining the feedback in response to a contour generation result for the blurred image obtained through a contour detection algorithm by executing one or more instructions stored in the memory (120). According to one embodiment, the processor (110) may perform at least one of the above-described operations.

[0260] An input image according to one embodiment may include a video image processed in a YUV color space and / or a graphic image processed in an RGB or RGBA color space.

[0261] A display (130) according to one embodiment outputs an image onto a screen. Specifically, the display (130) may output an image corresponding to video data or a video signal through an internally included display panel (not shown) so that a user can visually recognize the video data.

[0262] According to one embodiment, a display (130) can output a blurred image under the control of a processor (110). Color banding and contours can be minimized in the blurred image output to the display (130).

[0263] FIG. 16 is a detailed block diagram of a display device according to one embodiment of the present disclosure.

[0264] Referring to FIG. 16, the display device (100) may include a tuner unit (1640), a processor (110), a display (130), a communication unit (1650), a detection unit (1630), an input / output unit (1670), a video processing unit (1680), an audio processing unit (1685), an audio output unit (1660), a memory (120), and a power supply unit (1695).

[0265] A tuner unit (1640) according to one embodiment can select and tune only the frequency of a channel to be received by the display device (100) among many radio wave components through amplification, mixing, resonance, etc. of a broadcast signal received wired or wirelessly. The broadcast signal includes audio, video, and additional information (e.g., EPG (Electronic Program Guide)).

[0266] The tuner unit (1640) can receive broadcast signals from various sources, such as terrestrial broadcasting, cable broadcasting, satellite broadcasting, and Internet broadcasting. The tuner unit (1640) can also receive broadcast signals from sources, such as analog broadcasting or digital broadcasting.

[0267] The communication unit (1650) can transmit and receive data or signals with an external device or server. For example, the communication unit (1650) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, a LAN module, an Ethernet module, a wired communication module, etc. In this case, each communication module may be implemented in the form of at least one hardware chip.

[0268] The Wi-Fi module and Bluetooth module perform communication in the Wi-Fi and Bluetooth modes, respectively. When using the Wi-Fi module or Bluetooth module, various connection information such as the SSID and session key are first transmitted and received, and after establishing a communication connection using this, various information can be transmitted and received. The wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), and 5G (5th Generation).

[0269] According to one embodiment, a detection unit (1630) detects a user's voice, a user's image, or a user's interaction, and may include a microphone (1631), a camera unit (1632), and a light receiving unit (1633).

[0270] The microphone (1631) receives the user's spoken voice. The microphone (1631) can convert the received voice into an electrical signal and output it to the processor (110).

[0271] The optical receiver (1633) receives an optical signal (including a control signal) from an external control device through an optical window (not shown) of a bezel of the display unit (130), etc. The optical receiver (1633) can receive an optical signal corresponding to a user input (e.g., touch, pressing, touch gesture, voice, or motion) from the control device. A control signal can be extracted from the received optical signal under the control of the processor (110).

[0272] The input / output unit (1670) according to one embodiment can receive video (e.g., moving images, etc.), audio (e.g., voice, music, etc.), and additional information (e.g., EPG, etc.) from the outside of the display device (100). The input / output unit (1670) can include any one of a High-Definition Multimedia Interface (HDMI), a Mobile High-Definition Link (MHL), a Universal Serial Bus (USB), a Display Port (DP), a Thunderbolt, a Video Graphics Array (VGA) port, an RGB port, a D-subminiature (D-SUB), a Digital Visual Interface (DVI), a component jack, and a PC port.

[0273] A video processing unit (1680) according to one embodiment performs processing on video data received by the display device (100). The video processing unit (1680) can perform various image processing operations, such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion on the video data.

[0274] Additionally, the processor (110) may include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a VPU (Video Processing Unit). Alternatively, according to an embodiment, the processor (110) may be implemented in the form of a SoC (System On Chip) that integrates at least one of a CPU, a GPU, and a VPU. Alternatively, the processor (110) may further include an NPU (Neural Processing Unit).

[0275] According to one embodiment, the memory (120) can store various data, programs or applications for driving and controlling the display device (100).

[0276] Additionally, the program stored in the memory (120) may include one or more instructions. The program (one or more instructions) or application stored in the memory (120) may be executed by the processor (110).

[0277] According to one embodiment, the processor (110) may obtain an input image by executing one or more instructions stored in the memory (120). The input image may be an image previously stored in the memory (120) or an image received from an external device via the tuner unit (1640) or the communication unit (1650). In addition, the input image may be an image on which various image processing such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion are performed in the video processing unit (1680).

[0278] A display (130) according to one embodiment converts image signals, data signals, OSD signals, control signals, etc. processed by a processor (110) to generate a driving signal. The display (130) may be implemented as a PDP, LCD, OLED, flexible display, etc., and may also be implemented as a 3D display. In addition, the display (130) may be configured as a touch screen and may be used as an input device in addition to an output device.

[0279] The audio processing unit (1685) processes audio data. The audio processing unit (1685) may perform various processing operations, such as decoding, amplification, and noise filtering, on audio data. Meanwhile, the audio processing unit (1685) may include multiple audio processing modules to process audio corresponding to multiple contents.

[0280] The audio output unit (1660) outputs audio included in a broadcast signal received through the tuner unit (1640) under the control of the processor (110). The audio output unit (1660) can output audio (e.g., voice, sound) input through the communication unit (1650) or the input / output unit (1670). In addition, the audio output unit (1660) can output audio stored in the memory (120) under the control of the processor (110). The audio output unit (1660) can include at least one of a speaker, a headphone output terminal, or an S / PDIF (Sony / Philips Digital Interface:) output terminal.

[0281] The power supply unit (1695) supplies power input from an external power source to components inside the display device (100) under the control of the processor (110). In addition, the power supply unit (1695) can supply power output from one or more batteries (not shown) located inside the display device (100) to the internal components under the control of the processor (110).

[0282] The memory (120) can store various data, programs, or applications for driving and controlling the display device (100) under the control of the processor (110). The memory (120) can include a broadcast reception module (not shown), a channel control module, a volume control module, a communication control module, a voice recognition module, a motion recognition module, an optical reception module, a display control module, an audio control module, an external input control module, a power control module, a power control module for an external device connected wirelessly (e.g., Bluetooth), a voice database (DB), or a motion database (DB). The processor (110) can perform each function using the software stored in the memory (120).

[0283] A method of operating a display device (100) according to one embodiment of the present disclosure includes a step (310) of obtaining an input image, a step (320) of obtaining a contour prediction result including information predicting whether a contour occurs for the input image by inputting the input image to a contour prediction model (113), and a step (330) of performing at least one of a first blurring process and a second blurring process different from the first blurring process on the input image based on the contour prediction result.

[0284] The step (320) of obtaining a contour prediction result according to one embodiment of the present disclosure may include the step of outputting a contour occurrence probability for the input image through the contour prediction model (113), the step of performing the first blur processing on the input image when the contour occurrence probability is less than a predetermined probability, and the step of performing the second blur processing on the input image when the contour occurrence probability is greater than or equal to the predetermined probability.

[0285] The step (320) of obtaining the contour prediction result according to one embodiment of the present disclosure may include the step of converting the input image into a vector representation to generate an image embedding, the step of inputting the image embedding into the contour prediction model (113), and the step of outputting a contour prediction result for the input image through the contour prediction model (113).

[0286] A first blur processing according to one embodiment of the present disclosure may include downscaling, blurring, and upscaling operations on the input image.

[0287] The second blur processing according to one embodiment of the present disclosure may include a dimming operation that lowers the color gradation of the blurred image according to the first blur processing.

[0288] The second blur processing according to one embodiment of the present disclosure may further include a contour interpolation operation for a blurred image, and the contour interpolation operation may include an operation of detecting a boundary area in which a difference in grayscale between a first pixel and a second pixel included in the input image is greater than a preset value, and an operation of interpolating an intermediate grayscale between the grayscale of the first pixel and the grayscale of the second pixel in the boundary area.

[0289] A second blur processing according to one embodiment of the present disclosure includes downscaling, blurring, and upscaling operations on the input image, wherein a second blur intensity used in the second blur processing may be lower than a first blur intensity used in the first blur processing.

[0290] The method of operating a display device (100) according to one embodiment of the present disclosure may further include a step of providing learning data to the contour prediction model (113).

[0291] The step of providing learning data to the contour prediction model (113) according to one embodiment of the present disclosure may include a step of blurring a learning image to generate a blurred image (1210), a step of obtaining feedback on a contour generation result of the blurred image (1220), a step of converting the learning image into a vector representation to generate an image embedding (1230), and a step of providing a label corresponding to the image embedding and the feedback as learning data of the contour prediction model (113) (1240).

[0292] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0293] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. In the display device (100), A memory (120) storing one or more instructions; and comprising one or more processors (110) for executing one or more instructions stored in the memory (120); The one or more processors (110) execute the one or more instructions, Obtain the input image, By inputting the input image into the contour prediction model (113), a contour prediction result including information predicting whether a contour occurs in an image output by the display device (100) based on the input image is obtained, A display device (100) configured to perform at least one of a first blurring process and a second blurring process different from the first blurring process on the input image based on the contour prediction result.

2. In paragraph 1, The above contour prediction result includes a contour occurrence probability for an image output by the display device (100) based on the input image, The one or more processors (110) execute the one or more instructions, Since the probability of occurrence of the above contour is less than the threshold, the first blur processing is performed on the input image, A display device (100) configured to perform the second blur processing on the input image when the probability of occurrence of the contour is greater than or equal to the threshold.

3. In paragraph 1 or 2, The one or more processors (110) execute the one or more instructions, A display device (100) that converts the input image into a vector representation to generate an image embedding and inputs the image embedding into the contour prediction model (113), thereby obtaining the contour prediction result.

4. In any one of paragraphs 1 to 3, The above first blur processing includes down scaling, blurring, and upscaling operations for the input image, A display device (100), wherein the second blur processing includes a dimming operation that lowers the color gradation of a blurred image generated according to downscaling, blurring, and upscaling of the input image.

5. In any one of paragraphs 1 to 4, The second blur processing includes a contour interpolation operation for a blurred image generated according to blurring of the input image, A display device (100), wherein the contour interpolation operation includes an operation of detecting a boundary area in which a difference in grayscale between a first pixel and a second pixel included in the input image is greater than a predetermined value, and an operation of interpolating an intermediate grayscale between the grayscale of the first pixel and the grayscale of the second pixel in the boundary area.

6. In any one of paragraphs 1 to 5, The second blur processing includes down scaling, blurring, and upscaling operations on the input image. A display device (100), wherein the second blur intensity used for the second blur processing is lower than the first blur intensity used for the first blur processing.

7. In paragraph 6, A display device (100) in which a second scaling factor used for the second blur processing is smaller than a first scaling factor used for the first blur processing.

8. In paragraph 6 or 7, A display device (100) wherein the size of the second blur filter used for the second blur processing is smaller than the size of the first blur filter used for the first blur processing.

9. In any one of paragraphs 1 to 8, A display device (100), wherein, in the input image, the second blurred blur image has fewer contours than the first blurred blur image.

10. In any one of paragraphs 1 to 9, The one or more processors (110) execute the one or more instructions, Create a blurred image by blurring the training image, Obtain feedback on the contour generation result of the above blurred image, Convert the above learning image into a vector representation to generate image embedding, A display device (100) configured to provide the image embedding and the label corresponding to the feedback as learning data for the contour prediction model (113).

11. In clause 10, The one or more processors (110) execute the one or more instructions, An action of receiving user feedback via a user interface including an inquiry as to whether a contour has occurred in the above blurred image, and A display device (100) configured to perform at least one of the operations of obtaining the feedback in response to a contour generation result for the blurred image obtained through a contour detection algorithm.

12. In any one of paragraphs 1 to 11, The above input image is a display device (100) including a video image processed in a YUV color space and / or a graphic image processed in an RGB or RGBA color space.

13. In the operating method of the display device (100), Step of obtaining an input image (310); By inputting the input image into the contour prediction model (113), a step (320) of obtaining a contour prediction result including information predicting whether a contour occurs in an image output by the display device (100) based on the input image; and A method comprising the step (330) of performing at least one of a first blurring process and a second blurring process different from the first blurring process on the input image based on the contour prediction result.

14. In paragraph 13, The above contour prediction result includes a contour occurrence probability for an image output by the display device (100) based on the input image, The above method, A step of performing the first blur processing on the input image, as the contour occurrence probability is less than a predetermined probability; and A method further comprising the step of performing the second blur processing on the input image when the contour occurrence probability is greater than or equal to a predetermined probability.

15. A computer-readable recording medium having recorded thereon a program for performing the method described in Article 13 or Article 14 on a computer.

Images