Display device and operating method for the same

Abstract

The disclosed embodiment relates to a display device that performs image processing using a neural network comprising a plurality of layers, comprising a display, a memory that stores one or more instructions, and a processor that executes one or more instructions stored in the memory, wherein the processor acquires model information corresponding to each pixel based on object characteristics corresponding to each pixel included in a first image, acquires model information corresponding to each layer layer based on model information corresponding to each pixel input to the neural network and information regarding the time at which each pixel is processed in the neural network, updates parameters of each of the plurality of layers based on the acquired model information, and controls the display to acquire a second image and display the second image by performing operations based on the updated parameters in each of the plurality of layers.

Description

Technology Field

[0001] Various embodiments relate to a display device and a method of operating the same. More specifically, the invention relates to a display device and a method of operating the same that performs image processing using a neural network. Background Technology

[0002] Recently, various forms of learning-based image processing methods (e.g., upscaling) are being actively developed alongside the advancement of deep learning technology. For instance, learning-based upscaling refers to the process of transforming and enlarging low-quality, low-resolution images into high-quality, high-resolution images by using large-scale, high-complexity networks whose parameter values ​​are determined through learning. Here, the networks used can be designed with various structures and selected to suit the system by adjusting the network depth (number of layers) and the number of computational parameters (size of filter kernels). Learning-based upscaling methods demonstrate superior image restoration performance compared to existing signal processing or interpolation-based upscaling (Bicubic, Bilinear, Lanczos). However, the single-input, single-output structure shows limitations in specialized restoration for individual objects within an image that possess diverse characteristics. For example, the types of objects included in an image can include people, text, graphics, artifacts (buildings, sculptures, cars, etc.), and natural objects (animals, plants), and there are structurally distinguishable features between different types of objects. It is difficult to learn these object-specific features by distinguishing them through a single network; even if the training data is composed of multiple components, the network is trained to minimize overall error rather than reflecting these object-specific features during the learning process. Consequently, there is a problem in that the trained network outputs the average image quality of the target image. The problem to be solved

[0003] The disclosed embodiments may provide a display device and a method of operation thereof capable of changing parameters for each layer of a neural network based on object-specific characteristics included in an image and performing image processing using the changed parameters. means of solving the problem

[0004] A display device that performs image processing using a neural network comprising a plurality of layers according to one embodiment includes a display, a memory that stores one or more instructions, and a processor that executes the one or more instructions stored in the memory. The processor can control the display to acquire a second image and display the second image by obtaining model information corresponding to each of the pixels based on object characteristics corresponding to each of the pixels included in the first image, and obtaining model information corresponding to each of the layers layer by layer based on model information corresponding to each of the pixels input to the neural network and information regarding the time at which each of the pixels is processed in the neural network, and by updating parameters of each of the plurality of layers based on the obtained model information, and by performing operations based on the updated parameters in each of the plurality of layers.

[0005] Information regarding the time at which each of the pixels is processed in the neural network according to one embodiment may include at least one of information regarding the time at which each of the pixels is input to each of the plurality of layers, and information regarding the time at which each of the pixels is processed in each of the plurality of layers.

[0006] According to one embodiment, at least one of the time at which each of the pixels is input to each of the plurality of layers and the time at which each of the plurality of layers is processed may have a constant period.

[0007] Information regarding the time at which each of the pixels is processed in the neural network according to one embodiment can be determined based on the position information of each of the pixels in the first image.

[0008] According to one embodiment, the plurality of layers are connected in series, and the processor can control each of the pixels included in the first image to be sequentially input to the plurality of layers and sequentially output.

[0009] According to one embodiment, the processor can update the parameters of the first layer to parameters included in first model information corresponding to the first pixel based on the first pixel being input to the first layer, and, based on the first pixel being input to a second layer which is the next layer of the first layer and the second pixel being input to the first layer, update the parameters of the first layer to parameters included in second model information corresponding to the second pixel, and update the parameters of the second layer to parameters included in the first model information based on the first pixel being input to the second layer.

[0010] According to one embodiment, the processor may update the parameters of the first layer to the parameters included in the third model information corresponding to the third pixel, update the parameters of the second layer to the parameters included in the second model information, and update the parameters of the third layer to the parameters included in the first model information, based on the first pixel being input to the third layer, the second pixel being input to the second layer, and the third pixel being input to the first layer.

[0011] According to one embodiment, the processor detects object regions included in the first image and, based on model information corresponding to the object regions, can obtain model information corresponding to each pixel included in the first image.

[0012] According to one embodiment, the processor can obtain model information corresponding to each of the pixels based on a weighted sum of model information corresponding to the object regions.

[0013] According to one embodiment, the processor determines a weight of model information corresponding to each of the object regions based on the distance between a first pixel included in the first image and the center of each of the object regions, and can obtain model information corresponding to the first pixel based on the determined weight and the model information corresponding to each of the object regions.

[0014] A method of operation of a display device that performs image processing using a neural network comprising a plurality of layers according to one embodiment may include: receiving a first image; obtaining model information corresponding to each of the pixels based on object characteristics corresponding to each of the pixels included in the first image; obtaining model information corresponding to each of the layers layer by layer based on model information corresponding to each of the pixels input to the neural network and information regarding the time at which each of the pixels is processed in the neural network; updating parameters of each of the plurality of layers based on the obtained model information; generating a second image by performing operations in each of the plurality of layers based on the updated parameters; and displaying the second image. Effects of the invention

[0015] A display device according to one embodiment can acquire region-specific and pixel-specific object characteristics based on objects included in an image, and perform image processing using different model information according to the region-specific and pixel-specific object characteristics. Accordingly, the accuracy or performance of image processing can be improved.

[0016] A display device according to one embodiment can update parameters layer by layer without updating all parameters included in an image processing network with the same model information, thereby preventing delays in image processing. Brief explanation of the drawing

[0017] FIG. 1 is a drawing showing a display device according to one embodiment. FIG. 2 is a diagram showing the structure of an image processing network according to one embodiment. FIG. 3 is a flowchart illustrating the operation method of a display device according to one embodiment. FIG. 4 is a diagram illustrating a method for a display device according to one embodiment to acquire model information corresponding to a first image. FIG. 5 is a diagram showing pixel-specific model information and information about the time at which a pixel is processed in an image processing network according to one embodiment. FIGS. 6 and 7 are drawings illustrating a method for a display device according to one embodiment to update parameters of an image processing network based on parameter update information. FIG. 8 is a diagram illustrating a method for obtaining a plurality of model information according to one embodiment. FIG. 9 is a diagram illustrating a method for obtaining model information corresponding to a first image according to one embodiment. FIG. 10 is a diagram illustrating a method for obtaining model information corresponding to a first image according to one embodiment. FIG. 11 is a block diagram showing the configuration of a display device according to one embodiment. FIG. 12 is a block diagram showing the configuration of a display device according to another embodiment. Specific details for implementing the invention

[0018] The terms used in this specification will be briefly explained, and the invention will be described in detail.

[0019] The terms used in this invention have been selected based on currently widely used general terms, taking into account their functions within the invention; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should be defined not merely by their names, but based on their meanings and the overall content of the invention.

[0020] When a part of a specification is described as "including" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part" or "module" as used in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware or software, or as a combination of hardware and software.

[0021] The following describes embodiments with reference to the attached drawings so that those skilled in the art can easily implement the present invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0022] In the embodiments of this specification, the term "user" refers to a person who controls a system, function, or operation, and may include a developer, administrator, or installer.

[0023] Additionally, in the embodiments of this specification, 'image' or 'picture' may represent a still image, a video composed of a plurality of consecutive still images (or frames), or a video.

[0024] FIG. 1 is a drawing showing a display device according to one embodiment.

[0025] Referring to FIG. 1, a display device (100) according to one embodiment may be an electronic device that receives an image and performs image processing on the received image. At this time, the image processing may include upscaling, image quality processing, etc., but is not limited thereto.

[0026] The display device (100) can be implemented in various forms such as a TV, mobile phone, tablet PC, digital camera, camcorder, laptop computer, desktop, e-book reader, digital broadcasting terminal, PDA (Personal Digital Assistants), PMP (Portable Multimedia Player), navigation, MP3 player, wearable device, etc.

[0027] Additionally, the display device (100) may include a display and display an image on which image processing has been performed. Additionally, the display device (100) may be a fixed electronic device placed at a fixed location or a mobile electronic device having a portable form, and may be a digital broadcast receiver capable of receiving digital broadcasts. In particular, the embodiments can be easily implemented in an image processing device with a large display, such as a TV, but are not limited thereto.

[0028] A display device (100) according to one embodiment can perform upscaling or image quality enhancement processing of an image (10) received or input from an external device or external server, and display an upscaled or image quality enhanced image on a display. At this time, the display device (100) inputs the input image (10) into an image processing network (30) and performs operations in each of the plurality of layers included in the image processing network (30) to obtain an upscaled or image quality enhanced output image (20).

[0029] An image processing network (30) according to one embodiment may include a structure in which first to n layers (31, 32, 33, ..., 39) are connected in series. Accordingly, data input to the first layer (31) is processed through operations with parameters of the first layer (31), and the processed result is input to the second layer (32). Additionally, data input to the second layer (32) is processed through operations with parameters of the second layer (32), and the processed result may be input to the third layer (33). In this way, data input to the first layer (31) is processed sequentially in each of the first to n layers (31, 32, 33, ..., 39), so that final data can be output from the n layer (39).

[0030] According to one embodiment, an input image (10) may be input to an image processing network (30) in area units. For example, it may be input to the image processing network (30) in area units where each pixel included in the input image (10) is a central pixel. In one embodiment, the operation of inputting to the image processing network (30) may mean the operation of inputting to the first layer (31), which is the first layer among the layers included in the image processing network (30).

[0031] At this time, the size of the input area may be determined based on the size of the parameter (filter kernel) of the first layer (31). For example, if the size of the parameter of the first layer (31) is 3 x 3, the size of the area input to the first layer (31) may be 3 x 3 in order to perform an operation between the area input to the first layer (31) and the parameter. However, it is not limited thereto.

[0032] At this time, the input image (10) may include one or more objects. The input image (10) has distinguishable characteristics for each object region depending on the type of object included in the input image (10), and the model optimized for image processing changes accordingly. Accordingly, a display device (100) according to one embodiment can obtain object characteristics by region and by pixel based on the objects included in the input image (10), and perform image processing using different model information according to the object characteristics by region and by pixel.

[0033] A method for performing image processing using different model information for each region and pixel included in an input image (10) by a display device (100) according to one embodiment will be explained in detail with reference to FIG. 2.

[0034] FIG. 2 is a diagram showing the structure of an image processing network according to one embodiment.

[0035] Referring to FIG. 2, the image processing network (30) may include a structure in which a plurality of layers are connected in series. For example, the plurality of layers may include first to n layers (31, 32, 33, ..., 39).

[0036] The first image (210) input to the image processing network (30) can be input to the image processing network (30) in area units. For example, the first area (211), the second area (212), and the third area (213) of the first image (210) can be input to the image processing network (30) sequentially. At this time, the first area (211) or the center pixel of the first area (211) may have a first object characteristic (e.g., "face" characteristic), and the second area (212) or the center pixel of the second area (212) may have a second object characteristic (e.g., "text" characteristic). Additionally, the third area (213) or the center pixel of the third area (213) may have a third object characteristic (e.g., "background" characteristic).

[0037] In this case, the display device (100) can improve the accuracy or performance of image processing by performing image processing on the first to third regions (211, 212, 213) using different model information according to object characteristics rather than the same model information.

[0038] In order to perform image processing using different model information according to object characteristics for each region, the display device (100) acquires model information corresponding to object characteristics of the input region and can update the parameters of multiple layers included in the image processing network (30) according to the model information.

[0039] At this time, when the display device (100) simultaneously updates parameters for all of the multiple layers included in the image processing network (30), a delay occurs in processing the next area until the input area is sequentially processed in the first to nth layers (31, 32, 33, ..., 39). For example, if the display device (100) simultaneously updates all parameters of the first to nth layers (31, 32, 33, ..., 39) based on first model information corresponding to the first object characteristic of the first area (211) in order to process the first area (211) input to the image processing network (30), processing for the second area (212) cannot be performed until image processing for the first area (211) is completed in the first to nth layers (31, 32, 33, ..., 39).

[0040] Accordingly, a delay occurs in the image processing of the current input area until the image processing of the previously input area is completed. Additionally, as the number of layers included in the image processing network (30) increases (as the depth of the network increases), the delay time becomes longer.

[0041] Accordingly, a display device (100) according to one embodiment can update parameters layer by layer without updating all parameters included in the image processing network (30) with the same model information, based on model information corresponding to the area input to the image processing network (30). The display device (100) can obtain model information of the area input to the corresponding layer in each of the layers, and update parameters layer by layer based on the obtained model information.

[0042] A display device (100) according to one embodiment can update the parameters of each of the plurality of layers included in the image processing network (30) based on region-specific (pixel-specific) model information of an input image and time-stamped information processed by region-specific (pixel-specific) image processing network (30).

[0043] At this time, regions of the image input to the image processing network (30) are processed and output in the order in which they are input to the image processing network (30). For example, regions of the first image (210) are input to the first layer (31), and are processed sequentially in a pipeline manner through the first to nth layers (31, 32, 33,..., 39) and output from the nth layer (39).

[0044] Accordingly, since the area processed by the K-th layer at time T is processed by the K+1-th layer at time T+1, the parameters of the K+1-th layer at time T+1 (current time) can be updated based on the model information of the K-th layer at time T (previous time).

[0045] For example, as illustrated in FIG. 2, when the first region (211), second region (212), and third region (213) of the first image are sequentially input into the image processing network (30), at time T the first region (211) is processed in the first layer (31), at time T+1 the first region (211) is processed in the second layer (32), which is the next layer of the first layer (31), and the second region (212) is processed in the first layer (31). Additionally, at time T+2 the first region (211) is processed in the third layer (33), which is the next layer of the second layer (32), the second region (212) is processed in the second layer (32), and the third region (213) is processed in the first layer (31).

[0046] Accordingly, at time T, the parameters of the first layer (31) can be updated based on the first model information corresponding to the first region (211). Additionally, at time T+1, the parameters of the first layer (211) can be updated based on the second model information corresponding to the second region (212), and the parameters of the second layer (212) can be updated based on the first model information. Furthermore, at time T+2, the parameters of the first layer (31) can be updated based on the third model information corresponding to the third region (213), the parameters of the second layer (212) can be updated based on the second model information, and the parameters of the third layer (213) can be updated based on the first model information.

[0047] Additionally, a display device (100) according to one embodiment may perform calculations based on parameters updated in each of the layers and input the result values ​​calculated in each of the layers to the next layer. By repeating the parameter update operation and the calculation operation in each of the layers at preset intervals, the display device (100) may process the first image (210) and obtain the second image (220).

[0048] Hereinafter, with reference to the drawings, a method for a display device according to one embodiment to update the parameters of each of a plurality of layers will be described in detail.

[0049] FIG. 3 is a flowchart illustrating the operation method of a display device according to one embodiment.

[0050] Referring to FIG. 3, a display device (100) according to one embodiment can receive a first image (S310). For example, the display device (100) can receive or input the first image from an external device or an external server.

[0051] A display device (100) according to one embodiment can obtain model information corresponding to a received first image (S320).

[0052] The display device (100) can extract object characteristics corresponding to each pixel included in the first image and obtain model information corresponding to each pixel based on the object characteristics. For example, the display device (100) can detect one or more objects included in the first image and determine one or more object regions based on the detected objects. At this time, the display device (100) may use a method of representing the detected object regions included in the first image as bounding boxes (rectangles) and a segmentation method of representing them in pixel units. However, it is not limited thereto.

[0053] When one or more object regions are determined, the display device (100) can determine model information corresponding to the object characteristics of the object region as model information corresponding to the pixels included in the object region. For example, the display device (100) can assign model information corresponding to the face characteristics to the pixels included in the "face" region detected in the first image. Additionally, model information corresponding to the text characteristics can be assigned to the pixels included in the "text" region.

[0054] The display device (100) can obtain model information corresponding to each of the pixels included in the first image in the manner described above.

[0055] Meanwhile, although the above description explains that model information is acquired pixel by pixel, it is not limited thereto, and model information may also be acquired for each region included in the first image.

[0056] Additionally, the display device (100) can obtain model information for each pixel included in the first image based on a weighted sum of model information corresponding to object regions detected in the first image. For example, a weight for model information corresponding to an object region can be determined based on the distance between a first pixel included in the first image and the center of an object region detected in the first image, and model information for the first pixel can be obtained based on the determined weight. This will be described in detail later with reference to FIGS. 9 and FIGS. 10.

[0057] A display device (100) according to one embodiment can obtain model information corresponding to each layer based on model information corresponding to each pixel and information about the time when each pixel is processed in an image processing network (S330).

[0058] The display device (100) can obtain model information corresponding to each layer based on parameter update information mapped with pixel information included in the first image, model information corresponding to the pixel, and viewpoint information corresponding to the pixel. At this time, the viewpoint information corresponding to the pixel may be the time when an area with the corresponding pixel as the center pixel is input to the first layer of the image processing network, or the time when an area input to the first layer is processed in the first layer.

[0059] The display device (100) can obtain model information corresponding to the first layer based on the point in time when an area with each of the pixels as the center pixel is input to the first layer of the image processing network and the model information of each of the pixels.

[0060] Additionally, the display device (100) can obtain model information corresponding to the remaining layers, excluding the first layer of the image processing network, based on the model information used to set the parameters of the previous layer at a previous time.

[0061] A display device (100) according to one embodiment can update the parameters of each layer based on model information obtained for each layer (S340).

[0062] The display device (100) can update the parameters of the first layer to parameters included in the first model information corresponding to the first area based on the first area of ​​the first image being input to the first layer, and, based on the first area being input to the second layer which is the next layer of the first layer and the second area being input to the first layer, update the parameters of the first layer to parameters included in the second model information corresponding to the second area, and update the parameters of the second layer to parameters included in the first model information based on the first area being input to the second layer.

[0063] Additionally, based on the fact that the first region is input to the third layer, which is the next layer of the second layer, the second region is input to the second layer, and the third region is input to the first layer, the parameters of the first layer can be updated to the parameters included in the third model information corresponding to the third region, the parameters of the second layer can be updated to the parameters included in the second model information, and the parameters of the third layer can be updated to the parameters included in the first model information.

[0064] A display device (100) according to one embodiment can obtain a second image by performing operations on each of a plurality of layers based on updated parameters (S350).

[0065] For example, the display device (100) can perform operations on each of the plurality of layers based on updated parameters and input the result of the operation to the next layer. The display device (100) can obtain a second image by repeating the parameter update operation and the operation operation on each of the plurality of layers at preset intervals.

[0066] A display device (100) according to one embodiment can display a second image obtained (S360).

[0067] FIG. 4 is a diagram illustrating a method for a display device according to one embodiment to acquire model information corresponding to a first image.

[0068] Referring to FIG. 4, according to one embodiment, the display device (100) may include an object detection module. The object detection module according to one embodiment may include appropriate logic, circuits, interfaces, and / or code that are operated to detect an object in a first image (410).

[0069] An object detection module according to one embodiment can identify objects included in an image using an object detection network (420) as shown in FIG. 4, and obtain the type, size, location information, etc. of the identified objects.

[0070] An object detection network (420) according to one embodiment may be a neural network that receives an image and detects at least one object included in the input image. The object detection network (420) may use one or more neural networks to detect one or more objects from a first image (410) and output object information including an object class and an object location corresponding to the one or more detected objects.

[0071] Here, object detection involves determining where objects are located in a given image (object localization) and determining which category each object belongs to (object classification). Accordingly, the object detection network (420) may generally include three steps: selecting object candidate regions, extracting features from each candidate region, and applying a classifier to the extracted features to classify the type of object candidate region. Depending on the detection method, localization performance can be improved through subsequent post-processing such as bounding box regression.

[0072] An object detection network (420) according to one embodiment may be a deep neural network (DNN) having multiple internal layers that perform operations, and may be a convolutional neural network (CNN) in which the internal layers are composed of convolution layers that perform convolution operations, but is not limited thereto.

[0073] Referring to FIG. 4, an object detection network (420) according to one embodiment may include a region proposal module (421), a CNN (422), and a classifier module (423).

[0074] The region proposal module (421) can extract candidate regions from the first image (10). The number of candidate regions may be limited to a predetermined number, but is not limited thereto.

[0075] The CNN (422) can extract feature information from the region generated by the region proposal module (421).

[0076] The classifier module (423) can perform classification by receiving feature information extracted from the CNN (422) as input.

[0077] In order for a neural network to accurately output result data corresponding to input data, it must be trained according to the purpose. Here, 'training' refers to inputting various data into the neural network and training it so that it can independently discover or master methods for analyzing the input data, classifying the input data, and / or extracting features necessary for generating result data. Specifically, through the training process, the neural network can optimize and set its internal weight values ​​by training on training data (e.g., multiple different images). Then, by autonomously learning the input data through a neural network equipped with optimized weight values, it outputs the desired result.

[0078] For example, through training, weight values ​​within the object detection network (420) can be optimized so that the object detection network (420) detects at least one object included in the image input to the object detection network (420). At this time, the object detection network (420) can be trained to detect various types of object information in the image, such as a face (person), text, artifact, and natural object (background).

[0079] Accordingly, the object detection network (420) that has completed training receives an image as input, detects at least one object included in the image, and can output the detected result. For example, the object detection network (420) can detect various types of object regions, such as a face (person), text, artifact, and natural object (background), included in the first image (410).

[0080] As illustrated in FIG. 4, the image (430) output from the object detection network (420) may include information about objects detected in the input first image (410). The information about the objects may include information about the class of the detected objects and a bounding box (435) indicating the location of the detected objects. However, it is not limited thereto, and objects detected in the input first image (410) in various formats may be displayed in the output image (430).

[0081] Meanwhile, in FIG. 4, the detected object area is depicted as having the shape of a rectangular bounding box, but it is not limited thereto.

[0082] An object detection module according to one embodiment segments a first image in pixel units and can detect object regions based on the segmented regions. In this case, various types of object regions can be detected.

[0083] A display device (100) according to one embodiment can obtain model information corresponding to a first image (410) based on the object characteristics of the detected object regions. For example, a model A corresponding to face characteristics may be assigned to pixels included in the first object region (441) and the second object region (442). Additionally, a model B corresponding to text characteristics may be assigned to pixels included in the third object region (443), and a model C corresponding to background characteristics may be assigned to pixels included in the remaining area, the fourth object region (444).

[0084] FIG. 5 is a diagram showing pixel-specific model information and information about the time at which a pixel is processed in an image processing network according to one embodiment.

[0085] Referring to FIG. 5, a display device (100) according to one embodiment can acquire model information (520) corresponding to each of the pixels included in the first image (510). Since the method for acquiring model information (520) has been described in detail in FIG. 4, a detailed description will be omitted.

[0086] Additionally, a display device (100) according to one embodiment can obtain information (time point information) regarding the time point when each of the pixels included in the first image (510) is processed in the image processing network (30) according to one embodiment. At this time, the time point information may include information regarding the time point when an area with each of the pixels included in the first image as a center pixel is input to the image processing network (30). For example, as shown in FIG. 5, the time point when a first area with a first pixel (P1) as a center pixel is input to the image processing network (30) may be T1, and the time point when a second area with a second pixel (P2) as a center pixel is input to the image processing network (30) may be T2.

[0087] Alternatively, the time information according to one embodiment may include information regarding the time when an area with each of the pixels included in the first image as a center pixel is input to the first layer, or information regarding the time when an operation with a parameter is performed in the first layer on an area with each of the pixels included in the first image as a center pixel. However, it is not limited thereto.

[0088] Additionally, the timing information can be determined based on the order in which each of the regions included in the first image (510) is input to the image processing network (30), the input period, the processing period in each of the layers, etc.

[0089] For example, the time information is information about the time when regions are input into the image processing network (30), and the regions are input into the image processing network in the order of the first region and the second region, and when the regions are input into the image processing network with period P0, the time information T2 of the second region can be determined as T1 (time information of the first region) + P0.

[0090] Additionally, the time information is information about the time when regions are processed in the first layer (31), and is input to the image processing network in the order of the first region and the second region, and when the operation between the input region and the parameter in the first layer is performed in period P1, the time information T2 of the second region can be determined as T1 (time information of the first region) + P1.

[0091] Meanwhile, a display device (100) according to one embodiment can obtain model information (520) corresponding to each of the pixels and parameter update information (530) mapped to time information corresponding to each of the pixels. The parameter update information (530) according to one embodiment may appear in various forms.

[0092] FIGS. 6 and 7 are drawings illustrating a method for a display device according to one embodiment to update parameters of an image processing network based on parameter update information.

[0093] Referring to FIG. 6, a display device (100) according to one embodiment may include a parameter update unit (610) and a model information storage unit (620).

[0094] A parameter update unit (610) according to one embodiment may include appropriate logic, circuit, interface, and / or code that operates to update the parameters of each of the plurality of layers included in the image processing network (30) based on parameter update information.

[0095] According to one embodiment, the parameter update unit (610) can determine model information corresponding to a layer for each layer based on pixel-by-pixel model information included in the parameter update information. The parameter update unit (610) can obtain the determined model information from the model information storage unit (620) and update the parameters of the layer.

[0096] At this time, the model information storage unit (620) can store multiple model information received from an external device or an external server. The multiple model information may be parameter information of an image processing network that has been trained using a set of training images having different object characteristics and has completed training. A method for obtaining multiple model information will be described in detail later with reference to FIG. 8.

[0097] A clock signal that inputs a region or pixel to an image processing network (30) according to one embodiment, a clock signal applied to the image processing network (30), and a clock signal applied to the parameter update unit (610) can be synchronized.

[0098] The parameter update unit (610) receives parameter update information and can update the parameters of multiple layers based on the parameter update information. This will be explained in detail with reference to FIG. 7.

[0099] The parameter update information according to one embodiment may be information in which pixel information, model information corresponding to the pixel, and time point information corresponding to the pixel are mapped, as described in FIG. 5. In this case, the time point information corresponding to the pixel may be the time point when an area with the corresponding pixel as the center pixel is input to the first layer (31) or the time point when an area input to the first layer (31) is processed in the first layer (31). However, it is not limited thereto.

[0100] A parameter update unit (610) according to one embodiment can update the parameters of the first layer (31) based on the time when the model information of each pixel included in the parameter update information and the area having each pixel as a center pixel are input to the first layer (31).

[0101] For example, if the parameter update information according to one embodiment includes information that the model information corresponding to pixel P1 is model A and that the area centered on pixel P1 is input as a first layer at time T1, the parameter update unit (610) can set the parameters of the first layer (31) to the first parameters included in model A based on time T1, as shown in FIG. 7.

[0102] Additionally, if the parameter update information includes information that the model information corresponding to pixel P2 is model A, and that the area centered on pixel P2 is input as the first layer at time T2, the parameter update unit (610) can maintain the first parameter without updating the parameter of the first layer.

[0103] Additionally, if the parameter update information includes information that the model information corresponding to pixel P3 is model B, and that the area centered on pixel P3 is input as the first layer at time T3, the parameter update unit (610) can update the parameters of the first layer to the second parameters included in model B based on time T3.

[0104] Additionally, the parameter update unit (610) can update the parameters of the remaining layers, excluding the first layer (31), based on the model information used to set the parameters of the previous layer at a previous time.

[0105] For example, the parameter update unit (610) can set the parameters of the second layer (32) at time T2 to the third parameters included in Model A, which were used to set the parameters of the first layer (31), which is the previous layer, at time T1.

[0106] Additionally, at time T4, the parameters of the second layer (32) can be updated with the fourth parameters included in Model B, which were used to set the parameters of the first layer (31) at the previous time T3.

[0107] Additionally, the parameter update unit (610) can update parameters for the third layer (31) to the nth layer (39) in the same way as for the second layer (32).

[0108] In addition, operations on the updated parameters and the input area can be performed in each of the layers.

[0109] The time points according to one embodiment may have a preset period, wherein the period may be determined based on the period during which regions (pixels) are input to the image processing network (30), the period during which regions are input to and output to each of the layers, the time required for operations performed in each of the layers, but is not limited thereto.

[0110] Meanwhile, although FIG. 7 describes the time point information included in the parameter update information as the time point when each of the regions is input to the first layer (31), parameters for multiple layers can be updated in the same way even when each of the regions is processed in the first layer (31).

[0111] As described above, for a plurality of layers included in the image processing network (30), if parameters are updated according to the object characteristics of the area processed in each layer for each layer, the performance of image processing can be improved and image processing can be prevented from being delayed.

[0112] FIG. 8 is a diagram illustrating a method for obtaining a plurality of model information according to one embodiment.

[0113] Referring to FIG. 8, a plurality of model information according to one embodiment may be determined by an external device, wherein the external device may be a separate device from the display device (100) according to one embodiment. For example, the external device may determine parameters included in the image processing network (810) by training the image processing network (810) based on a training data set. The image processing network (810) of FIG. 8 may be a network having the same structure as the image processing network (30) described in FIG. 1 to 7.

[0114] The image processing network (810) may be a deep neural network (DNN) containing multiple layers, and the neural network must be trained according to the purpose in order for the neural network to accurately output result data corresponding to the input data. At this time, 'training' may mean inputting various data into the neural network and training the neural network so that the neural network can discover or learn on its own methods for analyzing the input data, methods for classifying the input data, and / or methods for extracting features necessary for generating result data from the input data. Specifically, through the training process, the neural network can learn the training data and optimize the parameters (weights, coefficients) within the neural network. The neural network configured with the optimized parameters can output the desired result by learning the input data on its own.

[0115] An image processing network (810) according to one embodiment may be a neural network that receives an image, performs image processing such as upscaling or image quality processing, and outputs an image processed image.

[0116] An external device can obtain multiple model information by training an image processing network (810) based on multiple training data sets. At this time, the multiple training data sets may be generated based on multiple training images having different object characteristics.

[0117] For example, if the image processing according to one embodiment is upscaling that improves resolution, the first training data set may include low-resolution face images and high-resolution face images. Additionally, the second training data set may include low-resolution text images and high-resolution text images. Additionally, the third training data set may include low-resolution animal images and high-resolution animal images, and the fourth training data set may include low-resolution background images and high-resolution background images.

[0118] The external device can determine the first model information (parameter information of Model A) by training the image processing network (810) based on the first training data set.

[0119] For example, an external device can determine first model information (parameter information of Model A) by inputting a low-resolution face image included in a first training data set into an image processing network (810) and updating parameters included in the image processing network (810) in a direction that minimizes the difference between the output image (output data) and the high-resolution face image. Accordingly, the image processing network (810) that has completed training based on the first training data set may be an upscaling model (Model A) optimized for face images.

[0120] In the same way, the external device can determine the second model information (parameter information of Model B) by training the image processing network (810) based on the second training data set. Accordingly, the image processing network (810) that has been trained based on the second training data set may be an upscaling model (Model B) optimized for text images.

[0121] Additionally, the external device can determine third model information (parameter information of Model C) by training the image processing network (810) based on the third training data set. Accordingly, the image processing network (810) that has been trained based on the third training data set may be an upscaling model (Model C) optimized for animal images.

[0122] Additionally, the external device can determine the fourth model information (parameter information of Model D) by training the image processing network (810) based on the fourth training data set. Accordingly, the image processing network (810) that has been trained based on the fourth training data set may be an upscaling model (Model D) optimized for the background image.

[0123] FIG. 9 is a diagram illustrating a method for obtaining model information corresponding to a first image according to one embodiment.

[0124] A display device (100) according to one embodiment can detect one or more objects included in a first image (910) and determine an object region. For example, the display device (100) can divide the first image (910) into a first object region (921) where a human face is detected and a second object region (922) where a human face is not detected. At this time, the display device (100) according to one embodiment can assign a first model (Model A) corresponding to face characteristics to pixels included in the first object region (921), and assign a second model (Model B) corresponding to background characteristics to pixels included in the second object region (922).

[0125] As described in FIGS. 1 to 7, the display device (100) can update parameters included in an image processing network based on model information corresponding to each of the pixels included in the first image (910), and perform image processing on an area centered on each of the pixels based on the updated parameters.

[0126] When the display device (100) processes the first image (910), at the boundary between the first object area (921) and the second object area (922), the model information is changed from the first model information to the second model information or from the second model information to the first model information. At this time, if the difference between the first model information and the second model information is large, a discontinuity may occur at the boundary between the first object area (921) and the second object area (922) of the second image output from the image processing network due to a sudden change in parameters.

[0127] Accordingly, a display device (100) according to one embodiment can obtain model information corresponding to each pixel included in the first image (910) based on a weighted sum of first model information corresponding to the first object area (921) and second model information corresponding to the second object area (922), so that model information between the first object area (921) and the second object area (922) is gradually changed.

[0128] For example, if the object regions included in the first image are two, a first object region and a second object region, the model information corresponding to each pixel included in the first image can be expressed by the following mathematical formula 1.

[0129] [Mathematical Formula 1]

[0130] Model Information = 1st Weight X 1st Model Information + (1 - 1st Weight) X 2nd Model Information

[0131] In addition, mathematical formula 1 can be expressed as mathematical formula 2 as follows.

[0132] [Mathematical Formula 2]

[0133] Model Information = 2nd Model Information + 1st Weight X (1st Model Information - 2nd Model Information)

[0134] In Equation 2, if the difference between the first model information and the second model information is defined as a delta model, Equation 2 can be simply expressed as Equation 3 as follows.

[0135] [Mathematical Formula 3]

[0136] Model Information = Second Model Information + First Weight X Delta Model

[0137] The delta model of mathematical formula 3 can be used to obtain model information without additional calculations by pre-calculating the difference between the first model information and the second model information.

[0138] A method for obtaining model information without using a delta model (e.g., Equation 1) requires twice the number of parameters included in the model, and if the number of parameters is 1 million, 2 million multiplication operations are required.

[0139] However, the method of obtaining model information using a delta model as shown in Equation 3 reduces the number of multiplication operations by half compared to the method as shown in Equation 1. Accordingly, additional power consumption can be minimized during network implementation. In addition, delta models represent differences between models, and the statistical distribution of parameters included in delta models tends to be concentrated in a small range. Therefore, it is advantageous for the quantization or compression of delta models.

[0140] According to one embodiment, when model information is obtained using a delta model, quantization precision can be reduced and power consumption can be minimized.

[0141] In addition, even when an image includes three or more objects, since a part of the image can be represented as a background area and one object area, the method of obtaining model information using a delta model according to one embodiment is applicable even when there are three or more object areas.

[0142] Referring to FIG. 9, the display device (100) can determine a first weight based on the distance between a first pixel (930) included in a first image (910) and the center (940) of a first object region (921). In this case, if the first object region (921) is in the form of a bounding box, the center (940) of the first object region (921) can be calculated through the following Equation 4.

[0143] [Mathematical Formula 4]

[0144] Center coordinates of the object area = (Top-left coordinate of the object area + Bottom-right coordinate of the object area) / 2

[0145] According to one embodiment, the first weight has a value of 0 or greater and 1 or less, and is inversely proportional to the distance between the first pixel (930) and the center (940) of the first object area (921). For example, the value decreases as the distance to the center (940) of the first object area (921) increases, and the value increases as the distance decreases.

[0146] When model information corresponding to each pixel included in the first image (910) appears as a weighted sum of the first model information and the second model information, the parameter update unit (610) according to one embodiment calculates the weighted sum of the first model information and the second model information based on the first weight, and can update the parameters of the layer based on the calculated model information.

[0147] FIG. 10 is a diagram illustrating a method for obtaining model information corresponding to a first image according to one embodiment.

[0148] Referring to FIG. 10, a first image (1010) according to one embodiment may be divided into three or more object regions. For example, the first image (1010) may be divided into first to fourth object regions. The first object region (1021) is an area where a face is detected and may include face characteristics. The second object region (1022) is an area where text is detected and may include text characteristics, the third object region (1023) is an area where an animal is detected and may include animal characteristics, and the fourth object region (1024) is the remaining area excluding the first to third object regions (1021, 1022, 1023) and may include background characteristics. However, it is not limited thereto.

[0149] A display device (100) according to one embodiment can obtain model information corresponding to each of the pixels included in a first image (1010) based on a weighted sum of first model information (A) corresponding to face characteristics, second model information (B) corresponding to text characteristics, third model information (C) corresponding to animal characteristics, and fourth model information (D) corresponding to background characteristics.

[0150] For example, model information for a first pixel (1030) included in a first image (1010) can be obtained by adding a value obtained by applying a first weight to the first model information, a value obtained by applying a second weight to the second model information, a value obtained by applying a third weight to the third model information, and a value obtained by applying a fourth weight to the fourth model information.

[0151] At this time, the first weight is a value inversely proportional to the distance between the first pixel (1030) and the center (1040) of the first object area (1021), the second weight is a value inversely proportional to the distance between the first pixel (1030) and the center (1050) of the second object area (1022), and the third weight is a value inversely proportional to the distance between the first pixel (1030) and the center (1060) of the third object area (1023). The first to third weights may be normalized values, and the fourth weight may be determined as a value obtained by subtracting the first to third weights from 1.

[0152] Although FIGS. 9 and 10 illustrate and describe the case where the object region is represented by a bounding box, when it is represented by a pixel-unit segmentation region, the center of the object region can be obtained by calculating the average of all pixel coordinates included in the object region.

[0153] In addition, although FIGS. 9 and 10 describe that the display device (100) uses only a weight determined by the distance between the pixel and the center of the object area to calculate model information corresponding to the pixel, it is not limited thereto and can be determined by further considering not only the distance to the object area, but also the area of ​​the object area, the ratio of the object area in the first image, the brightness of the object area, etc.

[0154] FIG. 11 is a block diagram showing the configuration of a display device according to one embodiment.

[0155] Referring to FIG. 11, a display device (100) according to one embodiment may include an image receiving unit (110), a processor (120), a memory (130), and a display (140).

[0156] A video receiving unit (110) according to one embodiment may include a communication interface, an input / output interface, etc. For example, the communication interface may transmit and receive data or signals with an external device or server. For example, the communication interface may include a Wi-Fi module, a Bluetooth module, an infrared communication module and a wireless communication module, a LAN module, an Ethernet module, a wired communication module, etc. In this case, each communication module may be implemented in the form of at least one hardware chip.

[0157] Wi-Fi modules and Bluetooth modules perform communication via Wi-Fi and Bluetooth methods, respectively. When using a Wi-Fi or Bluetooth module, various connection information such as SSID and session key is transmitted and received first; after establishing a communication connection using this information, various types of information can be transmitted and received. Wireless communication modules include Zigbee, 3G (3 rd Generation), 3GPP(3 rdGeneration Partnership Project), Long Term Evolution (LTE), LTE Advanced (LTE-A), 4G (4 th Generation), 5G(5 th It may include at least one communication chip that performs communication according to various wireless communication standards, such as Generation.

[0158] Alternatively, the input / output interface receives video (e.g., video, etc.), audio (e.g., voice, music, etc.), and additional information (e.g., EPG, etc.) from outside the display device (100). The input / output interface may include any one of HDMI (High-Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), DVI (Digital Visual Interface), component jack, and PC port.

[0159] An image receiving unit (110) according to one embodiment can receive one or more images.

[0160] A processor (120) according to one embodiment controls the overall operation of the display device (100) and the signal flow between the durable components of the display device (100), and performs the function of processing data.

[0161] The processor (120) may include a single core, dual core, triple core, quad core, and multiples thereof. Additionally, the processor (120) may include multiple processors. For example, the processor (120) may be implemented as a main processor (not shown) and a sub processor (not shown) operating in sleep mode.

[0162] In addition, the processor (120) It may include at least one of a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and a VPU (Video Processing Unit). Alternatively, depending on the embodiment, it may be implemented in the form of a System On Chip (SoC) integrating at least one of a CPU, a GPU, and a VPU.

[0163] A memory (130) according to one embodiment can store various data, programs, or applications for driving and controlling a display device (100).

[0164] Additionally, the program stored in memory (130) may include one or more instructions. The program (one or more instructions) or application stored in memory (130) may be executed by the processor (120).

[0165] A processor (120) according to one embodiment may include at least one of the object detection module described in FIG. 4 and the parameter update unit described in FIG. 6.

[0166] A processor (120) according to one embodiment can extract object characteristics corresponding to each of the pixels included in the received first image and obtain model information corresponding to each of the pixels based on the object characteristics. For example, the processor (120) can detect one or more objects included in the first image and determine one or more object regions based on the detected objects. At this time, the processor (120) may use a method of representing the detected object regions included in the first image as bounding boxes (rectangles) and a segmentation method of representing them in pixel units. However, it is not limited thereto.

[0167] When one or more object regions are determined, the processor (120) can determine model information corresponding to the object characteristics of the object region as model information corresponding to the pixels included in the object region. For example, the processor (120) can assign model information corresponding to face characteristics to pixels included in the "face" region detected in the first image. Additionally, model information corresponding to text characteristics can be assigned to pixels included in the "text" region. The processor (120) can obtain model information corresponding to each pixel included in the first image in the manner described above. Furthermore, the processor (120) can obtain model information for each region included in the first image.

[0168] Additionally, the processor (120) can obtain model information for each pixel included in the first image based on a weighted sum of model information corresponding to object regions detected in the first image. For example, the processor (120) can determine a weight for model information corresponding to an object region based on the distance between a first pixel included in the first image and the center of an object region detected in the first image, and obtain model information for the first pixel based on the determined weight.

[0169] The processor (120) can receive parameter update information mapped to pixel information included in the first image, model information corresponding to the pixel, and time point information corresponding to the pixel. At this time, the time point information corresponding to the pixel may be the time point when an area with the corresponding pixel as the center pixel is input to the first layer of the image processing network, or the time point when an area input to the first layer is processed in the first layer. The processor (120) can obtain model information corresponding to the first layer based on the time point when an area with each of the pixels as the center pixel is input to the first layer (31) of the image processing network and the model information of each of the pixels.

[0170] Additionally, the processor (120) can obtain model information corresponding to the remaining layers, excluding the first layer of the image processing network, based on the model information used to set the parameters of the previous layer at a previous time.

[0171] The processor (120) can update the parameters of each layer based on the model information obtained for each layer. For example, the processor (120) can update the parameters of the first layer to the parameters included in the first model information corresponding to the first area based on the first area of ​​the first image being input to the first layer, and, based on the first area being input to the second layer which is the next layer of the first layer and the second area being input to the first layer, update the parameters of the first layer to the parameters included in the second model information corresponding to the second area, and update the parameters of the second layer to the parameters included in the first model information based on the first area being input to the second layer. Additionally, the processor (120) can update the parameters of the first layer to the parameters included in the third model information corresponding to the third layer, update the parameters of the second layer to the parameters included in the second model information, and update the parameters of the third layer to the parameters included in the first model information, based on the first layer being input to the third layer which is the next layer of the second layer.

[0172] The processor (120) can perform operations on each of the multiple layers based on the updated parameters and input the results of the operations to the next layer. The processor (120) can obtain a second image by repeating the parameter update operation and the operation operation on each of the multiple layers at preset intervals.

[0173] The processor (120) can control the second image to be displayed on the display (140).

[0174] A display (140) according to one embodiment converts a video signal, data signal, OSD signal, control signal, etc. processed by a processor (120) to generate a driving signal. The display (140) can be implemented as a PDP, LCD, OLED, flexible display, etc., and can also be implemented as a 3D display. Additionally, the display (140) can be configured as a touch screen and used as an input device in addition to an output device.

[0175] A display (140) according to one embodiment can display a second image in which upscaling or image quality processing has been performed.

[0176] FIG. 12 is a block diagram showing the configuration of a display device according to another embodiment.

[0177] Referring to FIG. 12, the display device (1200) of FIG. 12 may be an embodiment of the display device (100) described with reference to FIG. 1 to FIG. 11.

[0178] Referring to FIG. 12, a display device (1200) according to one embodiment may include a tuner unit (1240), a processor (1210), a display unit (1220), a communication unit (1250), a detection unit (1230), an input / output unit (1270), a video processing unit (1280), an audio processing unit (1285), an audio output unit (1260), a memory (1290), and a power supply unit (1295).

[0179] The communication unit (1250) of FIG. 12 is configured to correspond to the communication interface included in the image receiving unit (110) of FIG. 11, the input / output unit (1270) of FIG. 12 is configured to correspond to the input / output interface included in the image receiving unit (110) of FIG. 11, the processor (1210) of FIG. 12 is configured to correspond to the processor (120) of FIG. 11, the memory (1290) of FIG. 12 is configured to correspond to the memory (130) of FIG. 11, and the display unit (1220) of FIG. 12 is configured to correspond to the display (140) of FIG. 11. Accordingly, content identical to that described above will be omitted.

[0180] A tuner unit (1240) according to one embodiment can select only the frequency of the channel to be received by the broadcast receiving device (100) from among many radio wave components by tuning through amplification, mixing, resonance, etc. of a broadcast signal received via wired or wireless means. The broadcast signal includes audio, video, and additional information (e.g., EPG (Electronic Program Guide)).

[0181] The tuner unit (1240) can receive broadcast signals from various sources such as terrestrial broadcasting, cable broadcasting, satellite broadcasting, internet broadcasting, etc. The tuner unit (1240) can also receive broadcast signals from sources such as analog broadcasting or digital broadcasting.

[0182] The detection unit (1230) detects the user's voice, the user's image, or the user's interaction, and may include a microphone (1231), a camera unit (1232), and an optical receiver (1233).

[0183] The microphone (1231) receives the user's uttered voice. The microphone (1231) can convert the received voice into an electrical signal and output it to the processor (1210). The user voice may include, for example, a voice corresponding to a menu or function of the display device (1200).

[0184] The camera unit (1232) can receive images (e.g., consecutive frames) corresponding to a user's motion including a gesture within the camera recognition range. The processor (1210) can use the received motion recognition result to select a menu displayed on the display device (1200) or perform control corresponding to the motion recognition result.

[0185] The light receiver (1233) receives a light signal (including a control signal) received from an external control device through a light window (not shown) in the bezel of the display unit (1220). The light receiver (1233) can receive a light signal corresponding to user input (e.g., touch, press, touch gesture, voice, or motion) from the control device. A control signal can be extracted from the received light signal under the control of the processor (1210).

[0186] The processor (1210) controls the overall operation of the display device (1400) and the signal flow between the internal components of the display device (1200), and performs the function of processing data. The processor (1210) can execute an OS (Operation System) and various applications stored in memory (1290) when there is user input or when a pre-set and stored condition is satisfied.

[0187] The processor (1210) may include RAM used as a storage area for storing signals or data input from outside the display device (1200) or for various tasks performed on the display device (1200), ROM stored as a control program for controlling the display device (1200), and a processor.

[0188] The video processing unit (1280) performs processing on video data received by the display device (1200). The video processing unit (1280) can perform various image processing on the video data, such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion.

[0189] The audio processing unit (1285) performs processing on audio data. Various processing such as decoding, amplification, and noise filtering on audio data can be performed in the audio processing unit (1285). Meanwhile, the audio processing unit (1285) may be equipped with multiple audio processing modules to process audio corresponding to multiple contents.

[0190] The audio output unit (1260) outputs audio included in a broadcast signal received through the tuner unit (1240) under the control of the processor (1210). The audio output unit (1260) can output audio (e.g., voice, sound) input through the communication unit (1250) or the input / output unit (1270). Additionally, the audio output unit (1260) can output audio stored in the memory (1290) under the control of the processor (1210). The audio output unit (1260) may include at least one of a speaker, a headphone output terminal, or an S / PDIF (Sony / Philips Digital Interface) output terminal.

[0191] The power supply unit (1295) supplies power input from an external power source to the components inside the display device (1200) under the control of the processor (1210). Additionally, the power supply unit (1295) can supply power output from one or more batteries (not shown) located inside the display device (1200) to the internal components under the control of the processor (1210).

[0192] The memory (1290) may store various data, programs, or applications for driving and controlling the display device (1200) under the control of the processor (1210). The memory (1290) may include a broadcast receiving module, a channel control module, a volume control module, a communication control module, a voice recognition module, a motion recognition module, an optical receiving module, a display control module, an audio control module, an external input control module, a power control module, a power control module for an external device connected wirelessly (e.g., Bluetooth), a voice database (DB), or a motion database (DB), which are not illustrated. The modules not illustrated and the database of the memory (1290) may be implemented in software form to perform broadcast reception control functions, channel control functions, volume control functions, communication control functions, voice recognition functions, motion recognition functions, optical reception control functions, display control functions, audio control functions, external input control functions, power control functions, or power control functions for an external device connected wirelessly (e.g., Bluetooth) in the display device (1200). The processor (1210) can perform each of these functions using the software stored in memory (1290).

[0193] Meanwhile, the block diagram of the display device (100, 1200) illustrated in FIGS. 11 and 12 is a block diagram for one embodiment. Each component of the block diagram may be integrated, added, or omitted according to the specifications of the actual implemented display device (100, 1200). That is, as needed, two or more components may be combined into a single component, or a single component may be subdivided into two or more components. Furthermore, the functions performed in each block are intended to explain the embodiments, and the specific operations or devices thereof do not limit the scope of the present invention.

[0194] A method of operation of a display device according to one embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the present invention, or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0195] In addition, the method of operation of the display device according to the disclosed embodiments may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product.

[0196] A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable app) that is electronically distributed through a manufacturer of an electronic device or an electronic market (e.g., Google Play Store, App Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a server of the manufacturer, a server of the electronic market, or a storage medium of a relay server that temporarily stores the software program.

[0197] A computer program product may include a storage medium of a server or a storage medium of a client device in a system composed of a server and a client device. Alternatively, if a third device (e.g., a smartphone) is communicationally connected to the server or client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S / W program itself, which is transmitted from the server to the client device or the third device, or from the third device to the client device.

[0198] In this case, one of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0199] For example, a server (e.g., a cloud server or an artificial intelligence server, etc.) can execute a computer program product stored on the server to control a client device connected to the server in communication to perform a method according to the disclosed embodiments.

[0200] Although the embodiments have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims also fall within the scope of the present invention.

Claims

Claim 1 A display device for performing image processing using a neural network comprising a plurality of layers, comprising: a display; a memory for storing one or more instructions; and a processor for executing the one or more instructions stored in the memory, wherein the processor acquires model information corresponding to each of the pixels based on object characteristics corresponding to each of the pixels included in a first image, acquires model information corresponding to each of the layers layer by layer based on model information corresponding to each of the pixels input to the neural network and information regarding the time at which each of the pixels is processed in the neural network, updates the parameters of the plurality of layers based on the acquired model information, and acquires a second image by processing the first image through the plurality of layers to which the updated parameters are applied, and controls the display to display the second image. Claim 2 A display device according to claim 1, wherein information regarding the time at which each of the pixels is processed in the neural network comprises at least one of information regarding the time at which each of the pixels is input to each of the plurality of layers and information regarding the time at which each of the pixels is processed in each of the plurality of layers. Claim 3 A display device according to paragraph 2, wherein at least one of the time at which each of the pixels is input to each of the plurality of layers and the time at which each of the plurality of layers is processed has a constant period. Claim 4 A display device according to claim 1, wherein information regarding the time at which each of the pixels is processed in the neural network is determined based on the position information of each of the pixels in the first image. Claim 5 A display device according to claim 1, wherein the plurality of layers are connected in series, and the processor controls each of the pixels included in the first image to be sequentially input to the plurality of layers and sequentially output. Claim 6 A display device according to claim 1, wherein the processor updates the parameters of the first layer to parameters included in first model information corresponding to the first pixel based on the first pixel being input to the first layer among the plurality of layers, and updates the parameters of the first layer to parameters included in second model information corresponding to the second pixel based on the first pixel being input to the second layer which is the next layer of the first layer and the second pixel being input to the first layer, and updates the parameters of the second layer to parameters included in the first model information. Claim 7 A display device according to claim 6, wherein the processor updates the parameters of the first layer to parameters included in the third model information corresponding to the third pixel, updates the parameters of the second layer to parameters included in the second model information, and updates the parameters of the third layer to parameters included in the first model information, based on the first pixel being input to the third layer which is the next layer of the second layer, the second pixel being input to the second layer, and the third pixel being input to the first layer. Claim 8 A display device according to claim 1, wherein the processor detects object regions included in the first image and, based on model information corresponding to the object regions, obtains model information corresponding to each pixel included in the first image. Claim 9 In claim 8, the display device, wherein the processor obtains model information corresponding to each of the pixels based on a weighted sum of model information corresponding to the object regions. Claim 10 A display device according to claim 9, wherein the processor determines a weight of model information corresponding to each of the object regions based on the distance between a first pixel included in the first image and the center of each of the object regions, and obtains model information corresponding to the first pixel based on the determined weight and the model information corresponding to each of the object regions. Claim 11 A method of operation of a display device that performs image processing using a neural network comprising a plurality of layers, comprising: receiving a first image; obtaining model information corresponding to each of the pixels based on object characteristics corresponding to each of the pixels included in the first image; obtaining model information corresponding to each of the layers layer by layer based on model information corresponding to each of the pixels input to the neural network and information regarding the time at which each of the pixels is processed in the neural network; updating parameters of the plurality of layers based on the obtained model information; generating a second image by processing the first image through the plurality of layers to which the updated parameters are applied; and displaying the second image. Claim 12 A method of operation of a display device according to claim 11, wherein information regarding the time at which each of the pixels is processed in the neural network comprises at least one of information regarding the time at which each of the pixels is input to each of the plurality of layers and information regarding the time at which each of the pixels is processed in each of the plurality of layers. Claim 13 A method of operation of a display device according to claim 12, wherein at least one of the time at which each of the pixels is input to each of the plurality of layers and the time at which each of the plurality of layers is processed has a constant period. Claim 14 A method of operation of a display device according to claim 11, wherein information regarding the time at which each of the pixels is processed in the neural network is determined based on position information of each of the pixels in the first image. Claim 15 A method of operation of a display device according to claim 11, wherein the plurality of layers are connected in series, and each pixel included in the first image is sequentially input to the plurality of layers and sequentially output. Claim 16 A method of operation of a display device according to claim 11, wherein the step of updating the parameters of the plurality of layers comprises: a step of updating the parameters of the first layer to parameters included in first model information corresponding to the first pixel based on the first pixel being input to the first layer among the plurality of layers; and a step of updating the parameters of the first layer to parameters included in second model information corresponding to the second pixel based on the first pixel being input to a second layer which is the next layer of the first layer, and the second pixel being input to the first layer, and updating the parameters of the second layer to parameters included in the first model information. Claim 17 A method of operation of a display device according to claim 16, wherein the step of updating the parameters of the plurality of layers comprises, based on the fact that the first pixel is input to the third layer which is the next layer of the second layer, the second pixel is input to the second layer, and the third pixel is input to the first layer, updating the parameters of the first layer to parameters included in the third model information corresponding to the third pixel, updating the parameters of the second layer to parameters included in the second model information, and updating the parameters of the third layer to parameters included in the first model information. Claim 18 In claim 11, the above-described method of operation further comprises the step of detecting object regions included in the first image, and the step of obtaining model information corresponding to each of the pixels comprises the step of obtaining model information corresponding to each of the pixels included in the first image based on the model information corresponding to the object regions. Claim 19 A method of operation of a display device according to claim 18, wherein the step of obtaining model information corresponding to each of the above pixels includes the step of obtaining model information corresponding to each of the above pixels based on a weighted sum of model information corresponding to the above object regions. Claim 20 ◈Claim 20 was abandoned upon payment of the registration fee.◈ In claim 19, the step of acquiring model information corresponding to each of the pixels comprises: a step of determining a weight of model information corresponding to each of the object regions based on the distance between a first pixel included in the first image and the center of each of the object regions; and a step of acquiring model information corresponding to the first pixel based on the determined weight of the model information corresponding to each of the object regions. Claim 21 One or more computer-readable recording media storing a program that enables the method of paragraph 11 to be performed.

Images